Repository: pandas-dev/pandas

Issue:
BUG: error in read_html when parsing badly-escaped HTML from an io object
#### Code Sample, a copy-pastable example if possible

Create `test.html`, with the contents:
```html
<!doctype html>
<html>
<body>
<table>
	<tr><td>poorly-escaped cell with an & oh noes</td></tr>
</table>
</body>
</html>
```

```py
>>> import pandas as pd
>>> pandas.__version__
'0.20.3'
>>> f = open('./test.html')
>>> pd.read_html(f)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    pd.read_html(f)
  File "/usr/lib/python3.6/site-packages/pandas/io/html.py", line 906, in read_html
    keep_default_na=keep_default_na)
  File "/usr/lib/python3.6/site-packages/pandas/io/html.py", line 743, in _parse
    raise_with_traceback(retained)
  File "/usr/lib/python3.6/site-packages/pandas/compat/__init__.py", line 344, in raise_with_traceback
    raise exc.with_traceback(traceback)
ValueError: No text parsed from document: <_io.TextIOWrapper name='/home/liam/test.html' mode='r' encoding='UTF-8'>
```
#### Problem description

Pandas attempts to invoke a series of parsers on HTML documents, returning when one produces a result, and continuing to the next on error. This works fine when passing a path or entire document to `read_html()`, but when an IO object is passed, the subsequent parsers will be reading from a file whose read cursor is at EOF, producing an inscrutable 'no text parsed from document' error.

This can easily be fixed by rewinding the file with `seek(0)` before continuing to the next parser (will add PR shortly).

#### Expected Output
```
[                                       0
0  poorly-escaped cell with an & oh noes]
```

#### Output of ``pd.show_versions()``

<details>

```
>>> pd.show_versions()
INSTALLED VERSIONS
------------------
commit: e1dabf37645f0fcabeed1d845a0ada7b32415606
python: 3.6.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.6-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0rc1+36.ge1dabf376.dirty
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.6.0
Cython: 0.27.2
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
```

</details>



Discussion:
Interestingly, when adding a test for the patch, I noticed that stuffing that test document into a `StringIO` seems to work? lxml still fails, but the fallback on html5lib/bs4 rewinds properly.

I'll investigate that further tomorrow.
Same issue: Cannot `read_html()` a webpage directly from an `urlopen()` result when `lxml` does not like it:
```python
>>> from urllib.request import urlopen
>>> import pandas as pd
>>> url = 'http://en.wikipedia.org/wiki/Matplotlib'
>>> assert pd.read_html(urlopen(url), 'Advantages', 'bs4')  # works with bs4 alone
>>> pd.read_html(urlopen(url), 'Advantages')
Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    pd.read_html(urlopen(url), 'Advantages')
  File "C:\Program Files\Python36\lib\site-packages\pandas\io\html.py", line 915, in read_html
    keep_default_na=keep_default_na)
  File "C:\Program Files\Python36\lib\site-packages\pandas\io\html.py", line 749, in _parse
    raise_with_traceback(retained)
  File "C:\Program Files\Python36\lib\site-packages\pandas\compat\__init__.py", line 367, in raise_with_traceback
    raise exc.with_traceback(traceback)
ValueError: No text parsed from document: <http.client.HTTPResponse object at 0x0000000005621358>
```
Note that one cannot do `.seek(0)` on the `urlopen` return value (so the fix needs to be more complex).

I think `lxml` does something slightly different with `StringIO`s. So here is a self-contained test case:
```python
>>> import pandas as pd
>>> from mock import Mock
>>> def mock_urlopen(data, url='http://spam'):
        return Mock(**{'geturl.return_value': url, 'read.side_effect': [data, '', '']})

>>> good = mock_urlopen('<table><tr><td>spam<br />eggs</td></tr></table>')
>>> bad = mock_urlopen('<table><tr><td>spam<wbr />eggs</td></tr></table>')
>>> assert pd.read_html(good)
>>> assert pd.read_html(bad, flavor='bs4')
>>> bad.reset_mock()
>>> pd.read_html(bad)
Traceback (most recent call last):
...
ValueError: No text parsed from document: <Mock id='85948960'>
>>> bad.mock_calls
[call.geturl(),
 call.tell(),
 call.read(4000),
 call.decode('ascii', 'strict'),
 call.decode().decode('ascii', 'strict'),
 call.decode().decode().find(':'),
 call.read()]
```
The second `.read()`-call is the one where `bs4` takes over and fails parsing the empty string.
Minimal amendment: `reset_mock()` does not rewind `read.side_effect` so here is the same with a fresh mock:
```python
>>> bad = mock_urlopen('<table><tr><td>spam<wbr />eggs</td></tr></table>')
>>> pd.read_html(bad)
Traceback (most recent call last):
...
ValueError: No text parsed from document: <Mock id='50837656'>
>>> bad.mock_calls
[call.geturl(),
 call.tell(),
 call.read(4000),
 call.read(3952),
 call.decode('ascii', 'strict'),
 call.decode().decode('ascii', 'strict'),
 call.decode().decode().find(':'),
 call.read()]
```
Again, the last `.read()`-call is from `bs4`
The only way to rewind a urlopen is re-requesting it or buffering it, unfortunately. This becomes a _much_ more complex patch, then :frowning: 
So i suppose that the try next parser should raise if we only have a filehandle (and not a path). would take that as a PR.
We can seek for some IO handles, though. I don't see any reason not to add something like
```py
if hasattr(io, 'seek'):
    io.seek(0)
```
and raise a warning if
```py
hasattr(io, 'read') and not hasattr(io, 'seek')
```
Sounds good to me. I think @jreback means that the `raise` (possibly after checking for `seek`) should occur in the branch after the first parser fails, so it makes the current behaviour more official/transparent (give a better error message).  The user can then select/try a different `flavor` (maybe the error message can hint at that) .
ah, you're talking about ditching the fallthrough to the next parser entirely?
I thought for io handles (possibly only non-seekable ones). Does not occur with file names, right?
Yep, since _read() reopens the file for each parser if you're passing in filenames.

On Sat, Oct 28, 2017 at 4:35 PM, Sebastian Bank <notifications@github.com>
wrote:

> I thought for io handles (possibly only non-seekable ones). Does not occur
> with file names, right?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/17975#issuecomment-340221351>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEa8SQpni0IETbGKuerchH4awQd7JEV5ks5sw54QgaJpZM4QFbJ2>
> .
>


Resolving patch:
diff --git a/pandas/io/html.py b/pandas/io/html.py
--- a/pandas/io/html.py
+++ b/pandas/io/html.py
@@ -742,6 +742,18 @@ def _parse(flavor, io, match, attrs, encoding, **kwargs):
         try:
             tables = p.parse_tables()
         except Exception as caught:
+            # if `io` is an io-like object, check if it's seekable
+            # and try to rewind it before trying the next parser
+            if hasattr(io, 'seekable') and io.seekable():
+                io.seek(0)
+            elif hasattr(io, 'seekable') and not io.seekable():
+                # if we couldn't rewind it, let the user know
+                raise ValueError('The flavor {} failed to parse your input. '
+                                 'Since you passed a non-rewindable file '
+                                 'object, we can\'t rewind it to try '
+                                 'another parser. Try read_html() with a '
+                                 'different flavor.'.format(flav))
+
             retained = caught
         else:
             break


Test patch:
diff --git a/pandas/tests/io/test_html.py b/pandas/tests/io/test_html.py
--- a/pandas/tests/io/test_html.py
+++ b/pandas/tests/io/test_html.py
@@ -967,3 +967,53 @@ def test_importcheck_thread_safety():
     while helper_thread1.is_alive() or helper_thread2.is_alive():
         pass
     assert None is helper_thread1.err is helper_thread2.err
+
+
+def test_parse_failure_unseekable():
+    # Issue #17975
+    _skip_if_no('lxml')
+
+    class UnseekableStringIO(StringIO):
+        def seekable(self):
+            return False
+
+    good = UnseekableStringIO('''
+        <table><tr><td>spam<br />eggs</td></tr></table>''')
+    bad = UnseekableStringIO('''
+        <table><tr><td>spam<foobr />eggs</td></tr></table>''')
+
+    assert read_html(good)
+    assert read_html(bad, flavor='bs4')
+
+    bad.seek(0)
+
+    with pytest.raises(ValueError,
+                       match='passed a non-rewindable file object'):
+        read_html(bad)
+
+
+def test_parse_failure_rewinds():
+    # Issue #17975
+    _skip_if_no('lxml')
+
+    class MockFile(object):
+        def __init__(self, data):
+            self.data = data
+            self.at_end = False
+
+        def read(self, size=None):
+            data = '' if self.at_end else self.data
+            self.at_end = True
+            return data
+
+        def seek(self, offset):
+            self.at_end = False
+
+        def seekable(self):
+            return True
+
+    good = MockFile('<table><tr><td>spam<br />eggs</td></tr></table>')
+    bad = MockFile('<table><tr><td>spam<foobr />eggs</td></tr></table>')
+
+    assert read_html(good)
+    assert read_html(bad)