Repository: pandas-dev/pandas

Issue:
read_html failing in many tests:  AssertionError: No tables found
typical error with 

```
======================================================================
FAIL: test_banklist (pandas.io.tests.test_html.TestReadHtmlBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 114, in test_banklist
    attrs={'id': 'table'})
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 67, in run_read_html
    return read_html(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/html.py", line 900, in read_html
    attrs)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/html.py", line 769, in _parse
    raise retained
AssertionError: No tables found
```



Discussion:
I guess another related failure

```
======================================================================
FAIL: pandas.io.tests.test_html.test_bs4_finds_tables
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.11.0+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_html.py", line 453, in test_bs4_finds_tables
    assert get_elements_from_url(filepath, 'table')
AssertionError
```

can you show ci/print_versions.py?

Here is what I have ATM (without running the build with adjusted
matplotlib backend and PYTHONPATH... which shouldn't matter -- just want
to make clear if I am not missing anything):

$> ci/print_versions.py 

## INSTALLED VERSIONS

Python: 2.7.5.final.0
OS: Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64
LC_ALL: None
LANG: en_US

Cython: 0.19
Numpy: 1.7.1
Scipy: 0.10.1
statsmodels: 0.4.2
    patsy: Not installed
scikits.timeseries: Not installed
dateutil: 1.5
pytz: 2012c
bottleneck: Not installed
PyTables: 2.3.1
    numexpr: 2.0.1
matplotlib: 1.1.1rc2
openpyxl: 1.6.1
xlrd: 0.6.1
xlwt: 0.7.4
sqlalchemy: 0.7.9
lxml: 3.2.0
bs4: 4.2.0
html5lib: 0.95-dev

On Thu, 11 Jul 2013, jreback wrote:

>    can you show ci/print_versions.py?
> 
>    —
>    Reply to this email directly or [1]view it on GitHub.
> 
> References
> 
>    Visible links
> 1. https://github.com/pydata/pandas/issues/4214#issuecomment-20822890

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

@cpcloud ?

@yarikoptic this is master correct?

7097368 should have fixed these issues

it looks like you're using an older version of html5lib. i believe 1.0b2 is out.

also, check out the [reading html gotchas](http://pandas.pydata.org/pandas-docs/dev/gotchas.html#html-table-parsing)

yes -- master 0.11.0+git43-g7b2eaa4

thanks for the fix -- I will check it out

html5lib -- well, I have the freshiest non-beta release ;)  thanks for the note though!

i'll try this on a vagrant box

the commit i referenced above is already in master

@yarikoptic is it of any significance that your version says 0.11.0?

``` sh
$ git describe
v0.12.0rc1-43-g7b2eaa4
```

@yarikoptic there are no guarantees that your `bs4` version will work...

see the [optional dependencies docs](http://pandas.pydata.org/pandas-docs/dev/install.html#optional-dependencies)

also your `lxml` version _might_ need to be upgraded.

i would say: leave html5lib alone for now that should be fine, but change your `bs4` from `4.2.0` to either `4.2.1`, `4.1.3` or `4.0.2`. then run tests. if `lxml` still fails upgrade to `3.2.1`

if none of this works then i'll mark it as a bug

there's a release of `bs4` that really shouldn't have existed, i believe it is `4.2.0`.

fyi travis installs `bs4` 4.0.2 and that works...i'm not sure how different ubuntu 12.04 is from debian sid, but i imagine they share a lot of similarity, i'm sure you know _much_ more than i about this

It think @cpcloud should setup a version dependency hotline, kind of like AA :)

Hello, this is `read_html`-one-one, what's your emergency?

ah -- thanks -- should have been indeed 0.12.0~rc1+git... as the Debian
perspective version -- fixed in my debian branch.

On Thu, 11 Jul 2013, Phillip Cloud wrote:

>    [1]@yarikoptic is it of any significance that your version says 0.11.0?
> 
>  $ git describe
>  v0.12.0rc1-43-g7b2eaa4

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

excellent!

does that mean `bs4==4.2.0` works for you?

yikes -- good to know... well at least in current Debian stable we have
4.1.0 -- is that good enough? ;-)

On Thu, 11 Jul 2013, Phillip Cloud wrote:

>    there's a release of bs4 that really shouldn't have existed, i believe it
>    is 4.2.0.

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

that should be ok, as long as it passes for you. does this also fix #4212? if so please close both, thanks for the report :smile: 

also the other issues you've raised today might be fixed now that your version is correct

sorry -- I guess my mental pipelining broke -- what should have fixed my issue? downgrade of bs4 to < 4.2.0?
that commit mentioned https://github.com/pydata/pandas/commit/7097368  has been in master for a while so was already included in the version I have built -- or am I wrong?

The bs4 downgrade should be the fix. 

ok -- will give it a try.  but as for the reports correction it would be better if pandas internally switched between backends/skipped the tests if broken bs4 is present; since otherwise it becomes impractical for me to build them across all debian/ubuntu releases having the broken bs4

btw -- what about bs4 4.2.1 -- should that be "good enough"?

Re bs4 skip good idea. I'll add a warning to the code as well since iirc 4.0.2 worked fine on my arch Linux box.

4.2.1 should be fine. 

I mean 4.2.0 worked fine. 

... so bs4 4.2.0 is not at fault? (anyways -- rebuilding/testing now with 4.2.1 installed)

no bs4 4.2.0 _is_ at fault

then what "I mean 4.2.0 worked fine. " is about? ;) or it is at fault only in some deployments?

works fine on my arch linux machine, but fails on debian distros, not sure exactly why

let me check that though...

wow -- with 4.2.1 only #4215 is left failing:

# 

## FAIL: test_invalid_colormap (pandas.tests.test_graphics.TestDataFrameGroupByPlots)

Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/build-area/pandas-0.12.0~rc1+git43-g7b2eaa4/debian/tmp/usr/lib/python2.7/dist-packages/pandas/tests/test_graphics.py", line 995, in test_invalid_colormap
    self.assertRaises(ValueError, df.plot, colormap='invalid_colormap')
AssertionError: ValueError not raised

---

Ran 3622 tests in 743.502s

FAILED (SKIP=83, failures=1)

On Thu, 11 Jul 2013, Phillip Cloud wrote:

>    let me check that though...

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

that good news! not sure where that is coming from...i'll see what i can do about it

@cpcloud ok....you are going to add a skip test / warning message ? if 4.2.0 is installed (maybe just raise in `read_html`...hey user your 4.2.0 is broken, this won't work.....

i'm going to actually raise if no tables found and `bs4==4.2.0`, since there's nothing else to fallback on after that. i've cooked up a fairly informative error message

hard to test for this...we don't have an extensive list of OSes that work with bs4


Resolving patch:
diff --git a/pandas/io/html.py b/pandas/io/html.py
--- a/pandas/io/html.py
+++ b/pandas/io/html.py
@@ -8,23 +8,18 @@
 import numbers
 import urllib2
 import urlparse
-import contextlib
 import collections
 
-
-try:
-    from importlib import import_module
-except ImportError:
-    import_module = __import__
+from distutils.version import LooseVersion
 
 import numpy as np
 
 from pandas import DataFrame, MultiIndex, isnull
-from pandas.io.common import _is_url
+from pandas.io.common import _is_url, urlopen
 
 
 try:
-    import_module('bs4')
+    import bs4
 except ImportError:
     _HAS_BS4 = False
 else:
@@ -32,7 +27,7 @@
 
 
 try:
-    import_module('lxml')
+    import lxml
 except ImportError:
     _HAS_LXML = False
 else:
@@ -40,7 +35,7 @@
 
 
 try:
-    import_module('html5lib')
+    import html5lib
 except ImportError:
     _HAS_HTML5LIB = False
 else:
@@ -119,7 +114,7 @@ def _read(io):
     """
     if _is_url(io):
         try:
-            with contextlib.closing(urllib2.urlopen(io)) as url:
+            with urlopen(io) as url:
                 raw_text = url.read()
         except urllib2.URLError:
             raise ValueError('Invalid URL: "{0}"'.format(io))
@@ -131,7 +126,8 @@ def _read(io):
     elif isinstance(io, basestring):
         raw_text = io
     else:
-        raise ValueError("Cannot read object of type '{0}'".format(type(io)))
+        raise TypeError("Cannot read object of type "
+                        "'{0.__class__.__name__!r}'".format(io))
     return raw_text
 
 
@@ -414,6 +410,7 @@ def _parse_tables(self, doc, match, attrs):
         element_name = self._strainer.name
         tables = doc.find_all(element_name, attrs=attrs)
         if not tables:
+            # known sporadically working release
             raise AssertionError('No tables found')
 
         mts = [table.find(text=match) for table in tables]
@@ -429,7 +426,8 @@ def _parse_tables(self, doc, match, attrs):
     def _setup_build_doc(self):
         raw_text = _read(self.io)
         if not raw_text:
-            raise AssertionError('No text parsed from document')
+            raise AssertionError('No text parsed from document: '
+                                 '{0}'.format(self.io))
         return raw_text
 
     def _build_doc(self):
@@ -721,6 +719,14 @@ def _parser_dispatch(flavor):
             raise ImportError("html5lib not found please install it")
         if not _HAS_BS4:
             raise ImportError("bs4 not found please install it")
+        if bs4.__version__ == LooseVersion('4.2.0'):
+            raise AssertionError("You're using a version"
+                                 " of BeautifulSoup4 (4.2.0) that has been"
+                                 " known to cause problems on certain"
+                                 " operating systems such as Debian. "
+                                 "Please install a version of"
+                                 " BeautifulSoup4 != 4.2.0, both earlier"
+                                 " and later releases will work.")
     else:
         if not _HAS_LXML:
             raise ImportError("lxml not found please install it")


Test patch:
diff --git a/pandas/io/tests/test_html.py b/pandas/io/tests/test_html.py
--- a/pandas/io/tests/test_html.py
+++ b/pandas/io/tests/test_html.py
@@ -2,18 +2,24 @@
 import re
 from cStringIO import StringIO
 from unittest import TestCase
-from urllib2 import urlopen
-from contextlib import closing
 import warnings
+from distutils.version import LooseVersion
 
 import nose
+from nose.tools import assert_raises
 
 import numpy as np
 from numpy.random import rand
 from numpy.testing.decorators import slow
 
-from pandas.io.html import read_html, import_module
-from pandas.io.html import _remove_whitespace
+try:
+    from importlib import import_module
+except ImportError:
+    import_module = __import__
+
+from pandas.io.html import read_html
+from pandas.io.common import urlopen
+
 from pandas import DataFrame, MultiIndex, read_csv, Timestamp
 from pandas.util.testing import (assert_frame_equal, network,
                                  get_data_path)
@@ -60,14 +66,26 @@ def assert_framelist_equal(list1, list2, *args, **kwargs):
         assert not frame_i.empty, 'frames are both empty'
 
 
+def test_bs4_version_fails():
+    _skip_if_no('bs4')
+    import bs4
+    if bs4.__version__ == LooseVersion('4.2.0'):
+        assert_raises(AssertionError, read_html, os.path.join(DATA_PATH,
+                                                              "spam.html"),
+                      flavor='bs4')
+
+
 class TestReadHtmlBase(TestCase):
     def run_read_html(self, *args, **kwargs):
-        self.try_skip()
         kwargs['flavor'] = kwargs.get('flavor', self.flavor)
         return read_html(*args, **kwargs)
 
     def try_skip(self):
         _skip_if_none_of(('bs4', 'html5lib'))
+        import bs4
+        if (bs4.__version__ == LooseVersion('4.2.0') and
+            self.flavor != ['lxml']):
+            raise nose.SkipTest
 
     def setup_data(self):
         self.spam_data = os.path.join(DATA_PATH, 'spam.html')
@@ -77,6 +95,7 @@ def setup_flavor(self):
         self.flavor = 'bs4'
 
     def setUp(self):
+        self.try_skip()
         self.setup_data()
         self.setup_flavor()
 
@@ -347,6 +366,7 @@ def test_pythonxy_plugins_table(self):
 
     @slow
     def test_banklist_header(self):
+        from pandas.io.html import _remove_whitespace
         def try_remove_ws(x):
             try:
                 return _remove_whitespace(x)
@@ -438,10 +458,9 @@ def test_invalid_flavor():
 def get_elements_from_url(url, element='table', base_url="file://"):
     _skip_if_none_of(('bs4', 'html5lib'))
     url = "".join([base_url, url])
-    from bs4 import BeautifulSoup, SoupStrainer
-    strainer = SoupStrainer(element)
-    with closing(urlopen(url)) as f:
-        soup = BeautifulSoup(f, features='html5lib', parse_only=strainer)
+    from bs4 import BeautifulSoup
+    with urlopen(url) as f:
+        soup = BeautifulSoup(f, features='html5lib')
     return soup.find_all(element)