REF/BUG/ENH/API: refactor read_html to use TextParser
State: closed

closes #4697 (refactor issue) (REF/ENH)
closes #4700 (header inconsistency issue) (API)
closes #5029 (comma issue, added this data set, ordering issue) (BUG)
closes #5048 (header type conversion issue) (BUG)
closes #5066 (index_col issue) (BUG)
- [x] figure out `skiprows`, `header`, and `index_col` interaction (a somewhat longstanding `MultiIndex` sorting issue, I just took the long way to get there :))
- ~~spam url not working anymore~~ (US gov "shutdown" is responsible for this, it correctly skips)
- ~~table ordering doc blurb/HTML gotchas~~ (was an actual "bug", now fixed in this PR)
- ~~add tests for rows with a different length~~ (this is already done by the existing tests)


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
:+1: 

yep looks good

also closes #4697 

@cancan101  yep thanks. added

@cpcloud Does this PR actually close #4679 ? That one is specific to Excel.

@jtratner Nope it doesn't, I think I must've mixed up 79 and 97. Thanks

this diff is very difficult to read ... sigh

a 5-fir!

I think you are trying to bump your stats with html files! lol

@jreback yep 

~~last issue is for me to make sure that `codec.open`  with my chosen errors param is correct~~ not using it anymore ... sticking with our old pal `open`. would be nice to handle encoding decoding ... but that's for the future

@jreback nah ... i'm trying to avoid slowing down the test suite with a bunch of `@network` tests

@jreback @jtratner comments?

I did a first pass. I'd probably like to go over it again and see what I see, but I'm sure it's good.

As an aside, can you help me understand how ordering works for the outputted tables? My assumption is that the ordering is deterministic. Does it follow the order in the HTML data that's passed in? (i.e., if you found all the line numbers of `<table>` elements, the order of outputted tables would be the same as the order of line numbers [unless a table isn't parseable])

@jtratner Re: ordering, see: https://github.com/pydata/pandas/issues/5029#issuecomment-25331119 and the subsequent comments.

@cpcloud Maybe also add that note about table ordering to the html gotchas?

@jreback @jtratner any more comments?

look ok

on tupleize_cols your explanation is odd - no other functions have it as true (by default)

`read_html` returns MI columns/index by default, to keep that behav had to default to `False`.

yep...that is default with read_csv too! (in 0.13)

ah ok .. 

@cpcloud FWIW This seems to run for 2 hours with no end in sight:

```
rez = pd.read_html("http://www.sec.gov/Archives/edgar/data/47217/000104746913006802/a2215416z10-q.htm")
```

Probably a cycle in the HTML parse tree. Does copy-pasting just the table work?

Surprised that site works...

i think maybe a timeout parameter might be useful

I interrupted and this is the trace:

``` python
/home/alex/git/pandas/pandas/io/html.py in read_html(io, match, flavor, header, index_col, skiprows, infer_types, attrs, parse_dates, tupleize_cols, thousands)
    838                          'data (you passed a negative value)')
    839     return _parse(flavor, io, match, header, index_col, skiprows, infer_types,
--> 840                   parse_dates, tupleize_cols, thousands, attrs)

/home/alex/git/pandas/pandas/io/html.py in _parse(flavor, io, match, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands, attrs)
    700 
    701         try:
--> 702             tables = p.parse_tables()
    703         except Exception as caught:
    704             retained = caught

/home/alex/git/pandas/pandas/io/html.py in parse_tables(self)
    172 
    173     def parse_tables(self):
--> 174         tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
    175         return (self._build_table(table) for table in tables)
    176 

/home/alex/git/pandas/pandas/io/html.py in _parse_tables(self, doc, match, attrs)
    396     def _parse_tables(self, doc, match, attrs):
    397         element_name = self._strainer.name
--> 398         tables = doc.find_all(element_name, attrs=attrs)
    399 
    400         if not tables:

/usr/local/lib/python2.7/dist-packages/bs4/element.pyc in find_all(self, name, attrs, recursive, text, limit, **kwargs)
   1165         if not recursive:
   1166             generator = self.children
-> 1167         return self._find_all(name, attrs, text, limit, generator, **kwargs)
   1168     findAll = find_all       # BS3
   1169     findChildren = find_all  # BS2

/usr/local/lib/python2.7/dist-packages/bs4/element.pyc in _find_all(self, name, attrs, text, limit, generator, **kwargs)
    483             # Optimization to find all tags with a given name.
    484             elif isinstance(name, basestring):
--> 485                 return [element for element in generator
    486                         if isinstance(element, Tag) and element.name == name]
    487             else:

/usr/local/lib/python2.7/dist-packages/bs4/element.pyc in descendants(self)
   1182         current = self.contents[0]
   1183         while current is not stopNode:
-> 1184             yield current
   1185             current = current.next_element
   1186 
```

Yep ... @jseabold had a similar issue ... let me find it

essentially the borked html is in a cycle, in this case a node goes to its child then when the current node (the child) goes to the next element, it's actually the previous node (its parent) and on and on ...

#4786 

Am I doing something wrong here:

```
pd.read_html("/home/alex/table.html",infer_types=False,header=[0])

---------------------------------------------------------------------------
TypeError                                 
----> 1 pd.read_html("/home/alex/table.html",infer_types=False,header=[0])

/home/alex/git/pandas/pandas/io/html.pyc in read_html(io, match, flavor, header, index_col, skiprows, infer_types, attrs, parse_dates, tupleize_cols, thousands)
    838                          'data (you passed a negative value)')
    839     return _parse(flavor, io, match, header, index_col, skiprows, infer_types,
--> 840                   parse_dates, tupleize_cols, thousands, attrs)

/home/alex/git/pandas/pandas/io/html.pyc in _parse(flavor, io, match, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands, attrs)
    710     return [_data_to_frame(table, header, index_col, skiprows, infer_types,
    711                            parse_dates, tupleize_cols, thousands)
--> 712             for table in tables]
    713 
    714 

/home/alex/git/pandas/pandas/io/html.pyc in _data_to_frame(data, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands)
    600                     skiprows=_get_skiprows(skiprows),
    601                     parse_dates=parse_dates, tupleize_cols=tupleize_cols,
--> 602                     thousands=thousands)
    603     df = tp.read()
    604 

/home/alex/git/pandas/pandas/io/parsers.pyc in TextParser(*args, **kwds)
   1173     """
   1174     kwds['engine'] = 'python'
-> 1175     return TextFileReader(*args, **kwds)
   1176 
   1177 

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, engine, **kwds)
    485             self.options['has_index_names'] = kwds['has_index_names']
    486 
--> 487         self._make_engine(self.engine)
    488 
    489     def _get_options_with_defaults(self, engine):

/home/alex/git/pandas/pandas/io/parsers.pyc in _make_engine(self, engine)
    601             elif engine == 'python-fwf':
    602                 klass = FixedWidthFieldParser
--> 603             self._engine = klass(self.f, **self.options)
    604 
    605     def _failover_to_python(self):

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, **kwds)
   1296         if len(self.columns) > 1:
   1297             self.columns, self.index_names, self.col_names, _ = self._extract_multi_indexer_columns(
-> 1298                 self.columns, self.index_names, self.col_names)
   1299         else:
   1300             self.columns = self.columns[0]

/home/alex/git/pandas/pandas/io/parsers.pyc in _extract_multi_indexer_columns(self, header, index_names, col_names, passed_names)
    736         # if we find 'Unnamed' all of a single level, then our header was too long
    737         for n in range(len(columns[0])):
--> 738             if all([ 'Unnamed' in c[n] for c in columns ]):
    739                 raise _parser.CParserError("Passed header=[%s] are too many rows for this "
    740                                            "multi_index of columns" % ','.join([ str(x) for x in self.header ]))

TypeError: argument of type 'float' is not iterable
```

This works fine:

```
pd.read_html("/home/alex/table.html",infer_types=False,header=0)
```

this is an issue with `TextParser`

if you need to pass only a single header just use the second version

it doesn't really make a whole lot of sense to pass a singleton list if you just want the first row anyway

only pass a list if you need more than 1 row as a `MultiIndex` header

i'll open an issue about `TextParser`

That might have been a poor example, I see this issue even with a proper list:

```
In [6]: pd.read_html("/home/alex/table.html",infer_types=False,header=[0,1])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-1b482817d4da> in <module>()
----> 1 pd.read_html("/home/alex/table.html",infer_types=False,header=[0,1])

/home/alex/git/pandas/pandas/io/html.pyc in read_html(io, match, flavor, header, index_col, skiprows, infer_types, attrs, parse_dates, tupleize_cols, thousands)
    838                          'data (you passed a negative value)')
    839     return _parse(flavor, io, match, header, index_col, skiprows, infer_types,
--> 840                   parse_dates, tupleize_cols, thousands, attrs)

/home/alex/git/pandas/pandas/io/html.pyc in _parse(flavor, io, match, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands, attrs)
    710     return [_data_to_frame(table, header, index_col, skiprows, infer_types,
    711                            parse_dates, tupleize_cols, thousands)
--> 712             for table in tables]
    713 
    714 

/home/alex/git/pandas/pandas/io/html.pyc in _data_to_frame(data, header, index_col, skiprows, infer_types, parse_dates, tupleize_cols, thousands)
    600                     skiprows=_get_skiprows(skiprows),
    601                     parse_dates=parse_dates, tupleize_cols=tupleize_cols,
--> 602                     thousands=thousands)
    603     df = tp.read()
    604 

/home/alex/git/pandas/pandas/io/parsers.pyc in TextParser(*args, **kwds)
   1173     """
   1174     kwds['engine'] = 'python'
-> 1175     return TextFileReader(*args, **kwds)
   1176 
   1177 

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, engine, **kwds)
    485             self.options['has_index_names'] = kwds['has_index_names']
    486 
--> 487         self._make_engine(self.engine)
    488 
    489     def _get_options_with_defaults(self, engine):

/home/alex/git/pandas/pandas/io/parsers.pyc in _make_engine(self, engine)
    601             elif engine == 'python-fwf':
    602                 klass = FixedWidthFieldParser
--> 603             self._engine = klass(self.f, **self.options)
    604 
    605     def _failover_to_python(self):

/home/alex/git/pandas/pandas/io/parsers.pyc in __init__(self, f, **kwds)
   1296         if len(self.columns) > 1:
   1297             self.columns, self.index_names, self.col_names, _ = self._extract_multi_indexer_columns(
-> 1298                 self.columns, self.index_names, self.col_names)
   1299         else:
   1300             self.columns = self.columns[0]

/home/alex/git/pandas/pandas/io/parsers.pyc in _extract_multi_indexer_columns(self, header, index_names, col_names, passed_names)
    736         # if we find 'Unnamed' all of a single level, then our header was too long
    737         for n in range(len(columns[0])):
--> 738             if all([ 'Unnamed' in c[n] for c in columns ]):
    739                 raise _parser.CParserError("Passed header=[%s] are too many rows for this "
    740                                            "multi_index of columns" % ','.join([ str(x) for x in self.header ]))

TypeError: argument of type 'float' is not iterable
```