Read html tables into DataFrames
State: closed

This PR adds new functionality for reading HTML tables from a URI, string, or file-like object into a DataFrame.
#3369


Labels:

Comments:
@y-p Ready for those notes whenever you are.

First of all, thanks for all the work. much appreciated especially being so thorough.
Here's some points, mostly I guess for consistency in API, we can discuss
what works best. I looked at the code a couple of days ago, my apologies if 
some points are now stale:
- [x] 1 `io` arg might be a url, in which case, should do the right thing. pd.read_csv does this
  ( I voted this down originally and I was wrong). see helper in `io.parsers._read` for free lunch.
- [x] 2- i agree with @jreback that you should return a list of dataframes, and my vote goes for 
  returning a singleton list even when a match is specified/only one table. I see this
  as `pd.read_html()`. The user is then free to index the returned list so I suggest jetissoning
  `table_number`.
- [x] 3- the docstring note on "expect some cleanup" is somewhat opaque. what does it mean?
- [x] 4 `df.applymap` currently bombs on duplicate labels (#2786), need to workaround for now.
- [x] 5_get_skiprows_iter is not actually used anywhere
- [x] 6 not clear what other legal values for `flavor` are.
- [x] 7- dedicating all kwargs to be passed off somewhere is a back-compat killer,
  since if we need to change something later on we can't add anything to the signature,
  make it a named dict arg instead (perhaps yanked by name from **kwargs).
  Also, doesn't make sense if you're using `lxml` directly. so if the user specified them
  but uses a diff engine, should probably raise in that case.
- [x] 8 -name bikeshedding:  s/convert/`infer_types` ?

1) urls are now handled properly, insofar as lxml can handle them (lxml only handles file, http, and ftp protocols) for bs4 there is a little bit of kludge around inferring the type of the input since bs4 doesn't handle it as transparently as lxml
2) TODO
3) TODO
4) not used anymore
5) used in _rows_to_frame
6) flavor is gone because there's no reason to use bs4 if you have lxml (that might be too opinionated :)
7) TODO
8) TODO

One thing that I realized is glaringly obvious is that I haven't done thorough testing using bs4, need to remove the ci line that installs lxml and run travis again.

1) the referenced `read_` wraps everything up into a .read()-able object, and abstracts that
away from teh underlying parse.

Are you aware that you can use detox/tox_prll.sh and `scripts/use_build_cache.py` to 
do all the testing locally and usually much faster?

Oh ok thanks. Didn't know that. Great.

see #3156.

read the instructions at the top of scripts/use_build_cache.py`(create dir, export envar)
run`scripts/use_build_cache.py` which will monkey patch setup.py, then run tox/tox_prll.sh/detox
to launch the build, second time around you only pay for running the tests.
runs py2.6/7/3.2/3.3.

I think I will print that out as a poster and frame it, it's so helpful.

The **kwargs in the case of lxml builds an expression that does what bs4 would do in that case see `_build_node_xpath_expr`. Is that a good idea? It's sort of a hack, but I thought that was a "nice-to-have" from bs4. Basically, `[@attr1='value1' and @attr2='value2' and ... and @attrN='valueN']` is what is built.

It's fine, it should be a dict that's passed in, and not take over the entire kwargs by definition,
because that paints us into a corner wrt future changes. Just a general rule.

so not:

```
def a(foo,**kwargs):
   my_delegate(**kwargs)
```

but

```
def a(foo,**kwargs):
   dkwds = kwargs.get("lxml_kwds",{})
   my_delegate(**dkwds)
```

@y-p Maybe I'm being a bit thick here, but `_read` tries to parse a delimited text file and thus fails when I pass in a url/string/HTML blob. Is there something I'm missing? It doesn't return a `read`able object it actually tries to do the reading.

Yeah, my bad, TextFileReader is actually a line-oriented, delimited file reader,
not a an IO source abstraction, sorry.

Oh wow, tox where have you been all my life :D

You've obviously spent considerable time on this and it shows: clean code, great docstrings. very nice.

in the "spam" example, I see that bs4 does a much better jobs then lxml in extracting
text from nested tags, while lxml's ".text" doesn't  go the extra mile.
I actually think the `flavor` argument was a good choice, since getting better data back trumps 
speed for many website scraping situations. It's great that you did both, so might as well let the 
user enjoy whichever solves his/her problem best.

Also, have you checked the license on all the HTML pages you included in the PR? 
Perhaps it's best to just leave everything as a @network test.

@y-p Cool. Thanks for the complements. Will add `flavor` back in. Also, tidying up the docstrings a bit, some of them are slightly inaccurate now. And yep, while bs4 is slower than lxml it's more intuitive and generally gives back "what you want". I also wanted the challenge of learning some xpath along the way so I thought why not do both?

[spam data set license](http://www.ars.usda.gov/Main/docs.htm?docid=6233#copyright)

failed banklist data set is [automagically part of the public domain](http://www.copyright.gov/title17/92chap1.html#105)

Anyway both are public domain, and I've only included those in the tests that are not `@network` so they're okay.

Build is failing because of failed HDFStore test. I haven't touched that code. Investigating...

This is especially strange since I don't get the failure locally (I've rebased on upstream/master)

I have seen an occasional failure (not on this particular one), but can never reproduce them....prob some wonkiness with travis..

just rebase (squash some commits together in any event), and try again (btw...add PTF in your commit message to make it work faster)

ok. just did `git commit --amend -C HEAD` to create a new hash a la @y-p's tip in #3156. @jreback What is PTF?

Program temporary fix?

Please Travis Faster....a y-p "hack" to make your whole testing run in about 4 min (it uses cached builds)....but have to put in a particular commit

ah ok. thanks.

I think I'm done twiddling here. Sorry about that, just wanted to be totally thorough.

no problem, thanks for addressing all the issues. marking for merge in  0.11.1.

need to tweak docstring to mention io accepts urls, suggest you make `bs4` the default for better results
out of the box and easier install, I think the typical use case for prasing HTML values quality
over speed.

calling with the [spam](http://ndb.nal.usda.gov/ndb/foods/show/1732?fg=&man=&lfacet=&format=&count=&max=25&offset=&sort=&qlookup=spam) url now works with lxml but fails with `flavor=bs4`.

@y-p can u post the error you're getting? The tests run successfully for both flavors over here. 

Also the io docstring mentions that it accepts urls. Should I add more detail? Would be happy to do that.