DOC: document bs4/lxml/html5lib issues
State: closed

to summarize this mess of deps and check my thoughts here
### if a user insists on using `lxml` (either with or without `bs4`)
- warning about its inability to deal with the modern web
- warning saying that the user should install `html5lib` and `bs4` so that a page will parse even if `lxml` barfs
- test coverage for failing and passing pages (things that would parse "correctly" before will now fail since the parser will be extremely strict) thus only pages validated by the DTD will even try to parse

(to be fair I was really enthusiastic about `lxml` because of how fast it is but now i'm sort of against it)
### what users should really do
- install `bs4`
- install `html5lib`
- happily parse things into `DataFrame`s with a low amount of stress
### anaconda + `lxml` (no `bs4`)
- no problems (modulo the above warnings)

@wesm maybe you could chime in about what (if anything) you did to `libxml2`/`libxslt` i wasn't clear on the details from the mailing list.
### anaconda + `bs4` + `lxml`
- make sure that you're using `bs4==4.2.1`
- make sure that you're using `lxml==3.2.1`
- workout the details of how to do this with `conda` (i did this already, but it was 2 or 3 AM so I'm a little foggy on the details)
### anaconda + `bs4` + `html5lib` (no `lxml`)
- happy parsing of HTML tables

this will be in a gotcha that will be linked to from a warning at the top of the read html section of io.rst


Labels:

Comments:
this looks good...are you going to add something about the valid installations (and invalid ones that we know about)? (or is that already in install?)

Yeah I just wanted to put it up to remind myself. Should be done by
tomorrow.
On Jun 3, 2013 8:49 PM, "jreback" notifications@github.com wrote:

> this looks good...are you going to add something about the valid
> installations (and invalid ones that we know about)? (or is that already in
> install?)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3751#issuecomment-18882997
> .

I'll put a link to the read HTML gotchas in the install and a mention of the anaconda issues, so that I can give a bit more detail in the gotcha. 

just let me know when you want to merge this

this warning is going to be gigantic. i hope this doesn't discourage people from trying this :(

hahah

they would try ti w/o the warning then go WTF!

![huge-warning](https://f.cloud.github.com/assets/417981/606990/8c6b1810-cd38-11e2-86e0-82f2988dbaa5.png)

I like it! +1

also update README.rst (unfort the warnings don't look quite as good there)

ah yes. thanks.

now to write a tome about anaconda...
- [x] list install proc for apt-get enabled systems

should probably wait 2 merge this until i've actually implemented the mentioned fallback to html5lib if lxml fails to parse

html parsing is amazing complicated (because the standard is not followed)

web sites should be REQUIRED to post csv of there table data!

when i first starting doing this i was like: "i'm going to implement [this](http://www.w3.org/TR/html5/tabular-data.html#table-model) in CYTHON". um yeah way too optimistic.

majority of tables are not semantic they are used for _formatting_ UGH

what i really need now is to operate my 4 different vagrant boxes thru tmux all at the same time so i can test this beast without repeating myself all the time!

you ARE a maniac!

i should have the fallback by tonight...or some time early in the am

shall i squash here?

yes....I almost always squash down to a few commits

if you want I can merge this with say 1/2 hour...that way it will be in today's docs.....up 2 u

when is the doc build? i'm about to participate in an eye movement experiment that will take 1/2 hour so i can't have these changes for another 45 min

make sure your eye are open!

i think sometime between 4-5 pm (changes show up after 5 est)

its no biggie.....can do whenever...when i make a big doc change i like to see it on the main site to read over again.....

it's there if u want to merge

thansk!

@jreback what do you think about removing bs4 + lxml support and have either lxml alone, or bs4 + html5lib. the point being why go through bs4 if you don't have to, if lxml can parse the page it should do it in the fastest way possible and going thru bs4 makes no sense. too much redundancy

more doc changes, but i don't mind making them shorter, and a whole lot of extra code surrounding imports and other cruft in `html.py` can be removed if i do this

@cpcloud docs are updated (small issue on Read HTML with attributes ?)

I thought you do lxml, then fallback to bs4 + html5lib ? only, right?