read_html: Handle colspan and rowspan
State: closed

This is essentially a rebased and squashed #17054 (mad props to @jowens
for doing all the hard thinking). My tweaks:

* test_computer_sales_page (see #17074) no longer tests for ParserError,
  because the ParserError was a bug caused by missing colspan support.
  Now, test that MultiIndex works as expected.
* I respectfully removed the fill_rowspan argument from #17073. Instead,
  the virtual cells created by rowspan/colspan are always copies of the
  real cells' text. This prevents _infer_columns() from naming virtual
  cells as "Unnamed: ..."
* I removed a small layer of abstraction to respect #20891 (multiple
  tbody support), which was implemented after @jowens' pull request.
  Now _HtmlFrameParser has _parse_thead_trs, _parse_tbody_trs and
  _parse_tfoot_trs, each returning a list of trs. That let me remove
  _parse_tr, Making All The Tests Pass.
* That caused a snowball effect. lxml does not fix malformed thead, as
  tested by spam.html. The previous hacky workaround was in
  _parse_raw_thead, but the new _parse_thead_trs signature returns nodes
  instead of text. The new hacky solution: return the thead itself,
  pretending it's a tr. This works in all the tests. A better solution
  is to use html5lib with lxml; but that might belong in a separate pull
  request.

- [x] closes #17054
- [x] closes #21641
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Hello @adamhooper! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on July 05, 2018 at 17:45 Hours UTC
Er, I mean, a rebased and squashed #17089.
You're a hero, @adamhooper.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=h1) Report
> Merging [#21487](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/62a0ebc5837a4a1f7cc925e13344be1adf1acc59?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `97.89%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21487/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21487      +/-   ##
==========================================
+ Coverage   91.92%   91.93%   +<.01%     
==========================================
  Files         158      158              
  Lines       49705    49716      +11     
==========================================
+ Hits        45693    45704      +11     
  Misses       4012     4012
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.3% <97.89%> (ø)` | :arrow_up: |
| #single | `41.97% <18.94%> (-0.03%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/21487/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `89.17% <97.89%> (+0.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=footer). Last update [62a0ebc...5fd863b](https://codecov.io/gh/pandas-dev/pandas/pull/21487?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback Thanks for the review! There was already an entry in whatsnew, but I rephrased it such that users will better be able to predict how the change will affect their data. Do you think that's enough, or should I add a subsection?
@jreback, @WillAyd I've implemented all the changes you requested. Could you please take a look?

@WillAyd I tested out an alternative algorithm, as you had concerns about the one in this pull request. The alternative is at https://github.com/adamhooper/pandas/commit/67dea69f604251d6de8185c8dd8c82f15eedfa2a. The diff is +53, -38. Maybe you find it more readable, though?
Need to clean up merge conflicts and maybe add whatsnew reference for proper indexing of tables with NA data. Otherwise lgtm
Urgh. I did a rebase on upstream/master and pushed with `--force` and now Appveyor is complaining. Was that a bad idea on my part? How should I fix?
I don't think the AppVeyor issue is a result of your rebase. That said, you'd be better off fetching master and merging it into your branch rather than rebase (maintains better comment history).

Looks like you have some file conflicts still anyway, so might was well resolve them locally and try merging from master into your branch this time before next push
Indeed -- I rebased over adamhooper/master instead of pandas/master. I've learned my lesson: merge, don't rebase.

Fixed by ... er ... rebasing. Correctly, this time.
^^ I've noticed a few other pull requests cause ResourceWarning failures on Travis. Does that error relate to this pull request?
Unrelated - that intermittently seems to pop up on Travis
@adamhooper can you fix merge conflict?

@jreback thoughts on your end?
Merged :)
@jreback Thanks -- added whatsnew entry.

Excited to see: is this last commit the final one?
thanks @adamhooper very nice!
Woo-hoo! Thank you all for your guidance. What a thrill :).
Hello, when will this be included into a release? I see it's being included in v0.24, but when will that happen? I'm having this exact problem and would love to benefit from this fix. Cheers!