ENH: GH17054: read_html() handles rowspan/colspan and infers headers
State: closed

 - [x] closes #17054
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@gfyoung can you help me reconcile failing tests in the CI checks vs. clean tests on my machine?

```
$ py.test-2.7 pandas/tests/io/test_html.py
============================= test session starts ==============================
platform darwin -- Python 2.7.13, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /Users/jowens/Documents/src/pandas-jowens, inifile: setup.cfg
collected 76 items

pandas/tests/io/test_html.py ............................................................................

========================== 76 passed in 49.13 seconds ==========================
```
@jowens : Judging from your Travis build output, here's what I see:

* You have a style-check error on the Python 2.7 machine.  However, all tests are passing.  That indicates your patch is working for Python 2.7.  You should patch this first to get that build working at least.

* Your changes are failing for Python 3.x, meaning that they are perhaps not compatible with Python 3.  I would suggest that you create a separate environment using Python 3 as your interpreter (I see you are using Python 2), install your branch, and rerun tests to see if you can replicate the failures.

If you can, then continue developing in the Python 3.x environment to get tests passing.  If you can't replicate, then that's going to be tricky to debug, but we can address that *if* we get there.
Can you point me to the "style-check error"?
The error can be found <a href="https://travis-ci.org/pandas-dev/pandas/jobs/258193317#L1818">here</a>.  BTW, you can also see the error yourself by running the style-check command in the PR description:

~~~
git diff upstream/master -u -- "*.py" | flake8 --diff
~~~
Yeah, I did that before checking in:

```
$ git diff upstream/master -u -- "*.py" | flake8-2.7 --diff
$
```

Not sure what went wrong. I'm on it!
Cool! @gfyoung what's next?
@gfyoung any chance of moving this forward? I'd prefer not to have a private build of Pandas for the rest of my life.
@jowens : Sorry about that!  My inbox is very full most of the time, so I think I just missed your comment about a month ago.

Let's do this:
* Rebase onto `master` (and fix the merge conflict) to see if tests still pass
* Ping @TomAugspurger for further review of the PR
@gfyoung FWIW the merge conflict is only "whatsnew"
@gfyoung because I *really* don't want to get this wrong and have a commit that's gonna be enormous, can you sketch out how I "rebase onto master" properly?
@jowens: True, but it's been a month since you've run the tests on Travis, Appveyor, etc.  I just want to make sure that tests still pass.
> @gfyoung because I really don't want to get this wrong and have a commit that's gonna be enormous, can you sketch out how I "rebase onto master" properly?

I would recommend to try using GitHub's UI editor for resolving the merge conflict.  If that doesn't work out, I can give you further suggestions, but they will be all local (and probably) less visual.
Don't worry about rebasing onto master yet.  Let's make sure this PR is merge-able first.
@gfyoung OK, looks like it's running and conflict-free. Stay tuned!
@TomAugspurger can you help me take the next step with this PR? Thanks @gfyoung!
I won't have a chance to review this in detail for a while, so if you think
it's good then feel free to merge @gfyoung.

On Tue, Aug 29, 2017 at 3:14 PM, John Owens <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/tomaugspurger> can you help me take
> the next step with this PR? Thanks @gfyoung <https://github.com/gfyoung>!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/17089#issuecomment-325787329>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIumSd6yu0bcha3VfFOCHko-9Fx8Aks5sdHErgaJpZM4Okn1p>
> .
>

@TomAugspurger @gfyoung could either one of you guys please take a look?
OK @gfyoung next step?
Hello @jowens! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on September 20, 2017 at 21:49 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=h1) Report
> Merging [#17089](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d4577911c750f2f48f760ce451d413116bed72da?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `94.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17089/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17089      +/-   ##
==========================================
- Coverage   91.15%   91.13%   -0.03%     
==========================================
  Files         163      163              
  Lines       49587    49623      +36     
==========================================
+ Hits        45203    45222      +19     
- Misses       4384     4401      +17
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.91% <94.5%> (-0.01%)` | :arrow_down: |
| #single | `40.23% <34.06%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `83.42% <94.5%> (-1.43%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.39% <0%> (-0.07%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `91.98% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.52% <0%> (+0.09%)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `96.88% <0%> (+0.22%)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `92.59% <0%> (+0.4%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=footer). Last update [d457791...39f7814](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=h1) Report
> Merging [#17089](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d4577911c750f2f48f760ce451d413116bed72da?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `94.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17089/graphs/tree.svg?width=650&src=pr&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17089      +/-   ##
==========================================
+ Coverage   91.15%   91.16%   +<.01%     
==========================================
  Files         163      163              
  Lines       49587    49658      +71     
==========================================
+ Hits        45203    45270      +67     
- Misses       4384     4388       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.95% <94.5%> (+0.02%)` | :arrow_up: |
| #single | `40.18% <34.06%> (-0.14%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `83.42% <94.5%> (-1.43%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/plotting/\_tools.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL190b29scy5weQ==) | `72.92% <0%> (-6.08%)` | :arrow_down: |
| [pandas/tseries/frequencies.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvZnJlcXVlbmNpZXMucHk=) | `96.11% <0%> (-0.4%)` | :arrow_down: |
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `82.52% <0%> (-0.17%)` | :arrow_down: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `97% <0%> (-0.15%)` | :arrow_down: |
| [pandas/core/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FycmF5LnB5) | `91.3% <0%> (-0.12%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.39% <0%> (-0.07%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.92% <0%> (-0.03%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=footer). Last update [d457791...092889a](https://codecov.io/gh/pandas-dev/pandas/pull/17089?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hit me with more changes, @gfyoung!
@gfyoung @TomAugspurger Next steps?
@jowens : Sorry for not looking at this sooner.  I made some additional comments, but it's definitely well on its way to getting merged.  Feel free to ping us again as soon as you finish addressing my comments.
@gfyoung I expect no issues with the build since it's documentation-changes-only. FWIW these are documentation issues that predate any of my changes (but I don't mind cleaning them up).
@jowens : Understood.  I suggested them because those docs are showing up in your diff, so they're fair game as far as I'm concerned. :smile: 
@gfyoung Any idea why Travis CI failed here? It passed before and all I changed recently was documentation.
@jowens : This was handled in in #17588.  Rebase on `master` and push again.
@TomAugspurger Next steps?
@TomAugspurger @gfyoung What can I do to move this forward?