ENH: Adding additional keywords to read_html for #13461
State: closed

- [x] closes #13461
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@gte620v The suggestion in #10534 was to add `dtype` argument (although the converters keyword will be able to achieve a similar goal). Can you have a look if it is easy to pass through that keyword as well?

In the same line, I think it would be useful to at once evaluate the other keywords to see which ones could be useful for html parsing.

> Can you have a look if it is easy to pass through that keyword as well?

My initial impression is that dtype can not be easily dropped in since it is not supported by the python engine parser: https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L463-L470

For reasons I haven't poked around enough to understand, the `read_html` function uses the `TextParser` function which uses the python engine: https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1600-L1601

> In the same line, I think it would be useful to at once evaluate the other keywords to see which ones could be useful for html parsing. 

Sure, I'll take a look.  It seems a bit cumbersome having to copy over args in 3 or 4 places to add them to `read_html`.  I poke through the code to see if there is a better way to add all applicable args.  Even without finding a better way to do things, it should be straightforward to use all of the `TextParser` args.  

Please let me know if you have any suggestions.

It would be easiest to just pass around kwargs from `TextParser` in https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1546-L1601

Is there an existing better way to do that other than explicitly calling out the arguments in each of these places?  If not, I can probably work about a better way to do it.  Let me know if you have any thoughts.
- https://github.com/pydata/pandas/blob/0c6226cbbc319ec22cf4c957bdcc055eaa7aea99/pandas/io/html.py#L719-L721
- https://github.com/pydata/pandas/blob/0c6226cbbc319ec22cf4c957bdcc055eaa7aea99/pandas/io/html.py#L743-L750
- https://github.com/pydata/pandas/blob/0c6226cbbc319ec22cf4c957bdcc055eaa7aea99/pandas/io/html.py#L757-L760

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13575?src=pr) is **84.53%**

> Merging [#13575](https://codecov.io/gh/pydata/pandas/pull/13575?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.21%**

``` diff
@@             master     #13575   diff @@
==========================================
  Files           138        141     +3   
  Lines         51157      51147    -10   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43132      43235   +103   
+ Misses         8025       7912   -113   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [ba82b51...2abb473](https://codecov.io/gh/pydata/pandas/compare/ba82b511c76d87421c8900348efebe4577548ec6...2abb473a6728b9d7037afa8e78afc39115144d53?src=pr)

@jorisvandenbossche I added support for a few more arguments: `keep_default_na`, `squeeze`, and `date_parser`.  

I also consolidated the argument definitions from being called in several places to only being enumerated in the `read_html` function.  This should make it easier to add more arguments to that function in the future.

I could not easily add dtype support since read_html uses the python engine.

Let me know if you want me to make more changes.

@gte620v The way of passing through looks good! Thanks

Regarding the extra keywords, maybe we should evaluate each keyword on its usefulness to add (sorry if I was a bit fast in pushing you to add more keywords):
- `keep_default_nan` -> certainly good to add (needed if we also add `na_values`, which was the original issue)
- `squeeze` / `date_parser` -> those are maybe less essential (the same effect can be easily obtained after parsing), so is maybe not worth the additional clutter in the long list of possible arguments

OK regarding not adding `dtype` for now (it's indeed not supported with the python engine), ~~but can you then remove the note that it closed that issue?~~ -> changed the PR title, so OK

Sure, will do!

Let me know definitively about squeeze and date_parser.  I can leave or remove.

@jreback Opinion on adding `squeeze` and `date_parser` keywords to `to_html`? 
("it's easy to support" (just passing through to TextParser) vs "not essential keywords so only cluttering long list of possible keywords" (in contrast to the NaN handling keywords, the effect of `squeeze` and `date_parser` are easily obtained manually after parsing))

`squeeze` should not be added; its purpose is to return a Series; while this method returns a list of dataframes (_always_), I would raise if its not-None.

`date_parser` is kind of useless. you already have `parse_dates` to auto parse most dates, otherwise you use `.to_datetime` (usually with a format). To be honest I would blow that away from the csv parsers, but that's another thing. So raise here if its passed as well (a `ValueError`)

@jreback Thanks.  I removed squeeze and date_parser.  I think it is good to go.  Let me know if I misunderstood.

Sure, I can do that.

I added examples to the docs.  Let me know if you want a modification or if I need to add something someplace else.

minor comment. ping when green.

@gte620v Thanks a lot!

My pleasure!