ENH: Add support for storage_options in pd.read_html
State: closed

- [x] closes #49944
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

Added support for the `storage_options` keyword for `pandas.read_html` , so that users may pass headers alongside the HTTP request when using `pandas.read_html` to read from a URL.

The solution involves two parts:
1. Adding `storage_options` as an optional parameter to `pandas.read_html` and propagating it down to the method(s) where an HTTP request is made (`get_handle` and `urlopen` in pandas.io.common)
2. If necessary, add support for `storage_options` in the function that makes an HTTP request and returns the response

### 1) Propagating `storage_options` from read_html down to the HTTP request
The sequence of functions leading to the HTTP request are as follows:
```
1. read_html                                    (pandas.io.html)
2. _parse                                       (pandas.io.html)
3. parser(args)         # initialize parser     (pandas.io.html)
4. parse_tables                                 (pandas.io.html)
5. _parse_tables                                (pandas.io.html)
6. _build_doc                                   (pandas.io.html)
  i)  urlopen                                   (pandas.io.common)
  ii) get_handle                                (pandas.io.common)
```

`storage_options` must be added as an optional argument to `pandas.read_html`, and should be passed down to each function (or object instance) until `urlopen` or `get_handle` is reached.

### 2) Adding support for  `storage_options` in the HTTP request functions
There exist two types of parsers that are used by `pandas.read_html`: `_BeautifulSoupHtml5LibFrameParser` and `_LxmlFrameParser`. Both of these classes implement the abstract methods of their parent class `_HtmlFrameParser`.

The two parsers implement `_build_doc` differently such that `_BeautifulSoupHtml5LibFrameParser` uses `get_handle` to make the GET request whereas `_LxmlFrameParser` uses `urlopen`.

`get_handle`, as it is defined in pandas.io.common, already supports the `storage_options` keyword, and thus doesn't require any modifying.

`urlopen`, however, does not expect or handle the `storage_options` keyword. Therefore, we must add logic such that if `urlopen` is called with `storage_options`, then the contents of `storage_options` should be passed as headers in the outgoing HTTP request. Otherwise, `urlopen` should follow the exact same logic it normally would.

For a more detailed problem analysis and solution description, please see [this notion document](https://sumptuous-hole-c97.notion.site/Support-storage_options-in-pandas-read_html-96c09eebdb254729a8074c7a9b599890).

Labels:
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@mroeschke do you know who I could ping for a review
Could you merge in main and resolve the merge conflict?
@mroeschke done
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Hi @mroeschke - I've attempted to address the failing tests and I've merged into main. Could you reopen this PR so that I can see the CI output? Thanks!
@mroeschke All should be good now. Please let me know if anything seems off, thanks!
Thanks for sticking with this @serhatgktp. Nice work