TYP: Add type hints to pd.read_html
State: closed


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> I think fine to use something like Optional[Union[int, Sequence[int]]

This would make mypy complain about arraylikes, .e.g. ndarray[int] (which doesn''t work anyway), so force people to convert those to e.g. lists to satisfy mypy.

I think I'd actually be ok with that where the sequence is expected to be small (for example in read_html), bu not everywhere (e.g. we wouldn't want to convert arrays to lists in ``iloc``). It should also be done consistently on all user-facing methods whick could be a large project. Maybe keep that separately from this PR?

I think should use the Sequence annotation here. You are correct that there is work to be done from a larger API perspective, but I don't think that changes things here for now
deprecate_nonkeyword_arguments is not preserving the signature, so these annotation don't add much value at the moment and mypy can't do much consistency checking.

> I think should use the Sequence annotation here.

why not Iterable?

the docs state list-like. we can't make braking change to the api, although I'm not sure exactly what list-like means from an end user perspective.

from the docstring of is_list_like in pandas\_libs\lib.pyx

Objects that are considered list-like are for example Python
    lists, tuples, sets, NumPy arrays, and Pandas Series.

so IMO we need to include these at least.

> This would make mypy complain about arraylikes, .e.g. ndarray[int] (which doesn''t work anyway), so force people to convert those to e.g. lists to satisfy mypy.

so would be Optional[Union[int, Iterable[int], AnyArrayLike] for now

and in-time when we make Series, Index and ExtensionArray Generic classes this would become

Optional[Union[int, Iterable[int], AnyArrayLike[int]]

unless we ensure that AnyArrayLike[int] is covered by Iterable[int]

can you rebase; @simonjayhawkins if you can review.
I'm happy with the type annotations. Just need discussion on docstring changes. IIUC we use different nomenclature in the docstrings for types.

Rebased and updated. @simonjayhawkins , I don't understand you wrt. the doc string, could you expand?
ping.
> I don't understand you wrt. the doc string, could you expand?

numpy has a glossary https://numpy.org/devdocs/glossary.html#term-array-like

for pandas, i'm not exactly sure what list-like is, but we use it a lot in our documentation. This PR is changing it in only one place.

We could switch to using sequence, if it is understood to be equivalent to typing.Sequence, using it everywhere where list-like is used where any sequence is accepted, and having proper testing. Although, I'm sure (but could be wrong) that in many places where we use list-like an iterable is accepted and therefore by changing the docstring, the documented api is changing.

As I said before, the typing annotations are fine. I'm just not sure about the docstring changes. @jorisvandenbossche wdyt? are the docstring changes OK?
The new doc strings mirror the annotations, so are correct, if the annotations are correct?

My understanding of list-like is anything that passes ``pd.api.types.is_list_like`` :-)
> My understanding of list-like is anything that passes `pd.api.types.is_list_like` :-)

without a definition for a pandas user, I would use this too as the definition. but there is then a discrepancy with sequence.

```
>>> pd.core.dtypes.common.is_list_like(i for i in [1, 2, 3])
True
>>>
>>> import collections
>>>
>>> isinstance((i for i in [1, 2, 3]), collections.abc.Sequence)
False
>>> isinstance((i for i in [1, 2, 3]), collections.abc.Iterable)
True
```

> The new doc strings mirror the annotations, so are correct, if the annotations are correct?

no guarantee that the annotations added are consistent with the rest of the codebase (or with the use of read_html in the wild).

(as an aside we have been tripped up a few times with this, the most recent I'm aware of being #34741)

deprecate_nonkeyword_arguments is not preserving the signature (no type annotations on deprecate_nonkeyword_arguments decorator) and read_html is not doing much apart from passing the parameters to _parse (which is not typed)

so these annotation don't add much value at the moment and mypy can't do much consistency checking.

I'm OK with the annotations as they stand though, since mypy is green and as we add more types, mypy will reveal any inconsistencies.

we need to be carefull to keep the addition of type annotations and potential api changes separate.
Regarding the discussion on "sequence" vs "list-like" in docstrings: personally I think it is more important to see that we consistently use a term in the docstrings, than to have the annotation and docstring match exactly (in the end, the docstring is meant to be user-readable, the annotation is often not). So meaning: if we almost anywhere use the term "list-like" and not "sequence" in the docstrings (I don't know if this is true, just assuming here), then I would use list-like here as well. 

But eg in the docstring guidelines, both are not mentioned (https://pandas.pydata.org/docs/dev/development/contributing_docstring.html#parameter-types, it mentions "iterable" and "array-like"). It would be nice to standardize this, make a glossary etc, but that's for a different issue (so for this PR I would just leave the docstring as it was and move on)





I'm ok with leaving the doc string as is, but I've made some clarifications on them though. Updated.
ping.
thanks @topper-123