TYP: Use Self instead of class-bound TypeVar in pyi files (remove Y019)
State: closed

Progress towards #51233. Next up will be to use `Self`  in the pandas code base.

xref #51309.

CC: @twoertwein


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> The `Self` changes look good to me. Whether `typing_extensions` should be a new dependency or not, should be judged by someone else.

+1 from me, I'm also not sure what is normal.

I looked at `typeshed`, and the project itself has `typing_extensions` as a test requirement, se the [requirements-tests.txt](https://github.com/python/typeshed/blob/main/requirements-tests.txt). OTOH, individual type stubs don't have it as a requirement AFAIKS, e.g. `types-protobuf` uses `typing_extensions` but has no dependencies...
This PR was made as a bit of a follow-up to #51309, where @jbrockmendel had a suggestion to use `TypeGuard` for a more precise typing experience. `TypeGuard` is only available from python 3.10, so he added `type_extensions` to `_typing` module. #51309 hasn't actually been pulled in yet, but I didn't sense that using `TypeGuard`/`typing_extensions` as controversial.

I was not not aware of the decision to not use `typing_extensions` (which is totally on me, I had a break from pandas so have not been aware of some thing happening in the community). Would it be possible to reconsider the decision to not use `typing_extensions`?

IMO `typing_extensions` will be very useful to use in Pandas, because:
* `typings_extensions` is just a backport of functionality from newer python version to allow mypy to type check using older python versions,
* it is a dependency of `mypy` itself already (so is already installed, if mypy is installed), is developed by Guido & company (so has good support), so it doesn't seem controversial to me to use it, and
*  Pandas is a very large code base, so more/better typing is IMO always helpful to our project.
?

> Would it be possible to reconsider the decision to not use `typing_extensions`?

When we had the discussion before I was arguing for including it. Now that we have pandas-stubs and typing in the codebase could be considered as a glorified linter there are probably less arguments for. Also, it has not always ended up been a blocker for using features supported in the latest mypy by implementing some trivial workarounds in some cases.

cc @TomAugspurger @jorisvandenbossche @jreback @WillAyd from previous discussions.
The previous decision was for a run-time dependency.

As the import is inside a TYPE_CHECKING block this implementation only makes it a dev dependency (just like the use of npt)?


> The previous decision was for a run-time dependency.
> 
> As the import is inside a TYPE_CHECKING block this implementation only makes it a dev dependency (just like the use of npt)?

Yes, this will only be active when type checking and it will not be a run time requirement.
scanning back over the historic typing_extensions discussions, the issue with having it as a dev-only dependency was the the need to wrap in TYPE_CHECKING and was also before typing_extensions was an required dependency of mypy in #27646

> IMO `typing_extensions` will be very useful to use in Pandas, because:
> 
> * `typings_extensions` is just a backport of functionality from newer python version to allow mypy to type check using older python versions,
> * it is a dependency of `mypy` itself already (so is already installed, if mypy is installed), is developed by Guido & company (so has good support), so it doesn't seem controversial to me to use it, and
> * Pandas is a very large code base, so more/better typing is IMO always helpful to our project.

adding to this list...

* pandas-stubs is not restricted by the minimum Python version support. It is desirable to keep the pandas-stubs and pandas type annotations in sync where possible/appropriate.
From looking at the linked issues, https://github.com/pandas-dev/pandas/issues/37119 might be the most recent-ish complete discussion issue about this (+ https://github.com/pandas-dev/pandas/issues/34869 as a precursor of it). We can maybe have the actual discussion about adding as dependency or not there?

From skimming that issue, what is still unclear to me is what the exact consequences are? Is this only a dev dependency (only contributors have to have it installed in their environment) and not a required runtime dependency? Or would it only be useful if it's actually a runtime dependency? 
(the issue has various conflicting messages about this, some say it's possible, others not, or that it limits its usefulness, ..) 
While from this PR it seems it is only a dev dependency, since it's not added to pyproject.toml.

Can someone summarize this over at https://github.com/pandas-dev/pandas/issues/37119?


I've answered in https://github.com/pandas-dev/pandas/issues/37119 regarding my experience using `typing_extensions` inside the `if TYPE_CHECKING` block.

Wrt. whether `typing_extensions` is/should be a run time dependency or not, I've made if NOT a run time dependency in this PR (i.e. only imported when `TYPE_CHECKING` is True).

If there's an argument for making this available at run time i'm open to arguments, but I just didn't see the need myself.  and it makes sense to always limit run time dependencies when possible.

I've merged main.
Wow great work @topper-123 
nice!
Can someone update https://github.com/pandas-dev/pandas/issues/37119 with the outcome? (and probably close it)