TYP: fix reportInvalidTypeVarUse from pyright
State: closed

This fixes the following warning from pyright:  `TypeVar "..." appears only once in generic function signature (reportInvalidTypeVarUse)`.

It would be great to enable pyright on the CI. It takes only ~90sec to run pyright on the entire codebase with the following configuration in pyproject.toml (ignores most type errors for now):
```
[tool.pyright]
pythonVersion = "3.8"
include = ["pandas"]
ignore = ["pandas/tests"]
reportOptionalMemberAccess = false
reportOptionalOperand = false
reportGeneralTypeIssues = false
reportMissingImports = false
reportUnboundVariable = false
reportMissingModuleSource = false
reportOptionalSubscript = false
reportOptionalCall = false
reportOptionalIterable = false
reportPropertyTypeMismatch = false
```
The following errors would need to be taken care of before then:
  pandas/pandas/core/frame.py:4575:9 - error: Method "_reindex_axes" cannot override final method defined in class "NDFrame"
  pandas/pandas/core/frame.py:10736:9 - error: Method "mask" cannot override final method defined in class "NDFrame"
  pandas/pandas/core/series.py:5449:9 - error: Method "mask" cannot override final method defined in class "NDFrame"
  pandas/pandas/core/computation/expr.py:268:26 - error: TypeVar bound type cannot be generic
  pandas/pandas/core/groupby/generic.py:676:9 - error: Method "describe" cannot override final method defined in class "GroupBy"
  pandas/pandas/core/groupby/generic.py:856:9 - error: Method "pct_change" cannot override final method defined in class "GroupBy"

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @twoertwein  for the PR. I have merged just merged #41955, so need to change `FrameOrSeriesUnion` -> `DataFrame | Series`

A bit of history here...

originally we just had the `FrameOrSeries` alias and later needed the `FrameOrSeriesUnion` alias

so some uses of `FrameOrSeries` remain where only `FrameOrSeriesUnion` was needed.

I think mypy does pick up some cases where `FrameOrSeries` (i.e. TypeVar) is used and is unbounded, but not all.
I should have replaced `FrameOrSeries` with `NDFrame` and not with `Series | DataFrame` as 
```py
FrameOrSeries = TypeVar("FrameOrSeries", bound="NDFrame")
```

If `FrameOrSeries` is only ever used for DataFrames and Series, it should probably be

```py
FrameOrSeries = TypeVar("FrameOrSeries", DataFrame, Series)
```


> If `FrameOrSeries` is only ever used for DataFrames and Series, it should probably be

originally, that was what was used. but that created issues with core.generic (and some of the groupby IIRC)

> I should have replaced `FrameOrSeries` with `NDFrame` and not with `Series | DataFrame` as

probably only in needed in  core.generic and maybe groupby. elsewhere `Series | DataFrame` is probably fine.
> I should have replaced `FrameOrSeries` with `NDFrame` and not with `Series | DataFrame` as
> 
> ```python
> FrameOrSeries = TypeVar("FrameOrSeries", bound="NDFrame")
> ```
> 
> If `FrameOrSeries` is only ever used for DataFrames and Series, it should probably be
> 
> ```python
> FrameOrSeries = TypeVar("FrameOrSeries", DataFrame, Series)
> ```

There is another subtle difference which maybe significant in some situations. `TypeVar("FrameOrSeries", bound="NDFrame")` maintains types for subclasses, whereas the `TypeVar("FrameOrSeries", DataFrame, Series)` variant does not. Internally, this is not really an issue, but for parts of the public api, this difference could be important to a group of users. 
> `TypeVar("FrameOrSeries", bound="NDFrame")` maintains types for subclasses, whereas the `TypeVar("FrameOrSeries", DataFrame, Series)` variant does not. Internally, this is not really an issue, but for parts of the public api, this difference could be important to a group of users.

Thanks for all the information!  I will prefer `NDFrame` in public functions and see whether that leaves any `DataFrame | Series` (currently it is the opposite way: `NDFrame` where necessary - I will change that).


I think this PR is ready from my perspective (happy to remove the pyright config in pyproject.toml). Let me know what you think @simonjayhawkins 
@twoertwein can you avoid squashing and force pushing, it makes reviewing updates more difficult.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Rebased (but this time not squashed)
@twoertwein whenever you have a second mind merging master
@alimcmaster1 rebased. Let me know if I should remove some parts that might be controversial (pyright settings in pyproject.toml?).
> It would be great to enable pyright on the CI. It takes only ~90sec to run pyright on the entire codebase with the following configuration in pyproject.toml (ignores most type errors for now):

@twoertwein can you setup this?
> > It would be great to enable pyright on the CI. It takes only ~90sec to run pyright on the entire codebase with the following configuration in pyproject.toml (ignores most type errors for now):
> 
> @twoertwein can you setup this?

I will look into that. Should that be part of this PR or a followup (it might take me some time to play around with the CI).

Where in the CI should pyright be integrated (I assume in "CI / Checks")?  Should pyright also be part of the pre-commit? There doesn't seem to be a pre-commit interface from pyright, except this third-party wrapper: https://github.com/necaris/pre-commit-pyright.

Other points to consider:

- Pyright depends on node.js
- Pyright fixes bugs at a rapid speed: it has ~1 release per week (probably need to pin the version)
- Mypy and pyright might disagree (I don't think this is a concern right now, since most type checking rules need to be disabled to pass pyright).

edit: pyright can apparently be easily integrated in pre-commit: https://github.com/microsoft/pyright/blob/main/docs/ci-integration.md
@twoertwein happy to merge this when ready
> @twoertwein happy to merge this when ready

I partially typed  `_get_grouper` in the latest commit (contains an `Any`) to keep the intended `TypeVar` as pointed out by @simonjayhawkins

Apart for double-checking that, it should be ready from my side.
Removed the pyright configuration (will be added in #43672). 
Rebased (includes #43672) and enabled checks for reportInvalidTypeVarUse and reportSelfClsParameterName.