TYP: Update type hints for ExtensionArray and ExtensionDtype
State: closed

Modifications to reduce error reports from `pyright --verifytypes` related to `ExtensionArray` and `ExtensionDtype`.

Some notes:
1. Had to use `Any` in a number of places, because an `ExtensionArray` can contain any type
2. Because `numpy` doesn't have a released version with types, `pyright` complains whenever we work with `numpy` types, so for type checking purposes, I redefine `np.dtype` and `np.ndarray` to a dummy type.  When `numpy` 1.20 is released, we can most likely remove that.  Alternatively, I can find out if there is a way to see if `pyright` has something like `TYPE_CHECKING` to bracket those changes.

I anticipate these changes might be controversial, so constructive criticism is welcome!


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
```diff
-        method: Optional[Literal["backfill", "bfill", "ffill", "pad"]] = None,
+        method: Optional[Literal[backfill, bfill, ffill, pad]] = None,
```
Looks like a bug in `no-string-hints`, sorry about that, will test + update!

---

Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update
> 2\. When `numpy` 1.20 is released

@Dr-Irv can you update
> > 2. When `numpy` 1.20 is released
> 
> @Dr-Irv can you update

Ha!  It was just released in pypl on 1/30/21, which is when I pushed this!  Waiting for it to be on conda and then will update.
> > 2. When `numpy` 1.20 is released
> 
> @Dr-Irv can you update

I did try this, and now `mypy pandas` reports more errors because `numpy` 1.20 has typing.  #39513 

I'm asking the `pyright` people to see if there is a different workaround.  Ideally, I'd like to tell pyright "process pandas, but ignore numpy"

> I'm asking the `pyright` people to see if there is a different workaround. Ideally, I'd like to tell pyright "process pandas, but ignore numpy"

You can now use `pyright` to find pandas symbols that are missing types by using `pyright --verifytypes pandas!` (The `!` means don't look at imported libraries).  That means it won't look at the `numpy` imports.  Based on that (and including this PR), we have 75.7% type completeness for pandas. 
> Looks like a bug in `no-string-hints`, sorry about that, will test + update!
> 
> Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update

@MarcoGorelli Has this been done yet?  I'm still getting the pre-commit failure
> > Looks like a bug in `no-string-hints`, sorry about that, will test + update!
> > Have updated and pushed a new tag, the version in pandas' pre-commit-config will get updated tomorrow morning as part of the scheduled weekly update
> 
> @MarcoGorelli Has this been done yet? I'm still getting the pre-commit failure

if you approve/merge #39524 and then rebase/merge, then it'll pass
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Nice. OK, so it seems this is a case where we don't need the extra overload we had to put in #40197 (which might only be necessary if have overloads for specific instances of the base type. I.e. if you overload `bool` with `Literal[True]` and `Literal[False]` you also need an overload for `bool`, but if you overload `Union[bool, int]` with `bool` and `int` then you don't need an extra overload) 

e.g.
```python
from typing import overload, Union, List, Any

MyType = List[str]


class Animal:
    
    
    @overload
    def foo(self, key: int) -> Any: ...
    
    @overload
    def foo(self, key: str) -> MyType: ...
    
    def foo(self, key: Union[int, str]) -> Union[MyType, Any]:
        ...

key: Union[int, str]
reveal_type(Animal().foo(key))  # Revealed type is 'Union[Any, builtins.list[builtins.str]]'
```

https://mypy-play.net/?mypy=latest&python=3.9&gist=c9bd0a94f79360506940aaf8274356dc
Ah, no - correction, it still doesn't fall back to the base type, it just applies some logic rules to deduce the return type from the overloads.

e.g.
```python
from typing import overload, Union, List, Any

MyType = List[str]


class Animal:
    
    
    @overload
    def foo(self, key: int) -> Any: ...
    
    @overload
    def foo(self, key: str) -> bool: ...
    
    def foo(self, key: Union[int, str]) -> Union[MyType, Any]:
        ...

key: Union[int, str]
reveal_type(Animal().foo(key))  # Revealed type is 'Union[Any, builtins.bool]', not 'Union[Any, builtins.list[builtins.str]]' !
```
https://mypy-play.net/?mypy=latest&python=3.9&gist=b45cd58a6aee05e41817ac9d0cc5935a

So, I guess there is some missing logic for it to deduce that `bool` is `Union[Literal[True], Literal[False]]`, which is why in that case we needed the extra overload... :exploding_head: 

----

from the [docs](https://mypy.readthedocs.io/en/stable/more_types.html?highlight=overload#type-checking-calls-to-overloads):

> if multiple variants match due to one or more of the arguments being a union, mypy will make the inferred type be the union of the matching variant returns
@simonjayhawkins Ugh the merge of #36092 raised a whole bunch of issues with this PR that I now have to address.
> @simonjayhawkins Ugh the merge of #36092 raised a whole bunch of issues with this PR that I now have to address.

@simonjayhawkins So in trying to fix these issues, if I change the declaration of `AnyArrayLike` and `ArrayLike` to be a `Union` instead of a `TypeVar`, I can get rid of a LOT of the `# type: ignore` stuff you added.  Before I start doing that, do you have an opinion on them being a `TypeVar` versus a `Union`
> do you have an opinion on them being a `TypeVar` versus a `Union`

done in #40379
> > do you have an opinion on them being a `TypeVar` versus a `Union`
> 
> done in #40379

Oh, so maybe I will wait for you to get that merged before I go further on this PR.

@simonjayhawkins Updated the PR based on your latest stuff with typing, so now `pandas/core/arrays/base.py` is fully typed and that allowed me to remove a bunch of `# type; ignore` things.   Main changes from last review is an overload for `astype` and fixing `isin`.  Looking forward to your review.
Thanks @Dr-Irv for working on this.

I would imagine that this PR will be easier to check if we limit the changes to `pandas/core/arrays/base.py` and the other changes are done in a separate PR(s).

There is a conflict here and I _think_ some other overlaps with other PRs removing the ignores.

some of the changes to remove ignores need more discussion than others. If ignores are being removed that's great, if ignores or changed or others added we need to review in more depth. 
> I'll push a commit merging master.

some new mypy errors and now also another conflict from a subsequent merge to master related to https://github.com/pandas-dev/pandas/pull/39501#discussion_r593767378
> I would imagine that this PR will be easier to check if we limit the changes to `pandas/core/arrays/base.py` and the other changes are done in a separate PR(s).

I'm not sure that it can be done alone, because `pandas/core/arrays/base.py` has the base `ExtensionArray` class, so you have to fix things upstream when fixing things there.  
> 
> There is a conflict here and I _think_ some other overlaps with other PRs removing the ignores.
> 
> some of the changes to remove ignores need more discussion than others. If ignores are being removed that's great, if ignores or changed or others added we need to review in more depth.

I removed ignores only based on when mypy said they were not needed any more.  I don't think I added any ignores.

Let me get the current version in a state that goes green.
> I'm not sure that it can be done alone, because `pandas/core/arrays/base.py` has the base `ExtensionArray` class, so you have to fix things upstream when fixing things there.
> 
> >

I think that while numpy types resolved to `Any` this PR adding types to  `pandas/core/arrays/base.py` was moreless ready.

all the other changes are related to numpy types now having some meaning. I'm sure these could be done separately (and in smaller chunks)

changes to 36 files takes some review, especially as changes are made as other PRs are merged.

typing PRs need to be atomic to be easily reviewed. 

> Main changes from last review is an overload for astype and fixing isin. Looking forward to your review.

so maybe a PR for the astype fixes and a PR for the isin fixes?
> all the other changes are related to numpy types now having some meaning. I'm sure these could be done separately (and in smaller chunks)

If I limit things to `pandas/core/arrays/base.py`, I'm pretty sure I can't get it to be green.  Is that OK?
@simonjayhawkins Latest commit https://github.com/pandas-dev/pandas/pull/39501/commits/5bb24d4aeeb44f5877a0b35d8f2389b6bf140b54 still has a variety of changes.  It just takes what you merged in and should make it mypy happy.
> so maybe a PR for the astype fixes and a PR for the isin fixes?

I'm going to create a separate PR that is effectively this one, MINUS the astype and isin.  
Things that have been approved/merged
- [x] `registry` as private  #40538
- [x] change `TypeVar` on `DatetimeLikeScalar` to be `Union`  #40540
- [x] `unique` and `repeat` #41260
- [x] simple functions #41255
- [x] Fix up typing in registry  #41203 
- [x] `searchsorted` and  `delete` #41513
- [x] `astype` overloads #41251
- [x] `getitem` overloads #41258

Awaiting approval/merges:


Things to do in `pandas/core/arrays.base.py` and waiting approval
- [ ] arguments for `isin`
- [ ] `to_numpy`

> Things that can be done independent of approval of #40421

Looks like you've made some progress on that checklist.  Is this PR still active?
> > Things that can be done independent of approval of #40421
> 
> Looks like you've made some progress on that checklist. Is this PR still active?

I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved.  Then when I finish the checklist, I'll close this one.


> I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved. Then when I finish the checklist, I'll close this one.

Is this still the master PR? if so could you merge master so we can monitor progress. and more easily see the outstanding review comments.

> > I will change it to draft status, as I'm using it to track the changes I would do after #40421 is approved. Then when I finish the checklist, I'll close this one.
> 
> Is this still the master PR? if so could you merge master so we can monitor progress. and more easily see the outstanding review comments.

Yes.  But I'd prefer to do the merge once  #41203,  #41251, #41258 and  #41513 are merged in so we can see where we are with respect to the comments.  
ok. will close for now. reopen when ready.