TYPING: Added types for some tests
State: closed

Working around a strange typing issue. See
https://github.com/pandas-dev/pandas/pull/28394#issuecomment-5456527290
for more, but the types on these were being inferred incorrectly by
mypy with just the addition of the `allows_duplicate_labels` kwarg.

cc @simonjayhawkins & @WillAyd. Hopefully I got things correct. I copied the use of TypeVar for FrameOrSeries with IndexOrSeries.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Any chance you can post the failures this was solving? Would be helpful as background
> Any chance you can post the failures this was solving? Would be helpful as background

```
pandas\tests\test_strings.py:206: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:240: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:379: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:391: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:431: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:444: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:498: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:600: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:660: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_strings.py:661: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:519: error: List item 0 has incompatible type "Type[Index]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:550: error: List item 0 has incompatible type "Type[Index]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:615: error: List item 0 has incompatible type "Type[Index]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:1093: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:1123: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:1147: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:1335: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\test_base.py:1399: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\io\json\test_json_table_schema.py:434: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\io\json\test_json_table_schema.py:441: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:518: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:544: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:566: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:786: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:798: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:836: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\indexing\test_coercion.py:868: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\dtypes\test_concat.py:43: error: List item 0 has incompatible type "Type[Index]"; expected "Type[PandasObject]"
pandas\tests\arithmetic\test_numeric.py:87: error: List comprehension has incompatible type List[PandasObject]; expected List[RangeIndex]
pandas\tests\arithmetic\test_numeric.py:101: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\arithmetic\test_numeric.py:126: error: List comprehension has incompatible type List[PandasObject]; expected List[RangeIndex]
pandas\tests\arithmetic\test_numeric.py:140: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
pandas\tests\arrays\test_array.py:275: error: List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
```
Here's a somewhat reduced example

(edit, simplified some more)

```python
# ok.py
from typing import List, Type, cast


class Base:
    pass


class Series(Base):
    def __init__(self, a=None):
        pass


class Index(Base):
    def __new__(cls, a=None, **kwargs) -> "Index":
        result = object.__new__(cls)
        return result


reveal_type([Index, Series])
[Index, Series]
cast(List[Type[Base]], [Index, Series])
```

```python
# bug.py
from typing import List, Type, cast


class Base:
    pass


class Series(Base):
    def __init__(self, a=None, z=None):
        pass


class Index(Base):
    def __new__(cls, a=None, **kwargs) -> "Index":
        result = object.__new__(cls)
        return result


reveal_type([Index, Series])
[Index, Series]
cast(List[Type[Base]], [Index, Series])
```

so the only difference is the new keyword. (`z=False`)


The outputs are

```bash
bug.py:20: note: Revealed type is 'builtins.list[def (a: Any =, Any =) -> bug.Base]'
bug.py:20: error: List item 0 has incompatible type "Type[Index]"; expected "Type[Base]"
bug.py:21: error: List item 0 has incompatible type "Type[Index]"; expected "Type[Base]"
ok.py:20: note: Revealed type is 'builtins.list[builtins.type*]'
```
A couple observations:

1. This goes away when there are the same number of kwargs (adding an unused argument to Index solves things).
2. Removing **kwargs from `Index.__new__` solves things.
May be onto something... This passes `mypy --strict` without any errors. I changed

* added a type to kwargs
* added a `type: Index` to the result of `Index.__new__`

```python
# bug.py
from typing import List, Type, cast, Any, Mapping, Hashable


class Base:
    pass


class Series(Base):
    def __init__(self, a: Any = None, z: bool=False) -> None:
        pass


class Index(Base):
    def __new__(cls, a: Any = None, **kwargs: Mapping[Hashable, Any]) -> "Index":
        result = object.__new__(cls)  # type: Index
        return result


reveal_type([Index, Series])
[Index, Series]
cast(List[Type[Base]], [Index, Series])

```
I asked about this on the typing Gitter and one of the devs shared this as where the "best" super type determination gets made:

https://github.com/python/mypy/blob/09c1fc7a4c19fe76c2ca9f0c58eaeccfe1ef9ed5/mypy/join.py#L340

Obviously here we have PandasObject and IndexOpsMixin as shared super types, and in this case it appears mypy infers PandasObject

Haven't stepped through the code but may provide some insight. In an case from comments seems like this behavior might be tenuous at best, so something to consider
might be clearer to use `return cast("Index", result)` instead of `  # type: Index`
Unfortunately, I can't get the suggestion in https://github.com/pandas-dev/pandas/pull/29205#issuecomment-546038272 to type `**kwargs` and `Index.__new__` working on my branch.
Given that this *seems* like a mypy bug, any objection to just adding `# type: ignore` to the lines causing issues in #28394? Hopefully that's OK since they're just test files...
my preference would be a `# type: ignore` to changing things if it maybe a mypy issue. 

`--warn-unused-ignores` can be used to remove if issues are fixed in mypy releases, so I don't think difficult to track.
> Given that this _seems_ like a mypy bug,

I don't think this is a bug. It's just ambiguous as to what our intent is here.

Does making the TypeVar point to PandasObject work? I guess having a TypeVar use a Mixin might not make a ton of sense anyway
another alternative to `# type: ignore` could be to simply replace `[Index, Series]` with `cast(Any, [Index, Series])` as an interim measure.
> I don't think this is a bug.

```
from typing import List, Type, cast


class Base:
    pass


class Series(Base):
    def __init__(self, a=None, z=None):
        pass


class Index(Base):
    def __new__(cls, a=None, **kwargs) -> "Index":
        result = object.__new__(cls)
        return result


reveal_type([Index, Series])

cast(List[Type[Base]], [Index, Series])

b: List[Type[Base]] = [Index, Series]
reveal_type(b)
```
```
$ mypy bug.py
bug.py:19: note: Revealed type is 'builtins.list[def (a: Any =, Any =) -> bug.Base]'
bug.py:19: error: List item 0 has incompatible type "Type[Index]"; expected "Type[Base]"
bug.py:24: note: Revealed type is 'builtins.list[Type[bug.Base]]'
(pandas-dev) 
```

The assignment to `b` does not fail. Is this not the type that mypy should be inferring?
That's a great example. I think the expression will always be ambiguous as to the intent whereas in the assignment we can clearly state what is expected. Can't we do that here in this PR but change `List[Type[IndexOrSeries]]` to just be `List[Type[PandasObject]]` then? I don't think it makes sense to use a Mixin here anyway, especially since these are really being used as constructors

A Literal `Type[Series]` or `Type[Index]` would probably be best, but we don't have that feature yet (though maybe should). If the `Type[PandasObject]` doesn't work then sure can ignore; I guess I'm just hesitant to do that because it's not clear what would need to happen to ever un-ignore
> I guess I'm just hesitant to do that because it's not clear what would need to happen to ever un-ignore

to be clear. mypy is infering a type for a list of two items, and then is immediately producing an error with the inferred type.

i.e. `reveal_type([Index, Series])` produces a error

the message in the error includes `expected "Type[Base]"` and yet explicity specifying the type as this type does not produce an error. i.e. `b: List[Type[Base]] = [Index, Series]`

so pretty sure this is a bug and what would need to happen to un-ignore is the bug to be fixed.
I think I'll close this. Will ignore as needed on my other PR.