Update type hints to python3 style (non-array files)
State: closed

- [X ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
Fixes #25790 (non-array part)

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
I'm unsure about the method I used to define TypeVars, so an instance method could refer to its own class type, e.g., in https://github.com/pandas-dev/pandas/pull/25802/commits/014c48e9e239deb6cc97d72c42f44cd109c950f9

Is there a more appropriate way or a better place for the definition?
@gwrome thanks! I don't think we need any TypeVars just yet - can you not just implicitly break lines within brackets for container types as required?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=h1) Report
> Merging [#25802](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/85c3f82dff73ede6d5d961611518e007c8ba0098?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25802/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25802      +/-   ##
==========================================
+ Coverage   91.27%   91.27%   +<.01%     
==========================================
  Files         173      173              
  Lines       53002    53014      +12     
==========================================
+ Hits        48375    48387      +12     
  Misses       4627     4627
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.84% <100%> (ø)` | :arrow_up: |
| #single | `41.78% <84%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.09% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9wZXJpb2QucHk=) | `92.06% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.52% <100%> (ø)` | :arrow_up: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `97.77% <100%> (ø)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.21% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `93.93% <100%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.8% <100%> (ø)` | :arrow_up: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `98.39% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.57% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=footer). Last update [85c3f82...0efb21d](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=h1) Report
> Merging [#25802](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d40446044fc240f4ec6904d12a7821f0091f3b29?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25802/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25802      +/-   ##
==========================================
- Coverage   91.48%   91.47%   -0.01%     
==========================================
  Files         175      175              
  Lines       52885    52872      -13     
==========================================
- Hits        48380    48366      -14     
- Misses       4505     4506       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.04% <100%> (-0.01%)` | :arrow_down: |
| #single | `41.81% <77.77%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `93.93% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.53% <100%> (ø)` | :arrow_up: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `97.77% <100%> (ø)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.21% <100%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <100%> (-0.01%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `98.39% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.57% <100%> (ø)` | :arrow_up: |
| [pandas/plotting/\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb21wYXQucHk=) | `86.66% <0%> (-0.84%)` | :arrow_down: |
| [pandas/io/formats/terminal.py](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvdGVybWluYWwucHk=) | `31.7% <0%> (-0.83%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/pandas-dev/pandas/pull/25802/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=footer). Last update [d404460...ce92ce8](https://codecov.io/gh/pandas-dev/pandas/pull/25802?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@WillAyd TIL about forward references in type hints, so that fixed what I was trying to address with TypeVars. They're all gone now.
Any guess as to why the MacOS 3.5 build fails? Did something change with respect to type hints between 3.5 and 3.6?
Can you run the test suite locally? The builds are pointing to a syntax error
Ah never mind I think the problem is that type annotations for class and instance variables came along later as part of [PEP 526](https://www.python.org/dev/peps/pep-0526/) in Python36, so for those probably best to stick with comment until we drop Py35 support
I returned the two variables class/instance variables to the previous style and left the rest of the changes intact.
we need a minimal run of mypy to validate things. it can be in this PR or a pre-cursor.
@jreback do you want that set up as part of CI or just validate output locally? I *think* the former is going to take a while as a pre-cursor so I figured switch over syntax first then clean up types, though open to however you want to approach
> we need a minimal run of mypy to validate things. it can be in this PR or a pre-cursor.

The bad news is that running mypy on the changed files throws a ton of errors. The less bad news is that it appears that most of the errors are outside the scope of the changes in this PR.

I have some idea of how to proceed with the issues created by these type hints. The broader problems, though, will probably take substantial work outside the scope of this PR.

All the commands below were run using these [numpy stubs](https://github.com/machinalis/mypy-data/tree/master/numpy-mypy).

# pandas/core/base.py
I think these type hints are fine, but there are other errors, including an issue related to numpy, a missing pandas stub (which I presume hasn't been written yet), and some attribute errors in this file.
<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/base.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/base.py:10: error: No library stub file for module 'pandas._libs.lib'
pandas/core/base.py:10: note: (Stub files are from https://github.com/python/typeshed)
pandas/core/base.py:182: error: Module has no attribute "sum"
pandas/core/base.py:183: error: Module has no attribute "max"
pandas/core/base.py:184: error: Module has no attribute "min"
pandas/core/base.py:191: error: Module has no attribute "all"
pandas/core/base.py:192: error: Module has no attribute "any"
pandas/core/base.py:193: error: Module has no attribute "sum"
pandas/core/base.py:194: error: Module has no attribute "nansum"
pandas/core/base.py:195: error: Module has no attribute "mean"
pandas/core/base.py:196: error: Module has no attribute "nanmean"
pandas/core/base.py:197: error: Module has no attribute "prod"
pandas/core/base.py:198: error: Module has no attribute "nanprod"
pandas/core/base.py:199: error: Module has no attribute "std"
pandas/core/base.py:200: error: Module has no attribute "nanstd"
pandas/core/base.py:201: error: Module has no attribute "var"
pandas/core/base.py:202: error: Module has no attribute "nanvar"
pandas/core/base.py:203: error: Module has no attribute "median"
pandas/core/base.py:204: error: Module has no attribute "nanmedian"
pandas/core/base.py:205: error: Module has no attribute "max"
pandas/core/base.py:206: error: Module has no attribute "nanmax"
pandas/core/base.py:207: error: Module has no attribute "min"
pandas/core/base.py:208: error: Module has no attribute "nanmin"
pandas/core/base.py:209: error: Module has no attribute "cumprod"
pandas/core/base.py:210: error: Module has no attribute "nancumprod"
pandas/core/base.py:211: error: Module has no attribute "cumsum"
pandas/core/base.py:212: error: Module has no attribute "nancumsum"
pandas/core/base.py:852: error: "IndexOpsMixin" has no attribute "_values"; maybe "_map_values"?
pandas/core/base.py:975: error: "IndexOpsMixin" has no attribute "values"
```
</details>

# pandas/core/common.py
The hints seem fine, but there are other errors, including an issue related to numpy, and a missing attribute error.
<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/common.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/common.py:133: error: Module has no attribute "any"
```

The remaining errors are similar to the other files.
</details>

# pandas/core/dtypes/base.py
This one has a hint-related error:
```
pandas/core/dtypes/base.py:212: error: Invalid type "pandas.core.dtypes.base.ExtensionDtype.type"
```
Importing Type from typing and replacing `type` with `Type` corrects the error.

It also has similar numpy and missing attribute errors.
<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/dtypes/base.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/dtypes/base.py:28: error: Module has no attribute "nan"
pandas/core/dtypes/base.py:212: error: Invalid type "pandas.core.dtypes.base.ExtensionDtype.type"
```
</details>

# pandas/core/frame.py
Miscellaneous errors unrelated to these hints.

<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/frame.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/frame.py:25: error: No library stub file for module 'numpy.ma'
pandas/core/frame.py:25: note: (Stub files are from https://github.com/python/typeshed)
pandas/core/frame.py:27: error: Module 'pandas._libs' has no attribute 'lib'
pandas/core/frame.py:27: error: Module 'pandas._libs' has no attribute 'algos'
pandas/core/frame.py:370: error: Need type annotation for '_accessors'
pandas/core/frame.py:397: error: No library stub file for module 'numpy.ma.mrecords'
pandas/core/frame.py:3790: error: Module has no attribute "nan"
pandas/core/frame.py:4716: error: No library stub file for module 'pandas._libs.hashtable'
```
</details>

# pandas/core/groupby/groupby.py
This one has a few relevant errors:
```
pandas/core/groupby/groupby.py:328: error: Need type annotation for '_apply_whitelist'
pandas/core/groupby/groupby.py:1047: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
pandas/core/groupby/groupby.py:1049: error: Incompatible return value type (got "Tuple[_ArrayLike[Any], Any]", expected "Tuple[ndarray[Any], Type[Any]]")
pandas/core/groupby/groupby.py:1052: error: Incompatible return value type (got "_ArrayLike[Any]", expected "ndarray[Any]")
pandas/core/groupby/groupby.py:1749: error: Incompatible types in assignment (expression has type "str", variable has type "Optional[Type[int64]]")
pandas/core/groupby/groupby.py:1750: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
pandas/core/groupby/groupby.py:1762: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
```
Almost all are a mismatch between `_ArrayLike` and `ndarray`. I guess those can be fixed by swapping the hints to expecting `np._ArrayLike[Any]`. Will need to make sure that's an accurate expectation.

Line 1749's error can be corrected by adding `str` to the second element of the return type, since the code can return `np.int64`, `'datetime64[ns]'`, or `None`.

<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/groupby/groupby.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/groupby/groupby.py:328: error: Need type annotation for '_apply_whitelist'
pandas/core/groupby/groupby.py:1047: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
pandas/core/groupby/groupby.py:1047: error: Module has no attribute "bool"
pandas/core/groupby/groupby.py:1049: error: Incompatible return value type (got "Tuple[_ArrayLike[Any], Any]", expected "Tuple[ndarray[Any], Type[Any]]")
pandas/core/groupby/groupby.py:1049: error: Module has no attribute "bool"
pandas/core/groupby/groupby.py:1052: error: Incompatible return value type (got "_ArrayLike[Any]", expected "ndarray[Any]")
pandas/core/groupby/groupby.py:1749: error: Incompatible types in assignment (expression has type "str", variable has type "Optional[Type[int64]]")
pandas/core/groupby/groupby.py:1750: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
pandas/core/groupby/groupby.py:1750: error: Module has no attribute "float"
pandas/core/groupby/groupby.py:1762: error: Incompatible types in assignment (expression has type "_ArrayLike[Any]", variable has type "ndarray[Any]")
```
</details>

# pandas/core/indexes/base.py
Miscellaneous errors unrelated to these hints.

<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/indexes/base.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/indexes/base.py:9: error: Module 'pandas._libs' has no attribute 'algos'
pandas/core/indexes/base.py:9: error: Module 'pandas._libs' has no attribute 'index'
pandas/core/indexes/base.py:9: error: Module 'pandas._libs' has no attribute 'join'
pandas/core/indexes/base.py:11: error: No library stub file for module 'pandas._libs.lib'
pandas/core/indexes/base.py:11: note: (Stub files are from https://github.com/python/typeshed)
pandas/core/indexes/base.py:12: error: Module 'pandas._libs.tslibs' has no attribute 'OutOfBoundsDatetime'
pandas/core/indexes/base.py:12: error: Module 'pandas._libs.tslibs' has no attribute 'Timedelta'
pandas/core/indexes/base.py:12: error: Module 'pandas._libs.tslibs' has no attribute 'Timestamp'
pandas/core/indexes/base.py:13: error: No library stub file for module 'pandas._libs.tslibs.timezones'
pandas/core/indexes/base.py:150: error: Too many arguments for "dtype"
pandas/core/indexes/base.py:1784: error: Module has no attribute "nan"
```
</details>

# pandas/core/indexes/datetimelike.py
Miscellaneous errors unrelated to these hints.

<details>
<p>
```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/indexes/datetimelike.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/indexes/datetimelike.py:136: error: Decorated property not supported
pandas/core/indexes/datetimelike.py:206: error: Name '_box_values' already defined on line 71
pandas/core/indexes/datetimelike.py:701: error: Need type annotation for '_raw_methods'
pandas/core/indexes/datetimelike.py:703: error: Need type annotation for '_raw_properties'
```
</details>

# pandas/core/internals/blocks.py
Miscellaneous errors unrelated to these hints.

<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/internals/blocks.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
pandas/core/internals/blocks.py:77: error: Module has no attribute "concatenate"
pandas/core/internals/blocks.py:2217: error: Definition of "_can_hold_na" in base class "ExtensionBlock" is incompatible with definition in base class "DatetimeBlock"
```
</details>

# pandas/core/internals/managers.py
Numpy errors.

<details>
<p>

```
$ MYPYPATH=stubs/ mypy --follow-imports skip ./pandas/core/internals/managers.py
stubs/numpy/__init__.pyi:48: error: Class numpy.flatiter has abstract attributes "__next__"
stubs/numpy/__init__.pyi:48: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
```
</details>
> @jreback do you want that set up as part of CI or just validate output locally? I think the former is going to take a while as a pre-cursor so I figured switch over syntax first then clean up types, though open to however you want to approach

we absolutely need CI for this, which excludes non-fixed files.
@gwrome can you merge master?
You got it, @WillAyd 
thanks @gwrome