[WIP] fix --check-untyped-defs for MyPy
State: closed

**tl;dr No need for review!**

no intention of merging this so using patterns that probably won't be accepted.

PR is for visibility and reference only.

using http://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/ as a suitable template for a project plan. resolution of #27568 would come before too much work on adding type hints in general.

There is quite a bit of work involved to enable check-untyped-defs globally so need to keep the phases described in "Fully annotating a large codebase" section as out-of-scope for now.

I do not expect to complete this in the near term, so am using Python 3.6 type annotations for variables (better formatting with black and no need to initialize variables to None) and mypy 0.720 for checking (less false positives)

One of the advantages of enabling check-untyped-defs globally, is the additional checking of PRs.

However, for as long as I can keep this branch in sync with master, this additional checking is effectively being performed.

The goal is to cover as much ground in the shortest possible time. The piecemeal approach of submitting separate PRs was, IMO, not allowing us to see the wood from the trees. The extent of the work involved (and changes required) won't be known till the majority of modules have been covered.

So far, nothing major has surfaced. If latent bugs are discovered could always break bits off and submit for review along the way.




Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel I would appreciate another pair of eyes on https://github.com/pandas-dev/pandas/pull/28339/commits/a398d83f96c4b3c92006308c51b6bedaebe5e91b
_get_counts is already typed 
```
def _get_counts(
    values_shape: Tuple[int],
    mask: Optional[np.ndarray],
    axis: Optional[int],
    dtype=float,
) -> Union[int, np.ndarray]:
```
> I would appreciate another pair of eyes on a398d83

The edits in the tests look good.

The annotation for nanmean looks right.  I _think_ `values` should be `Union[np.ndarray, ExtensionArray]`.

_FuncType and _F make things marginally _less_ clear to me, but I trust that you know what you're doing on that front.
nanmean is decorated by disallow and bottleneck_switch, so this is needed to preserve the types. see https://mypy.readthedocs.io/en/latest/generics.html#declaring-decorators
> The annotation for nanmean looks right. I _think_ `values` should be `Union[np.ndarray, ExtensionArray]`.

added.
> @jbrockmendel I would appreciate another pair of eyes on [a398d83](https://github.com/pandas-dev/pandas/commit/a398d83f96c4b3c92006308c51b6bedaebe5e91b)

Thanks.
Improved check_untyped_defs in mypy 0.740

from the blog http://mypy-lang.blogspot.com/2019/10/mypy-0740-released.html ...

"The self argument to methods without annotations is now correctly given the object’s type (instead of Any) when using check_untyped_defs (PR 7530).

This allows substantially better checking of untyped methods as well as enabling inference of attribute types in some situations where they would not have been determined in the past.

Additionally, we now suppress most “needs type annotation” errors caused by assigning empty collections (i.e, data = []), in untyped functions."

this "substantially better checking" increases the number of errors reported significantly. so the number of outstanding errors to resolve is now...

<details>

    121 pandas\tseries\offsets.py
     67 pandas\core\indexes\base.py
     58 pandas\plotting\_matplotlib\core.py
     51 pandas\core\base.py
     49 pandas\core\generic.py
     44 pandas\core\arrays\base.py
     39 pandas\io\pytables.py
     34 pandas\io\parsers.py
     29 pandas\core\arrays\datetimelike.py
     25 pandas\io\stata.py
     22 pandas\core\internals\blocks.py
     19 pandas\util\testing.py
     19 pandas\core\indexes\datetimelike.py
     18 pandas\core\internals\managers.py
     17 pandas\core\indexes\category.py
     17 pandas\core\computation\pytables.py
     16 pandas\core\indexes\interval.py
     13 pandas\io\json\_json.py
     13 pandas\core\apply.py
     12 pandas\core\computation\ops.py
     11 pandas\io\sas\sas_xport.py
     11 pandas\core\internals\construction.py
     11 pandas\core\groupby\generic.py
     10 pandas\io\excel\_openpyxl.py
      8 pandas\core\indexes\range.py
      8 pandas\core\groupby\base.py
      8 pandas\_version.py
      7 pandas\core\indexes\period.py
      6 pandas\core\resample.py
      6 pandas\core\indexes\datetimes.py
      6 pandas\core\groupby\groupby.py
      6 pandas\core\arrays\sparse\array.py
      5 pandas\core\arrays\interval.py
      5 pandas\_config\config.py
      4 pandas\io\packers.py
      4 pandas\io\excel\_xlsxwriter.py
      4 pandas\core\window\ewm.py
      3 pandas\plotting\_matplotlib\misc.py
      3 pandas\io\html.py
      3 pandas\io\excel\_base.py
      3 pandas\core\indexes\multi.py
      3 pandas\core\groupby\grouper.py
      3 pandas\compat\chainmap.py
      2 pandas\core\indexing.py
      2 pandas\core\indexes\timedeltas.py
      2 pandas\core\indexes\numeric.py
      2 pandas\core\computation\scope.py
      2 pandas\core\computation\expr.py
      2 pandas\core\algorithms.py
      1 pandas\plotting\_misc.py
      1 pandas\io\sas\sas7bdat.py
      1 pandas\io\formats\style.py
      1 pandas\io\formats\csvs.py
      1 pandas\io\excel\_xlwt.py
      1 pandas\io\common.py
      1 pandas\core\window\common.py
      1 pandas\core\reshape\merge.py
      1 pandas\core\groupby\ops.py
      1 pandas\core\frame.py
      1 pandas\core\arrays\string_.py
      1 pandas\core\arrays\period.py
      1 pandas\core\arrays\numpy_.py

</details>

does dropping py35 make this actionable?
I think could start breaking bits off.. the issue is the mypy version i'm using.

mypy wasn't pinned before #29431, so could not know when conda would resolve to newer version. so used latest to keep ahead of the curve.

The problem is using 0.740 this task has become virtually impossible to achieve without additional (community) support https://github.com/pandas-dev/pandas/pull/28339#issuecomment-543684846

started with around 400 errors to invesigate, couple of months to get just over half way, with 0.740 went up to 800.
The first step is to enable --check-untyped-defs globally and lockdown the passing modules. @WillAyd has just given the green light https://gitter.im/pydata/pandas?at=5dc5b425ca778c1fbfc9f3cc
looks like this is green.  does that mean ready for review?
@simonjayhawkins I think would be great to push a PR with things you have typed in this one 
> @simonjayhawkins I think would be great to push a PR with things you have typed in this one

first step is to get #29493 merged to lockdown passing modules.
ci no longer running on this draft branch. @datapythonista any idea?
> ci no longer running on this draft branch. @datapythonista any idea?

I haven't worked with draft PRs, is CI expected to run in them? 
I

> I haven't worked with draft PRs, is CI expected to run in them?

it had been. I think on a couple of occasions in the past it may have not triggered, but the last 3 pushes in a row have not triggered.

The last push that triggered was 200 commits.

maybe later today if I get the chance, I'll create a new dummy PR as draft and see what happens if I don't see any other draft PRs triggering ci. 
@datapythonista ci working again.
closing to reduce noise on notifications when keeping in sync with master
reopening temporarily see https://github.com/pandas-dev/pandas/pull/33274#issuecomment-611932062