Typing: Support New Mypy Semantic Analyzer
State: closed

- [x] closes #27018


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Does this cause Mypy failures? Wasn’t think we’d add this as a setting as much as just fix the errors that using it would cause
^ I'm just checking that it does cause failures. This way we can verify i've actually fixed the issues
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=h1) Report
> Merging [#27070](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d94146cf5d5d9f899a31308d0a4d9e5d132f6442?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27070/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #27070      +/-   ##
==========================================
- Coverage   92.04%   92.03%   -0.01%     
==========================================
  Files         180      180              
  Lines       50714    50714              
==========================================
- Hits        46680    46675       -5     
- Misses       4034     4039       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.67% <ø> (ø)` | :arrow_up: |
| #single | `41.86% <ø> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=footer). Last update [d94146c...c098d1b](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=h1) Report
> Merging [#27070](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a173cacbffbd49c0aecafda76ace7aed9b0af60c?src=pr&el=desc) will **increase** coverage by `0.16%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27070/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #27070      +/-   ##
==========================================
+ Coverage   91.87%   92.03%   +0.16%     
==========================================
  Files         180      180              
  Lines       50834    50714     -120     
==========================================
- Hits        46703    46675      -28     
+ Misses       4131     4039      -92
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.67% <ø> (+0.16%)` | :arrow_up: |
| #single | `41.86% <ø> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/io/excel/\_openpyxl.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL19vcGVucHl4bC5weQ==) | `84.71% <0%> (-3.23%)` | :arrow_down: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `96.3% <0%> (-1.32%)` | :arrow_down: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.95% <0%> (-0.8%)` | :arrow_down: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `90.72% <0%> (-0.34%)` | :arrow_down: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `94.19% <0%> (-0.31%)` | :arrow_down: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `95.8% <0%> (-0.28%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.92% <0%> (-0.21%)` | :arrow_down: |
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `88.82% <0%> (-0.2%)` | :arrow_down: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.17% <0%> (-0.16%)` | :arrow_down: |
| ... and [40 more](https://codecov.io/gh/pandas-dev/pandas/pull/27070/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=footer). Last update [a173cac...04b49ed](https://codecov.io/gh/pandas-dev/pandas/pull/27070?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Gotcha. Just FYI you can also run `mypy --new-semantic-analyzer pandas` locally to see
first two errors removed, unsure of how to deal with the mypy error
```
pandas/io/json/__init__.py:5: error: Name 'json' is not defined
```

Due to the fact we have: ( in our __init__.py)
```
del json, normalize, table_schema  # noqa
```

Any advice @WillAyd ?

Does converting the imports to abolute help at all? Seems like kind of a strange __init__ file - maybe modeling it after pandas.io.excel would help and allow for removal of noqa
Hello @alimcmaster1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-07 20:44:20 UTC
Just remove the noqa - shouldn't need them 
Latest test failures seem unrelated: ( currently looking )

```
pandas\tests\indexes\multi\test_format.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <contextlib._GeneratorContextManager object at 0x000002299F625860>
type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               AssertionError: Did not see expected warning of class 'FutureWarning'.
```
Code changes look good. Can you remove changes to mypy.ini and any requirements files?

@jreback any objection to renaming of files in json folder?
no objections
Thanks for the review @WillAyd . 

Just for my understanding what’s your motivation for removing the changes to mypy.ini it’s whats validating the mypy semantic analysis in CI - to avoid anyone introducing further errors?

Thanks 
> Just for my understanding what’s your motivation for removing the changes to mypy.ini it’s whats validating the mypy semantic analysis in CI - to avoid anyone introducing further errors?

Because we don't need to really pin a dependency here. Any of these changes are backwards compatible we are just preventing future mypy versions from breaking; no need to force a min version
@alimcmaster1 isn't the commit adding the semantic analyzer removed? IOW this doesn't look like it is triggering it.
I asked to remove the semantic analyzer from the config because it naturally becomes the default in the next Mypy release. Figured no need to set then and not setting maintains better backwards compat 
> I asked to remove the semantic analyzer from the config because it naturally becomes the default in the next Mypy release. Figured no need to set then and not setting maintains better backwards compat

ok, then we should set a min on mypy (when this comes out)? or at least on our CI do that?
We don’t have to. The new analyzer is more strict so anything done here works with or without it. 

But yea if we wanted to play it safe could set min version to next release when it comes out

Sent from my iPhone

> On Jul 5, 2019, at 11:51 AM, Jeff Reback <notifications@github.com> wrote:
> 
> I asked to remove the semantic analyzer from the config because it naturally becomes the default in the next Mypy release. Figured no need to set then and not setting maintains better backwards compat
> 
> ok, then we should set a min on mypy (when this comes out)? or at least on our CI do that?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

Updated as per your comments and green @jreback 
thanks @alimcmaster1