MyPy.ini Make Import Machinery Explicit
State: closed

Now that we've wound down a lot of the mistyped annotations in the code base the next logical review point is our import machinery. Previously we used `ignore_missing_imports` for everything in the code base, but this is suggested by Mypy only as a last resort:

https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports

This PR makes the missing imports explicit. Third party libraries I think we'll have to keep until they become available in typeshed. Internal imports may be resolved through either stubfiles or internal import machinery (would need review as follow ups)

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @gwrome @vaibhavhrt 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=h1) Report
> Merging [#26645](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c07d71d13b21e0b6e22146f0f546f1f8e24a64b3?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26645/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26645      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46575    46572       -3     
- Misses       4117     4120       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <100%> (ø)` | :arrow_up: |
| #single | `41.8% <100%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `78.12% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.91% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=footer). Last update [c07d71d...78e0cbf](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=h1) Report
> Merging [#26645](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c07d71d13b21e0b6e22146f0f546f1f8e24a64b3?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26645/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26645      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46575    46572       -3     
- Misses       4117     4120       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.41% <100%> (ø)` | :arrow_up: |
| #single | `41.8% <100%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `78.12% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26645/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.91% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=footer). Last update [c07d71d...78e0cbf](https://codecov.io/gh/pandas-dev/pandas/pull/26645?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Side-note: will mypy look for config in setup.cfg?  A lot of tools are moving to support that, and its nice to keep it all in one place.
Yea we could migrate this to that file as well:

https://mypy.readthedocs.io/en/latest/config_file.html
Yeah keeping all configs at same place i.e. `setup.cfg` is a good idea.
Whew a lot of external dependencies needs to be ignored and typeshed repo doesn't looks too active. Maybe I will add one of the our deps to typeshed. Let me know if you have one in mind(a small and simple one preferably).
why is this a good idea at this time? shouldn't we type more things first?
I’m not totally against deferring but figured since mypy doesn’t suggest the previous method we had of totally ignoring missing imports I would rather make that explicit now so as to not silently introduce any new import errors with new annotations or changes

Sent from my iPhone

> On Jun 6, 2019, at 10:42 AM, Jeff Reback <notifications@github.com> wrote:
> 
> why is this a good idea at this time? shouldn't we type more things first?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.

Is this actually the recommended solution by mypy? it seems very strange since no libraries (that we need) actually have stubs.
Calling it the recommended solution might be a little strong but our current method is the discouraged approach. Citing link from the OP:

> If the module is a third party library, but you cannot find any existing type hints nor have time to write your own, you can silence the errors:

> 1. To silence a single missing import error, add a # type: ignore at the end of the line containing the import.
> 2. To silence all missing import imports errors from a single library, add a section to your mypy config file for that library setting ignore_missing_imports to True. For example, suppose your codebase makes heavy use of an (untyped) library named foobar. You can silence all import errors associated with that library and that library alone by adding the following section to your config file:
> ...
> 3. To silence all missing import errors for all libraries in your codebase, invoke mypy with the --ignore-missing-imports command line flag or set the ignore_missing_imports config file option to True in the global section of your mypy config file:
> ...
> **We recommend using this approach only as a last resort: it’s equivalent to adding a # type: ignore to all unresolved imports in your codebase.**

So this PR would essentially move us from option 3 to option 2 above

I am probably ok with option 2 for external libraries

but what about our internal ones? (the _libs) these we should ‘fix’
Yea agreed on internal. Not saying we just keep these as is but we are ignoring them with the current mypy file. This PR makes it explicit and gives a checklist of items we can prioritize with the community to tackle
Closing for now to keep queue clean. Can always revisit