Repository: urllib3/urllib3

Issue:
Start shipping our own type stubs
Typeshed houses very rudimentary type stubs for urllib3 but they're hidden within `requests.packages.urllib3` so Mypy can't take advantage of them when using urllib3 on its own. We could also probably improve on what is already in typeshed by adding better types than `Any` everywhere.


Discussion:
Hi @sethmlarson! I'm at Recurse Center and got introduced to his by @shazow. Could I clarify, as per the title, what you have in mind is to have the `.pyi` files alongside `.py` files? Would the first step simply involve porting the one in [requests](https://github.com/python/typeshed/tree/master/third_party/2and3/requests/packages/urllib3), and then expanding on the `Any`?

I've actually been a little stuck getting the stubs hidden in [requests](https://github.com/python/typeshed/tree/master/third_party/2and3/requests/packages/urllib3) to work with `mypy`, even after importing `requests`.

For additional context, I spent some time in my previous making py2 code compatible with py2/py3 and adding type comments. The thought was there's less friction not having to review separate `.pyi` files when refactoring.
@savarin Glad you're interested in helping us here.  I was thinking starting with the type stubs that typeshed has (pyi) and making the `Any` more specific.  We can do this work incrementally, to be easily reviewable. 

Let me know if you have additional questions!
@sethmlarson, would you prefer that we modify the stubs in typeshed or would you rather have the .pyi files in this repo checked in alongside the .py files? OR, would you rather add the type info into the .py files?
I'd like the pyi files shipped with urllib3.
Cool, sounds like a good starting point. Two questions, stemming from my lack of familiarity with stubs and/or `typeshed`.

1. How do you run a type check with the existing `typeshed` [stubs](https://github.com/python/typeshed/tree/master/third_party/2and3/requests/packages/urllib3) of `urllib3` since it's in `requests`? Detailed run below, importing `requests` doesn't seem to help.

```
$ cat run.py

import requests
import urllib3

def pool():
    return urllib3.PoolManager()

$ mypy run.py
run.py:3: error: Skipping analyzing 'urllib3': found module but no type hints or library stubs
run.py:3: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
```

2. I presume all the changes would be in the `typeshed` [repo](https://github.com/python/typeshed/tree/master/third_party/2and3/requests/packages/urllib3) alone. What's the anticipated flow to check these changes are correct (since `typeshed` normally comes with `mypy`), i.e. clone `typeshed`, move to `MYPYPATH`, run `mypy` on `urrlib3` tests (since stubs won't check the code [itself](https://github.com/python/mypy/issues/5028))?
+1 for shipping our own type stubs. `.pyi` files aren't ideal, but types as comments aren't either. Hopefully in 2021 we'll be able to drop Python 2 support in urllib3 2.0.

I have one question though: does anyone understand how providing types for `requests.packages.urllib3` in typeshed achieves anything?

urllib3 used to be vendored in requests, but it's no longer the case: requests now directly imports urllib3, without going through [requests.package.urllib3](https://github.com/psf/requests/blob/master/requests/packages.py) which only exists for backwards-compatibility: it allows others to continue importing urllib3 through `requests.package`.
@pquentin I'm guessing it's for backwards compatibility :)

@savarin All changes would be made in the urllib3 repo, not typeshed. I'd recommend the following approach:

- Add the type stubs from typeshed to urllib3 and add necessary changes to ensure pyi files are added to dists

- File by file update all `Any` usages with specific types

- :tada: Profit!
> urllib3 used to be vendored in requests, but it's no longer the case

@pquentin Ah this makes more sense now, nullifies questions I had above.

@sethmlarson This is helpful, thank you! After the port, what would the check for correctness be? As above, my understanding is stubs can't run checks on the code so I presume it'll just be on `urllib3/test`.
Mypy can check that types are consistent within themselves.

Maybe the approach I took with Eland could be useful for us? https://github.com/elastic/eland/blob/master/noxfile.py#L75
Including the dependency [tree](https://svgshare.com/i/NtH.svg) from https://github.com/urllib3/urllib3/pull/1924 to this issue.
@pquentin Didn't mean to cause confusion deleting a prior comment. I'm getting some strange behavior (e.g. not getting a `mypy` error when I comment out https://github.com/urllib3/urllib3/commit/33c4c13e4d3a5ce77b449b825d8f7214df2e2806, even after introducing a breakpoint and looking through all the errors) which I can't quite explain. I suspect it's something to do with the `mypy` cache, will fiddle around a bit more.
Agreed that something weird is going on. I'm not familiar enough with mypy to help, sorry!
@pquentin No worries, just wanted to check it's something you're seeing as well.

I've tested this out a bit more, as discussed commenting out https://github.com/urllib3/urllib3/commit/33c4c13e4d3a5ce77b449b825d8f7214df2e2806 doesn't trigger any mypy errors with `nox -s lint`. I got it to trigger by importing creating a file that imports from `_implementation.py` (compare test passing in https://github.com/urllib3/urllib3/commit/41f39577364206a79c104ef2f320655b38031720 vs failing in https://github.com/urllib3/urllib3/commit/207a5698297dadfbce19b3d057f27a0373b47e72). There's a discussion on a similar issue [here](https://stackoverflow.com/questions/37537194/using-mypy-local-stubs).

@sethmlarson Let me know if you have any thoughts. I'll continue adding stubs and testing it on toy imports where possible. It's a little unsettling that the existing mypy tests aren't raising errors (without the toy import).

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -4,41 +4,6 @@
 
 import nox
 
-# Whenever type-hints are completed on a file it should be added here so that
-# this file will continue to be checked by mypy. Errors from other files are
-# ignored.
-TYPED_FILES = {
-    "src/urllib3/contrib/__init__.py",
-    "src/urllib3/contrib/_securetransport/bindings.py",
-    "src/urllib3/contrib/_securetransport/low_level.py",
-    "src/urllib3/contrib/ntlmpool.py",
-    "src/urllib3/contrib/pyopenssl.py",
-    "src/urllib3/contrib/securetransport.py",
-    "src/urllib3/contrib/socks.py",
-    "src/urllib3/__init__.py",
-    "src/urllib3/connection.py",
-    "src/urllib3/connectionpool.py",
-    "src/urllib3/exceptions.py",
-    "src/urllib3/_collections.py",
-    "src/urllib3/fields.py",
-    "src/urllib3/filepost.py",
-    "src/urllib3/poolmanager.py",
-    "src/urllib3/request.py",
-    "src/urllib3/response.py",
-    "src/urllib3/util/connection.py",
-    "src/urllib3/util/proxy.py",
-    "src/urllib3/util/queue.py",
-    "src/urllib3/util/response.py",
-    "src/urllib3/util/ssl_.py",
-    "src/urllib3/util/ssl_match_hostname.py",
-    "src/urllib3/util/ssltransport.py",
-    "src/urllib3/util/url.py",
-    "src/urllib3/util/request.py",
-    "src/urllib3/util/retry.py",
-    "src/urllib3/util/timeout.py",
-    "src/urllib3/util/util.py",
-    "src/urllib3/util/wait.py",
-}
 SOURCE_FILES = [
     "docs/",
     "dummyserver/",
@@ -146,27 +111,7 @@ def mypy(session):
     session.install("mypy==0.812")
     session.install("idna>=2.0.0")
     session.run("mypy", "--version")
-
-    session.log("mypy --strict src/urllib3")
-    all_errors, errors = [], []
-    process = subprocess.run(
-        ["mypy", "--strict", "src/urllib3"],
-        env=session.env,
-        text=True,
-        stdout=subprocess.PIPE,
-        stderr=subprocess.STDOUT,
-    )
-    # Ensure that mypy itself ran successfully
-    assert process.returncode in (0, 1)
-
-    for line in process.stdout.split("\n"):
-        all_errors.append(line)
-        filepath = line.partition(":")[0]
-        if filepath.replace(".pyi", ".py") in TYPED_FILES:
-            errors.append(line)
-    session.log(f"all errors count: {len(all_errors)}")
-    if errors:
-        session.error("\n" + "\n".join(sorted(set(errors))))
+    session.run("mypy", "--strict", "src/urllib3")
 
 
 @nox.session


Test patch: