Repository: pypa/pip

Issue:
Network failure leads to TypeError in retry
I got this traceback after a network failure with urllib3 v1.10 on Linux:

```
Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 135, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/connection.py", line 66, in create_connection
    for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
  File "/usr/lib64/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 782, in _validate_conn
    conn.connect()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 215, in connect
    conn = self._new_conn()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 144, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f86e18d8198>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/usr/lib64/python3.4/site-packages/pip/commands/install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 461, in _prepare_file
    req_to_install.populate_link(finder, self.upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_install.py", line 250, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 486, in find_requirement
    all_versions = self._find_all_versions(req.name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 404, in _find_all_versions
    index_locations = self._get_index_urls_locations(project_name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 378, in _get_index_urls_locations
    page = self._get_page(main_index_url)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 818, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 928, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib64/python3.4/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib64/python3.4/site-packages/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/lib64/python3.4/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 226, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
```

Somehow pip has a Retry object which is incompatible.
I reported this first with urllib3 in https://github.com/shazow/urllib3/issues/742 who say it is a bug in pip (or requests).



Discussion:
I used pip3.4 from Gentoo repos.

I used pip to install a local egg directory, which loaded some dependencies off the web (sqlalchemy to be precise)

I did not trim the traceback.

@JohannesBuchner: It looks like the pip dependencies have been debundled. It would be interesting to know what version of the dependencies you have compared to what pip 7.1.2 expects: https://github.com/pypa/pip/blob/7.1.2/pip/_vendor/vendor.txt

I think Gentoo does not debundle

the following is from the ebuild file

# Check pip/_vendor/vendor.txt for this

VENDOR_DEPEND="
        >=dev-python/distlib-0.2.1[${PYTHON_USEDEP}]
        >=dev-python/html5lib-0.999999[${PYTHON_USEDEP}]
        >=dev-python/six-1.9[${PYTHON_USEDEP}]
        >=dev-python/colorama-0.3.3[${PYTHON_USEDEP}]
        >=dev-python/requests-2.7.0[${PYTHON_USEDEP}]
        >=dev-python/CacheControl-0.11.5[${PYTHON_USEDEP}]
        >=dev-python/lockfile-0.10.2[${PYTHON_USEDEP}]
        >=dev-python/progress-1.2[${PYTHON_USEDEP}]
        >=dev-python/packaging-15.3[${PYTHON_USEDEP}]
        >=dev-python/retrying-1.3.3[${PYTHON_USEDEP}]
        virtual/python-ipaddress[${PYTHON_USEDEP}]
"
RDEPEND="${VENDOR_DEPEND}
        >=dev-python/setuptools-18.2[${PYTHON_USEDEP}]
"

@xavfernandez ah I had missed that. I was more focused on whether urllib3 had been debundled from requests, not whether pip's dependencies had been debundled from pip.

@JohannesBuchner can you tell us what version of requests was installed? And they definitely have been debundled. I can tell by the traceback.

---

I'm looking at pip and https://github.com/pypa/pip/blob/c20ed50a82d7f8af793c9754f3c936794aa0b56d/pip/download.py#L323 seems to be the only place it creates a `Retry` object. I have to wonder if the `isinstance` check in urllib3 will not work when `requests` is debundled.

I have requests 2.8.1

So yeah, I'm willing to bet that because of the [vendored](https://github.com/pypa/pip/blob/7.1.2/pip/_vendor/__init__.py#L29..L38) function in `pip._vendor` we have a problem where `isinstance` checks fail. I wonder if this would be fixed by appending to that list

``` py
    vendored('requests.packages')
    vendored('requests.packages.urllib3')
    # urllib3's subpackages
```

@JohannesBuchner to confirm my suspicion can you run the following code?

``` py
from pip._vendor.requests.packages import urllib3
from requests.packages import urllib3 as _urllib3

r = urllib3.Retry(total=10)
if isinstance(r, _urllib3.Retry):
    print('@sigmavirus24 is wrong')
else:
    print('@sigmavirus24 is right')
```

If that ends up being correct, I'm going to write a quick patch and see if applying it helps you @JohannesBuchner because this will be a problem for other distros too (to some extent) probably.

(FWIW, my guess about needing to specify `requests.packages.*.*.*` is only because I see `six`, `six.moves`, and `six.moves.urllib` there so I'm extrapolating just a tiny bit.)

The code output says you are right.

I doubt I will be able to reproduce the conditions of the network failure.

Ah, that's true. @Lukasa thoughts on how to potentially reproduce the network failure?

Easiest way is to patch the socket module to force it to throw that particular error from getaddrinfo.


Resolving patch:
diff --git a/pip/_vendor/__init__.py b/pip/_vendor/__init__.py
--- a/pip/_vendor/__init__.py
+++ b/pip/_vendor/__init__.py
@@ -64,3 +64,37 @@ def vendored(modulename):
     vendored("progress")
     vendored("retrying")
     vendored("requests")
+    vendored("requests.packages")
+    vendored("requests.packages.urllib3")
+    vendored("requests.packages.urllib3._collections")
+    vendored("requests.packages.urllib3.connection")
+    vendored("requests.packages.urllib3.connectionpool")
+    vendored("requests.packages.urllib3.contrib")
+    try:
+        vendored("requests.packages.urllib3.contrib.ntlmpool")
+    except ImportError:
+        pass
+    try:
+        vendored("requests.packages.urllib3.contrib.pyopenssl")
+    except ImportError:
+        pass
+    vendored("requests.packages.urllib3.exceptions")
+    vendored("requests.packages.urllib3.fields")
+    vendored("requests.packages.urllib3.filepost")
+    vendored("requests.packages.urllib3.packages")
+    vendored("requests.packages.urllib3.packages.ordered_dict")
+    vendored("requests.packages.urllib3.packages.six")
+    vendored("requests.packages.urllib3.packages.ssl_match_hostname")
+    vendored("requests.packages.urllib3.packages.ssl_match_hostname."
+             "_implementation")
+    vendored("requests.packages.urllib3.poolmanager")
+    vendored("requests.packages.urllib3.request")
+    vendored("requests.packages.urllib3.response")
+    vendored("requests.packages.urllib3.util")
+    vendored("requests.packages.urllib3.util.connection")
+    vendored("requests.packages.urllib3.util.request")
+    vendored("requests.packages.urllib3.util.response")
+    vendored("requests.packages.urllib3.util.retry")
+    vendored("requests.packages.urllib3.util.ssl_")
+    vendored("requests.packages.urllib3.util.timeout")
+    vendored("requests.packages.urllib3.util.url")


Test patch: