Repository: pypa/pip

Issue:
pip 10 fails silently after retrying extra index urls if those hosts return non-2xx status
* Pip version: 10.0.0
* Python version: 3.6.3
* Operating system: Ubuntu 17.10

### Description:

At our company we have a local PyPI server (PyShop v1.2.3).
When trying to install public package (not cached in our pypi) pip successfully finds
it at pypi.org but receives `500 internal server error` from our proxy.
It retries several times and then fails silently (just an exit status reports there was an error),
although the package is available at pypi.org.

Previous versions of pip was just ignoring this 500s.

### What I've run:

```bash
# using pip v10:
$ pip -vvv install -i https://pypi.python.org/simple --extra-index-url http://pypi.company/simple --trusted-host pypi.company hiredis
Created temporary directory: /tmp/pip-ephem-wheel-cache-41drbbbv                                                                                                                                                                                                               
Created temporary directory: /tmp/pip-install-cypbxh8o                                                                                                                                                                                                                         
Looking in indexes: https://pypi.python.org/simple, http://pypi.company/simple                                                                                                                                                                                          
Collecting hiredis                                                                                                                                                                                                                                                             
  2 location(s) to search for versions of hiredis:                                                                                                                                                                                                                             
  * https://pypi.python.org/simple/hiredis/                                                                                                                                                                                                                                    
  * http://pypi.company/simple/hiredis/                                                                                                                                                                                                                                 
  Getting page https://pypi.python.org/simple/hiredis/                                                                                                                                                                                                                         
  Looking up "https://pypi.python.org/simple/hiredis/" in the cache                                                                                                                                                                                                            
  Current age based on date: 98771                                                                                                                                                                                                                                             
  Freshness lifetime from max-age: 600                                                                                                                                                                                                                                         
  Freshness lifetime from request max-age: 600                                                                                                                                                                                                                                 
  The cached response is "stale" with no etag, purging                                                                                                                                                                                                                         
  Starting new HTTPS connection (1): pypi.python.org                                                                                                                                                                                                                           
  https://pypi.python.org:443 "GET /simple/hiredis/ HTTP/1.1" 302 98                                                                                                                                                                                                           
  Status code 302 not in (200, 203, 300, 301)                                                                                                                                                                                                                                  
  Looking up "https://pypi.org/simple/hiredis/" in the cache                                                                                                                                                                                                                   
  No cache entry available                                                                                                                                                                                                                                                     
  Starting new HTTPS connection (1): pypi.org                                                                                                                                                                                                                                  
  https://pypi.org:443 "GET /simple/hiredis/ HTTP/1.1" 200 1183                                                                                                                                                                                                                
  Updating cache with response from "https://pypi.org/simple/hiredis/"                                                                                                                                                                                                         
  Caching due to etag                                                                                                                                                                                                                                                          
  Analyzing links from page https://pypi.org/simple/hiredis/                                                                                                                                                                                                                   
    Found link https://files.pythonhosted.org/packages/ee/f6/8f06c6a7f7eef2e23b651ed600cea37543bd7cdaa89d64c8e0541cc654fa/hiredis-0.0.1.tar.gz#sha256=f88444cb393a9fd4a3da4daabd432507df2aa5a728900963796be96f221c1a3e (from https://pypi.org/simple/hiredis/), version: 0.0.1 
    Found link https://files.pythonhosted.org/packages/79/3c/76d46e33591c14735fc3416862df80fd44cb5fe34a4a9a6ea2b491def1e7/hiredis-0.0.3.tar.gz#sha256=5dcd00fc25dd96bcd7ba5d5da712014bf4aa6c2313b82e5773254a16977db4f4 (from https://pypi.org/simple/hiredis/), version: 0.0.3 
    Found link https://files.pythonhosted.org/packages/b3/16/fba970b7bac5821038d6d0bc2ab84236363c6001e1d3df1596622471d93b/hiredis-0.0.4.tar.gz#sha256=5195b943a5ae885f6a759ce532b06bb5490b875f98c2040c92473898d5b26bcf (from https://pypi.org/simple/hiredis/), version: 0.0.4 
    Found link https://files.pythonhosted.org/packages/84/c4/a4584c3af1f75a93b056dd54eeae1337ac1bf6a02d44d83e836d7430dfab/hiredis-0.1.0.tar.gz#sha256=06648b06d6f8380c31c3e7e0e6790c2b3ddf1e3f74ac8998fef7b2dc46da82f2 (from https://pypi.org/simple/hiredis/), version: 0.1.0 
    Found link https://files.pythonhosted.org/packages/66/d2/ee4495855b5b487446d56bba06763e86770ff7777d5fb69cf0b6e485a3f3/hiredis-0.1.1.tar.gz#sha256=ca7b3840d12f836dabfd73708bd64c0ac31204b22a926279b6515e6433e50ffc (from https://pypi.org/simple/hiredis/), version: 0.1.1 
    Found link https://files.pythonhosted.org/packages/c2/72/97ea0dc6ce9b942bf5e0442008df8c2cd9ae70615af5f6741d1f544758cc/hiredis-0.1.2.tar.gz#sha256=c44df954d9dcc9f57682bf8fabf172a112edf53ff35bb5ce8f8fff1b1985abb1 (from https://pypi.org/simple/hiredis/), version: 0.1.2 
    Found link https://files.pythonhosted.org/packages/52/64/717becc19cffd5f6844c1dfda799429677c772136d011e649f3b08ad81cd/hiredis-0.1.3.tar.gz#sha256=77b7b31fb81d74d356206fae891a18d6095279611bc1d93e927900208173fca1 (from https://pypi.org/simple/hiredis/), version: 0.1.3 
    Found link https://files.pythonhosted.org/packages/74/34/a57fcb52ce295b8744e0813d1cf671943a13a75f88c319c4fe9ae7698c08/hiredis-0.1.4.tar.gz#sha256=eb78e5b2f9f3b5f460b1da24b2d5e6179b3f918295697f8e3a71e982619d1a11 (from https://pypi.org/simple/hiredis/), version: 0.1.4 
    Found link https://files.pythonhosted.org/packages/54/63/c017be3f439eeb1199b05bafdf4c8f3cbd0f2f7ab3702e75bbf4ff291234/hiredis-0.1.5.tar.gz#sha256=82faae39716e80769501e33e2ae37dac84891b9b5fed19ed3e0c580573619f70 (from https://pypi.org/simple/hiredis/), version: 0.1.5 
    Found link https://files.pythonhosted.org/packages/27/d2/5e4d2ac0b3161f00ce7251f51c7cb86dfc51a6fe309e550cabddd6926f65/hiredis-0.1.6.tar.gz#sha256=b6df262a0231d8172fe1941e589f04e51974b273209bae6d0d8e235daa3b6a35 (from https://pypi.org/simple/hiredis/), version: 0.1.6 
    Found link https://files.pythonhosted.org/packages/1b/98/4766d85124b785ff1989ee1c79631a1b6ecfcb444ff39999a87877b2027e/hiredis-0.2.0.tar.gz#sha256=ca958e13128e49674aa4a96f02746f5de5973f39b57297b84d59fd44d314d5b5 (from https://pypi.org/simple/hiredis/), version: 0.2.0 
  Getting page http://pypi.company/simple/hiredis/                                                                                                                                                                                                                      
  Starting new HTTP connection (1): pypi.company                                                                                                                                                                                                                                
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110                                                                                                                                                                                                                
  Incremented Retry for (url='/simple/hiredis/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)                                                                                                                                                          
  Retry: /simple/hiredis/                                                                                                                                                                                                                                                      
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110
  Incremented Retry for (url='/simple/hiredis/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
  Retry: /simple/hiredis/
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110
  Incremented Retry for (url='/simple/hiredis/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
  Retry: /simple/hiredis/
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110
  Incremented Retry for (url='/simple/hiredis/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
  Retry: /simple/hiredis/
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110
  Incremented Retry for (url='/simple/hiredis/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
  Retry: /simple/hiredis/
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110

Traceback (most recent call last):
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/requests/adapters.py", line 440, in send
    timeout=timeout
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  [Previous line repeated 1 more times]
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/urllib3/connectionpool.py", line 712, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
pip._vendor.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='pypi.company', port=80): Max retries exceeded with url: /simple/hiredis/ (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 243, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 307, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/index.py", line 484, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/index.py", line 442, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/index.py", line 587, in _get_pages
    page = self._get_page(location)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/index.py", line 705, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/index.py", line 814, in get_page
    "Cache-Control": "max-age=600",
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_internal/download.py", line 397, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "~/venv/tmp/lib/python3.6/site-packages/pip/_vendor/requests/adapters.py", line 499, in send
    raise RetryError(e, request=request)
pip._vendor.requests.exceptions.RetryError: HTTPConnectionPool(host='pypi.company', port=80): Max retries exceeded with url: /simple/hiredis/ (Caused by ResponseError('too many 500 error responses',))
```
```bash
# using pip v9.0.3
$ pip -vvv install -i https://pypi.python.org/simple --extra-index-url http://pypi.company/simple --trusted-host pypi.company hiredis
Collecting hiredis                                                                                                                                                                                                                                                             
  2 location(s) to search for versions of hiredis:                                                                                                                                                                                                                             
  * https://pypi.python.org/simple/hiredis/                                                                                                                                                                                                                                    
  * http://pypi.company/simple/hiredis/                                                                                                                                                                                                                                 
  Getting page https://pypi.python.org/simple/hiredis/                                                                                                                                                                                                                         
  Looking up "https://pypi.python.org/simple/hiredis/" in the cache                                                                                                                                                                                                            
  No cache entry available                                                                                                                                                                                                                                                     
  Starting new HTTPS connection (1): pypi.python.org                                                                                                                                                                                                                           
  https://pypi.python.org:443 "GET /simple/hiredis/ HTTP/1.1" 302 98                                                                                                                                                                                                           
  Status code 302 not in [200, 203, 300, 301]                                                                                                                                                                                                                                  
  Looking up "https://pypi.org/simple/hiredis/" in the cache                                                                                                                                                                                                                   
  Cache entry deserialization failed, entry ignored                                                                                                                                                                                                                            
  Starting new HTTPS connection (1): pypi.org                                                                                                                                                                                                                                  
  https://pypi.org:443 "GET /simple/hiredis/ HTTP/1.1" 200 1183                                                                                                                                                                                                                
  Updating cache with response from "https://pypi.org/simple/hiredis/"                                                                                                                                                                                                         
  Caching due to etag                                                                                                                                                                                                                                                          
  Analyzing links from page https://pypi.org/simple/hiredis/                                                                                                                                                                                                                   
    Found link https://files.pythonhosted.org/packages/ee/f6/8f06c6a7f7eef2e23b651ed600cea37543bd7cdaa89d64c8e0541cc654fa/hiredis-0.0.1.tar.gz#sha256=f88444cb393a9fd4a3da4daabd432507df2aa5a728900963796be96f221c1a3e (from https://pypi.org/simple/hiredis/), version: 0.0.1 
    Found link https://files.pythonhosted.org/packages/79/3c/76d46e33591c14735fc3416862df80fd44cb5fe34a4a9a6ea2b491def1e7/hiredis-0.0.3.tar.gz#sha256=5dcd00fc25dd96bcd7ba5d5da712014bf4aa6c2313b82e5773254a16977db4f4 (from https://pypi.org/simple/hiredis/), version: 0.0.3 
    Found link https://files.pythonhosted.org/packages/b3/16/fba970b7bac5821038d6d0bc2ab84236363c6001e1d3df1596622471d93b/hiredis-0.0.4.tar.gz#sha256=5195b943a5ae885f6a759ce532b06bb5490b875f98c2040c92473898d5b26bcf (from https://pypi.org/simple/hiredis/), version: 0.0.4 
    Found link https://files.pythonhosted.org/packages/84/c4/a4584c3af1f75a93b056dd54eeae1337ac1bf6a02d44d83e836d7430dfab/hiredis-0.1.0.tar.gz#sha256=06648b06d6f8380c31c3e7e0e6790c2b3ddf1e3f74ac8998fef7b2dc46da82f2 (from https://pypi.org/simple/hiredis/), version: 0.1.0 
    Found link https://files.pythonhosted.org/packages/66/d2/ee4495855b5b487446d56bba06763e86770ff7777d5fb69cf0b6e485a3f3/hiredis-0.1.1.tar.gz#sha256=ca7b3840d12f836dabfd73708bd64c0ac31204b22a926279b6515e6433e50ffc (from https://pypi.org/simple/hiredis/), version: 0.1.1 
    Found link https://files.pythonhosted.org/packages/c2/72/97ea0dc6ce9b942bf5e0442008df8c2cd9ae70615af5f6741d1f544758cc/hiredis-0.1.2.tar.gz#sha256=c44df954d9dcc9f57682bf8fabf172a112edf53ff35bb5ce8f8fff1b1985abb1 (from https://pypi.org/simple/hiredis/), version: 0.1.2 
    Found link https://files.pythonhosted.org/packages/52/64/717becc19cffd5f6844c1dfda799429677c772136d011e649f3b08ad81cd/hiredis-0.1.3.tar.gz#sha256=77b7b31fb81d74d356206fae891a18d6095279611bc1d93e927900208173fca1 (from https://pypi.org/simple/hiredis/), version: 0.1.3 
    Found link https://files.pythonhosted.org/packages/74/34/a57fcb52ce295b8744e0813d1cf671943a13a75f88c319c4fe9ae7698c08/hiredis-0.1.4.tar.gz#sha256=eb78e5b2f9f3b5f460b1da24b2d5e6179b3f918295697f8e3a71e982619d1a11 (from https://pypi.org/simple/hiredis/), version: 0.1.4 
    Found link https://files.pythonhosted.org/packages/54/63/c017be3f439eeb1199b05bafdf4c8f3cbd0f2f7ab3702e75bbf4ff291234/hiredis-0.1.5.tar.gz#sha256=82faae39716e80769501e33e2ae37dac84891b9b5fed19ed3e0c580573619f70 (from https://pypi.org/simple/hiredis/), version: 0.1.5 
    Found link https://files.pythonhosted.org/packages/27/d2/5e4d2ac0b3161f00ce7251f51c7cb86dfc51a6fe309e550cabddd6926f65/hiredis-0.1.6.tar.gz#sha256=b6df262a0231d8172fe1941e589f04e51974b273209bae6d0d8e235daa3b6a35 (from https://pypi.org/simple/hiredis/), version: 0.1.6 
    Found link https://files.pythonhosted.org/packages/1b/98/4766d85124b785ff1989ee1c79631a1b6ecfcb444ff39999a87877b2027e/hiredis-0.2.0.tar.gz#sha256=ca958e13128e49674aa4a96f02746f5de5973f39b57297b84d59fd44d314d5b5 (from https://pypi.org/simple/hiredis/), version: 0.2.0 
  Getting page http://pypi.company/simple/hiredis/                                                                                                                                                                                                                      
  Starting new HTTP connection (1): pypi.company                                                                                                                                                                                                                                
  http://pypi.company:80 "GET /simple/hiredis/ HTTP/1.1" 500 110                                                                                                                                                                                                                
  Could not fetch URL http://pypi.company/simple/hiredis/: 500 Server Error: Internal Server Error for url: http://pypi.company/simple/hiredis/ - skipping
Using version 0.2.0 (newest of versions: 0.0.1, 0.0.3, 0.0.4, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.2.0)                                                                                                                                                        
  Looking up "https://files.pythonhosted.org/packages/1b/98/4766d85124b785ff1989ee1c79631a1b6ecfcb444ff39999a87877b2027e/hiredis-0.2.0.tar.gz" in the cache
...
```


Discussion:
After digging a bit I think this is related to this change:
https://github.com/pypa/pip/pull/4473/files#diff-34481ab2fefef95e8820a46e86004ae7R353

Adding `requests.RetryError` to this code should fix the issue:
https://github.com/pypa/pip/blob/275ab7c/src/pip/_internal/index.py#L834-L845
```diff
        except requests.HTTPError as exc:
            cls._handle_fail(link, exc, url)
+       except requests.RetryError as exc:
+           cls._handle_fail(link, exc, url)
        except SSLError as exc:
            reason = "There was a problem confirming the ssl certificate: "
            reason += str(exc)
            cls._handle_fail(link, reason, url, meth=logger.info)
        except requests.ConnectionError as exc:
            cls._handle_fail(link, "connection error: %s" % exc, url)
        except requests.Timeout:
            cls._handle_fail(link, "timed out", url)
        else:
            return inst
```
Confirm that this happens with pip 10 and pypi-server.
Maybe this is intentional and pip developers have the opinion that custom pypi-like servers should not return 500 for a missing package? It would be nice to know where to start fixing the problem.
Hey @Juanlu001 and @popravich!

I think that non-PyPI servers should return 404s for packages that are missing, like PyPI.

How pip should behave when servers don't do that is something I don't have a good feel for. Maybe @dstufft or @xavfernandez would have something to say here.

Thanks @pradyunsg! I've reported the issue upstream in the meantime.
Hi, @pradyunsg 

> I think that non-PyPI servers should return 404s for packages that are missing, like PyPI.

I agree, however I still believe that its just a small missed part in terms of implementing retry support in
v10: when retry limit is reached requests raises exception that is not a subclass of `HttpError`.
So I think the retry error should be also handled, otherwise it looks like any error causing a server
(including PyPI) to respond with code 500 (for instance broken db connection) will make pip silently fail
even if requested package can be or already found in another index.
I'm personally fine with the patch in https://github.com/pypa/pip/issues/5270#issuecomment-382020265. If someone files a PR for it, I'm not going to block it (or merge it FWIW, since I'm not super familiar with this code).

Resolving patch:
diff --git a/src/pip/_internal/index.py b/src/pip/_internal/index.py
--- a/src/pip/_internal/index.py
+++ b/src/pip/_internal/index.py
@@ -16,7 +16,7 @@
 from pip._vendor.packaging import specifiers
 from pip._vendor.packaging.utils import canonicalize_name
 from pip._vendor.packaging.version import parse as parse_version
-from pip._vendor.requests.exceptions import SSLError
+from pip._vendor.requests.exceptions import RetryError, SSLError
 from pip._vendor.six.moves.urllib import parse as urllib_parse
 from pip._vendor.six.moves.urllib import request as urllib_request
 
@@ -163,6 +163,8 @@ def _get_html_page(link, session=None):
         inst = HTMLPage(resp.content, resp.url, resp.headers)
     except requests.HTTPError as exc:
         _handle_get_page_fail(link, exc, url)
+    except RetryError as exc:
+        _handle_get_page_fail(link, exc, url)
     except SSLError as exc:
         reason = "There was a problem confirming the ssl certificate: "
         reason += str(exc)


Test patch:
diff --git a/tests/unit/test_index.py b/tests/unit/test_index.py
--- a/tests/unit/test_index.py
+++ b/tests/unit/test_index.py
@@ -1,11 +1,13 @@
+import logging
 import os.path
 
 import pytest
-from pip._vendor import html5lib
+from mock import Mock
+from pip._vendor import html5lib, requests
 
 from pip._internal.download import PipSession
 from pip._internal.index import (
-    Link, PackageFinder, _determine_base_url, egg_info_matches,
+    Link, PackageFinder, _determine_base_url, _get_html_page, egg_info_matches,
 )
 
 
@@ -190,3 +192,28 @@ def test_egg_info_matches(egg_info, search_name, expected):
     link = None     # Only used for reporting.
     version = egg_info_matches(egg_info, search_name, link)
     assert version == expected
+
+
+def test_request_http_error(caplog):
+    caplog.set_level(logging.DEBUG)
+    link = Link('http://localhost')
+    session = Mock(PipSession)
+    session.get.return_value = resp = Mock()
+    resp.raise_for_status.side_effect = requests.HTTPError('Http error')
+    assert _get_html_page(link, session=session) is None
+    assert (
+        'Could not fetch URL http://localhost: Http error - skipping'
+        in caplog.text
+    )
+
+
+def test_request_retries(caplog):
+    caplog.set_level(logging.DEBUG)
+    link = Link('http://localhost')
+    session = Mock(PipSession)
+    session.get.side_effect = requests.exceptions.RetryError('Retry error')
+    assert _get_html_page(link, session=session) is None
+    assert (
+        'Could not fetch URL http://localhost: Retry error - skipping'
+        in caplog.text
+    )