Requests stops respecting configured proxies

There is an open bug report on pip (pypa/pip#1805) after some back and forth this appears to be a legit requests (or maybe urllib3) bug.

Using requests 2.3.0 I had one of the people run a script like this:

``` python
import requests
session = requests.Session()
for i in range(100):
    print(i)
    session.get("https://pypi.python.org/simple/").content
```

They said they had both `http_proxy` and `https_proxy` environment variables set up, and when they ran it they got this output: https://dpaste.de/doVg/raw

On the pip ticket people have said they got this while configuring the proxy via `--proxy` which translates to ``session.proxies ={"https": "<value>", "http": "<value>"}.


I wonder if this has anything to do with Connection Pooling. @shazow?


In fact, the weird increments this is happening at suggest it. Your ticket has it happening at ~16, this one is coming at 51. Very suspicious.


That said, we use a pool size of 10 by default, which isn't really close to either of those.


For the record, there is roughly 2-3 HTTP requests made for every package installed, so that 16 is more like 32-48 or so. It might even be 50 depending on different factors.


Interesting...


Right, so in the trivial case I can't reproduce this using `mitmproxy`, though `mitmproxy` is a bad example in this case because it obviously MITMs the request.


Might be interesting to get the person who can repro this bug to turn connection pooling off (create a `HTTPAdapter` with the pool size set to 1).


They ate kibad on irc. I think they joined the requests irc channel

> On Jun 24, 2014, at 1:10 PM, Cory Benfield notifications@github.com wrote:
> 
> Might be interesting to get the person who can repro this bug to turn connection pooling off (create a HTTPAdapter with the pool size set to 1).
> 
> —
> Reply to this email directly or view it on GitHub.


Let's try this then:

``` python
import requests
from requests.adapters import HTTPAdapter
session = requests.Session()
s.mount('http://', HTTPAdapter(pool_maxsize=1))
s.mount('https://', HTTPAdapter(pool_maxsize=1))
for i in range(100):
    print(i)
    session.get("https://pypi.python.org/simple/").content
```

I'd be interested to see how this affects the result.


Changing the authority (from PyPI to elsewhere) seems to fix the bug. This suggests an interaction between PyPI and the user's proxy. We'll close this for now, but we should be alert to the idea that this issue might come back to us.


It could be proxy-retry-related. I'm not sure off the top of my head how we handle retrying when proxies are at play (do we retry proxy timeouts?), and not sure if Requests does its own thing for that too. (Just a thought.)


Thanks for quickly helping in nailing down this issue.


Configured local pypi mirror devpi-server and tried creating venv using tox. Ended up with read time out error. Log can be found here https://dpaste.de/d8yw/raw


## Should've finished reading that long pypa/pip thread, sorry for the noise…

This emulates what devpi does, and fails reliably _after_ index 50 (i.e. in the 52nd request). With "Connection: close" added, it does so in the 2nd request, which might hint at some relation to persistent connections, and the "magic" number (here: 51±1) could be MaxKeepAliveRequests or similar in the proxy.

Also, GET requests (for the PyPI index, and BLOBs) on the same installation pose no problem at all – it's just the XMLRPC POSTs.

``` py
import sys, os

try:
    import xmlrpc.client as xmlrpc
except ImportError:
    import xmlrpclib as xmlrpc

import requests
from requests.adapters import HTTPAdapter

agent = "devpi-%s/%s" % ("devpi", "2.0.2")
agent += " (py%s; %s)" % (sys.version.split()[0], sys.platform)
session = requests.Session()
session.headers["user-agent"] = agent
session.headers["content-type"] = "text/xml"
session.headers["Accept"] = "text/xml"
#session.headers["Connection"] = "close"
#session.mount('http://', HTTPAdapter(pool_maxsize=1))
#session.mount('https://', HTTPAdapter(pool_maxsize=1))

for i in range(55):
    payload = xmlrpc.dumps((1201322,), "changelog_since_serial")
    reply = session.post("https://pypi.python.org/pypi/", data=payload, stream=False, timeout=5.0)
    print i, len(reply.content)
```


I know this is old & closed, but the problem persists. Please tell me what you mean by "changing the authority (from PyPI to elsewhere) seems to fix the bug".  I'm just trying to install packages via pip3.  Can I "change the authority"?

The authority is the broader term of the URI which includes user information/authentication, host/IPv4/IPv6, and port. The point about changing the authority was that there were no issues communicating with a different service than PyPI (changing the authority) and that indicated an issue between the proxy and PyPI.