regression over 2.6.0 with session behind proxy

Hello everyone

Here is a piece of code that used to work until Requests 2.6.0

``` python

    def __init__(self, proxy=None):
        super(Model, self).__init__()
        self.br = requests.Session()
        if proxy is not None:
            self.br.proxies = {"http": proxy, "https": proxy}

    def __open_http(self, encoding, root_url):
        if codecs.lookup(encoding).name == "shift_jis":
            encoding = "cp932"
        def open_http(method, url, values):
            val = {x: y.encode(encoding) for (x, y) in values if y}
            url = urljoin(root_url, url)
            if method == "GET":
                return self.br.get(url=url, params=val)
            elif method == "POST":
                return self.br.post(url=url, params=val)
        return open_http

    def login(self, user, password):
        url = "http://mysite.html"
        data = {"aACTION": "mailbox"}
        reply = self.br.get(url=url, params=data)
        doc = lxml.html.fromstring(reply.text)
        form = doc.forms[1]
        form.fields["aEMAIL"] = user
        form.fields["aPASSWORD"] = password
        response = lxml.html.submit_form(form, open_http=self.__open_http(encoding=reply.encoding, root_url=reply.url))
```

Since I upgraded to 2.6.1, I noticed that this login function would fail when I'm behind a proxy.
I did some debugging in the console, with 2.6.2, and could get the following.

1) the code works just fine with no proxy
2) when calling self.br.post(url=url, params=val), I always need to call it twice before having a valid reply. The first time results in a ResponseNotReady exception
3) when calling directly from requests.post instead of session.post, it works from the first time, even behind a proxy

Here is also the stacktrace I get

``` python
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 374, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python34\lib\http\client.py", line 1164, in getresponse
    raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 374, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python34\lib\http\client.py", line 1164, in getresponse
    raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 370, in send
    timeout=timeout
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\util\retry.py",line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\packages\six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 374, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python34\lib\http\client.py", line 1164, in getresponse
    raise ResponseNotReady(self.__state)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ResponseNotReady('Request-sent',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\MyApp\MyApp.py", line 148, in updateModel
    data_model.login(myaddress, mypassword)
  File "F:\MyApp\model.py", line 108, in login
    response = lxml.html.submit_form(form, open_http=self.__open_http(encoding=reply.encoding, root_url=reply.url))
  File "C:\Python34\lib\site-packages\lxml\html\__init__.py", line 953, in submit_form
    return open_http(form.method, url, values)
  File "F:\MyApp\model.py", line 97, in open_http
    return self.br.post(url=url, params=val)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 508, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))
```


Are you sure that it doesn't work the _first_ time in your sample code, and fail thereafter?


Indeed you are right... Sorry for the confusion.
In my case the first get is indeed sucessful, then the post fails, and i
need to send it a second time to make things work correctly.


Ok, so the problem we have is that the httplib state machine believes there's an outstanding web response that we haven't consumed. Most likely this occurs because `HTTPResponse.isclosed()` on the old response is returning `False`.

Working out exactly why is going to be tricky. Can I see the response headers, please?


Which ones would you want ?
The ones after the first successful response ?


I doubt you can get the headers for those, I want the first one.


ok here we go
this is what I get after the first GET request

{'transfer-encoding': 'chunked', 'x-powered-by': 'PleskLin', 'server':
'Apache',
 'set-cookie': 'jgid_ip=91.103.42.50; path=/, jgid_time=1429857762; path=/,
geo_c=0; path=/, geo_country_c=FR; path=/, jg_test=set; expires=Sun,
24-May-2015 06:42:42 GMT; path=/',
'content-type':'text/html;charset=shift-jis', 'date': 'Fri, 24 Apr 2015
06:42:42 GMT', 'ms-author-via': 'DAV', 'connection': 'Keep-Alive'}


Chunked is exactly what I expected.

Can you try hitting any URL that does not return chunked data and check whether you still see this problem? @sigmavirus24 @shazow Looks like the chunked-encoding logic has a problem with proxy connections.


I'm starting to expect that https://github.com/kennethreitz/requests/issues/2568 and this are related and it's because Tomas, while using internal attributes and methods on the httplib response, did not try to catch all the exceptions that can be raised by them. Looks like my re-factor will need to be a lot more thorough than we thought.

Proposal: 2.6.3 is released reverting the changes to urllib3 and only including the changes to `requests/packages/__init__.py` since this otherwise was working fine.


Sadly, I think we may have to do that.

Ugh.


I believe this is the same as #2568, so I'm closing to focus conversation there.


Fwiw, it seems to me that if the first response doesn't include a content-length this could cripple the state of the client connection.


@wbolster All responses must do one of the following: have a content-length, have a chunked transfer encoding, or close the connection when they're done. We know we have a bug with chunked, but when that's resolved there should be no problems. =)


Yeah, I was hit by that bug. Omitting a Content-Type header from the response (on a server within my control) caused the wsgi server to go with chunked encoding... which triggered the `requests` bug.