Getting tunnel CONNECT response headers

Hi all,

   There was a issue with sending https requests via the rotating proxy service named ProxyMesh. Here is the code snippet.

```
  >>> import requests
  >>> auth = requests.auth.HTTPProxyAuth('foo', 'PASSWORD')
  >>> proxies = {'https': 'http://us-ca.proxymesh.com:31280'}
  >>> response = requests.get('https://example.com', proxies=proxies, auth=auth)
```

  I couldn't get the tunnel `CONNECT` responses headers in the final response. While it contains the service-specified header X-ProxyMesh-IP which could be useful to requests-rate control.

`http://proxymesh.com/blog/pages/proxy-server-headers.html#https`

  I have read source code related to https request handling, and found that requests mainly use the underlaying http library urllib3 to process http/https connections. It uses the ssl.wrap_socket() to wrap ssl suites on http connection and do some subsequent operations on the SSL Tunnel. 

  ref: https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/util/ssl_.py

  Is there any methods could inject or attach the CONNECT responses headers into final response returned by https requests? or some suggestions on using requests with rotating proxy services like ProxyMesh crawlera?

  I wonder is there anyone have ever thought how to get the CONNECT response headers?


Unfortunately, getting the headers on CONNECT responses is not easy to do, because the underlying HTTP library hides that request away from us. Changing that fact either requires monkey patching the underlying library, totally rewriting the way we handle tunnelling to do it explicitly ourselves, or using a new library. Any of those is a substantial and dangerous chunk of work, and it's just not really of enough value to justify at this time. 

What do you need from the headers?


@Lukasa It contains the service-specified header X-ProxyMesh-IP which could be used to control request-rate. I could manage a proxies-count to do deeper requests-handling.

I have searched for a long time, found nothing could help me to figure it out. Could you offer me some advices on getting tunnel CONNECT responses headers. It is so significant to my application.


Or tell me some helpful libraries could do the stuff. The urllib3 uses ssl to wrap socket which is hard to do monkey-patch. If I insists on using requests to get the specified header. It will require me to do some hard work on it. 


@Lukasa https://bugs.python.org/issue24964 Here is a patch to python httplib which is same to mine. However it was python 3 patch not python 2.7. On the other hand, I think the rotating proxy service design was not easy handful. Oops...  I have no idea how could I works with the proxy in a convenient way. 


Wrapping the socket isn't important here: you want to look for the `HTTPConnection._tunnel` method, which you could monkey patch. 


@Lukasa Awesome. Thanks for your suggestion. I'm so sorry for making mistake the purpose of ssl.wrap_socket() method. I will try to do some monkey-patch hacking based on your reply. 


There's no need to apologise! =)


Yeah, I'm so appreciate for your answer. Thanks a lot. 


@ryanchou1991 Do you have some result? I'm with exactly same problem. 


Yeah. You could inherit the `httplib.HTTPSConnection` to re-implement the `connect` method to do the stuff to get the IP.  And it could work well.


@ryanchou1991 actually i got the IP making a http request before all the flow. I think the propouse is taking an valid IP from proxy, am i right? Is not a problem anymore.

The problem that I have now is to make proxymesh works with a https URL. I can't make this works. It always response a 407 error. 

Can you help me?


@moacirmoda Yeah. I have tried to work it with the requests. While it couldn't work well. I emailed to the Proxymesh author. He offered me a solution.  You could email him for the solution. : ) He would give U a short code snippet for U.


Nice man! Thanks for the help


@moacirmoda You're welcome.  I originally mail the issue to the maintainer. While I re-post to him some code snippets. and we worked together to figure it out. I finally re-implement by inheriting the `httplib.HTTPSConnection` to do the stuff based on the code snippet from the ProxyMesh maintainer. 

While my code snippets was so complicated so that I couldn't give you a clear example for u to work with. You could refer the example by email to him. Hope it could help you


Yeah... I will try to work in `httplib.HTTPSConnection`, and I already send and email to proxymesh support. Thanks for the help.


@ryanchou1991 
Could you show me some examples from the snippets? Me emails with the proxymesh are not so efficiently.


@moacirmoda Sorry, I haven't checked my github notifications yesterday. Have you email him? Or I could offer the code snippets I worked together with him. Here is the code snippet link 

https://gist.github.com/ryanchou1991/e95d6ef76fbc2ed8d395


@moacirmoda I remind of a discussion on stackoverflow. It may help you work it out with the issue.

Here is the discussion `https://stackoverflow.com/questions/14665064/using-python-requests-with-existing-socket-connection`


Hi @ryanchou1991 ,

Their support send me a snippet like these that you send me.

Look. I did implement your snippet and doesn't work:

https://gist.github.com/moacirmoda/122ab1557a062943161e723f8a8c37ad

If you look in the output, on the headers, there's no 'X-ProxyMesh-IP', it means that proxy doesn't work, I'm right?


@ryanchou1991 

Look, I made even better: I access `https://www.whatismyip.com` under the proxy:

https://gist.github.com/moacirmoda/122ab1557a062943161e723f8a8c37ad

Look the output:

```
HTTP/1.1 200 Connection established
X-ProxyMesh-IP: 162.213.38.164


Your IP Address Is: 177.148.142.34
```

`177.148.142.34` is my real ip address. Means not work, right?


I ran into this issue as well. I couldn't find any nice way to modify `HTTPResponse` to include the proxy headers separately, so instead I just merged them into the final headers dictionary.

The below code is copied from my [StackOverflow question](https://stackoverflow.com/questions/39068998/reading-connect-headers):

```
import socket
import httplib
import requests

from requests.packages.urllib3.connection import HTTPSConnection
from requests.packages.urllib3.connectionpool import HTTPSConnectionPool
from requests.packages.urllib3.poolmanager import ProxyManager

from requests.adapters import HTTPAdapter


class ProxyHeaderHTTPSConnection(HTTPSConnection):
    def __init__(self, *args, **kwargs):
        super(ProxyHeaderHTTPSConnection, self).__init__(*args, **kwargs)
        self._proxy_headers = []

    def _tunnel(self):
        self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self._tunnel_host, self._tunnel_port))

        for header, value in self._tunnel_headers.iteritems():
            self.send("%s: %s\r\n" % (header, value))

        self.send("\r\n")

        response = self.response_class(self.sock, strict=self.strict, method=self._method)
        version, code, message = response._read_status()

        if version == "HTTP/0.9":
            # HTTP/0.9 doesn't support the CONNECT verb, so if httplib has
            # concluded HTTP/0.9 is being used something has gone wrong.
            self.close()
            raise socket.error("Invalid response from tunnel request")

        if code != 200:
            self.close()
            raise socket.error("Tunnel connection failed: %d %s" % (code, message.strip()))

        self._proxy_headers = []

        while True:
            line = response.fp.readline(httplib._MAXLINE + 1)

            if len(line) > httplib._MAXLINE:
                raise LineTooLong("header line")

            if not line or line == '\r\n':
                break

            # The line is a header, save it
            if ':' in line:
                self._proxy_headers.append(line)

    def getresponse(self, buffering=False):
        response = super(ProxyHeaderHTTPSConnection, self).getresponse(buffering)
        response.msg.headers.extend(self._proxy_headers)

        return response


class ProxyHeaderHTTPSConnectionPool(HTTPSConnectionPool):
    ConnectionCls = ProxyHeaderHTTPSConnection


class ProxyHeaderProxyManager(ProxyManager):
    def _new_pool(self, scheme, host, port):
        assert scheme == 'https'

        return ProxyHeaderHTTPSConnectionPool(host, port, **self.connection_pool_kw)


class ProxyHeaderHTTPAdapter(HTTPAdapter):
    def proxy_manager_for(self, proxy, **proxy_kwargs):
        if proxy in self.proxy_manager:
            manager = self.proxy_manager[proxy]
        else:
            proxy_headers = self.proxy_headers(proxy)
            manager = self.proxy_manager[proxy] = ProxyHeaderProxyManager(
                proxy_url=proxy,
                proxy_headers=proxy_headers,
                num_pools=self._pool_connections,
                maxsize=self._pool_maxsize,
                block=self._pool_block,
                **proxy_kwargs)

        return manager
```

You can then install the adapter onto a session and pretend the proxy injected the headers into the proxied response, like for HTTP:

```
session = requests.Session()
session.mount('https://', ProxyHeaderHTTPAdapter())

response = session.get('https://example.com', proxies={...})
```

@moacirmoda The gist you linked to didn't work for me either. Mine does, however: https://gist.github.com/Blender3D/b8763f7b12099198fe1d947613c2739a