senko/tornado-proxy — pull request #10 by schlamar (2014-10-15 14:33:18+00:00)
Title: Add support for upstream proxy

This PR adds support to handle a secondary upstream proxy. All requests are forwarded to the proxy defined in the `http_proxy` and `https_proxy` environment variables.

GET/POST requests use the `tornado.curl_httpclient.CurlAsyncHTTPClient` to handle an upstream proxy (which requires pycurl). A CONNECT request is manually proxied.

Here is some example output from the proxy:

$ export HTTPS_PROXY="http://upstream_proxy:4128/"
$ export HTTP_PROXY="http://upstream_proxy:4128/"
$ python tornado_proxy/proxy.py 17001
DEBUG:tornado_proxy:Handle GET request to http://github.com/
DEBUG:tornado_proxy:Forward request via upstream proxy http://upstream_proxy:4128/
DEBUG:tornado.general:GET http://github.com/
DEBUG:tornado.general:Hostname was NOT found in DNS cache
DEBUG:tornado.general:Trying upstream_proxy...
DEBUG:tornado.general:Connected to upstream_proxy (upstream_proxy) port 4128 (#0)
DEBUG:tornado.general:> GET http://github.com/ HTTP/1.1
DEBUG:tornado.general:> Accept-Encoding: gzip,deflate
DEBUG:tornado.general:> Proxy-Connection: Keep-Alive
DEBUG:tornado.general:> Accept: */*
DEBUG:tornado.general:> User-Agent: curl/7.30.0
DEBUG:tornado.general:> Host: github.com
DEBUG:tornado.general:>
DEBUG:tornado.general:< HTTP/1.1 301 Moved Permanently
DEBUG:tornado.general:< Content-length: 0
DEBUG:tornado.general:< Location: https://github.com/
DEBUG:tornado.general:< Connection: keep-alive
DEBUG:tornado.general:<
DEBUG:tornado.general:Connection #0 to host upstream_proxy left intact
INFO:tornado.access:301 GET http://github.com/ (127.0.0.1) 332.00ms
DEBUG:tornado_proxy:Start CONNECT to github.com:443
DEBUG:tornado_proxy:Connected to upstream proxy http://upstream_proxy:4128/
DEBUG:tornado_proxy:CONNECT tunnel established to github.com:443

And the curl request producing this output (a few things are stripped):

$ export HTTPS_PROXY="http://127.0.0.1:17001/"
$ export HTTP_PROXY="http://127.0.0.1:17001/"
$ curl -v -L http://github.com/ -s -o /dev/null
* About to connect() to proxy 127.0.0.1 port 17001 (#0)
* Trying 127.0.0.1...
...
* Connected to 127.0.0.1 (127.0.0.1) port 17001 (#0)
> GET http://github.com/ HTTP/1.1
> User-Agent: curl/7.30.0
> Host: github.com
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 15 Oct 2014 14:13:40 GMT
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< Location: https://github.com/
* Server TornadoServer/4.0.2 is not blacklisted
< Server: TornadoServer/4.0.2
<
* Connection #0 to host 127.0.0.1 left intact
* Issue another request to this URL: 'https://github.com/'
* Found bundle for host github.com: 0x4d36f0
* About to connect() to proxy 127.0.0.1 port 17001 (#1)
* Trying 127.0.0.1...
...
* Connected to 127.0.0.1 (127.0.0.1) port 17001 (#1)
* Establish HTTP proxy tunnel to github.com:443
> CONNECT github.com:443 HTTP/1.1
> Host: github.com:443
> User-Agent: curl/7.30.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
<
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
* CAfile: C:\msysgit\mingw\bin\curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
...
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: github.com
> Accept: */*
>
< HTTP/1.1 200 OK
* Server GitHub.com is not blacklisted
< Server: GitHub.com
< Date: Wed, 15 Oct 2014 14:13:41 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Status: 200 OK
...
<
{ [data not shown]
* Connection #1 to host 127.0.0.1 left intact