Requests 2.0.0 breaks SSL proxying via https_port of Squid.

For a Squid configuration of:

```
http_port 3128
https_port 3129 cert=/usr/local/opt/squid/etc/ssl/squid.crt key=/usr/local/opt/squid/etc/ssl/squid.key
```

And test names where:

The 'http_port' annotation indicates whether Squid HTTP port was used.

The 'https_port' annotation indicates whether Squid HTTPS port was used.

The 'scheme' annotation in test names refers to whether the scheme was part of proxy definition in proxies dictionary passed to requests

```
no_scheme --> { 'https': 'localhost:3129' }
http_scheme --> { 'https': 'http://localhost:3129' }
https_scheme --> { 'https': 'https://localhost:3129' }
```

Results for different requests versions are:

```
                    requests        0.9.3   1.2.3   2.0.0

                    no_proxy        PASS    PASS    PASS
         http_port_no_scheme        PASS    FAIL    FAIL
  http_port_with_http_scheme        PASS    PASS    PASS
 http_port_with_https_scheme        FAIL    FAIL    PASS
        https_port_no_scheme        FAIL    PASS    FAIL
 https_port_with_http_scheme        FAIL    FAIL    FAIL
https_port_with_https_scheme        PASS    PASS    FAIL
```

The problem one I am concerned about is https_port_with_https_scheme as this no longer works any more.

I fully realise that http_port_with_https_scheme now works and that this presumably would use CONNECT as is the safest option, but we would have to notify all customers relying on https_port_with_https_scheme that what they used before no longer works and they will need to change their configuration they have. If they don't heed that instruction, then we will start see failed connections and complaints.

BTW, it would be really good if the documentation explained the differences between all the combinations, what mechanisms they use and what represents best practice for being the safest/best to use.

Test used for https_port_with_https_scheme is:

```
import unittest
import requests

PROXY_HOST = 'localhost'
PROXY_HTTP_PORT = 3128
PROXY_HTTPS_PORT = 3129

REMOTE_URL = 'https://pypi.python.org/pypi'

class TestProxyingOfSSLRequests(unittest.TestCase):

    def test_proxy_via_squid_https_port_with_https_scheme(self):
        proxies = { 'https': 'https://%s:%s' % (PROXY_HOST, PROXY_HTTPS_PORT) }

        response = requests.get(REMOTE_URL, proxies=proxies)

        self.assertTrue(len(response.content) != 0)

if __name__ == '__main__':
    unittest.main()
```

For full set of tests and results see:
- https://dl.dropboxusercontent.com/u/22571016/requests-ssl-proxy.tar

Cheat sheet for setting up Squid on MacOS X with SSL support from our own cheat sheet is:

To test proxying via SSL, the easiest thing to do is install 'squid' via 'brew' under MacOSX, but avoid the standard recipe and instead use:

```
brew install https://raw.github.com/mxcl/homebrew/a7bf4c381f4e38c24fb23493a92851ea8339493e/Library/Formula/squid.rb
```

This will install 'squid' with SSL support.

You can then generate a self signed certificate to use:

```
cd /usr/local/opt/squid/etc
mkdir ssl
cd ssl

openssl genrsa -des3 -out squid.key 1024
openssl req -new -key squid.key -out squid.csr
cp squid.key squid.key.org
openssl rsa -in squid.key.org -out squid.key
openssl x509 -req -days 365 -in squid.csr -signkey squid.key -out squid.crt
```

Then edit the the squid configuration file at '/usr/local/opt/squid/etc/squid.conf', adding:

```
https_port 3129 cert=/usr/local/opt/squid/etc/ssl/squid.crt key=/usr/local/opt/squid/etc/ssl/squid.key
```

Then use configuration of:

```
proxy_host = localhost
proxy_port = 3129
```


Unless someone else beats me to it, I'll probably take a look at duplicating this weekend. I have pretty much no context on the proxy work though so if anyone with some background on that wants to pair on this let me know.


To explain more about what has been broken. Consider the following;

requests - 1.2.3

http_port_no_scheme (FAIL) - As no scheme is provided, requests will use the scheme of the target URL, which in these tests is a 'https' request. It therefore attempts to connect to http_port using a SSL connection, which will fail as Squid isn't expecting that on that port. Expected result.

http_port_with_http_scheme (PASS) - Scheme provided was 'http', which means that requests connects to Squid over a non SSL connection. It then issues a 'GET https//..'. Squid will then create a SSL connection for the outgoing request. Both the request and response content between requests and Squid aren't encrypted and can be sniffed. This is undesirable and why a change to CONNECT was being attempted. Expected result.

http_port_with_https_scheme (FAIL) - Scheme provided was 'https' which means that requests connects to http_port using a SSL connection, which will fail as Squid isn't expecting that on that port. Expected result.

https_port_no_scheme (PASS) - As no scheme is provided, requests will use the scheme of the target URL, which in these tests is a 'https' request. It therefore attempts to connect to https_port using a SSL connection, which is all fine because that is what it is expecting. Within that encrypted connection, it then issues a 'GET https://..'. Squid will then create a SSL connection for the outgoing request. Both the request and response content between requests and Squid aren't encrypted within the SSL connection, so cannot be sniffed. Expected result.

https_port_with_http_scheme (FAIL) - Scheme provided was 'http', which means that requests connects to Squid over a non SSL connection, which will fails as Squid was expecting a SSL connection on https_port. Expected result.

https_port_with_https_scheme (PASS) - Scheme provided was 'https'. It therefore attempts to connect to https_port using a SSL connection, which is all fine because that is what it is expecting. Within that encrypted connection, it then issues a 'GET https://..'. Squid will then create a SSL connection for the outgoing request. Both the request and response content between requests and Squid aren't encrypted within the SSL connection, so cannot be sniffed. Expected result.

requests - 2.0.0

http_port_with_http_scheme (PASS) - Scheme provided was 'http', which means that requests connects to Squid over a non SSL connection. It then issues a CONNECT instead of GET and tunnels the encrypted request over that connection. Expected result.

http_port_with_https_scheme (PASS) - Scheme provided was 'https'. The requests module though ignores what the scheme is and connects using a non SSL connection. It then issues a CONNECT instead of GET and tunnels the encrypted request over that connection. It works, but breaks the one way that used to sort of work with old requests versions. See below.

https_port_with_http_scheme (FAIL) - Scheme provided was 'http', which means that requests connects to Squid over a non SSL connection, which will fails as Squid was expecting a SSL connection on https_port. Expected result.

https_port_with_https_scheme (FAIL) - Scheme provided was 'https'. The requests module though ignores what the scheme is and connects using a non SSL connection. This fails because Squid was expecting a SSL connection on https_port. Unfortunately this was the closest there was to getting things working prior to requests 2.0.0, so breaking this may break some existing setups and they will be forced to change configurations.

Summary:

Prior to requests 2.0.0 the only way of having encryption traffic over the network was by using https_port_with_https_scheme. There were two issues with this. It meant that if you wanted host SSL certification validation, then Squid had to do it because there were actually two SSL connections. The first between requests and Squid and the other between Squid and the target server. The second is that since Squid decrypted in the middle, then technically a modified Squid could capture the traffic that was being sent.

Using CONNECT resolves this as a SSL connection is tunnelled the whole way. CONNECT though is done over a connection which is initially non SSL, so when doing https_port_with_https_scheme it will not actually be a SSL connection.

What you break therefore is those people who may have been getting by by using https_port_with_https_scheme to encrypt traffic over local network as well as public network beyond Squid. There isn't likely though any way of dealing with that automatically and still wouldn't be as secure.

End result is that you are unlikely to make any changes as you obviously want the best level of security. What you do need though is a dedicated page in the documentation about SSL security issues and what all the combinations are in regard to proxies so that users have a clue, especially since the majority of users have no idea about this stuff.


This is a good catch, and thanks so much for the bug report. The problem is almost certainly in the underlying urllib3 implementation of the proxy behaviour though. Let's not raise an issue there just yet until we can isolate the problem, but I think the fix will have to be in that library.

Hey @schlamar, can you take a look at this?


@GrahamDumpleton Could you please run these tests against plain urllib3, too (see https://gist.github.com/schlamar/5080598#file-test_proxy-py for example code)?


It is most likely due to the introduction of HTTPSConnectionPool._prepare_conn() in 'requests/packages/urllib3/connectionpool.py'. This is called at the end of HTTPSConnectionPool._new_conn(). It is called whenever a proxy is being used for a HTTPS connection, but doesn't take into consideration the scheme that was specified for talking to the proxy. Thus it tries to connect to a SSL port as non SSL and then create a tunnel. It fails as SSL connection expected.

What appears to work, is to change:

```
    return self._prepare_conn(connection)
```

at the end of _new_conn() to:

```
    if self.proxy.scheme != 'https':
        return self._prepare_conn(connection)
    else:
        return connection
```

I need to retest properly all cases, but am guessing that result of everything would then be:

```
                    no_proxy        PASS
         http_port_no_scheme        FAIL
  http_port_with_http_scheme        PASS
 http_port_with_https_scheme        FAIL
        https_port_no_scheme        FAIL
 https_port_with_http_scheme        FAIL
https_port_with_https_scheme        PASS
```

This makes a bit more sense in that https scheme against http_port will fail and http against https_port will fail. Is only when scheme matches port type that will work.

You still have the issue that for https_port_with_https_scheme it is two separate SSL connections and a modified proxy could intercept any data being sent via the proxy, plus host certificate validation for target URL has to be done by the proxy, but does at least preserve how it worked before and less surprises.

The end result is that the safest option is a http_port on the proxy with http scheme. For a https target URL, then CONNECT is still used in this case and everything is encrypted end to end.


Out of interest, why can't we send CONNECT over the SSL connection to the squid proxy? That feels like it's probably the right thing to do.


You would be tunnelling SSL over a connection which is already SSL. That is a bit strange to start with. Am not even sure whether a proxy server would allow that. I don't know enough about urllib3/httplib to even work out how to make it even attempt it.

Anyway, I have tested that change in our actual product and it appears to work in all the cases I can think of that need to work.


So to give some perspective, my worry is about security. Specifically, if you do this:

``` python
r = requests.get(some_https_url, proxies={'https': 'https://127.0.0.1:8888/'})
```

It's not clear from any of that that the proxy will be terminating the SSL and starting a new connection. It seems bizarre to have this line be more secure:

``` python
r = requests.get(some_https_url, proxies={'https': 'http://127.0.0.1:8888/'})
```


Technically tunnelling on top of SSL is possible:
- http://stackoverflow.com/questions/6594604/connect-request-to-a-forward-http-proxy-over-an-ssl-connection

How widely it is supported by proxies may be an issue.

As I had already pointed out, the https_port_with_https_scheme does have the man in the middle problem if the proxy is compromised. But then am sure that some corporates probably set it up that way so they can see your data.

This is why I said back at the start that probably didn't expect a change as I figured that someone decided to make it as secure as possible and deliberately break this specific scenario so it didn't work.

In making that decision though, they have broken something that used to work, even though the level of security is questionable.

This is why I called out that what at least should be done is decent documentation on all the security implications of using proxies and SSL connections. The details in the proxies section is very scant and it isn't clear in that what is a good setup and what isn't. People are forced to look elsewhere for details, and I am not even sure where to go search out a good reference for the information. I only understand what is happening from putting a TCP proxy between the client and proxy and observing what happens.


I agree that we're going to have to document our decision-making process here, whatever we decide to do. I'm just trying to think about what the best procedure would be, both from a Requests and a urllib3 perspective. =)


@GrahamDumpleton I'm pretty sure the only issue is your setup. Please check first how curl behaves (I expect exactly like requests 2.x).

> they have broken something that used to work

That's wrong. Proxy support was broken before, it is fixed with 2.x. 

`http_port_with_https_scheme` is now exactly how it should work. There is just a TCP connection to the proxy (doesn't matter if port 80 or port 443). All HTTP data is encrypted with the cert of the target. There is no security issue here (you can check with curl/wireshark).

`https_port_with_https_scheme` should fail at cert validation because squid is configured as MITM proxy. Adding your custom cert to the request call should work.


I have supplied all the test scripts and log output from running the tests at the URL I gave. All the variations are laid out in the tests and as summarised in the tables above. This isn't a case of having a single specific setup wrong. All the possibilities were enumerated for connecting to a target web server with a https URL and what occurs simply described for comparison.

Comparing to other proxy aware clients is certainly worthwhile for comparison as well, but the point of the comparisons to this point was to illustrate how requests itself has changed compared to prior versions to make it clear for anyone who cares.

You can say that proxy support was broken before, but even so, some of us still had to find ways to make that work even though requests was broken, and https_port_with_https_scheme was the closest one could get to at least having stuff going over the network encrypted, even if the proxy itself was susceptible to a MITM as I quite clearly acknowledged.

I did not say that http_port_with_https_scheme itself is now a security issue. I was inferring that the ignoring of the scheme for everything as was done in this case, also meant that https_port_with_https_scheme would now break because the proxy was still going to expect a SSL connection and requests would always use a non SSL connection.

As to https_port_with_https_scheme failing at cert validation, it doesn't even get that far as the proxy is listening on a SSL port and requests is connecting to it over a non SSL connection.

The main point of me pointing all this out is that the documentation is far from adequate. It doesn't explain well to an audience who have no clue about this stuff, what the implications are of different settings.

In the mean time I still need to work out how to deal with customers who were relying on the closest thing that worked from prior broken versions of requests. As I already said, we will likely just work out who they are tell them they are forced to change their configuration so they no longer use the https_port of their proxy as they were.

That I even indicated what code could be changed was only to illustrate where the issue arise. I said back at the beginning I wasn't expecting to have people agree to change anything as someone likely felt how it now works is better even though it breaks one variation of the configuration which used to work even though it wasn't fully secure.


So, let's just clarify what we expect `{'https': 'https://127.0.0.1/'}` to actually do.Should we be expecting an SSL connection to the proxy followed by CONNECT, or a non SSL connection followed by CONNECT?


It attempts a non SSL connection and then a CONNECT. This is all it can do, as you cannot know the other end is actually going to talk SSL or not anymore based on the scheme because it has to ignore the scheme for http_port_with_https_scheme to work. This is why https_port_with_https_scheme cannot ever work if the intent it is to support CONNECT for http_port_with_https_scheme. So that is reality and just needs to be documented which combinations will and wont work.


> This isn't a case of having a single specific setup wrong.

Nope, I assume that the _general_ squid setup does not work like intended. I think that squid does not allow CONNECT on your HTTPS port. Please verify that with curl/browser, e.g.:

```
$ export https_proxy=https://127.0.0.1:3129/
$ curl -vv https://google.com
```


> Technically tunnelling on top of SSL is possible: [...] How widely it is supported by proxies may be an issue.

Definitely not by squid (and by requests/urllib3).

> The CONNECT method is a way to tunnel any kind of connection through an HTTP proxy

from http://wiki.squid-cache.org/Features/HTTPS


One last comment

> https_port_with_https_scheme (PASS) - Scheme provided was 'https'. It therefore attempts to connect to https_port using a SSL connection, which is all fine because that is what it is expecting. Within that encrypted connection, it then issues a 'GET https://..'. Squid will then create a SSL connection for the outgoing request. Both the request and response content between requests and Squid aren't encrypted within the SSL connection, so cannot be sniffed. Expected result.

This is **not** the expected result, because doing `GET` to an HTTPS resource via a proxy is just wrong (which was fixed in requests 2.x).

> The main point of me pointing all this out is that the documentation is far from adequate. It doesn't explain well to an audience who have no clue about this stuff, what the implications are of different settings.

There are no different settings. Tunnel is tunnel no matter what scheme you have set on your proxy. This is mainly an issue for version 1.x. There is really an security issue because HTTPS connections via HTTP proxy are not encrypted (which I have pointed out [a long time ago](https://github.com/kennethreitz/requests/issues/655#issuecomment-10281268)).


You miss the point of those descriptions. It was simply giving the observed behaviour for that particular version of the requests module (1.2.3) and whether for how that version of requests/urllib3 was implemented, whether it behaved as it would be expected from that code at that time. It was not telling you how it should overall work in giving the description of what it did in 1.2.3. I did not say that the behaviour that occurred with 1.2.3 was correct and should supersede what 2.0.0 does. The descriptions explained why for old versions it was insecure. I already acknowledged back in the initial description that CONNECT is the better way and use of it would was the correct way, thus why I expected that no change would be made. Please stop confusing a description of the state of things as they were for the older version as somehow being a statement that I believe that that old behaviour should prevail, I have not said that.


> Please stop confusing a description of the state of things as they were for the older version as somehow being a statement that I believe that that old behaviour should prevail.

I didn't think (nor said) that :) But it was not clear if you understood why your code was failing at https_port_with_https_scheme because in general it works if the TCP endpoint at https_port supports HTTP connect. This implies that you have not to document which combinations work because all combinations work if the TCP endpoint works.

So: what's your point after all? :)


I believe the point is that we should better document the behaviour of various proxy settings. =)


> I believe the point is that we should better document the behaviour of various proxy settings.

Which is always the same on 2.x, so there is IMO no need for it. :)


Let me quote again what I said before:

"""
https_port_with_https_scheme (FAIL) - Scheme provided was 'https'. The requests module though ignores what the scheme is and connects using a non SSL connection. This fails because Squid was expecting a SSL connection on https_port.
"""

So in this case requests is going to contact the https_port on Squid. Squid is expecting a SSL connection to be established. The requests library IS NOT establishing a SSL connection, it is establishing a NON SSL connection. Within that NON SSL connection it is then sending:

```
CONNECT pypi.python.org:443 HTTP/1.0
```

This is in the clear. It can be seen by monitoring the network traffic or using tcpwatch as an intermediary to capture traffic. It is not within an encrypted connection.

In the case of:

"""
http_port_with_https_scheme (PASS) - Scheme provided was 'https'. The requests module though ignores what the scheme is and connects using a non SSL connection. It then issues a CONNECT instead of GET and tunnels the encrypted request over that connection. It works, but breaks the one way that used to sort of work with old requests versions.
"""

Squid is expecting to receive a non SSL connection. Requests was told though that the scheme for the proxy was htttps. In prior versions of requests and in any sane interpretation of what https means for connecting to the proxy that I have seen, it means that requests should have connected to the proxy using a SSL connection. It did not this, requests used a NON SSL connection instead which meant that this variation worked where as it previously failed with older requests because in the older requests it honoured the proxy scheme and connected using a SSL connection.

In summary, requests is now totally ignoring the proxy scheme and in all cases is connecting using a NON SSL connection and trying to issue a CONNECT within that. This effectively means there is no difference any more between http://proxy and https://proxy.

So it means absolutely nothing as to whether the proxy supports CONNECT after establishing the connection, because a connection cannot be established for https_port_with_https_scheme because requests is not creating a SSL connection and fails on the initial connection setup.

Since you seem to still believe there is no problem, please confirm the following statement as correct then.

That the proxy scheme, ie., the 'http' and 'https' in http://proxy:port and https://proxy.port is totally irrelevant any more because requests 2.0.0. ignores it and always connects using a NON SSL connection anyway.

If you disagree with that statement, then requests is broken, because that is what it is doing as has been described.

If you agree with that statement, then it breaks what I have generally always seen as the meaning of having the scheme in the proxy URL. That is, that 'https' implies a SSL connection is used to the proxy. In that case, you need to document clearly that the scheme is no longer taken into consideration.


You have linked to a SO question above, maybe you should read all answers to that question, especially http://stackoverflow.com/a/14905773/851737

Plus:

> Unfortunately, popular modern browsers do not permit configuration of TLS/SSL encrypted proxy connections. There are open bug reports against most of those browsers now, waiting for support to appear. If you have any interest, please assist browser teams with getting that to happen.
> 
> Meanwhile, tricks using stunnel or SSH tunnels are required to encrypt the browser-to-proxy connection before it leaves the client machine.

from http://wiki.squid-cache.org/Features/HTTPS

> Squid is expecting to receive a non SSL connection. Requests was told though that the scheme for the proxy was htttps. In prior versions of requests

This was totally wrong implemented in 1.x It never really worked. You have a wrong assumption here.

> and in any sane interpretation of what https means for connecting to the proxy that I have seen,

Again: have a look at curl. It behaves exactly like requests 2.x. No client except Chrome is supporting proxy via HTTPS. And even Chrome falls back to HTTP connect if the proxy endpoint does not support SSL.

> In summary, requests is now totally ignoring the proxy scheme and in all cases is connecting using a NON SSL connection and trying to issue a CONNECT within that. This effectively means there is no difference any more between http://proxy and https://proxy.

This is the behavior for every HTTP client around (see above).


Then why did you ever bother making the scheme mandatory in the proxies map when you ignore it? You could have still accepted host:port alone and just always made it use http anyway.

Anyway, what it comes down to is the documentation needs to state then that the scheme is totally irrelevant and that http:// is the only one that makes sense, because https:// is a lie.


> Then why did you ever bother making the scheme mandatory in the proxies map when you ignore it? You could have still accepted host:port aline and just always made it use http anyway.

I have asked that myself. Because curl and other HTTP clients do support `http_proxy=host:port` without scheme. @Lukasa ?

> Anyway, what it comes down to is the documentation needs to state then that the scheme is totally irrelevant and that http:// is the only one that makes sense, because https:// is a lie.

[subjective opinion] The documentation shows only examples with http:// so this kind of implies that. I think you cannot expect CONNECT to work via SSL while it is not explicitly stated. 


Ugh. The answer is because the work to make proxy schemes explicit was actually implemented before we had CONNECT support.


Allowing proxy without scheme, would that be an API breaking change? If not we might should support that, so we are compatible with curl in terms of proxy handling. Probably in form of undocumented feature.


Uh, we made this change in [2.0](http://docs.python-requests.org/en/latest/community/updates/#id1):

> Proxy URLs now must have an explicit scheme. A `MissingSchema` exception will be raised if they don’t.


Ah, it looks like requests 1.x added the proxy scheme based on target URL (which is horribly wrong...): https://github.com/kennethreitz/requests/commit/840540b6b1f07ef87faab73392c03fbef0dcc9fe I guess this is the reason for this change. 

Would be adding `http`  if no scheme was given an acceptable change for 2.x?


@schlamar Sorry I let this slip past me. The answer there is 'maybe'. @t-8ch was one of the people strongly pushing for explicit proxy schemes: do you have an opinion?