Repository: mitmproxy/mitmproxy

Issue:
Support removing SSL/TLS from upstream https connection
I'm trying to use an upstream proxy that expects plain HTTP bytes instead of TLS-encapsulated HTTP for HTTPS URLs. I've tried to modify the scheme and other things but mitmproxy still continues to send a TLS client hello after the initial upstream CONNECT. 

It appears that this is currently unsupported in mitmproxy?

##### Steps to reproduce the problem:

uplink.py:

```
from mitmproxy import ctx

def http_connect(flow):
        flow.request.scheme = 'http'
        flow.request.port = 80
        ctx.log.info('script: connect '+repr(flow))

def request(flow):
        flow.request.scheme = 'http'
        flow.request.port = 80
        ctx.log.info('script: request '+repr(flow))
```


1. socat -v tcp-listen:8082,reuseaddr,fork  TCP:upstreamproxy:999
2. mitmproxy -p 8081 -s uplink.py -U http://localhost:8082 --insecure --no-upstream-cert
3. curl -v -k -x http://127.0.0.1:8081 https://www.google.com

The output of socat is:

```
> 2017/05/11 10:11:56.527763  length=119 from=0 to=118
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com:443\r
User-Agent: curl/7.51.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 10:11:56.720058  length=116 from=0 to=115
HTTP/1.1 200 OK\r
Content-Length: 0\r
Date: Thu, 11 May 2017 08:11:56 GMT\r
Content-Type: text/plain; charset=utf-8\r
\r
> 2017/05/11 10:11:56.753419  length=164 from=119 to=282
............i..RF+,Q..2.=.......
oA5T].Lt....2.,.+.$.#.
.       .0./.(.'.........k.g.9.3.....=.<.5./.....@.\v.......
.......#...\r. ........................................< 2017/05/11 10:11:56.753987  length=82 from=116 to=197
HTTP/1.1 502 Uplink Bad Gateway\r
Content-Length: 131\r
Content-Type: text/plain\r
\r
< 2017/05/11 10:11:56.754159  length=131 from=198 to=328
malformed HTTP request "\\x16\\x03\\x01\\x00\\x9f\\x01\\x00\\x00\\x9b\\x03\\x03\\xd2i\\xe3\\xacRF+,Q\\xa4\\xec2\\xfe=\\x11\\x11\\xf1\\xfe\\xcf\\x0e\\u007f"^C⏎
```

As you can see above, the TLS client hello `\\x16\\x03\\x01\\x00` is sent. The upstream proxy doesn't like it and replies with error 502.

##### Expected behavior

If configured/scripted, mitmproxy should send upstream HTTP bytes for a HTTPS URL.

##### Any other comments? What have you tried so far?

I also tried some variants of uplink.py, also the script from issue #1731 for example. I also tried switching off `--insecure` or `--no-upstream-cert`. I'm currently using those flags because Burp (which I actually intend to use instead of curl) apparently doesn't send SNI [properly](https://support.portswigger.net/customer/portal/questions/16827939-burp-does-not-set-sni-on-the-outgoing-connection-to-an-ssl-enabled-web-server).

##### System information

```
Mitmproxy version: 2.0.2 (1.2.0dev0000-0x8cba0352e)
Python version: 3.6.1
Platform: Darwin-16.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0e  16 Feb 2017
Mac version: 10.12.4 ('', '', '') x86_64
```
Support removing SSL/TLS from upstream https connection
I'm trying to use an upstream proxy that expects plain HTTP bytes instead of TLS-encapsulated HTTP for HTTPS URLs. I've tried to modify the scheme and other things but mitmproxy still continues to send a TLS client hello after the initial upstream CONNECT. 

It appears that this is currently unsupported in mitmproxy?

##### Steps to reproduce the problem:

uplink.py:

```
from mitmproxy import ctx

def http_connect(flow):
        flow.request.scheme = 'http'
        flow.request.port = 80
        ctx.log.info('script: connect '+repr(flow))

def request(flow):
        flow.request.scheme = 'http'
        flow.request.port = 80
        ctx.log.info('script: request '+repr(flow))
```


1. socat -v tcp-listen:8082,reuseaddr,fork  TCP:upstreamproxy:999
2. mitmproxy -p 8081 -s uplink.py -U http://localhost:8082 --insecure --no-upstream-cert
3. curl -v -k -x http://127.0.0.1:8081 https://www.google.com

The output of socat is:

```
> 2017/05/11 10:11:56.527763  length=119 from=0 to=118
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com:443\r
User-Agent: curl/7.51.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 10:11:56.720058  length=116 from=0 to=115
HTTP/1.1 200 OK\r
Content-Length: 0\r
Date: Thu, 11 May 2017 08:11:56 GMT\r
Content-Type: text/plain; charset=utf-8\r
\r
> 2017/05/11 10:11:56.753419  length=164 from=119 to=282
............i..RF+,Q..2.=.......
oA5T].Lt....2.,.+.$.#.
.       .0./.(.'.........k.g.9.3.....=.<.5./.....@.\v.......
.......#...\r. ........................................< 2017/05/11 10:11:56.753987  length=82 from=116 to=197
HTTP/1.1 502 Uplink Bad Gateway\r
Content-Length: 131\r
Content-Type: text/plain\r
\r
< 2017/05/11 10:11:56.754159  length=131 from=198 to=328
malformed HTTP request "\\x16\\x03\\x01\\x00\\x9f\\x01\\x00\\x00\\x9b\\x03\\x03\\xd2i\\xe3\\xacRF+,Q\\xa4\\xec2\\xfe=\\x11\\x11\\xf1\\xfe\\xcf\\x0e\\u007f"^C⏎
```

As you can see above, the TLS client hello `\\x16\\x03\\x01\\x00` is sent. The upstream proxy doesn't like it and replies with error 502.

##### Expected behavior

If configured/scripted, mitmproxy should send upstream HTTP bytes for a HTTPS URL.

##### Any other comments? What have you tried so far?

I also tried some variants of uplink.py, also the script from issue #1731 for example. I also tried switching off `--insecure` or `--no-upstream-cert`. I'm currently using those flags because Burp (which I actually intend to use instead of curl) apparently doesn't send SNI [properly](https://support.portswigger.net/customer/portal/questions/16827939-burp-does-not-set-sni-on-the-outgoing-connection-to-an-ssl-enabled-web-server).

##### System information

```
Mitmproxy version: 2.0.2 (1.2.0dev0000-0x8cba0352e)
Python version: 3.6.1
Platform: Darwin-16.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0e  16 Feb 2017
Mac version: 10.12.4 ('', '', '') x86_64
```


Discussion:
Thanks for the fantastic bug report! 😃 

This seems to be a bug. Here is the socat output on master with another mitmproxy instance running on port 999:
```
mitmproxy -p 8081 -s uplink.py --mode upstream:http://localhost:8082 --ssl-insecure --set upstream_cert=false
```
```
> 2017/05/11 13:12:03.064050  length=119 from=0 to=118
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com:443\r
User-Agent: curl/7.47.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 13:12:03.120659  length=39 from=0 to=38
HTTP/1.1 200 Connection established\r
\r
> 2017/05/11 13:12:03.426827  length=239 from=119 to=357
...........2...?6n.dm*w`]..qm.{.|.u...q..]<..B.+.,.	.#.
.$.\b./.0...'...(......./.<.5.=.A...
.....3.g.9.k.E.........{.........www.google.com.\v.......
<snip>
socat[1498] E write(6, 0x15294a0, 31): Broken pipe


> 2017/05/11 13:12:03.762322  length=115 from=0 to=114
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com\r
User-Agent: curl/7.47.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 13:12:03.846242  length=39 from=0 to=38
HTTP/1.1 200 Connection established\r
\r
> 2017/05/11 13:12:03.849285  length=78 from=115 to=192
GET / HTTP/1.1\r
Host: www.google.com\r
User-Agent: curl/7.47.0\r
Accept: */*\r
\r
< 2017/05/11 13:12:03.988207  length=243 from=39 to=281
HTTP/1.1 302 Found\r
<snip>
```

This looks like we initially establish a TLS connection, but then immediately disconnect and open a new (plaintext) connection. Looking at the code, we always establish a connection [here](https://github.com/mitmproxy/mitmproxy/blob/6af72160bf98b58682b8f9fc5aabf51928d2b1d3/mitmproxy/proxy/protocol/http.py#L220). I'm not exactly sure why that is the case, maybe we just need to call `self.set_server(address)` at this point instead. 

@Kriechi @ujjwal96: Can one of you verify if that works? I'm moving this weekend, so I won't get to this before the end of next week.
Need this feature too.

Like the post I send:

https://discourse.mitmproxy.org/t/upstream-mode-always-send-connect-to-upstream-server-when-using-https/412
The issue, which @icese7en linked, is the same I'm having.
Thanks for the fantastic bug report! 😃 

This seems to be a bug. Here is the socat output on master with another mitmproxy instance running on port 999:
```
mitmproxy -p 8081 -s uplink.py --mode upstream:http://localhost:8082 --ssl-insecure --set upstream_cert=false
```
```
> 2017/05/11 13:12:03.064050  length=119 from=0 to=118
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com:443\r
User-Agent: curl/7.47.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 13:12:03.120659  length=39 from=0 to=38
HTTP/1.1 200 Connection established\r
\r
> 2017/05/11 13:12:03.426827  length=239 from=119 to=357
...........2...?6n.dm*w`]..qm.{.|.u...q..]<..B.+.,.	.#.
.$.\b./.0...'...(......./.<.5.=.A...
.....3.g.9.k.E.........{.........www.google.com.\v.......
<snip>
socat[1498] E write(6, 0x15294a0, 31): Broken pipe


> 2017/05/11 13:12:03.762322  length=115 from=0 to=114
CONNECT www.google.com:80 HTTP/1.1\r
Host: www.google.com\r
User-Agent: curl/7.47.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2017/05/11 13:12:03.846242  length=39 from=0 to=38
HTTP/1.1 200 Connection established\r
\r
> 2017/05/11 13:12:03.849285  length=78 from=115 to=192
GET / HTTP/1.1\r
Host: www.google.com\r
User-Agent: curl/7.47.0\r
Accept: */*\r
\r
< 2017/05/11 13:12:03.988207  length=243 from=39 to=281
HTTP/1.1 302 Found\r
<snip>
```

This looks like we initially establish a TLS connection, but then immediately disconnect and open a new (plaintext) connection. Looking at the code, we always establish a connection [here](https://github.com/mitmproxy/mitmproxy/blob/6af72160bf98b58682b8f9fc5aabf51928d2b1d3/mitmproxy/proxy/protocol/http.py#L220). I'm not exactly sure why that is the case, maybe we just need to call `self.set_server(address)` at this point instead. 

@Kriechi @ujjwal96: Can one of you verify if that works? I'm moving this weekend, so I won't get to this before the end of next week.
Need this feature too.

Like the post I send:

https://discourse.mitmproxy.org/t/upstream-mode-always-send-connect-to-upstream-server-when-using-https/412
The issue, which @icese7en linked, is the same I'm having.

Resolving patch:
diff --git a/mitmproxy/log.py b/mitmproxy/log.py
--- a/mitmproxy/log.py
+++ b/mitmproxy/log.py
@@ -4,6 +4,9 @@ def __init__(self, msg, level):
         self.msg = msg
         self.level = level
 
+    def __repr__(self):
+        return "LogEntry({}, {})".format(self.msg, self.level)
+
 
 class Log:
     """
diff --git a/mitmproxy/proxy/root_context.py b/mitmproxy/proxy/root_context.py
--- a/mitmproxy/proxy/root_context.py
+++ b/mitmproxy/proxy/root_context.py
@@ -70,8 +70,16 @@ def _next_layer(self, top_layer):
                 top_layer.server_tls,
                 top_layer.server_conn.address[0]
             )
-        if isinstance(top_layer, protocol.ServerConnectionMixin) or isinstance(top_layer, protocol.UpstreamConnectLayer):
+        if isinstance(top_layer, protocol.ServerConnectionMixin):
             return protocol.TlsLayer(top_layer, client_tls, client_tls)
+        if isinstance(top_layer, protocol.UpstreamConnectLayer):
+            # if the user manually sets a scheme for connect requests, we use this to decide if we
+            # want TLS or not.
+            if top_layer.connect_request.scheme:
+                tls = top_layer.connect_request.scheme == "https"
+            else:
+                tls = client_tls
+            return protocol.TlsLayer(top_layer, client_tls, tls)
 
         # 3. In Http Proxy mode and Upstream Proxy mode, the next layer is fixed.
         if isinstance(top_layer, protocol.TlsLayer):


Test patch:
diff --git a/mitmproxy/test/taddons.py b/mitmproxy/test/taddons.py
--- a/mitmproxy/test/taddons.py
+++ b/mitmproxy/test/taddons.py
@@ -17,6 +17,8 @@ def __init__(self, master):
     def trigger(self, event, *args, **kwargs):
         if event == "log":
             self.master.logs.append(args[0])
+        elif event == "tick" and not args and not kwargs:
+            pass
         else:
             self.master.events.append((event, args, kwargs))
         super().trigger(event, *args, **kwargs)
diff --git a/test/mitmproxy/proxy/test_server.py b/test/mitmproxy/proxy/test_server.py
--- a/test/mitmproxy/proxy/test_server.py
+++ b/test/mitmproxy/proxy/test_server.py
@@ -1,28 +1,27 @@
 import os
 import socket
 import time
-import pytest
 from unittest import mock
 
-from mitmproxy.test import tutils
-from mitmproxy import options
-from mitmproxy.addons import script
-from mitmproxy.addons import proxyauth
-from mitmproxy import http
-from mitmproxy.proxy.config import HostMatcher
+import pytest
+
 import mitmproxy.net.http
-from mitmproxy.net import tcp
-from mitmproxy.net import socks
 from mitmproxy import certs
 from mitmproxy import exceptions
+from mitmproxy import http
+from mitmproxy import options
+from mitmproxy.addons import proxyauth
+from mitmproxy.addons import script
+from mitmproxy.net import socks
+from mitmproxy.net import tcp
 from mitmproxy.net.http import http1
+from mitmproxy.proxy.config import HostMatcher
+from mitmproxy.test import tutils
 from pathod import pathoc
 from pathod import pathod
-
 from .. import tservers
 from ...conftest import skip_appveyor
 
-
 """
     Note that the choice of response code in these tests matters more than you
     might think. libcurl treats a 304 response code differently from, say, a
@@ -1009,6 +1008,40 @@ def test_change_upstream_proxy_connect(self):
         assert len(self.chain[0].tmaster.state.flows) == 0
         assert len(self.chain[1].tmaster.state.flows) == 1
 
+    def test_connect_https_to_http(self):
+        """
+        https://github.com/mitmproxy/mitmproxy/issues/2329
+
+        Client <- HTTPS -> Proxy <- HTTP -> Proxy <- HTTPS -> Server
+        """
+        self.proxy.tmaster.addons.add(RewriteToHttp())
+        self.chain[1].tmaster.addons.add(RewriteToHttps())
+        p = self.pathoc()
+        with p.connect():
+            resp = p.request("get:'/p/418'")
+
+        assert self.proxy.tmaster.state.flows[0].client_conn.tls_established
+        assert not self.proxy.tmaster.state.flows[0].server_conn.tls_established
+        assert not self.chain[1].tmaster.state.flows[0].client_conn.tls_established
+        assert self.chain[1].tmaster.state.flows[0].server_conn.tls_established
+        assert resp.status_code == 418
+
+
+class RewriteToHttp:
+    def http_connect(self, f):
+        f.request.scheme = "http"
+
+    def request(self, f):
+        f.request.scheme = "http"
+
+
+class RewriteToHttps:
+    def http_connect(self, f):
+        f.request.scheme = "https"
+
+    def request(self, f):
+        f.request.scheme = "https"
+
 
 class UpstreamProxyChanger:
     def __init__(self, addr):
diff --git a/test/mitmproxy/test_log.py b/test/mitmproxy/test_log.py
--- a/test/mitmproxy/test_log.py
+++ b/test/mitmproxy/test_log.py
@@ -1 +1,6 @@
-# TODO: write tests
+from mitmproxy import log
+
+
+def test_logentry():
+    e = log.LogEntry("foo", "info")
+    assert repr(e) == "LogEntry(foo, info)"