Repository: mitmproxy/mitmproxy

Issue:
Wrong value for flow address on successive requests
Hi,

I encountered a problem which I think is similar to #2567.

I could easily reproduce it with my client (Alamofire on iOS), although it took me some time to reproduce it with `curl`.

When my client tries to access `https://my.api.com` through `mitmproxy`, I want requests to be transferred to my server running on `http://localhost:8080`.

To do so, I have the following script:
```python
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host == "my.api.com":
        original_host = flow.request.pretty_host
        original_scheme = flow.request.scheme

        flow.request.scheme = "http"
        flow.request.host = "localhost"
        flow.request.port = 8080

        flow.request.headers["Host"] = original_host
        flow.request.headers["X-Forwarded-Proto"] = original_scheme

    if flow.request.pretty_host == "localhost":
        print("bad!")
```

Now, what I started to observe, is that for successive requests (made on the same socket), the first request was correctly transferred to `http://localhost:8080`, but when my script is called during the second one, its scheme, host and port are oddly `https://localhost:8080`, and I eventually see in the logs the following error:
```
 << Cannot establish TLS with localhost:8080 (sni: localhost): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",)
```

By digging a bit more, I found out that in `protocol/http.py`, on line 305 `f.request.host = self.__initial_server_conn.address.host`, `__initial_server_conn` was set to `localhost:8080` for the second request, which I imagine is wrong.

Now, I can only reproduce the problem when `_client_tls and establish_server_tls_now` is set to `False` in `tls.py`. I thought about setting this manually after having a look at the logs:

verbose logs excerpt (happens with my client). Error !
```
192.168.0.162:63994: clientconnect
192.168.0.162:63994: Set new server address: ('my.api.com', 443)
192.168.0.162:63994: Establish TLS with client
192.168.0.162:63994: request
192.168.0.162:63994: Set new server address: localhost:8080
192.168.0.162:63994: serverconnect
192.168.0.162:63994: response
192.168.0.162:63994: GET http://localhost:8080/path
192.168.0.162:63994: request
192.168.0.162:63994: serverdisconnect
192.168.0.162:63994: Set new server address: localhost:8080
192.168.0.162:63994: serverconnect
192.168.0.162:63994: Establish TLS with server
192.168.0.162:63994: GET https://localhost:8080/path
 << Cannot establish TLS with localhost:8080 (sni: localhost): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",)
192.168.0.162:63994: serverdisconnect
192.168.0.162:63994: clientdisconnect
```

verbose logs excerpt (happens when I use curl). No Error
```
127.0.0.1:56369: clientconnect
127.0.0.1:56369: Set new server address: ('my.api.com', 443)
127.0.0.1:56369: serverconnect
127.0.0.1:56369: Establish TLS with server
127.0.0.1:56369: ALPN selected by server: http/1.1
127.0.0.1:56369: Establish TLS with client
127.0.0.1:56369: ALPN for client: b'http/1.1'
127.0.0.1:56369: request
127.0.0.1:56369: serverdisconnect
127.0.0.1:56369: Set new server address: localhost:8080
127.0.0.1:56369: serverconnect
127.0.0.1:56369: response
127.0.0.1:56369: GET http://localhost:8080/path
127.0.0.1:56369: request
127.0.0.1:56369: response
127.0.0.1:56369: GET http://localhost:8080/path
127.0.0.1:56369: serverdisconnect
127.0.0.1:56369: clientdisconnect
```

##### Steps to reproduce the problem:

1. Set `establish_server_tls_now` to `False` in `tls.py`, e.g.:
```python
        # ...
        # if self._client_tls and establish_server_tls_now:
        if False:
            self._establish_tls_with_client_and_server()
        # ...
```
2. run a dummy server on `localhost:8080`
3. run mitmdump with the script (e.g. replacing `my.api.com` with `google.com`)
```
mitmdump -v -p 9999 --no-http2 -s ./redirect_host.py
``` 
4. make 2 requests on the same connection with curl
```
curl --cacert ~/.mitmproxy/mitmproxy-ca.pem --proxy http://localhost:9999 https://google.com https://google.com
```
5. observe error on second request
```
404 page not found # <- first request, this is expected
<html>
            <head>
                <title>502 Bad Gateway</title>
            </head>
            <body>
            <h1>502 Bad Gateway</h1>
            <p>TlsProtocolException(&#x27;Cannot establish TLS with localhost:8080 (sni: localhost): TlsException(&quot;SSL handshake error: Error([(\&#x27;SSL routines\&#x27;, \&#x27;ssl3_get_record\&#x27;, \&#x27;wrong version number\&#x27;)],)&quot;,)&#x27;,)</p>
            </body>
        </html>
```

------

Do you have any idea where this can come from?

Let me know if you need more info!

##### System information
```
$ mitmdump --version
Mitmproxy version: 2.0.2 (release version)
Python version: 3.6.3
Platform: Darwin-17.0.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0f  25 May 2017
Mac version: 10.13 ('', '', '') x86_64
```
Wrong value for flow address on successive requests
Hi,

I encountered a problem which I think is similar to #2567.

I could easily reproduce it with my client (Alamofire on iOS), although it took me some time to reproduce it with `curl`.

When my client tries to access `https://my.api.com` through `mitmproxy`, I want requests to be transferred to my server running on `http://localhost:8080`.

To do so, I have the following script:
```python
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host == "my.api.com":
        original_host = flow.request.pretty_host
        original_scheme = flow.request.scheme

        flow.request.scheme = "http"
        flow.request.host = "localhost"
        flow.request.port = 8080

        flow.request.headers["Host"] = original_host
        flow.request.headers["X-Forwarded-Proto"] = original_scheme

    if flow.request.pretty_host == "localhost":
        print("bad!")
```

Now, what I started to observe, is that for successive requests (made on the same socket), the first request was correctly transferred to `http://localhost:8080`, but when my script is called during the second one, its scheme, host and port are oddly `https://localhost:8080`, and I eventually see in the logs the following error:
```
 << Cannot establish TLS with localhost:8080 (sni: localhost): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",)
```

By digging a bit more, I found out that in `protocol/http.py`, on line 305 `f.request.host = self.__initial_server_conn.address.host`, `__initial_server_conn` was set to `localhost:8080` for the second request, which I imagine is wrong.

Now, I can only reproduce the problem when `_client_tls and establish_server_tls_now` is set to `False` in `tls.py`. I thought about setting this manually after having a look at the logs:

verbose logs excerpt (happens with my client). Error !
```
192.168.0.162:63994: clientconnect
192.168.0.162:63994: Set new server address: ('my.api.com', 443)
192.168.0.162:63994: Establish TLS with client
192.168.0.162:63994: request
192.168.0.162:63994: Set new server address: localhost:8080
192.168.0.162:63994: serverconnect
192.168.0.162:63994: response
192.168.0.162:63994: GET http://localhost:8080/path
192.168.0.162:63994: request
192.168.0.162:63994: serverdisconnect
192.168.0.162:63994: Set new server address: localhost:8080
192.168.0.162:63994: serverconnect
192.168.0.162:63994: Establish TLS with server
192.168.0.162:63994: GET https://localhost:8080/path
 << Cannot establish TLS with localhost:8080 (sni: localhost): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",)
192.168.0.162:63994: serverdisconnect
192.168.0.162:63994: clientdisconnect
```

verbose logs excerpt (happens when I use curl). No Error
```
127.0.0.1:56369: clientconnect
127.0.0.1:56369: Set new server address: ('my.api.com', 443)
127.0.0.1:56369: serverconnect
127.0.0.1:56369: Establish TLS with server
127.0.0.1:56369: ALPN selected by server: http/1.1
127.0.0.1:56369: Establish TLS with client
127.0.0.1:56369: ALPN for client: b'http/1.1'
127.0.0.1:56369: request
127.0.0.1:56369: serverdisconnect
127.0.0.1:56369: Set new server address: localhost:8080
127.0.0.1:56369: serverconnect
127.0.0.1:56369: response
127.0.0.1:56369: GET http://localhost:8080/path
127.0.0.1:56369: request
127.0.0.1:56369: response
127.0.0.1:56369: GET http://localhost:8080/path
127.0.0.1:56369: serverdisconnect
127.0.0.1:56369: clientdisconnect
```

##### Steps to reproduce the problem:

1. Set `establish_server_tls_now` to `False` in `tls.py`, e.g.:
```python
        # ...
        # if self._client_tls and establish_server_tls_now:
        if False:
            self._establish_tls_with_client_and_server()
        # ...
```
2. run a dummy server on `localhost:8080`
3. run mitmdump with the script (e.g. replacing `my.api.com` with `google.com`)
```
mitmdump -v -p 9999 --no-http2 -s ./redirect_host.py
``` 
4. make 2 requests on the same connection with curl
```
curl --cacert ~/.mitmproxy/mitmproxy-ca.pem --proxy http://localhost:9999 https://google.com https://google.com
```
5. observe error on second request
```
404 page not found # <- first request, this is expected
<html>
            <head>
                <title>502 Bad Gateway</title>
            </head>
            <body>
            <h1>502 Bad Gateway</h1>
            <p>TlsProtocolException(&#x27;Cannot establish TLS with localhost:8080 (sni: localhost): TlsException(&quot;SSL handshake error: Error([(\&#x27;SSL routines\&#x27;, \&#x27;ssl3_get_record\&#x27;, \&#x27;wrong version number\&#x27;)],)&quot;,)&#x27;,)</p>
            </body>
        </html>
```

------

Do you have any idea where this can come from?

Let me know if you need more info!

##### System information
```
$ mitmdump --version
Mitmproxy version: 2.0.2 (release version)
Python version: 3.6.3
Platform: Darwin-17.0.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0f  25 May 2017
Mac version: 10.13 ('', '', '') x86_64
```


Discussion:
Thanks for tracking this down and submitting such a detailed report! :smiley: I'm quite busy this week, but will try to take a look as soon as possible.
Thanks for tracking this down and submitting such a detailed report! :smiley: I'm quite busy this week, but will try to take a look as soon as possible.

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/base.py b/mitmproxy/proxy/protocol/base.py
--- a/mitmproxy/proxy/protocol/base.py
+++ b/mitmproxy/proxy/protocol/base.py
@@ -96,15 +96,7 @@ def __call__(self):
     def __init__(self, server_address=None):
         super().__init__()
 
-        self.server_conn = None
-        if self.config.options.spoof_source_address and self.config.options.upstream_bind_address == '':
-            self.server_conn = connections.ServerConnection(
-                server_address, (self.ctx.client_conn.address[0], 0), True)
-        else:
-            self.server_conn = connections.ServerConnection(
-                server_address, (self.config.options.upstream_bind_address, 0),
-                self.config.options.spoof_source_address
-            )
+        self.server_conn = self.__make_server_conn(server_address)
 
         self.__check_self_connect()
 
@@ -125,6 +117,16 @@ def __check_self_connect(self):
                     "The proxy shall not connect to itself.".format(repr(address))
                 )
 
+    def __make_server_conn(self, server_address):
+        if self.config.options.spoof_source_address and self.config.options.upstream_bind_address == '':
+            return connections.ServerConnection(
+                server_address, (self.ctx.client_conn.address[0], 0), True)
+        else:
+            return connections.ServerConnection(
+                server_address, (self.config.options.upstream_bind_address, 0),
+                self.config.options.spoof_source_address
+            )
+
     def set_server(self, address):
         """
         Sets a new server address. If there is an existing connection, it will be closed.
@@ -146,11 +148,7 @@ def disconnect(self):
         self.server_conn.close()
         self.channel.tell("serverdisconnect", self.server_conn)
 
-        self.server_conn = connections.ServerConnection(
-            address,
-            (self.server_conn.source_address[0], 0),
-            self.config.options.spoof_source_address
-        )
+        self.server_conn = self.__make_server_conn(address)
 
     def connect(self):
         """
diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py
--- a/mitmproxy/proxy/protocol/http.py
+++ b/mitmproxy/proxy/protocol/http.py
@@ -165,7 +165,7 @@ class HttpLayer(base.Layer):
     def __init__(self, ctx, mode):
         super().__init__(ctx)
         self.mode = mode
-        self.__initial_server_conn = None
+        self.__initial_server_address = None  # type: tuple
         "Contains the original destination in transparent mode, which needs to be restored"
         "if an inline script modified the target server for a single http request"
         # We cannot rely on server_conn.tls_established,
@@ -177,7 +177,7 @@ def __init__(self, ctx, mode):
     def __call__(self):
         if self.mode == HTTPMode.transparent:
             self.__initial_server_tls = self.server_tls
-            self.__initial_server_conn = self.server_conn
+            self.__initial_server_address = self.server_conn.address
         while True:
             flow = http.HTTPFlow(
                 self.client_conn,
@@ -313,8 +313,8 @@ def _process_flow(self, f):
             # Setting request.host also updates the host header, which we want
             # to preserve
             host_header = f.request.host_header
-            f.request.host = self.__initial_server_conn.address[0]
-            f.request.port = self.__initial_server_conn.address[1]
+            f.request.host = self.__initial_server_address[0]
+            f.request.port = self.__initial_server_address[1]
             f.request.host_header = host_header  # set again as .host overwrites this.
             f.request.scheme = "https" if self.__initial_server_tls else "http"
         self.channel.ask("request", f)


Test patch: