Repository: mitmproxy/mitmproxy

Issue:
Replay via upstream proxy uses CONNECT instead of original request method for HTTP connections
#### Problem Description
Replaying a request that was routed through a upstream proxy server fails as the preceding CONNECT method is not supported.

#### Steps to reproduce the behavior:
1. Run mitmpoxy in upstream mode (e.g. route traffic through tinyproxy server)
2. Execute any request
3. Replay this request
4. The replay fails as it is executed with a preceding CONNECT request

#### System Information
Mitmproxy: 8.0.0.dev (+148, commit 79f464b)
Python:    3.8.10
OpenSSL:   OpenSSL 1.1.1l  24 Aug 2021
Platform:  Linux-5.11.0-38-generic-x86_64-with-glibc2.29




Discussion:
This needs concrete repro steps. The following setup creates two perfectly replayable requests for me:
```
mitmproxy -p 8001
```
```
mitmproxy --mode upstream:localhost:8001 --set ssl_insecure
```
```
curl -x localhost:8080 -k https://example.com http://example.com
```


My setup is practically equal to yours, except I use `tinyproxy` as proxy. The original request works like a charm, replay fails with "Upstream proxy 127.0.0.1:8001 refused HTTP CONNECT request: 403 Access violation".
Might be also be a tinyproxy bug but the CONNECT method is not required to replay the original request. 
Ah, I see. Can you confirm that this only applies to plaintext http:// requests? For HTTPS you do need `CONNECT`. In this case what we are doing is not wrong per se, but I agree that it would be nicer to proxy absolute-form HTTP requests and not `CONNECT` before that.
Yes, I can confirm, that this only applies for raw HTTP.
 With HTTPS it works.
Thanks. The problem here likely is the interaction between replaying in transparent mode:

https://github.com/mitmproxy/mitmproxy/blob/1c10abef000ba2f112bc00119bcdb6707d6ff08e/mitmproxy/addons/clientplayback.py#L81

while checking whether we need to CONNECT here:

https://github.com/mitmproxy/mitmproxy/blob/1c10abef000ba2f112bc00119bcdb6707d6ff08e/mitmproxy/proxy/layers/http/__init__.py#L834

I think we need to be smarter about setting the HTTP mode for client replay, it may just be good enough to set this to upstream when we are in upstream mode. I won't find any time to work on this anytime soon, but if you want to take a closer look I'd be happy to merge a PR.

Resolving patch:
diff --git a/mitmproxy/addons/clientplayback.py b/mitmproxy/addons/clientplayback.py
--- a/mitmproxy/addons/clientplayback.py
+++ b/mitmproxy/addons/clientplayback.py
@@ -78,7 +78,10 @@ def __init__(self, flow: http.HTTPFlow, options: Options) -> None:
 
         super().__init__(context)
 
-        self.layer = layers.HttpLayer(context, HTTPMode.transparent)
+        if options.mode.startswith("upstream:"):
+            self.layer = layers.HttpLayer(context, HTTPMode.upstream)
+        else:
+            self.layer = layers.HttpLayer(context, HTTPMode.transparent)
         self.layer.connections[client] = MockServer(flow, context.fork())
         self.flow = flow
         self.done = asyncio.Event()


Test patch:
diff --git a/test/mitmproxy/addons/test_clientplayback.py b/test/mitmproxy/addons/test_clientplayback.py
--- a/test/mitmproxy/addons/test_clientplayback.py
+++ b/test/mitmproxy/addons/test_clientplayback.py
@@ -31,13 +31,13 @@ async def handler(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
             writer.close()
             handler_ok.set()
             return
+        req = await reader.readline()
         if mode == "upstream":
-            conn_req = await reader.readuntil(b"\r\n\r\n")
-            assert conn_req == b'CONNECT address:22 HTTP/1.1\r\n\r\n'
-            writer.write(b"HTTP/1.1 200 Connection Established\r\n\r\n")
+            assert req == b'GET http://address:22/path HTTP/1.1\r\n'
+        else:
+            assert req == b'GET /path HTTP/1.1\r\n'
         req = await reader.readuntil(b"data")
         assert req == (
-            b'GET /path HTTP/1.1\r\n'
             b'header: qvalue\r\n'
             b'content-length: 4\r\n'
             b'\r\n'
@@ -59,6 +59,8 @@ async def handler(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
             flow.request.content = b"data"
             if mode == "upstream":
                 tctx.options.mode = f"upstream:http://{addr[0]}:{addr[1]}"
+                flow.request.authority = f"{addr[0]}:{addr[1]}"
+                flow.request.host, flow.request.port = 'address', 22
             else:
                 flow.request.host, flow.request.port = addr
             cp.start_replay([flow])
@@ -70,6 +72,37 @@ async def handler(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
                 assert flow.response.status_code == 204
 
 
+@pytest.mark.asyncio
+async def test_playback_https_upstream():
+    handler_ok = asyncio.Event()
+
+    async def handler(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
+        conn_req = await reader.readuntil(b"\r\n\r\n")
+        assert conn_req == b'CONNECT address:22 HTTP/1.1\r\n\r\n'
+        writer.write(b"HTTP/1.1 502 Bad Gateway\r\n\r\n")
+        await writer.drain()
+        assert not await reader.read()
+        handler_ok.set()
+
+    cp = ClientPlayback()
+    ps = Proxyserver()
+    with taddons.context(cp, ps) as tctx:
+        tctx.configure(cp)
+        async with tcp_server(handler) as addr:
+            cp.running()
+            flow = tflow.tflow()
+            flow.request.scheme = b"https"
+            flow.request.content = b"data"
+            tctx.options.mode = f"upstream:http://{addr[0]}:{addr[1]}"
+            cp.start_replay([flow])
+            assert cp.count() == 1
+            await asyncio.wait_for(cp.queue.join(), 5)
+            await asyncio.wait_for(handler_ok.wait(), 5)
+            cp.done()
+            assert flow.response is None
+            assert str(flow.error) == f'Upstream proxy {addr[0]}:{addr[1]} refused HTTP CONNECT request: 502 Bad Gateway'
+
+
 @pytest.mark.asyncio
 async def test_playback_crash(monkeypatch):
     async def raise_err():