Repository: mitmproxy/mitmproxy

Issue:
Smoke Tests
weebly.com
`curl -L -k -s -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0' --proxy http://localhost:8080 http://weebly.com/ -v`
gives:
```
127.0.0.1:60757: clientconnect
127.0.0.1:60757: GET http://weebly.com/
              << 301 Moved Permanently 230b
127.0.0.1:60757: GET http://www.weebly.com/
              << 302 Found 0b
127.0.0.1:60757: GET http://www.weebly.com/dehttp://www.weebly.com:80/
              << 301 Moved Permanently 661b
```
after which the client connection just hangs. This seems to be related to first-line format being absolute.

Same for bbc.co.uk

------------

addthis.com
`curl -L -k -s -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0' --proxy http://localhost:8080 http://addthis.com/ -v`
gives:
```
127.0.0.1:61559: clientconnect
127.0.0.1:61559: GET http://addthis.com/
              << 400 Bad Request 0b
127.0.0.1:61559: clientdisconnect
```
```
> GET http://addthis.com/ HTTP/1.1
> Host: addthis.com
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 400 Bad Request
< Date: Fri, 18 Nov 2016 18:09:02 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: cloudflare-nginx
< CF-RAY: 303d5f05901763df-FRA
<
```
same for soundcloud.com and www.huffingtonpost.com

ref #1754 


Discussion:

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py
--- a/mitmproxy/proxy/protocol/http.py
+++ b/mitmproxy/proxy/protocol/http.py
@@ -268,6 +268,11 @@ def _process_flow(self, f):
 
         self.log("request", "debug", [repr(request)])
 
+        # set first line format to relative in regular mode,
+        # see https://github.com/mitmproxy/mitmproxy/issues/1759
+        if self.mode is HTTPMode.regular and request.first_line_format == "absolute":
+            request.first_line_format = "relative"
+
         # update host header in reverse proxy mode
         if self.config.options.mode == "reverse":
             f.request.headers["Host"] = self.config.upstream_server.address.host
diff --git a/pathod/pathod.py b/pathod/pathod.py
--- a/pathod/pathod.py
+++ b/pathod/pathod.py
@@ -144,6 +144,7 @@ def handle_http_request(self, logger):
             path = req.path
             http_version = req.http_version
             headers = req.headers
+            first_line_format = req.first_line_format
 
             clientcert = None
             if self.clientcert:
@@ -167,6 +168,7 @@ def handle_http_request(self, logger):
                     sni=self.sni,
                     remote_address=self.address(),
                     clientcert=clientcert,
+                    first_line_format=first_line_format
                 ),
                 cipher=None,
             )


Test patch:
diff --git a/test/mitmproxy/test_server.py b/test/mitmproxy/test_server.py
--- a/test/mitmproxy/test_server.py
+++ b/test/mitmproxy/test_server.py
@@ -282,6 +282,21 @@ def test_stream_modify(self):
         assert d.content == b"bar"
         self.master.addons.remove(s)
 
+    def test_first_line_rewrite(self):
+        """
+        If mitmproxy is a regular HTTP proxy, it must rewrite an absolute-form request like
+            GET http://example.com/foo HTTP/1.0
+        to
+            GET /foo HTTP/1.0
+        when sending the request upstream. While any server should technically accept
+        the absolute form, this is not the case in practice.
+        """
+        req = "get:'%s/p/200'" % self.server.urlbase
+        p = self.pathoc()
+        with p.connect():
+            assert p.request(req).status_code == 200
+            assert self.server.last_log()["request"]["first_line_format"] == "relative"
+
 
 class TestHTTPAuth(tservers.HTTPProxyTest):
     def test_auth(self):