Repository: mitmproxy/mitmproxy

Issue:
Omitting a forward slash from http domain produces Bad HTTP request line
#### Problem Description
Running a GET request for "http://wikipedia.com" will produce the following output:
`35.176.198.120:43784: clientconnect
 << Bad HTTP request line: b'GET http://wikipedia.com HTTP/1.1'
::ffff:35.176.198.120:43784: request
  -> HTTP protocol error in client request: Bad HTTP request line: b'GET http://wikipedia.com HTTP/1.1'
35.176.198.120:43784: clientdisconnect`

The request status, headers and body are as follows for the client:
`400
<CIMultiDictProxy('Server': 'mitmproxy 5.3.0', 'Connection': 'close', 'Content-Length': '306', 'Content-Type': 'text/html')>
b'<html>\n            <head>\n                <title>400 Bad Request</title>\n            </head>\n            <body>\n            <h1>400 Bad Request</h1>\n            <p>HttpSyntaxException(&quot;Bad HTTP request line: b&#x27;GET http://wikipedia.com HTTP/1.1&#x27;&quot;)</p>\n            </body>\n        </html>`

Running a GET request for "http://wikipedia.com/" will not produce this error. Nor will "https://wikipedia.com".
Curl runs the problematic URL fine as it appends the missing trailing slash before running the request which can be seen by printing flow.request.url.

#### Steps to reproduce the behavior:
1. Start mitmproxy using "mitmdump -s proxy_server.py --set block_global=false"
2. Run a GET request with mitm proxy as the request proxy for "http://wikipedia.com".

#### System Information
Mitmproxy: 5.3.0
Python:    3.7.9
OpenSSL:   OpenSSL 1.1.1h  22 Sep 2020
Platform:  Linux-4.14.203-156.332.amzn2.x86_64-x86_64-with-glibc2.2.5



Discussion:
Thanks, I can reproduce this on master. Fixes welcome!
Hey, @mhils! I'd like to take this issue. I'm expecting to send a PR this week.
Hey, @mhils! I've done some research on the topic:

[HTTP 1.1 spec](https://tools.ietf.org/html/rfc2616#section-5.1.2) says that Request-Line `GET http://wikipedia.com HTTP/1.1` is actually a malformed one: 
```Note that the absolute path cannot be empty; if none is present in the original URI, it MUST be given as "/" (the server root).```
It also says: ```A transparent proxy MUST NOT rewrite the "abs_path" part of the received Request-URI when forwarding it to the next inbound server, except as noted above to replace a null abs_path with "/".```

Having this, I think we should fix it by appending a "/" to request URI when it doesn't have one.

What are your thoughts?
This generally sounds good to me. Have you checked if there is more current guidance in any of the 72xx HTTP RFCs?

Resolving patch:
diff --git a/mitmproxy/net/http/http1/read.py b/mitmproxy/net/http/http1/read.py
--- a/mitmproxy/net/http/http1/read.py
+++ b/mitmproxy/net/http/http1/read.py
@@ -111,7 +111,7 @@ def _read_request_line(line: bytes) -> Tuple[str, int, bytes, bytes, bytes, byte
                 raise ValueError
         else:
             scheme, rest = target.split(b"://", maxsplit=1)
-            authority, path_ = rest.split(b"/", maxsplit=1)
+            authority, _, path_ = rest.partition(b"/")
             path = b"/" + path_
             host, port = url.parse_authority(authority, check=True)
             port = port or url.default_port(scheme)


Test patch:
diff --git a/test/mitmproxy/net/http/http1/test_read.py b/test/mitmproxy/net/http/http1/test_read.py
--- a/test/mitmproxy/net/http/http1/test_read.py
+++ b/test/mitmproxy/net/http/http1/test_read.py
@@ -137,6 +137,8 @@ def t(b):
             ("foo", 42, b"CONNECT", b"", b"foo:42", b"", b"HTTP/1.1"))
     assert (t(b"GET http://foo:42/bar HTTP/1.1") ==
             ("foo", 42, b"GET", b"http", b"foo:42", b"/bar", b"HTTP/1.1"))
+    assert (t(b"GET http://foo:42 HTTP/1.1") ==
+            ("foo", 42, b"GET", b"http", b"foo:42", b"/", b"HTTP/1.1"))
 
     with pytest.raises(ValueError):
         t(b"GET / WTF/1.1")