Repository: benoitc/gunicorn

Issue:
Bad Request when sending utf-8 encoded http path under python3
This is the sent data:
```
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 8080))
print(s.send('GET /à%20k HTTP/1.0\r\n\r\n'.encode('utf-8')))
print(s.recv(100000).decode('ascii'))
```

This is the response
```
HTTP/1.1 400 Bad Request
Connection: close
Content-Type: text/html
Content-Length: 181

<html>
  <head>
    <title>Bad Request</title>
  </head>
  <body>
    <h1><p>Bad Request</p></h1>
    Invalid HTTP Version 'Invalid HTTP Version: '%20k HTTP/1.0''
  </body>
</html>
```

This is because the request line is first decoded as latin1 using the _compat:bytes_to_str this causes the "à" to be returned as "\xc3\xa0", then the request line is split using line.split(None, 2) which will consider the \xa0 (non breaking space) as whitespace and strip it, thus rendering the request line invalid.

A first attempt would be to use line.split(' ', 2) but then the split will no longer eat up all consecutive whitespaces and may introduce other bugs.

I'm not sure what would be the best solution here.


Discussion:
Note that python2 is unaffected because bytes_to_str is a no-op in this case:

python2:
```
>>> 'GET /\xc3\xa0%20 HTTP'.split(None, 2)
['GET', '/\xc3\xa0%20', 'HTTP']
```

python3:
```
>>> b'GET /\xc3\xa0%20 HTTP'.decode('latin1').split(None, 2)
['GET', '/Ã', '%20 HTTP']
```
Could we split before decoding? `[part.decode('latin1') for part in b'GET /\xc3\xa0%20 HTTP'.split(None, 2)]`?
That would at least preserve the python2 behavior, i don't know if there's an encoding that might break if split on space, i guess not.
I would think not or else the HTTP request line would be malformed, since the encoding there is pretty strict.
Thanks for raising this issue. Would you be willing to make a PR for it?
mmm shouldn't the start line be encoded in us-ascii though ? 

http://httpwg.org/specs/rfc7230.html#rfc.section.3
This section describe exactly the problem gunicorn is having in this case:

`A recipient MUST parse an HTTP message as a sequence of octets in an encoding that is a superset of US-ASCII [USASCII]. Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A). String-based parsers can only be safely used within protocol elements after the element has been extracted from the message, such as within a header field-value after message parsing has delineated the individual fields.`

In any case the parsing of the request line should be done on bytes.

It's debatable if the request line should ascii-only. The spec is a bit unclear. However there are clients in the wild that just send those request lines and I'd rather handle them in the application which can do further processing to handle them properly.

Resolving patch:
diff --git a/gunicorn/http/message.py b/gunicorn/http/message.py
--- a/gunicorn/http/message.py
+++ b/gunicorn/http/message.py
@@ -173,7 +173,7 @@ def parse(self, unreader):
             buf.write(rbuf)
             line, rbuf = self.read_line(unreader, buf, self.limit_request_line)
 
-        self.parse_request_line(bytes_to_str(line))
+        self.parse_request_line(line)
         buf = BytesIO()
         buf.write(rbuf)
 
@@ -301,10 +301,10 @@ def parse_proxy_protocol(self, line):
             "proxy_port": d_port
         }
 
-    def parse_request_line(self, line):
-        bits = line.split(None, 2)
+    def parse_request_line(self, line_bytes):
+        bits = [bytes_to_str(bit) for bit in line_bytes.split(None, 2)]
         if len(bits) != 3:
-            raise InvalidRequestLine(line)
+            raise InvalidRequestLine(bytes_to_str(line_bytes))
 
         # Method
         if not METH_RE.match(bits[0]):
@@ -325,7 +325,7 @@ def parse_request_line(self, line):
         try:
             parts = urlsplit(self.uri)
         except ValueError:
-            raise InvalidRequestLine(line)
+            raise InvalidRequestLine(bytes_to_str(line_bytes))
         self.path = parts.path or ""
         self.query = parts.query or ""
         self.fragment = parts.fragment or ""


Test patch:
diff --git a/tests/requests/valid/027.http b/tests/requests/valid/027.http
new file mode 100644
--- /dev/null
+++ b/tests/requests/valid/027.http
@@ -0,0 +1,2 @@
+GET /à%20k HTTP/1.0\r\n
+\r\n
diff --git a/tests/requests/valid/027.py b/tests/requests/valid/027.py
new file mode 100644
--- /dev/null
+++ b/tests/requests/valid/027.py
@@ -0,0 +1,8 @@
+request = {
+    "method": "GET",
+    "uri": uri("/\xc3\xa0%20k"),
+    "version": (1, 0),
+    "headers": [
+    ],
+    "body": ''
+}
diff --git a/tests/treq.py b/tests/treq.py
--- a/tests/treq.py
+++ b/tests/treq.py
@@ -78,8 +78,11 @@ def send_lines(self):
             yield lines
 
     def send_bytes(self):
-        for d in str(self.data.decode("latin1")):
-            yield bytes(d.encode("latin1"))
+        for d in self.data:
+            if six.PY3:
+                yield bytes([d])
+            else:
+                yield d
 
     def send_random(self):
         maxs = round(len(self.data) / 10)