Repository: tornadoweb/tornado

Issue:
Empty reply on malformed HTTP request
When an incorrect HTTP request is performed, tornado makes no reply (log + return in code). I suggest returning an 400 error code.

For example, when working behind an nginx proxy-server, it causes a 502 error code to the end-user and a record in nginx's error log, which is incorrect (should be 400 and a warning).



Discussion:
The assumption is that a client sending a malformed HTTP request is actually speaking some non-HTTP protocol and attempting to send an HTTP response will just confuse it further.  It's worrying if you're seeing nginx generate or forward malformed requests; one of the reasons to use nginx is to let it reject malformed requests on its own.  What exactly is the problem with the requests you're seeing?  Should Tornado be more lenient to accept whatever nginx is accepting to proxy?

In my setup, the following query is considered to be correct by nginx, but it raises _BadRequestException  in tornado, with an empty response:

``` bash
$ curl 'http://localhost:9001/res?data= test'
curl: (52) Empty reply from server

$ curl 'http://localhost/res?data= test'
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.2.6</center>
</body>
</html>
```

There is obviously a typo in this request, and I'd like to answer the requester with a proper diagnostics (that the request contains a mistake).

#559 is also about spaces in URLs.  They might be common enough that we should just support them, even though they are not allowed by the HTTP spec as far as I can tell.

> The assumption is that a client sending a malformed HTTP request is actually
> speaking some non-HTTP protocol and attempting to send an HTTP response will
> just confuse it further.

My point is that once a request was handled with the HTTP1Connection class, it should be processed as an HTTP request. I didn't go deep into the code to find, where the decision about handlers is made. But I think that once such decision (that the request is HTTP one) was made, the http1connection should consider all requests to be HTTP.

> They might be common enough that we should just support them, even though they
> are not allowed by the HTTP spec as far as I can tell.

I suggest to answer with correct HTTP codes once an incorrect request was performed. A question whether (and how) to support spaces in the request is another one.

We consider data on the wire to be an HTTP request if its first line contains exactly three space-separated tokens and the third one starts with "HTTP/".  Spaces in the URL violate that assumption so we decide that it is not an HTTP request and reject the connection before it even starts.  

Maybe, I don't understand the reason to think about non-HTTP clients while developing an HTTP server. The operations we are speaking about are performed in the HTTP-server module. For example, in my application it is simply done in a following manner:

``` python
self.server = tornado.httpserver.HTTPServer(self.application)
```

By writing this I define that I'm creating an HTTP server. I don't care about clients who may try use a non-HTTP protocol. I think, that as an HTTP server, it is reasonable for an application to answer with a proper (for HTTP protocol) response.

upd: So, in my opinion, an HTTP server should consider all requests to be HTTP.

The general issue is called a cross-protocol attack; much of the complexity in the websocket protocol is dedicated to preventing them.  The major concern with cross-protocol attacks is that javascript can send http requests to any service even behind firewalls, so it it can craft an http request that looks sufficiently like a request in some other protocol it can cause problems.  The other direction, sending non-http requests to an http server, is much less of a concern in practice but I don't want to ignore it.  (I'm also concerned about response splitting and alignment problems when you start to get clever about determining what constitutes an http request).

If a client cannot send a minimally-valid http request, it's safest to assume that it will not be able to consume an http response. Anything else is a courtesy to users of buggy clients; since there are security implications (even if small), I'm inclined to be cautious about making any changes like this.

This just got me too.... 

[RFC 7230 Section 3.1.1](https://tools.ietf.org/html/rfc7230#section-3.1.1) which came out about the time of this issue suggests:

> Recipients of an invalid request-line SHOULD respond with either a 400 (Bad Request) error or a 301 ...
Interesting. I was not aware that the RFC has guidance on this subject. That mitigates my concern about cross-protocol attacks, so returning a 400 looks like the right thing to do here.
@bdarnell I've looked through the code some, but I don't have line of sight on a good way to implement a fix for this.
I believe the line to change is [this one](https://github.com/tornadoweb/tornado/blob/a73bfc9ddd7515489508f194759e71b92e3b7ff7/tornado/http1connection.py#L254). The fix may be as simple as inserting `yield self.stream.write(b'HTTP/1.1 400 Bad Request\r\n\r\n')` before the `self.close()` (I'm guessing we need to write a raw response instead of using `self.write_headers` because that method requires some data from the request). 
@bdarnell I quickly got lost/sidetracked on weather or not all `HTTPInputError` should behave this way , or just the one raised by `httputil.parse_request_start_line()` (I think this one error needs to be a new narrow `HTTP400InputError`). I also think this case is scoped just to `is_client=True` but i'm not sure if there is also a case where `self.stream` would be None.

Resolving patch:
diff --git a/tornado/http1connection.py b/tornado/http1connection.py
--- a/tornado/http1connection.py
+++ b/tornado/http1connection.py
@@ -250,6 +250,8 @@ def _read_message(self, delegate):
         except httputil.HTTPInputError as e:
             gen_log.info("Malformed HTTP message from %s: %s",
                          self.context, e)
+            if not self.is_client:
+                yield self.stream.write(b'HTTP/1.1 400 Bad Request\r\n\r\n')
             self.close()
             raise gen.Return(False)
         finally:
diff --git a/tornado/httputil.py b/tornado/httputil.py
old mode 100644
new mode 100755
--- a/tornado/httputil.py
+++ b/tornado/httputil.py
@@ -829,6 +829,8 @@ def parse_request_start_line(line):
     try:
         method, path, version = line.split(" ")
     except ValueError:
+        # https://tools.ietf.org/html/rfc7230#section-3.1.1
+        # invalid request-line SHOULD respond with a 400 (Bad Request)
         raise HTTPInputError("Malformed HTTP request line")
     if not re.match(r"^HTTP/1\.[0-9]$", version):
         raise HTTPInputError(


Test patch:
diff --git a/tornado/test/httpserver_test.py b/tornado/test/httpserver_test.py
--- a/tornado/test/httpserver_test.py
+++ b/tornado/test/httpserver_test.py
@@ -29,18 +29,19 @@
 
 def read_stream_body(stream, callback):
     """Reads an HTTP response from `stream` and runs callback with its
-    headers and body."""
+    start_line, headers and body."""
     chunks = []
 
     class Delegate(HTTPMessageDelegate):
         def headers_received(self, start_line, headers):
             self.headers = headers
+            self.start_line = start_line
 
         def data_received(self, chunk):
             chunks.append(chunk)
 
         def finish(self):
-            callback((self.headers, b''.join(chunks)))
+            callback((self.start_line, self.headers, b''.join(chunks)))
     conn = HTTP1Connection(stream, True)
     conn.read_response(Delegate())
 
@@ -217,7 +218,7 @@ def raw_fetch(self, headers, body, newline=b"\r\n"):
                              [utf8("Content-Length: %d" % len(body))]) +
                 newline + newline + body)
             read_stream_body(stream, self.stop)
-            headers, body = self.wait()
+            start_line, headers, body = self.wait()
             return body
 
     def test_multipart_form(self):
@@ -406,7 +407,15 @@ def test_empty_request(self):
         self.io_loop.add_timeout(datetime.timedelta(seconds=0.001), self.stop)
         self.wait()
 
-    def test_malformed_first_line(self):
+    def test_malformed_first_line_response(self):
+        self.stream.write(b'asdf\r\n\r\n')
+        read_stream_body(self.stream, self.stop)
+        start_line, headers, response = self.wait()
+        self.assertEqual('HTTP/1.1', start_line.version)
+        self.assertEqual(400, start_line.code)
+        self.assertEqual('Bad Request', start_line.reason)
+
+    def test_malformed_first_line_log(self):
         with ExpectLog(gen_log, '.*Malformed HTTP request line'):
             self.stream.write(b'asdf\r\n\r\n')
             # TODO: need an async version of ExpectLog so we don't need
@@ -438,7 +447,7 @@ def test_chunked_request_body(self):
 
 """.replace(b"\n", b"\r\n"))
         read_stream_body(self.stream, self.stop)
-        headers, response = self.wait()
+        start_line, headers, response = self.wait()
         self.assertEqual(json_decode(response), {u'foo': [u'bar']})
 
     def test_chunked_request_uppercase(self):
@@ -457,7 +466,7 @@ def test_chunked_request_uppercase(self):
 
 """.replace(b"\n", b"\r\n"))
         read_stream_body(self.stream, self.stop)
-        headers, response = self.wait()
+        start_line, headers, response = self.wait()
         self.assertEqual(json_decode(response), {u'foo': [u'bar']})
 
     def test_invalid_content_length(self):
@@ -627,7 +636,7 @@ def test_unix_socket_bad_request(self):
             self.stream.write(b"garbage\r\n\r\n")
             self.stream.read_until_close(self.stop)
             response = self.wait()
-        self.assertEqual(response, b"")
+        self.assertEqual(response, b"HTTP/1.1 400 Bad Request\r\n\r\n")
 
 
 class KeepAliveTest(AsyncHTTPTestCase):
@@ -1036,24 +1045,26 @@ def test_small_body(self):
     def test_large_body_buffered(self):
         with ExpectLog(gen_log, '.*Content-Length too long'):
             response = self.fetch('/buffered', method='PUT', body=b'a' * 10240)
-        self.assertEqual(response.code, 599)
+        self.assertEqual(response.code, 400)
 
     def test_large_body_buffered_chunked(self):
         with ExpectLog(gen_log, '.*chunked body too large'):
             response = self.fetch('/buffered', method='PUT',
                                   body_producer=lambda write: write(b'a' * 10240))
-        self.assertEqual(response.code, 599)
+        # this test is flaky on windows; accept 400 (expected) or 599
+        self.assertIn(response.code, [400, 599])
 
     def test_large_body_streaming(self):
         with ExpectLog(gen_log, '.*Content-Length too long'):
             response = self.fetch('/streaming', method='PUT', body=b'a' * 10240)
-        self.assertEqual(response.code, 599)
+        self.assertEqual(response.code, 400)
 
     def test_large_body_streaming_chunked(self):
         with ExpectLog(gen_log, '.*chunked body too large'):
             response = self.fetch('/streaming', method='PUT',
                                   body_producer=lambda write: write(b'a' * 10240))
-        self.assertEqual(response.code, 599)
+        # this test is flaky on windows; accept 400 (expected) or 599
+        self.assertIn(response.code, [400, 599])
 
     def test_large_body_streaming_override(self):
         response = self.fetch('/streaming?expected_size=10240', method='PUT',
@@ -1090,14 +1101,14 @@ def test_body_size_override_reset(self):
             stream.write(b'PUT /streaming?expected_size=10240 HTTP/1.1\r\n'
                          b'Content-Length: 10240\r\n\r\n')
             stream.write(b'a' * 10240)
-            headers, response = yield gen.Task(read_stream_body, stream)
+            start_line, headers, response = yield gen.Task(read_stream_body, stream)
             self.assertEqual(response, b'10240')
             # Without the ?expected_size parameter, we get the old default value
             stream.write(b'PUT /streaming HTTP/1.1\r\n'
                          b'Content-Length: 10240\r\n\r\n')
             with ExpectLog(gen_log, '.*Content-Length too long'):
                 data = yield stream.read_until_close()
-            self.assertEqual(data, b'')
+            self.assertEqual(data, b'HTTP/1.1 400 Bad Request\r\n\r\n')
         finally:
             stream.close()