Repository: sanic-org/sanic

Issue:
Bad char cause parse exception  and how to log the url when request can't be parsed?
Using: sanic 0.5.1  (While I don't see which change with 0.5.4 in changelog)

In our code we using : 

```
@app.exception(Exception)
def server_error_handler(request, exception):
    if request is not None:
        msg = request.url + ' ' + traceback.format_exc()
    elif isinstance(exception, RequestTimeout):
        return response.text('timeout', status=504)
    else:
        msg = str(exception) + ' ' + traceback.format_exc()
    asyncio.ensure_future(fluent.write_debug('error', msg))
``` 
to log exception.  We find error has : 

 
> Bad Request Traceback (most recent call last):
>   File "httptools/parser/parser.pyx", line 247, in httptools.parser.parser.cb_on_header_field (httptools/parser/parser.c:4007)
>   File "httptools/parser/parser.pyx", line 109, in httptools.parser.parser.HttpParser._on_header_field (httptools/parser/parser.c:1893)
>   File "httptools/parser/parser.pyx", line 105, in httptools.parser.parser.HttpParser._maybe_call_on_header (httptools/parser/parser.c:1822)
>   File "/usr/local/lib/python3.5/dist-packages/sanic/server.py", line 157, in on_header
>     self.headers.append((name.decode().casefold(), value.decode()))
> AttributeError: 'NoneType' object has no attribute 'decode'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/sanic/server.py", line 144, in data_received
>     self.parser.feed_data(data)
>   File "httptools/parser/parser.pyx", line 171, in httptools.parser.parser.HttpParser.feed_data (httptools/parser/parser.c:2721)
> httptools.parser.errors.HttpParserCallbackError: the on_header_field callback failed

At this moment the request have not been created. So is there a method can I log the access url ?





Discussion:
A different exception, But the similar situation Using python3 to request:


```
#!/usr/bin/env python
# coding=utf-8

import requests

url = 'http://127.0.0.1:9000/my_blueprint/foo'
headers = {'User-Agent': b'Mozilla/5.0 (Linux; Android 5.0; \xd6wn Smart Build/LRX21M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/37.0.0.0 Mobile Safari/537.36'}

response = requests.get(url, headers=headers)
print(response.content)

```

sanic will show:

> 2017-06-01 17:38:37,644: ERROR: Traceback (most recent call last):
>   File "httptools/parser/parser.pyx", line 247, in httptools.parser.parser.cb_on_header_field (httptools/parser/parser.c:4007)
>   File "httptools/parser/parser.pyx", line 109, in httptools.parser.parser.HttpParser._on_header_field (httptools/parser/parser.c:1893)
>   File "httptools/parser/parser.pyx", line 105, in httptools.parser.parser.HttpParser._maybe_call_on_header (httptools/parser/parser.c:1822)
>   File "/home/jiamo/.pyenv/versions/new_service/lib/python3.5/site-packages/sanic/server.py", line 157, in on_header
>     self.headers.append((name.decode().casefold(), value.decode()))
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 33: invalid continuation byte
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/home/jiamo/.pyenv/versions/new_service/lib/python3.5/site-packages/sanic/server.py", line 144, in data_received
>     self.parser.feed_data(data)
>   File "httptools/parser/parser.pyx", line 171, in httptools.parser.parser.HttpParser.feed_data (httptools/parser/parser.c:2721)
> httptools.parser.errors.HttpParserCallbackError: the on_header_field callback failed
> 

Or any method to ignore the bad char?


Yes, I find the same issue in sanic 0.5.4. But i dont know what caused it. Can it be fixed next version?    
```
2017-08-21 22:50:05 ERROR handlers.py:104 - Traceback (most recent call last):
  File "httptools/parser/parser.pyx", line 247, in httptools.parser.parser.cb_on_header_field (httptools/parser/parser.c:4007)
  File "httptools/parser/parser.pyx", line 109, in httptools.parser.parser.HttpParser._on_header_field (httptools/parser/parser.c:1893)
  File "httptools/parser/parser.pyx", line 105, in httptools.parser.parser.HttpParser._maybe_call_on_header (httptools/parser/parser.c:1822)
  File "/home/bot/services/pyvenv3/lib/python3.6/site-packages/sanic/server.py", line 164, in on_header
    self.headers.append((name.decode().casefold(), value.decode()))
AttributeError: 'NoneType' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bot/services/pyvenv3/lib/python3.6/site-packages/sanic/server.py", line 151, in data_received
    self.parser.feed_data(data)
  File "httptools/parser/parser.pyx", line 171, in httptools.parser.parser.HttpParser.feed_data (httptools/parser/parser.c:2721)
httptools.parser.errors.HttpParserCallbackError: the on_header_field callback failed
```
I think httptools.parser.parser.HttpParser.feed_data can not parse utf-8 charset, someone know how we can fix this problem in httptools?

Resolving patch:
diff --git a/sanic/server.py b/sanic/server.py
--- a/sanic/server.py
+++ b/sanic/server.py
@@ -189,10 +189,12 @@ def on_header(self, name, value):
                     and int(value) > self.request_max_size:
                 exception = PayloadTooLarge('Payload Too Large')
                 self.write_error(exception)
-
+            try:
+                value = value.decode()
+            except UnicodeDecodeError:
+                value = value.decode('latin_1')
             self.headers.append(
-                    (self._header_fragment.decode().casefold(),
-                     value.decode()))
+                    (self._header_fragment.decode().casefold(), value))
 
             self._header_fragment = b''
 


Test patch: