Repository: encode/django-rest-framework

Issue:
Invalid Unicode byte in Authorization token raises an DjangoUnicodeDecodeError
When we sent an invalid/unicode byte in token authentication we got a `500 Internal Server Error` (`DjangoUnicodeDecodeError`) instead of a `401 Unauthorized`:

``` python
headers = {'Authorization':'token cfbcf941f2fa06a0647b7dc4cb7839199b495c2b¸'}  # <-- last char is an invalid unicode char
```

Raises the following traceback:

```
Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 440, in dispatch
    self.initial(request, *args, **kwargs)
  File "rest_framework/views.py", line 354, in initial
    self.perform_authentication(request)
  File "rest_framework/views.py", line 292, in perform_authentication
    request.user
  File "rest_framework/request.py", line 491, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "rest_framework/request.py", line 266, in user
    self._authenticate()
  File "rest_framework/request.py", line 454, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "rest_framework/authentication.py", line 167, in authenticate
    return self.authenticate_credentials(auth[1])
  File "rest_framework/authentication.py", line 171, in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
  File "django/db/models/query.py", line 325, in get
    clone = self.filter(*args, **kwargs)
  File "django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "django/db/models/sql/query.py", line 1304, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "django/db/models/sql/query.py", line 1331, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "django/db/models/sql/query.py", line 1203, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "django/db/models/sql/query.py", line 1096, in build_lookup
    return final_lookup(lhs, rhs)
  File "django/db/models/lookups.py", line 96, in __init__
    self.rhs = self.get_prep_lookup()
  File "django/db/models/lookups.py", line 134, in get_prep_lookup
    return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
  File "django/db/models/fields/__init__.py", line 727, in get_prep_lookup
    return self.get_prep_value(value)
  File "django/db/models/fields/__init__.py", line 1125, in get_prep_value
    return self.to_python(value)
  File "django/db/models/fields/__init__.py", line 1121, in to_python
    return smart_text(value)
  File "django/utils/encoding.py", line 56, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "django/utils/encoding.py", line 102, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
```



Discussion:
Does this happen if the extra bytes are encoded ahead of time?

And is that character allowed in the `Authorization` header? [Related httpie ticket.](https://github.com/jakubroztocil/httpie/issues/212)

@kevin-brown even though we should gracefully return the issue.

@osantana do you have a test case for this issue or was it discovered on a running server ?

I've a testcase but it's an application that we're developing/deploying privately for a company, so, we did not have a public API to provide.

But I can reproduce this issue and send you more information about it. The stacktrace provided is the full stacktrace and we don't use any custom Authentication method.

``` python
# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'apps.core.permissions.ModelPermission',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',)
}
```

@osantana I gave it a try to reproduce the issue but so far I've not been able to do it within a testcase.

If we can't replicate this, there's no way we'll be able to progress the issue.
Going to close the ticket off for now, but if there's a demonstrable way to replicate it, then we'll reopen.

Sorry, I forgot to mention a _very_ important thing: this project runs with Python 3.4 (and this exception raises only in Py3).

Besides that I've forked DRF-tutorial project and created a test to reproduce this issue:

https://github.com/osantana/rest-framework-tutorial/blob/master/snippets/tests.py

You can just clone this repository, install requirements and run `./manage.py tests`.

@osantana will give it a try tonight and reopen if I can reproduce it. Thanks for the details.

News about this issue?

@osantana at the moment no. I unsuccessfully tried to reproduce it.

Even with the tests that I created for [rest-framework-tutorial](https://github.com/osantana/rest-framework-tutorial/blob/master/snippets/tests.py)? Have you tried with Python 3.4? This is a Py3-related issue...

Sorry, I missed that part and didn't have the to test against your test case yet.

@osantana here's the result on my box:

```
bash-3.2$ pip freeze -l
You are using pip version 6.1.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Django==1.8.2
djangorestframework==3.1.2
Markdown==2.6.2
Pygments==2.0.2
requests==2.7.0
bash-3.2$ python --version
Python 3.4.3
bash-3.2$ python manage.py test
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.790s

OK
Destroying test database for alias 'default'...
bash-3.2$ 
```

What OS are you using and what are your locals ?

I'm using Ubuntu 14.04 and locale(?) is set with en_US.UTF-8 but I've other language packs installed and generated.

**Edited**: I'm using 14.04! :)

But I can try to reproduce this issue on other environment... (I've a Mac OS X 10.10 to make tests)

@osantana I'm not saying this issue doesn't exist or is void. I'm simply pointing out that it doesn't look like a simple py3 related issue - at least not an obvious one at all.

Ok, I understand this... What I'm trying to do is to know _if_ you managed to reproduce the issue or if I need provide more information to help you :)

It's don't even know if it's a "big issue" but I believe that people can use it to generate a DoS attack in affected APIs...

@osantana At this point I think there might be more involved that needs to be looked at.

If you can replicate the test within the DRF repository and create a (failing) pull request, that might be better than trying to reproduce it outside of the repository.


Resolving patch:
diff --git a/rest_framework/authentication.py b/rest_framework/authentication.py
--- a/rest_framework/authentication.py
+++ b/rest_framework/authentication.py
@@ -170,7 +170,13 @@ def authenticate(self, request):
             msg = _('Invalid token header. Token string should not contain spaces.')
             raise exceptions.AuthenticationFailed(msg)
 
-        return self.authenticate_credentials(auth[1])
+        try:
+            token = auth[1].decode()
+        except UnicodeError:
+            msg = _('Invalid token header. Token string should not contain invalid characters.')
+            raise exceptions.AuthenticationFailed(msg)
+
+        return self.authenticate_credentials(token)
 
     def authenticate_credentials(self, key):
         try:


Test patch:
diff --git a/tests/test_authentication.py b/tests/test_authentication.py
--- a/tests/test_authentication.py
+++ b/tests/test_authentication.py
@@ -1,3 +1,5 @@
+# coding: utf-8
+
 from __future__ import unicode_literals
 from django.conf.urls import patterns, url, include
 from django.contrib.auth.models import User
@@ -162,6 +164,12 @@ def test_post_form_passing_token_auth(self):
         response = self.csrf_client.post('/token/', {'example': 'example'}, HTTP_AUTHORIZATION=auth)
         self.assertEqual(response.status_code, status.HTTP_200_OK)
 
+    def test_fail_post_form_passing_invalid_token_auth(self):
+        # add an 'invalid' unicode character
+        auth = 'Token ' + self.key + "¸"
+        response = self.csrf_client.post('/token/', {'example': 'example'}, HTTP_AUTHORIZATION=auth)
+        self.assertEqual(response.status_code, status.HTTP_401_UNAUTHORIZED)
+
     def test_post_json_passing_token_auth(self):
         """Ensure POSTing form over token auth with correct credentials passes and does not require CSRF"""
         auth = "Token " + self.key