Repository: encode/django-rest-framework

Issue:
Request.data empty when multipart/form-data POSTed
Django==1.9.2
djangorestframework==3.3.2

This is essentially a repost of #3814.  There is already some research and discussion attached to that issue that I will not rehash, but the author closed out the item once a workaround was found.

Is that workaround the long term solution?  Is the answer to never touch the Django Request before DRF has a chance to do its thing?  This can be difficult to control in projects with third party middleware.

The issue seems to be that, when POSTing data, Django is exhausting the underlying request stream if the WSGIRequest is accessed prior to initialize_request in the DRF View dispatcher.  PUTs seem to be unaffected.

A totally contrived example that demonstrates the issue.  First example, presuming no middleware Request meddling...

``` python
class TestView(APIView):

    def post(self, request, *args, **kwargs):
        # request.data, .FILES, and .POST are populated
        # request._request.FILES and .POST are empty
        return Response()

    def put(self, request, *args, **kwargs):
        # request.data, .FILES, and .POST are populated
        # request._request.FILES and .POST are empty
        return Response()
```

Second example, forcibly evaluating the Django request before dispatching.

``` python
class TestView(APIView):

    def dispatch(self, request, *args, **kwargs):
        p = request.POST  # Force evaluation of the Django request
        return super().dispatch(request, *args, **kwargs)

    def post(self, request, *args, **kwargs):
        # request.data, .FILES, and .POST are empty
        # request._request.FILES and .POST are populated
        return Response()

    def put(self, request, *args, **kwargs):
        # request.data, .FILES, and .POST are populated
        # request._request.FILES and .POST are empty
        return Response()
```

Cheers!



Discussion:
+1 here. The workaround for the previously closed issue is not a solution for the issue at hand. The truth is that there very well is a use case for accessing the Django POST QueryDict in middleware before DRF can consume it.

As there is for streaming content too which breaks the use of middleware introspecting the request content.
Moreover, if a middleware do look at the request content, it means it'll bypasses at least partially the content negotiation. At this point, it's probably a good option not to use DRF views and explicitly feed the serializer if required.

I'd also be interested in learning a bit more in your solution about this issue - in particular how to preserve the compatibility with the current behavior.

The closest I've come to a "solution" is to add an additional check in the `_parse` method of the DRF `Request`:

``` python
...
        try:
            parsed = parser.parse(stream, media_type, self.parser_context)
            # If Django got to the POST/FILES data before we did, just use theirs.
            if (len(parsed.data) + len(parsed.files)) < 1:
                parsed.data = self._request.POST
                parsed.files = self._request.FILES
        except:
            # If we get an exception during parsing, fill in empty data and
...
```

I'm not sure how dangerous it is to use Django's POST data directly like this (rather than making a copy).  They seem to be essentially read only once parsed.

While this retains the current Django WSGIRequest behavior, it leaves it in an inconsistent state insofar as POSTs may or may not have POST/FILE data depending on whether or not the request was touched prior to DRF Request parsing.

Just hit this on production today...
As workaround, I've put a check in our base GenericAPIView instead of custom middleware
(for compatibility must keep both override methods)

``` python
    def initialize_request(self, request, *args, **kwargs):
       request = super().initialize_request(request, *args, **kwargs)
       if request.path.startswith("/api/3/") and request.method == "POST":
          request.method = request.META.get("HTTP_X_HTTP_METHOD_OVERRIDE", request.POST.get("_method", request.method)).upper()
       return request
```

Also running into this problem.  It appears to have been introduced in `3.2` (was not experiencing it while using `3.1.1`).

Does anyone know if there is a way to reset the `request` object to its original state after examination or possibly duplicate the object without exhausting the POST data stream?

It feels like DRF should be capable of using an already-examined `WSGIRequest`.

+1
I have run into a similar issue.  I created a new `nested_route` decorator that acts a lot like `list_route` and `detail_route`.  I usually redirect to another viewset to handle the nested path.  The issue was that the stream was being read too early (by the parent viewset `dispatch`), so I needed to ensure that the `initialize_request` function was only called once for a given request.

Working off of @rsalmaso's comment above, I overrode the `initialize_request` method in the children viewsets with the following.

```
def initialize_request(self, request, *args, **kwargs):
        if not isinstance(request, Request):
            request = super().initialize_request(request, *args, **kwargs)
        return request
```

This works fine, but I think having some sort of attribute for either ignoring the `initialize_request` function within `dispatch` or having a global check to only run `initialize_request` iff `isinstance(request, rest_framework.request.Request) == False`.  I'm happy to prepare the PR with tests if that could be acceptable.

I have the same issue. For now I have reverted to drf 3.2

I've spent almost 3 hours searching about why my data wouldn't be parsed correctly. I first thought the issue was on my Angular App code. And I finally found this issue. Damn, almost banged my head on my desk.

The only work around working for me is getting back to 3.2.5. I'm surprised this issue has no real fix ?

This shows what [open source really is](http://www.commitstrip.com/en/2014/05/07/the-truth-behind-open-source-apps/).

To get this fixed, the first step is to have someone take time and have a deep analysis of when / why and how the behavior change has been introduced. An initial analysis was performed on #3814 though it requires confirmation that the mentioned change did affect the behavior and then identify how the behavior change has been introduced.

Before that, I won't be able to consider the various workarounds as a solution.

Here's what I've managed to track down.

The change definitely occurred in commit 566812a as part of removing method and content overriding.  Prior to that change, in [request.py](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py), the [_load_data_and_files](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L264) method called [_load_method_and_content_type](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L279) which called [_perform_form_overloading](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L316).  Within [_perform_form_overloading](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L316) was a [bit of code](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L336) that seeded the DRF Request._data attribute with Request._request.POST (which is the Django WSGIRequest POST).

``` python
        # At this point we're committed to parsing the request as form data.
        self._data = self._request.POST
        self._files = self._request.FILES
        self._full_data = self._data.copy()
        self._full_data.update(self._files)
```

This means that, back in [_load_data_and_files](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L264), [_parse](https://github.com/tomchristie/django-rest-framework/blob/721efa28176507434a2f4b006f2f31b9a33ca2aa/rest_framework/request.py#L360) was probably never called for multipart POSTs (at least not in the default configuration) and thus DRF did not attempt to re-read the request stream.

In more recent versions of DRF, [_load_data_and_files](https://github.com/tomchristie/django-rest-framework/blob/0c42c742290400e3ee742e77268681b6911604ed/rest_framework/request.py#L238) no longer calls _load_method_and_content_type as that method no longer exists, so now execution is falling through to the call to [_parse](https://github.com/tomchristie/django-rest-framework/blob/0c42c742290400e3ee742e77268681b6911604ed/rest_framework/request.py#L269).  Herein lies the problem.

For multipart POSTs, [_parse](https://github.com/tomchristie/django-rest-framework/blob/0c42c742290400e3ee742e77268681b6911604ed/rest_framework/request.py#L269) is using a [DRF MultiPartParser](https://github.com/tomchristie/django-rest-framework/blob/0c42c742290400e3ee742e77268681b6911604ed/rest_framework/parsers.py#L90) which is using a [Django MultiPartParser](https://github.com/django/django/blob/dafddb6b8c0eb778072bec1ccd536bafad0eb936/django/http/multipartparser.py#L45) which is attempting to re-read the request stream which has previously been exhausted.

So, another potential solution would be to just allow Django to handle the parsing of multipart forms.  This could be achieved by replacing the [MultiPartParser.parse](https://github.com/tomchristie/django-rest-framework/blob/0c42c742290400e3ee742e77268681b6911604ed/rest_framework/parsers.py#L97) method with something more along the lines of:

``` python
    def parse(self, stream, media_type=None, parser_context=None):
        """
        Parses the incoming bytestream as a multipart encoded form,
        and returns a DataAndFiles object.

        `.data` will be a `QueryDict` containing all the form parameters.
        `.files` will be a `QueryDict` containing all the form files.
        """
        parser_context = parser_context or {}
        request = parser_context['request']
        _request = request._request
        if _request.method == 'POST':
            return DataAndFiles(_request.POST, _request.FILES)
        encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
        meta = request.META.copy()
        meta['CONTENT_TYPE'] = media_type
        upload_handlers = request.upload_handlers

        try:
            parser = DjangoMultiPartParser(meta, stream, upload_handlers, encoding)
            data, files = parser.parse()
            return DataAndFiles(data, files)
        except MultiPartParserError as exc:
            raise ParseError('Multipart form parse error - %s' % six.text_type(exc))
```

This essentially grabs the WSGIRequest that underlies the DRF Request and allows it to perform the request parsing.  What I don't know is, are there any conditions under which Django would use another parser when DRF is expecting a MultiPartParser?

Edit:  Updated "fix" code.  Forgot that Django really only parses POSTs for multipart forms.  Also wanted to note that I'm working on tests and eventually a pull request.

I'm depending on this fix. Are there already plans on when to release it?

I experience the same in 3.3.3. Had to roll back to 3.2.4 on Django 1.8.13. A patch for this issue is highly anticipated.

I met the same issue today. Doing PUT instead of POST is really the solution for now.

I'll put it on the list, but no promises if it'll make the 3.4.0 cut or not. We'll see.

I did some digging on this too as I ran into it as well.

(This is django 1.8)
The DjangoMultiPartParser loops through the upload_handlers and called handle_raw_input which should return something, but none of them do. 

The MemoryFileUploadHandler just set's activated to True and the TemporaryFileUploadHandler doesn't even implement it.

So maybe the handlers being passed are the wrong one, or rest framework is using the DjangoMultiPartParser wrong.

+1 Answering @alukach, I have the same issue but only when updating to `3.3.0`. With `3.2.0` works fine (I had to downgrade)

Ran in to this issue as well :-( Had to drop back to version 3.2

Anything I (we as a company) can do to help with this?
This is preventing us from upgrading to drf >= 3.3 and thus is also blocking our django upgrade.

Well, it's on the list, and being re-raised by folks, so I'd expect to deal with it prior to 3.5, hopefully towards the end of this month.

> Anything I (we as a company) can do to help with this?

This, this, this...

https://fund.django-rest-framework.org/topics/funding/

(We're just a few % short of what we need for baseline sustainability right now, getting over the mark will mean I can focus exclusively on the project itself)

So, we're (Zupr) now funding you to work on DRF 🎉 

This issue has the label Needs information and there is a pull request #4026 with a label needs further review.

Now the pull request by @CaffeineFueled currently does not merge cleanly (but only conflicts in the test file) and was created against 3.3.2 (which was the version that had the issue show up for the first time).

I used the pull request version with our code and ran the test suite (1024 tests, ~84% coverage) and all the test passed. 


Resolving patch:
diff --git a/rest_framework/request.py b/rest_framework/request.py
--- a/rest_framework/request.py
+++ b/rest_framework/request.py
@@ -15,6 +15,7 @@
 from django.conf import settings
 from django.http import QueryDict
 from django.http.multipartparser import parse_header
+from django.http.request import RawPostDataException
 from django.utils import six
 from django.utils.datastructures import MultiValueDict
 
@@ -263,10 +264,20 @@ def _load_stream(self):
 
         if content_length == 0:
             self._stream = None
-        elif hasattr(self._request, 'read'):
+        elif not self._request._read_started:
             self._stream = self._request
         else:
-            self._stream = six.BytesIO(self.raw_post_data)
+            self._stream = six.BytesIO(self.body)
+
+    def _supports_form_parsing(self):
+        """
+        Return True if this requests supports parsing form data.
+        """
+        form_media = (
+            'application/x-www-form-urlencoded',
+            'multipart/form-data'
+        )
+        return any([parser.media_type in form_media for parser in self.parsers])
 
     def _parse(self):
         """
@@ -274,8 +285,18 @@ def _parse(self):
 
         May raise an `UnsupportedMediaType`, or `ParseError` exception.
         """
-        stream = self.stream
         media_type = self.content_type
+        try:
+            stream = self.stream
+        except RawPostDataException:
+            if not hasattr(self._request, '_post'):
+                raise
+            # If request.POST has been accessed in middleware, and a method='POST'
+            # request was made with 'multipart/form-data', then the request stream
+            # will already have been exhausted.
+            if self._supports_form_parsing():
+                return (self._request.POST, self._request.FILES)
+            stream = None
 
         if stream is None or media_type is None:
             empty_data = QueryDict('', encoding=self._request._encoding)


Test patch:
diff --git a/tests/test_parsers.py b/tests/test_parsers.py
--- a/tests/test_parsers.py
+++ b/tests/test_parsers.py
@@ -7,11 +7,16 @@
 from django.core.files.uploadhandler import (
     MemoryFileUploadHandler, TemporaryFileUploadHandler
 )
+from django.http.request import RawPostDataException
 from django.test import TestCase
 from django.utils.six.moves import StringIO
 
 from rest_framework.exceptions import ParseError
-from rest_framework.parsers import FileUploadParser, FormParser
+from rest_framework.parsers import (
+    FileUploadParser, FormParser, JSONParser, MultiPartParser
+)
+from rest_framework.request import Request
+from rest_framework.test import APIRequestFactory
 
 
 class Form(forms.Form):
@@ -122,3 +127,39 @@ def test_get_encoded_filename(self):
 
     def __replace_content_disposition(self, disposition):
         self.parser_context['request'].META['HTTP_CONTENT_DISPOSITION'] = disposition
+
+
+class TestPOSTAccessed(TestCase):
+    def setUp(self):
+        self.factory = APIRequestFactory()
+
+    def test_post_accessed_in_post_method(self):
+        django_request = self.factory.post('/', {'foo': 'bar'})
+        request = Request(django_request, parsers=[FormParser(), MultiPartParser()])
+        django_request.POST
+        assert request.POST == {'foo': ['bar']}
+        assert request.data == {'foo': ['bar']}
+
+    def test_post_accessed_in_post_method_with_json_parser(self):
+        django_request = self.factory.post('/', {'foo': 'bar'})
+        request = Request(django_request, parsers=[JSONParser()])
+        django_request.POST
+        assert request.POST == {}
+        assert request.data == {}
+
+    def test_post_accessed_in_put_method(self):
+        django_request = self.factory.put('/', {'foo': 'bar'})
+        request = Request(django_request, parsers=[FormParser(), MultiPartParser()])
+        django_request.POST
+        assert request.POST == {'foo': ['bar']}
+        assert request.data == {'foo': ['bar']}
+
+    def test_request_read_before_parsing(self):
+        django_request = self.factory.put('/', {'foo': 'bar'})
+        request = Request(django_request, parsers=[FormParser(), MultiPartParser()])
+        django_request.read()
+        with pytest.raises(RawPostDataException):
+            request.POST
+        with pytest.raises(RawPostDataException):
+            request.POST
+            request.data