Repository: encode/django-rest-framework

Issue:
JSONField webform-input always parsed as string
Hello Tom - we spoke on IRC yesterday, I apologize for the initial confusion since I hadn't quite figured out how this was happening at the time and I had some custom extensions which I needed to disable to be sure that this was actually general to rest-framework. 
## Checklist
- [X] I have verified that that issue exists against the `master` branch of Django REST framework.
  9b56dda91850a07cfaecbe972e0f586434b965c3@master
- [X] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
  https://github.com/tomchristie/django-rest-framework/issues/3884
  
  https://github.com/tomchristie/django-rest-framework/issues/3884#issuecomment-216495045:
  
  > Closing due to lack of strictly delimited, actionable issue.
  
  Opening with substantially more detail. 
- [X] I have reduced the issue to the simplest possible case.
- [X] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)
## Current Behaviour (web-form input - incorrect)

![screenshot from 2016-05-21 12-01-22](https://cloud.githubusercontent.com/assets/10236920/15446850/38af4d46-1f5f-11e6-9bd6-f9f4116822b4.png)
Result (note that json_field is a string literal):

``` python
In [47]: TestModel.objects.all()[0].attributes
Out[47]:
{'_state': <django.db.models.base.ModelState at 0x7f90a85b9390>,
 'account_id': 38,
 'created_at': datetime.datetime(2016, 5, 21, 4, 1, 17, 543922, tzinfo=<UTC>),
 'id': 71,
 'json_field': '{"a": true} ',
 'updated_at': datetime.datetime(2016, 5, 21, 4, 1, 17, 545692, tzinfo=<UTC>)}
```
## Current Behaviour (curl -X POST - correct)

```
curl 'http://127.0.0.1:8000/api/test_models/' -H 'Pragma: no-cache' -H 'Cookie: sessionid=cw85hy8pelhvhyb4z3xtngfapyeokf1v' -H "Content-Type: application/json" -X POST -d '{"json_field":{"a":true}}'
```

Result (note that json_field is the parsed object of the JSON submitted via POST):

``` python
In [48]: TestModel.objects.all()[0].attributes
Out[48]:
{'_state': <django.db.models.base.ModelState at 0x7f90a8855630>,
 'account_id': 35,
 'created_at': datetime.datetime(2016, 5, 21, 4, 6, 10, 65872, tzinfo=<UTC>),
 'id': 73,
 'json_field': {'a': True},
 'updated_at': datetime.datetime(2016, 5, 21, 4, 6, 10, 67324, tzinfo=<UTC>)}
```
## WebForm-Input (Here is what a POST using the webform-input looks like)

Request Headers:

```
POST /api/test_models/ HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://127.0.0.1:8000/api/test_models/
Connection: keep-alive
```

Request Payload:

```
Content-Type: multipart/form-data; boundary=---------------------------991790026648295083338748743
Content-Length: 339

-----------------------------991790026648295083338748743
Content-Disposition: form-data; name="csrfmiddlewaretoken"

wNJTnxROcxtm9nXmK6sv0sN4MhY7PfC2
-----------------------------991790026648295083338748743
Content-Disposition: form-data; name="json_field"

{"a": true}
-----------------------------991790026648295083338748743--
```

It seems that any `Content-Disposition: form-data` will be parsed as a string input. Technically it should be possible\* to tag a portion of a `multipart/form-data` with `Content-Type` e.g. `Content-Type: application/json` or `Content-Type: text/json` but even if we do add it Django seems to ignore this.

[0] http://www.ietf.org/rfc/rfc2388.txt

I think the best solution would be to modify `get_value` to check `html.is_html_input(dictionary)` and json.loads the string if it is. Therefore non-quoted inputs would be parsed as JSON objects e.g. 
![screenshot from 2016-05-21 13-59-14](https://cloud.githubusercontent.com/assets/10236920/15446854/49e46fa6-1f5f-11e6-90d0-456c212f6a7c.png)

One could still express a JSON string literal e.g. 
![screenshot from 2016-05-21 14-30-50](https://cloud.githubusercontent.com/assets/10236920/15446909/db9faf7c-1f60-11e6-8a1f-5554cb005f7c.png)



Discussion:
> Technically it should be possible\* to tag a portion of a multipart/form-data with Content-Type

That's not possible. Form data only supports string and file upload primitives.
This behaviour is simply a limitation of what HTML forms do and don't support.
You'll need to use the Raw data input for endpoints that contain JSON or other nested/complex datatypes.

> That's not possible. Form data only supports string and file upload primitives.

according to the RFC 

> As with all multipart MIME types, each part has an optional "Content-Type", which defaults to text/plain.

That implementations e. g.  Django ignore it seems to be the case,  but it's not obvious to me that this is entirely correct or at least that parsing other content-types would be 'less correct' - otherwise what's the sense of the RFC mentioning optional content-types? 

In django-rest-framework case the current behaviour makes web-input for JSON 100% non-usable (i don't think any one is putting jsonb into their database schema for string literals).  If you are against parsing the JSON provided in webform-input submitting the webform entirely as a seralized JSON would be preferable (although it may be a rather large change).  

Otherwise it would be nice if at least the limitations of the current behavior were documented - since it's somewhat misleading and not very obvious. 

Thanks for django-rest-framework,  it's an awesome tool. 

Some food for thought here : certainly agree that itd be nice to either improve the behaviour or the documentation, but it's not exactly clear how to do this best.

HTML forms are a limitation here, not just for JSON field but for any nested structures, but I'll reopen this for some consideration, even if that only ends up being documentation or similar.

(Specilative: we could mark fields that don't support plain HTML form input, and ensure that the browsable API renders some appropriate messaging)

> It seems that any Content-Disposition: form-data will be parsed as a string input. Technically it should be possible\* to tag a portion of a multipart/form-data with Content-Type e.g. Content-Type: application/json or Content-Type: text/json but even if we do add it Django seems to ignore this.

I was going to say I disagree here and would blame browsers for not supporting it.

However, things are starting to rage in my brain and pandora box is now opened.

Do we have a test case for a multipart/form-data with say JSON content and files linked ? I'm sorry, I just read http://shazwazza.com/post/uploading-files-and-json-data-in-the-same-request-with-angular-js/ which does something similar. DRF parser would not be able to cope with that but I've no idea how Django would do.

> which does something similar.

That example is just sending the JSON portion as `text/plain`, as far as I can see.

I've never seen any multipart implementations that handle primitives other than text and file uploads, and the content-type if set ought to indicate the media type of the uploaded file, as per here...

> If the contents of a file are returned via filling out a form, then the file input is identified as the appropriate media type, if known, or "application/octet-stream".

Reasonably strongly of the opinion that treating multipart as supporting anything other than text and file uploads isn't something we want to do. 

Just out of curiosity, can't the framework itself handle the serialization. For eg., If serializer field is a JSONField and the json data passed is a string (from Browsable API), framework can convert it to json by itself when serialization is done in the background.

I am just a beginner, so if this idea seems vague, please ignore it.

Milestoning for review.


Resolving patch:
diff --git a/rest_framework/fields.py b/rest_framework/fields.py
--- a/rest_framework/fields.py
+++ b/rest_framework/fields.py
@@ -1594,9 +1594,21 @@ def __init__(self, *args, **kwargs):
         self.binary = kwargs.pop('binary', False)
         super(JSONField, self).__init__(*args, **kwargs)
 
+    def get_value(self, dictionary):
+        if html.is_html_input(dictionary) and self.field_name in dictionary:
+            # When HTML form input is used, mark up the input
+            # as being a JSON string, rather than a JSON primative.
+            class JSONString(six.text_type):
+                def __new__(self, value):
+                    ret = six.text_type.__new__(self, value)
+                    ret.is_json_string = True
+                    return ret
+            return JSONString(dictionary[self.field_name])
+        return dictionary.get(self.field_name, empty)
+
     def to_internal_value(self, data):
         try:
-            if self.binary:
+            if self.binary or getattr(data, 'is_json_string', False):
                 if isinstance(data, six.binary_type):
                     data = data.decode('utf-8')
                 return json.loads(data)


Test patch:
diff --git a/tests/test_fields.py b/tests/test_fields.py
--- a/tests/test_fields.py
+++ b/tests/test_fields.py
@@ -1756,6 +1756,19 @@ class TestJSONField(FieldValues):
     ]
     field = serializers.JSONField()
 
+    def test_html_input_as_json_string(self):
+        """
+        HTML inputs should be treated as a serialized JSON string.
+        """
+        class TestSerializer(serializers.Serializer):
+            config = serializers.JSONField()
+
+        data = QueryDict(mutable=True)
+        data.update({'config': '{"a":1}'})
+        serializer = TestSerializer(data=data)
+        assert serializer.is_valid()
+        assert serializer.validated_data == {'config': {"a": 1}}
+
 
 class TestBinaryJSONField(FieldValues):
     """