Repository: Kinto/kinto

Issue:
JSON Merge Patch deserialization is broken 
Merge-patch operations are broken. It looks like a deserialization problem, which may have been introduced with  #790.  Also, we should definitely include some view tests for this content-type. My bad :/ 
  
```
echo '{"data": {"aaa": "bbb"}}' | http patch localhost:8888/v1/buckets/b1 Content-Type:application/merge-patch+json -a aaa:
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 331
Content-Type: application/json; charset=UTF-8
Date: Sun, 11 Dec 2016 23:47:26 GMT
Server: waitress

{
    "code": 400,
    "details": [
        {
            "description": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality.",
            "location": "body",
            "name": ""
        }
    ],
    "errno": 107,
    "error": "Invalid parameters",
    "message": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality."
}
```
JSON Merge Patch deserialization is broken 
Merge-patch operations are broken. It looks like a deserialization problem, which may have been introduced with  #790.  Also, we should definitely include some view tests for this content-type. My bad :/ 
  
```
echo '{"data": {"aaa": "bbb"}}' | http patch localhost:8888/v1/buckets/b1 Content-Type:application/merge-patch+json -a aaa:
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 331
Content-Type: application/json; charset=UTF-8
Date: Sun, 11 Dec 2016 23:47:26 GMT
Server: waitress

{
    "code": 400,
    "details": [
        {
            "description": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality.",
            "location": "body",
            "name": ""
        }
    ],
    "errno": 107,
    "error": "Invalid parameters",
    "message": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality."
}
```
JSON Merge Patch deserialization is broken 
Merge-patch operations are broken. It looks like a deserialization problem, which may have been introduced with  #790.  Also, we should definitely include some view tests for this content-type. My bad :/ 
  
```
echo '{"data": {"aaa": "bbb"}}' | http patch localhost:8888/v1/buckets/b1 Content-Type:application/merge-patch+json -a aaa:
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 331
Content-Type: application/json; charset=UTF-8
Date: Sun, 11 Dec 2016 23:47:26 GMT
Server: waitress

{
    "code": 400,
    "details": [
        {
            "description": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality.",
            "location": "body",
            "name": ""
        }
    ],
    "errno": 107,
    "error": "Invalid parameters",
    "message": "\"{\"data\": {\"aaa\": \"bbb\"}}\n\" is not a mapping type: Does not implement dict-like functionality."
}
```


Discussion:
Thanks for reporting! 

It is strange that no test was broken oO
Yes. It is.
I guess it's because the `record` tests in the core scope override the `validated` field, thus skipping deserialization and validation.
https://github.com/Kinto/kinto/blob/master/tests/core/resource/test_record.py

Thanks for reporting! 

It is strange that no test was broken oO
Yes. It is.
I guess it's because the `record` tests in the core scope override the `validated` field, thus skipping deserialization and validation.
https://github.com/Kinto/kinto/blob/master/tests/core/resource/test_record.py

Thanks for reporting! 

It is strange that no test was broken oO
Yes. It is.
I guess it's because the `record` tests in the core scope override the `validated` field, thus skipping deserialization and validation.
https://github.com/Kinto/kinto/blob/master/tests/core/resource/test_record.py


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -21,7 +21,7 @@ def read_file(filename):
 REQUIREMENTS = [
     'colander',
     'colorama',
-    'cornice >= 2.1',
+    'cornice >= 2.3',
     'jsonschema',
     'jsonpatch',
     'python-dateutil',


Test patch:
diff --git a/tests/core/resource/test_record.py b/tests/core/resource/test_record.py
--- a/tests/core/resource/test_record.py
+++ b/tests/core/resource/test_record.py
@@ -228,9 +228,67 @@ def test_patch_record_returns_updated_fields(self):
         self.assertEquals(self.stored['id'], self.result['id'])
         self.assertEquals(self.result['position'], 10)
 
+    def test_record_timestamp_is_not_updated_if_none_for_missing_field(self):
+        self.resource.request.json = {'data': {'polo': None}}
+        result = self.resource.patch()['data']
+        self.assertEquals(self.result['last_modified'],
+                          result['last_modified'])
+
+    def test_record_timestamp_is_not_updated_if_no_field_changed(self):
+        self.resource.request.json = {'data': {'position': 10}}
+        result = self.resource.patch()['data']
+        self.assertEquals(self.result['last_modified'],
+                          result['last_modified'])
+
+    def test_collection_timestamp_is_not_updated_if_no_field_changed(self):
+        self.resource.request.json = {'data': {'position': 10}}
+        self.resource.patch()
+        self.resource = self.resource_class(request=self.get_request(),
+                                            context=self.get_context())
+        self.resource.collection_get()['data']
+        last_modified = int(self.last_response.headers['ETag'][1:-1])
+        self.assertEquals(self.result['last_modified'], last_modified)
+
+    def test_timestamp_is_not_updated_if_no_change_after_preprocessed(self):
+        with mock.patch.object(self.resource, 'process_record') as mocked:
+            mocked.return_value = self.result
+            self.resource.request.json = {'data': {'position': 20}}
+            result = self.resource.patch()['data']
+            self.assertEquals(self.result['last_modified'],
+                              result['last_modified'])
+
+    def test_returns_changed_fields_among_provided_if_behaviour_is_diff(self):
+        self.resource.request.json = {'data': {'unread': True, 'position': 15}}
+        self.resource.request.headers['Response-Behavior'] = 'diff'
+        with mock.patch.object(self.resource.model, 'update_record',
+                               return_value={'unread': True, 'position': 0}):
+            result = self.resource.patch()['data']
+        self.assertDictEqual(result, {'position': 0})
+
+    def test_returns_changed_fields_if_behaviour_is_light(self):
+        self.resource.request.json = {'data': {'unread': True, 'position': 15}}
+        self.resource.request.headers['Response-Behavior'] = 'light'
+        with mock.patch.object(self.resource.model, 'update_record',
+                               return_value={'unread': True, 'position': 0}):
+            result = self.resource.patch()['data']
+        self.assertDictEqual(result, {'unread': True, 'position': 0})
+
+
+class MergePatchTest(BaseTest):
+    def setUp(self):
+        super(MergePatchTest, self).setUp()
+        self.stored = self.model.create_record({})
+        self.resource.record_id = self.stored['id']
+        self.headers = self.resource.request.headers
+        self.headers['Content-Type'] = 'application/merge-patch+json'
+
+        class ArticleSchema(ResourceSchema):
+            unread = colander.SchemaNode(colander.Boolean(), missing=colander.drop)
+            position = colander.SchemaNode(colander.Int(), missing=colander.drop)
+
+        self.resource.schema = ArticleSchema
+
     def test_merge_patch_updates_attributes_recursively(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/merge-patch+json'
         self.resource.request.json = {'data': {'a': {'b': 'bbb',
                                                      'c': 'ccc'}}}
         self.resource.patch()
@@ -240,8 +298,6 @@ def test_merge_patch_updates_attributes_recursively(self):
         self.assertEqual(result['a']['b'], 'aaa')
 
     def test_merge_patch_removes_attribute_if_none(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/merge-patch+json'
         self.resource.request.json = {'data': {'field': 'aaa'}}
         self.resource.patch()
         self.resource.request.json = {'data': {'field': None}}
@@ -251,8 +307,6 @@ def test_merge_patch_removes_attribute_if_none(self):
         self.assertNotIn('field', result)
 
     def test_merge_patch_removes_attributes_recursively_if_none(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/merge-patch+json'
         self.resource.request.json = {'data': {'a': {'b': 'aaa'}}}
         self.resource.patch()
         self.resource.request.json = {'data': {'a': {'b': None}}}
@@ -266,8 +320,6 @@ def test_merge_patch_removes_attributes_recursively_if_none(self):
         self.assertNotIn('cc', result['aa']['bb'])
 
     def test_merge_patch_doesnt_remove_attribute_if_false(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/merge-patch+json'
         self.resource.request.json = {'data': {'field': 0}}
         result = self.resource.patch()['data']
         self.assertIn('field', result)
@@ -279,8 +331,7 @@ def test_merge_patch_doesnt_remove_attribute_if_false(self):
         self.assertIn('field', result)
 
     def test_patch_doesnt_remove_attribute_if_not_merge_header(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/json'
+        self.headers['Content-Type'] = 'application/json'
         self.resource.request.json = {'data': {'field': 'aaa'}}
         self.resource.patch()
         self.resource.request.json = {'data': {'field': None}}
@@ -290,63 +341,17 @@ def test_patch_doesnt_remove_attribute_if_not_merge_header(self):
         self.assertIn('field', result)
 
     def test_merge_patch_doesnt_remove_previously_inserted_nones(self):
-        header = self.resource.request.headers
-        header['Content-Type'] = 'application/json'
+        self.headers['Content-Type'] = 'application/json'
         self.resource.request.json = {'data': {'field': 'aaa'}}
         result = self.resource.patch()['data']
         self.resource.request.json = {'data': {'field': None}}
         result = self.resource.patch()['data']
         self.assertIn('field', result)
-        header['Content-Type'] = 'application/merge-patch+json'
+        self.headers['Content-Type'] = 'application/merge-patch+json'
         self.resource.request.json = {'data': {'position': 10}}
         result = self.resource.patch()['data']
         self.assertIn('field', result)
 
-    def test_record_timestamp_is_not_updated_if_none_for_missing_field(self):
-        self.resource.request.json = {'data': {'polo': None}}
-        result = self.resource.patch()['data']
-        self.assertEquals(self.result['last_modified'],
-                          result['last_modified'])
-
-    def test_record_timestamp_is_not_updated_if_no_field_changed(self):
-        self.resource.request.json = {'data': {'position': 10}}
-        result = self.resource.patch()['data']
-        self.assertEquals(self.result['last_modified'],
-                          result['last_modified'])
-
-    def test_collection_timestamp_is_not_updated_if_no_field_changed(self):
-        self.resource.request.json = {'data': {'position': 10}}
-        self.resource.patch()
-        self.resource = self.resource_class(request=self.get_request(),
-                                            context=self.get_context())
-        self.resource.collection_get()['data']
-        last_modified = int(self.last_response.headers['ETag'][1:-1])
-        self.assertEquals(self.result['last_modified'], last_modified)
-
-    def test_timestamp_is_not_updated_if_no_change_after_preprocessed(self):
-        with mock.patch.object(self.resource, 'process_record') as mocked:
-            mocked.return_value = self.result
-            self.resource.request.json = {'data': {'position': 20}}
-            result = self.resource.patch()['data']
-            self.assertEquals(self.result['last_modified'],
-                              result['last_modified'])
-
-    def test_returns_changed_fields_among_provided_if_behaviour_is_diff(self):
-        self.resource.request.json = {'data': {'unread': True, 'position': 15}}
-        self.resource.request.headers['Response-Behavior'] = 'diff'
-        with mock.patch.object(self.resource.model, 'update_record',
-                               return_value={'unread': True, 'position': 0}):
-            result = self.resource.patch()['data']
-        self.assertDictEqual(result, {'position': 0})
-
-    def test_returns_changed_fields_if_behaviour_is_light(self):
-        self.resource.request.json = {'data': {'unread': True, 'position': 15}}
-        self.resource.request.headers['Response-Behavior'] = 'light'
-        with mock.patch.object(self.resource.model, 'update_record',
-                               return_value={'unread': True, 'position': 0}):
-            result = self.resource.patch()['data']
-        self.assertDictEqual(result, {'unread': True, 'position': 0})
-
 
 class JsonPatchTest(BaseTest):
     def setUp(self):
diff --git a/tests/test_views_records.py b/tests/test_views_records.py
--- a/tests/test_views_records.py
+++ b/tests/test_views_records.py
@@ -301,6 +301,49 @@ def test_records_can_be_created_after_deletion(self):
                           headers=headers, status=201)
 
 
+class RecordsViewMergeTest(BaseWebTest, unittest.TestCase):
+
+    collection_url = '/buckets/beers/collections/barley/records'
+    _record_url = '/buckets/beers/collections/barley/records/%s'
+
+    def setUp(self):
+        super(RecordsViewMergeTest, self).setUp()
+        self.app.put_json('/buckets/beers', MINIMALIST_BUCKET,
+                          headers=self.headers)
+        self.app.put_json('/buckets/beers/collections/barley',
+                          MINIMALIST_COLLECTION,
+                          headers=self.headers)
+        record = MINIMALIST_RECORD.copy()
+        record['data'] = {}
+        record['data']['grain'] = {'one': 1}
+        resp = self.app.post_json(self.collection_url,
+                                  record,
+                                  headers=self.headers)
+        self.record = resp.json['data']
+        self.record_url = self._record_url % self.record['id']
+
+    def test_merge_patch(self):
+        headers = self.headers.copy()
+        headers['Content-Type'] = 'application/merge-patch+json'
+        json = {'data': {'grain': {'two': 2}}}
+        resp = self.app.patch_json(self.record_url,
+                                   json,
+                                   headers=headers,
+                                   status=200)
+        self.assertEquals(resp.json['data']['grain']['one'], 1)
+        self.assertEquals(resp.json['data']['grain']['two'], 2)
+
+    def test_merge_patch_remove_nones(self):
+        headers = self.headers.copy()
+        headers['Content-Type'] = 'application/merge-patch+json'
+        json = {'data': {'grain': {'one': None}}}
+        resp = self.app.patch_json(self.record_url,
+                                   json,
+                                   headers=headers,
+                                   status=200)
+        self.assertNotIn('one', resp.json['data']['grain'])
+
+
 class RecordsViewPatchTest(BaseWebTest, unittest.TestCase):
 
     collection_url = '/buckets/beers/collections/barley/records'