Repository: Kinto/kinto

Issue:
Consistency on PUT with mandatory schema fields
While working on #790 I realize that there is something not clear in our specifications.

Currently, if a resource has a mandatory field (eg. groups `members`), then we cannot do a `PUT` with just the `permissions` values. This is because a PUT can lead to a creation, and the `members` fields has to be provided.

On other resources, which have no mandatory field, it is perfectly possible to only provide `permissions`.

But, I believe we should make every resources behave the same way.

For example, when we'll implement the edition of permissions in Kinto-admin, we don't want to have to pass the `data` if it was not changed.

Two solutions:
- Add a default value (`[]`) for the groups members attribute (_my prefered one, trivial and not absurd_)
- Allow `data` to be omitted only when the `PUT` replaces an existing object (_more complex to implement, but would work for any resource with mandatory fields_)

Consistency on PUT with mandatory schema fields
While working on #790 I realize that there is something not clear in our specifications.

Currently, if a resource has a mandatory field (eg. groups `members`), then we cannot do a `PUT` with just the `permissions` values. This is because a PUT can lead to a creation, and the `members` fields has to be provided.

On other resources, which have no mandatory field, it is perfectly possible to only provide `permissions`.

But, I believe we should make every resources behave the same way.

For example, when we'll implement the edition of permissions in Kinto-admin, we don't want to have to pass the `data` if it was not changed.

Two solutions:
- Add a default value (`[]`) for the groups members attribute (_my prefered one, trivial and not absurd_)
- Allow `data` to be omitted only when the `PUT` replaces an existing object (_more complex to implement, but would work for any resource with mandatory fields_)



Discussion:
Yes I agree that it should be consistent and I agree with your proposed solution.

Let's go with the trivial solution (at least for now): add a default value...

I would like to work on this for my outreachy contribution.  Can you assign to me?

Yes go ahead :)

@anna-liao would you need help to get started?

Thanks!  I still need to look into how resources work in Kinto.  I understand what the output should be for a given input, but need to figure out which modifications to the code need to be made.  Primarily the advice I received from glasserc is to look through the docs for resource and also in kinto/views/foo.py... which, now that I'm looking it up, I don't see a foo.py, unless I read that too literally.  If you have any suggestions of where to look, that would be helpful.

As magopian said:

> Let's go with the trivial solution (at least for now): add a default value...

So basically, all you need to do is allowing to create groups with a `PUT` request that has no body. For that you'll have to change the group validation schema in views files for groups, and add a default value for the `members` attribute (empty list `[]`).

You'll have to update the group views tests to reflect that change if any test fails, and a add a new one that specifies the behavior.

From the tests the following operation should work:

``` py
self.app.put("/buckets/bid", headers=self.headers, status=201)
self.app.put("/buckets/bid/groups/atest", headers=self.headers, status=201)
```

And from the command-line, with httpie installed:

```
$ http PUT :8888/v1/buckets/bid --auth token:blah
$ http PUT :8888/v1/buckets/bid/groups/atest --auth token:blah
```

With the current code, you'll that it complains because `members` is not provided.

@leplatrem Thanks for the info!  Understandably, this contribution will be too late for Outreachy, but I still want to work on it.  I'm going to look into fixing it and also writing a test for it.

Has this been fixed already?  I just checked and it is no longer throwing a "400 Bad Request" error.

http PUT :8888/v1/buckets/bid/groups/atest --auth token:blah
HTTP/1.1 201 Created
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 156
Content-Type: application/json; charset=UTF-8
Date: Fri, 28 Oct 2016 21:34:13 GMT
Etag: "1477690453885"
Last-Modified: Fri, 28 Oct 2016 21:34:13 GMT
Server: waitress

{
    "data": {
        "id": "atest",
        "last_modified": 1477690453885
    },
    "permissions": {
        "write": [
            "basicauth:12e00e7821f216ae3dac21c3f996b2fd123ed72c6b8245cea6cd3e7c9dde7a7e"
        ]
    }
}

Oh...

I don't see any default value for members : https://github.com/Kinto/kinto/blob/master/kinto/views/groups.py#L13-L16

So this means that the upgrade to Cornice 2 ( #790 ) changed something here. 

There is still something to fix. The group should **always** have a `members` attribute, with an empty list if not specified.

Thanks for catching this!

I think the next step should be to start a PR adding a test for the behavior we want and the one we don't and then we can elaborate from there.

@Natim Is there an example of a similar test in the repo?  I have used unit tests, but not sure how the test structure works in this repo.

oh, nevermind, I see the tests now under kinto/tests so will take a look through that.

https://github.com/Kinto/kinto/blob/master/tests/test_views_buckets.py#L145-L148

@anna-liao I don't have the same behaviour with a PUT and an empty body:

```
http PUT :8888/v1/buckets/t/groups/g --auth t:t
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 159
Content-Type: application/json; charset=UTF-8
Date: Thu, 01 Dec 2016 15:40:34 GMT
Server: waitress

{
    "code": 400, 
    "details": [
        {
            "description": "data is missing", 
            "location": "body", 
            "name": "data"
        }
    ], 
    "errno": 107, 
    "error": "Invalid parameters", 
    "message": "data is missing"
}

```


Are you still interested to look into it?
@leplatrem Thanks for the feedback.  I would still like to look into this.  Sorry for the delay.
Please, no, don't be sorry! 

First, there is no emergency :) Second I suppose that you would do that on your personal free time, so it should never be a burden! 

I was asking because sometimes our interests change over time, and someone would be glad to take over. 

So, please, it's all yours, take your time, and I hope you'll enjoy hacking on it :)
thanks, appreciate the nice comment to clarify and really appreciate all of you for taking the time to work with newbies.  I feel like the structure of how this all works is probably quite standard, but new to me.  I would like to learn, and hopefully once I have spent some time with it, will be able to contribute more to this project.  I have extra time during the holidays, so going to spend some quality time looking into this.  Cheers!
Yes I agree that it should be consistent and I agree with your proposed solution.

Let's go with the trivial solution (at least for now): add a default value...

I would like to work on this for my outreachy contribution.  Can you assign to me?

Yes go ahead :)

@anna-liao would you need help to get started?

Thanks!  I still need to look into how resources work in Kinto.  I understand what the output should be for a given input, but need to figure out which modifications to the code need to be made.  Primarily the advice I received from glasserc is to look through the docs for resource and also in kinto/views/foo.py... which, now that I'm looking it up, I don't see a foo.py, unless I read that too literally.  If you have any suggestions of where to look, that would be helpful.

As magopian said:

> Let's go with the trivial solution (at least for now): add a default value...

So basically, all you need to do is allowing to create groups with a `PUT` request that has no body. For that you'll have to change the group validation schema in views files for groups, and add a default value for the `members` attribute (empty list `[]`).

You'll have to update the group views tests to reflect that change if any test fails, and a add a new one that specifies the behavior.

From the tests the following operation should work:

``` py
self.app.put("/buckets/bid", headers=self.headers, status=201)
self.app.put("/buckets/bid/groups/atest", headers=self.headers, status=201)
```

And from the command-line, with httpie installed:

```
$ http PUT :8888/v1/buckets/bid --auth token:blah
$ http PUT :8888/v1/buckets/bid/groups/atest --auth token:blah
```

With the current code, you'll that it complains because `members` is not provided.

@leplatrem Thanks for the info!  Understandably, this contribution will be too late for Outreachy, but I still want to work on it.  I'm going to look into fixing it and also writing a test for it.

Has this been fixed already?  I just checked and it is no longer throwing a "400 Bad Request" error.

http PUT :8888/v1/buckets/bid/groups/atest --auth token:blah
HTTP/1.1 201 Created
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 156
Content-Type: application/json; charset=UTF-8
Date: Fri, 28 Oct 2016 21:34:13 GMT
Etag: "1477690453885"
Last-Modified: Fri, 28 Oct 2016 21:34:13 GMT
Server: waitress

{
    "data": {
        "id": "atest",
        "last_modified": 1477690453885
    },
    "permissions": {
        "write": [
            "basicauth:12e00e7821f216ae3dac21c3f996b2fd123ed72c6b8245cea6cd3e7c9dde7a7e"
        ]
    }
}

Oh...

I don't see any default value for members : https://github.com/Kinto/kinto/blob/master/kinto/views/groups.py#L13-L16

So this means that the upgrade to Cornice 2 ( #790 ) changed something here. 

There is still something to fix. The group should **always** have a `members` attribute, with an empty list if not specified.

Thanks for catching this!

I think the next step should be to start a PR adding a test for the behavior we want and the one we don't and then we can elaborate from there.

@Natim Is there an example of a similar test in the repo?  I have used unit tests, but not sure how the test structure works in this repo.

oh, nevermind, I see the tests now under kinto/tests so will take a look through that.

https://github.com/Kinto/kinto/blob/master/tests/test_views_buckets.py#L145-L148

@anna-liao I don't have the same behaviour with a PUT and an empty body:

```
http PUT :8888/v1/buckets/t/groups/g --auth t:t
HTTP/1.1 400 Bad Request
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 159
Content-Type: application/json; charset=UTF-8
Date: Thu, 01 Dec 2016 15:40:34 GMT
Server: waitress

{
    "code": 400, 
    "details": [
        {
            "description": "data is missing", 
            "location": "body", 
            "name": "data"
        }
    ], 
    "errno": 107, 
    "error": "Invalid parameters", 
    "message": "data is missing"
}

```


Are you still interested to look into it?
@leplatrem Thanks for the feedback.  I would still like to look into this.  Sorry for the delay.
Please, no, don't be sorry! 

First, there is no emergency :) Second I suppose that you would do that on your personal free time, so it should never be a burden! 

I was asking because sometimes our interests change over time, and someone would be glad to take over. 

So, please, it's all yours, take your time, and I hope you'll enjoy hacking on it :)
thanks, appreciate the nice comment to clarify and really appreciate all of you for taking the time to work with newbies.  I feel like the structure of how this all works is probably quite standard, but new to me.  I would like to learn, and hopefully once I have spent some time with it, will be able to contribute more to this project.  I have extra time during the holidays, so going to spend some quality time looking into this.  Cheers!

Resolving patch:
diff --git a/kinto/views/groups.py b/kinto/views/groups.py
--- a/kinto/views/groups.py
+++ b/kinto/views/groups.py
@@ -13,7 +13,8 @@ def validate_member(node, member):
 class GroupSchema(resource.ResourceSchema):
     members = colander.SchemaNode(colander.Sequence(),
                                   colander.SchemaNode(colander.String(),
-                                                      validator=validate_member))
+                                                      validator=validate_member),
+                                  missing=[])
 
 
 @resource.register(name='group',


Test patch:
diff --git a/tests/test_views_groups.py b/tests/test_views_groups.py
--- a/tests/test_views_groups.py
+++ b/tests/test_views_groups.py
@@ -34,6 +34,31 @@ def test_groups_can_be_posted_without_id(self):
         self.assertIn('id', resp.json['data'])
         self.assertEqual(resp.json['data']['members'], ['fxa:user'])
 
+    def test_groups_can_be_put_with_empty_body(self):
+        self.app.put('/buckets/beers/groups/simple', headers=self.headers)
+
+    def test_groups_can_be_created_with_just_permissions(self):
+        group = {'permissions': {'write': ['github:me']}}
+        resp = self.app.put_json('/buckets/beers/groups/g',
+                                 group,
+                                 headers=self.headers)
+        self.assertNotIn('members', resp.json['data'])
+
+    def test_groups_can_be_create_without_members_attribute(self):
+        group = {'data': {'alias': 'admins'}}
+        resp = self.app.put_json(self.record_url,
+                                 group,
+                                 headers=self.headers)
+        self.assertEqual(resp.json['data']['members'], [])
+
+    def test_groups_can_be_patched_without_members_attribute(self):
+        group = {'data': {'alias': 'admins'}}
+        resp = self.app.patch_json(self.record_url,
+                                   group,
+                                   headers=self.headers)
+        self.assertEqual(resp.json['data']['members'], ['fxa:user'])
+        self.assertEqual(resp.json['data']['alias'], 'admins')
+
     def test_groups_can_be_put_with_simple_name(self):
         self.assertEqual(self.record['id'], 'moderators')
 
@@ -230,28 +255,3 @@ def test_groups_data_is_optional_with_patch(self):
         self.app.patch_json(self.group_url,
                             valid,
                             headers=self.headers)
-
-
-class InvalidGroupTest(BaseWebTest, unittest.TestCase):
-
-    group_url = '/buckets/beers/groups/moderators'
-
-    def setUp(self):
-        super().setUp()
-        self.create_bucket('beers')
-
-    def test_groups_data_is_required_with_put(self):
-        invalid = {'permissions': {'write': ['github:me']}}
-        resp = self.app.put_json(self.group_url,
-                                 invalid,
-                                 headers=self.headers,
-                                 status=400)
-        self.assertEqual(resp.json['message'], "data in body: Required")
-
-    def test_groups_must_have_members_attribute(self):
-        invalid = {'data': {'alias': 'admins'}}
-        resp = self.app.put_json(self.group_url,
-                                 invalid,
-                                 headers=self.headers,
-                                 status=400)
-        self.assertIn('data.members in body', resp.json['message'])