Repository: Kinto/kinto

Issue:
Add custom (meta) data on buckets and collections 
For some use-cases, it might become useful to be able to store some custom attributes in buckets or collections (e.g. metadata like application version, contact email or whatever).

Currently both Collection and Bucket resources do not define extra fields in their schema, and Cliquet drops unknown fields if not explicitly allowed.

We can either:
- Allow unknown fields in collection and buckets schemas
- Add a specific root level field (along `data` and `permissions`)
- Add a specific field (called `meta` for example) in the schema that could receive anything.

The advantage of the latter is that custom fields do not interfere with anything in the protocol, and are trivial to implement. The inconvenient is having to put `{data: {metadata: {email: "toto@yo.com"}}` in the payload.

Thoughts ?



Discussion:
> Taken from #161:

When considering this proposal, @QuentinRoy was suggesting that we use "metadata" as a field name for `last_modified` and other internal properties, and store the collection data in "data".

I am not sure I got what is the inconvenient you talked about. But I don't think the `metadata`object/container/property have to be inside `data`. I would have put them in 2 separate places: `{data: {}, metadata: {email: "ilove@pancakes.com"}`. `metadata` would be a read-only system-managed property container.

> I would have put them in 2 separate places: {data: {}, metadata: {email: "ilove@pancakes.com"}. metadata would be a read-only system-managed property container.

I agree that this should be the path forward for that.

I would also let people set `metadata` in their payload for validation.

i.e, the system-side generated value should be equal to the provided one for the server to accept the payload. (email/signing)

@Natim why not but this is orthogonal to what we're proposing this IMO. 

Doing the rename of the current "data" in "metadata" will break the protocol, though, so we should issue a new version of it. Since we don't have anything in production just yet that relies on the current version of the protocol, it might be the right timing to change it.

Yes I think it is the good time to break the protocol.

The validation is orthogonal it was just me having some ideas about the read-only proposal.

The question is what about the id?

On a PUT should it be on both data and metadata? Or should we add it back on data on server side if we need it (let say for signing for instance)

I just realised that this post is about allowing data on buckets and collections I thought this was already the case.

```
$ echo '{"data": {"signing": "hash"}}' | http PUT https://kinto.dev.mozaws.net/v1/buckets/default/collections/tasks --auth user:pass
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length
Connection: keep-alive
Content-Length: 156
Content-Type: application/json; charset=UTF-8
Date: Mon, 24 Aug 2015 13:18:31 GMT
ETag: "1440422311022"
Last-Modified: Mon, 24 Aug 2015 13:18:31 GMT
Server: nginx/1.4.6 (Ubuntu)

{
    "data": {
        "id": "tasks", 
        "last_modified": 1440422311022
    }, 
    "permissions": {
        "write": [
            "basicauth:631c2d625ee5726172cf67c6750de10a3e1a04bcd603bc9ad6d6b196fa8257a6"
        ]
    }
}
```

Signing is lost during the operation.

IMHO the server should never put, alter, use or rely on any data stored in the `data` container (if we retain the `data`/`metadata` scheme we were talking about). It should be entirely managed by the user and should not cause any action from the server (except recording, pull events and sync stuffs of course).
If the server needs an `id`, even provided by the user at some point, it should be in `metadata`. If the user wants to use its own id and doesn't want to rely on the server-managed one (in `metadata` where he may not be able to set it or change it), he can do whatever he wants inside `data` without any risk of side effects from the server.

That's exactly the scope of the data/metadata thing: _metadata_ is data kinto relies on internally, whereas _data_ is meant to be used by the client.

Yes :+1: 


Resolving patch:
diff --git a/kinto/views/collections.py b/kinto/views/collections.py
--- a/kinto/views/collections.py
+++ b/kinto/views/collections.py
@@ -28,6 +28,9 @@ def deserialize(self, cstruct=colander.null):
 class CollectionSchema(resource.ResourceSchema):
     schema = JSONSchemaMapping(missing=colander.drop)
 
+    class Options:
+        preserve_unknown = True
+
 
 @resource.register(name='collection',
                    collection_methods=('GET',),
diff --git a/kinto/views/records.py b/kinto/views/records.py
--- a/kinto/views/records.py
+++ b/kinto/views/records.py
@@ -7,7 +7,7 @@
 
 
 class RecordSchema(schema.ResourceSchema):
-    class Options():
+    class Options:
         preserve_unknown = True
 
 


Test patch:
diff --git a/kinto/tests/test_views_collections.py b/kinto/tests/test_views_collections.py
--- a/kinto/tests/test_views_collections.py
+++ b/kinto/tests/test_views_collections.py
@@ -64,6 +64,18 @@ def test_wrong_create_permissions_cannot_be_added_on_collections(self):
                           headers=self.headers,
                           status=400)
 
+    def test_collections_can_handle_arbitrary_attributes(self):
+        collection = MINIMALIST_COLLECTION.copy()
+        fingerprint = "5866f245a00bb3a39100d31b2f14d453"
+        collection['data'] = {'fingerprint': fingerprint}
+        resp = self.app.put_json('/buckets/beers/collections/barley',
+                                 collection,
+                                 headers=self.headers,
+                                 status=200)
+        data = resp.json['data']
+        self.assertIn('fingerprint', data)
+        self.assertEqual(data['fingerprint'], fingerprint)
+
 
 class CollectionDeletionTest(BaseWebTest, unittest.TestCase):