Repository: Kinto/kinto

Issue:
Paginating permissions crashes the server
It seems impossible to paginate permissions using the `/permissions` endpoint.

To reproduce, set `paginate_by` to `10`. Then:

```
$ http -a x: PUT :8888/v1/buckets/b1
$ http -a x: PUT :8888/v1/buckets/b2
$ http -a x: PUT :8888/v1/buckets/b3
$ http -a x: PUT :8888/v1/buckets/b4
$ http -a x: PUT :8888/v1/buckets/b5
$ http -a x: PUT :8888/v1/buckets/b6
$ http -a x: PUT :8888/v1/buckets/b7
$ http -a x: PUT :8888/v1/buckets/b8
$ http -a x: PUT :8888/v1/buckets/b9
$ http -a x: PUT :8888/v1/buckets/b10
$ http -a x: PUT :8888/v1/buckets/b11
```

The first retrieval succeeds:

```json
$ http -a x: :8888/v1/permissions?_sort=-last_modified

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Pragma, Cache-Control, Total-Records, Next-Page, Last-Modified, ETag, Alert, Content-Length, Expires, Retry-After, Backoff
Cache-Control: no-cache, no-store
Content-Length: 1596
Content-Type: application/json
Date: Mon, 13 Mar 2017 17:41:13 GMT
Etag: "1489426873181"
Last-Modified: Mon, 13 Mar 2017 17:41:13 GMT
Next-Page: http://localhost:8888/v1/permissions?_token=eyJsYXN0X3JlY29yZCI6e30sIm9mZnNldCI6MTB9&_sort=-last_modified&_limit=10
Server: waitress
Total-Records: 11
X-Content-Type-Options: nosniff

{
    "data": [
        {
            "bucket_id": "b9", 
...
```

But if you take the value of the `Next-Page` header and GET it:

```
$ http -a x: 'http://localhost:8888/v1/permissions?_token=eyJsYXN0X3JlY29yZCI6e30sIm9mZnNldCI6MTB9&_sort=-last_modified&_limit=10'

HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Pragma, Cache-Control, Total-Records, Next-Page, Last-Modified, ETag, Alert, Content-Length, Expires, Retry-After, Backoff
Content-Length: 177
Content-Type: application/json
Date: Mon, 13 Mar 2017 17:41:39 GMT
Server: waitress
X-Content-Type-Options: nosniff

{
    "code": 500, 
    "errno": 999, 
    "error": "Internal Server Error", 
    "info": "https://github.com/Kinto/kinto/issues/", 
    "message": "A programmatic error occured, developers have been informed."
}
```

The stack trace on the server is:

```python
ERROR:root:list index out of range
Traceback (most recent call last):
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/tweens.py", line 22, in excview_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 119, in tm_tween
    reraise(*exc_info)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 98, in tm_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/router.py", line 155, in handle_request
    view_name
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/view.py", line 612, in _call_view
    response = view_callable(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/config/views.py", line 181, in __call__
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 389, in attr_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 367, in predicate_wrapper
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 300, in secured_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 438, in rendered_view
    result = view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 147, in _requestonly_view
    response = view(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/cornice/service.py", line 491, in wrapper
    response = view_()
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 279, in collection_get
    limit, sorting)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 1055, in _extract_pagination_rules_from_token
    filters = self._build_pagination_rules(sorting, last_record)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 1021, in _build_pagination_rules
    field, direction = sorting[-1]
IndexError: list index out of range
```

Using latest Kinto server master at the time of posting this issue. 
Paginating permissions crashes the server
It seems impossible to paginate permissions using the `/permissions` endpoint.

To reproduce, set `paginate_by` to `10`. Then:

```
$ http -a x: PUT :8888/v1/buckets/b1
$ http -a x: PUT :8888/v1/buckets/b2
$ http -a x: PUT :8888/v1/buckets/b3
$ http -a x: PUT :8888/v1/buckets/b4
$ http -a x: PUT :8888/v1/buckets/b5
$ http -a x: PUT :8888/v1/buckets/b6
$ http -a x: PUT :8888/v1/buckets/b7
$ http -a x: PUT :8888/v1/buckets/b8
$ http -a x: PUT :8888/v1/buckets/b9
$ http -a x: PUT :8888/v1/buckets/b10
$ http -a x: PUT :8888/v1/buckets/b11
```

The first retrieval succeeds:

```json
$ http -a x: :8888/v1/permissions?_sort=-last_modified

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Pragma, Cache-Control, Total-Records, Next-Page, Last-Modified, ETag, Alert, Content-Length, Expires, Retry-After, Backoff
Cache-Control: no-cache, no-store
Content-Length: 1596
Content-Type: application/json
Date: Mon, 13 Mar 2017 17:41:13 GMT
Etag: "1489426873181"
Last-Modified: Mon, 13 Mar 2017 17:41:13 GMT
Next-Page: http://localhost:8888/v1/permissions?_token=eyJsYXN0X3JlY29yZCI6e30sIm9mZnNldCI6MTB9&_sort=-last_modified&_limit=10
Server: waitress
Total-Records: 11
X-Content-Type-Options: nosniff

{
    "data": [
        {
            "bucket_id": "b9", 
...
```

But if you take the value of the `Next-Page` header and GET it:

```
$ http -a x: 'http://localhost:8888/v1/permissions?_token=eyJsYXN0X3JlY29yZCI6e30sIm9mZnNldCI6MTB9&_sort=-last_modified&_limit=10'

HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Pragma, Cache-Control, Total-Records, Next-Page, Last-Modified, ETag, Alert, Content-Length, Expires, Retry-After, Backoff
Content-Length: 177
Content-Type: application/json
Date: Mon, 13 Mar 2017 17:41:39 GMT
Server: waitress
X-Content-Type-Options: nosniff

{
    "code": 500, 
    "errno": 999, 
    "error": "Internal Server Error", 
    "info": "https://github.com/Kinto/kinto/issues/", 
    "message": "A programmatic error occured, developers have been informed."
}
```

The stack trace on the server is:

```python
ERROR:root:list index out of range
Traceback (most recent call last):
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/tweens.py", line 22, in excview_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 119, in tm_tween
    reraise(*exc_info)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 98, in tm_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/router.py", line 155, in handle_request
    view_name
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/view.py", line 612, in _call_view
    response = view_callable(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/config/views.py", line 181, in __call__
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 389, in attr_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 367, in predicate_wrapper
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 300, in secured_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 438, in rendered_view
    result = view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 147, in _requestonly_view
    response = view(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/cornice/service.py", line 491, in wrapper
    response = view_()
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 279, in collection_get
    limit, sorting)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 1055, in _extract_pagination_rules_from_token
    filters = self._build_pagination_rules(sorting, last_record)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto/core/resource/__init__.py", line 1021, in _build_pagination_rules
    field, direction = sorting[-1]
IndexError: list index out of range
```

Using latest Kinto server master at the time of posting this issue. 


Discussion:
It seems like permissions entries don't have a `last_modified` property, which actually makes sense. Maybe that's an erroneous use of the API by the client, but still, this should not crash the server.
I confirm the server doesn't crash and paginates properly when the `_sort` parameter on a non existent property is omitted.
Refs https://github.com/Kinto/kinto/pull/816 ?
Yeah looks a lot like it's the same issue. FWIW the fix in kinto-http.js looks like this https://github.com/Kinto/kinto-http.js/pull/170 (brr)

Edit: I mean this

![](http://i.imgur.com/5IqGySr.png)
It seems like permissions entries don't have a `last_modified` property, which actually makes sense. Maybe that's an erroneous use of the API by the client, but still, this should not crash the server.
I confirm the server doesn't crash and paginates properly when the `_sort` parameter on a non existent property is omitted.
Refs https://github.com/Kinto/kinto/pull/816 ?
Yeah looks a lot like it's the same issue. FWIW the fix in kinto-http.js looks like this https://github.com/Kinto/kinto-http.js/pull/170 (brr)

Edit: I mean this

![](http://i.imgur.com/5IqGySr.png)

Resolving patch:
diff --git a/kinto/views/permissions.py b/kinto/views/permissions.py
--- a/kinto/views/permissions.py
+++ b/kinto/views/permissions.py
@@ -4,6 +4,7 @@
 
 from kinto.authorization import PERMISSIONS_INHERITANCE_TREE
 from kinto.core import utils as core_utils, resource
+from kinto.core.storage import Sort
 from kinto.core.storage.memory import extract_record_set
 
 
@@ -172,6 +173,10 @@ def _extract_sorting(self, limit):
         result = super()._extract_sorting(limit)
         without_last_modified = [s for s in result
                                  if s.field != self.model.modified_field]
+        # For pagination, there must be at least one sort criteria.
+        # We use ``uri`` because its values are unique.
+        if "uri" not in [s.field for s in without_last_modified]:
+            without_last_modified.append(Sort("uri", -1))
         return without_last_modified
 
     def _extract_filters(self):


Test patch:
diff --git a/tests/test_views_permissions.py b/tests/test_views_permissions.py
--- a/tests/test_views_permissions.py
+++ b/tests/test_views_permissions.py
@@ -100,6 +100,31 @@ def test_permissions_list_do_not_crash_with_preconditions(self):
         headers = {'If-None-Match': '"123"', **self.headers}
         self.app.get('/permissions', headers=headers)
 
+    def test_permissions_can_be_paginated(self):
+        for i in range(10):
+            self.app.put_json('/buckets/beers/collections/barley/records/r-{}'.format(i),
+                              MINIMALIST_RECORD,
+                              headers=self.headers)
+        resp = self.app.get('/permissions?resource_name=record&_limit=7', headers=self.headers)
+        page1 = resp.json["data"]
+        next_page = resp.headers["Next-Page"].replace("http://localhost/v1", "")
+        resp = self.app.get(next_page, headers=self.headers)
+        page2 = resp.json["data"]
+        assert len(page1 + page2) == 11  # see setup().
+
+    def test_permissions_can_be_paginated_with_uri_in_sorting(self):
+        for i in range(10):
+            self.app.put_json('/buckets/beers/collections/barley/records/r-{}'.format(i),
+                              MINIMALIST_RECORD,
+                              headers=self.headers)
+        resp = self.app.get('/permissions?resource_name=record&_limit=7&_sort=uri',
+                            headers=self.headers)
+        page1 = resp.json["data"]
+        next_page = resp.headers["Next-Page"].replace("http://localhost/v1", "")
+        resp = self.app.get(next_page, headers=self.headers)
+        page2 = resp.json["data"]
+        assert len(page1 + page2) == 11  # see setup().
+
 
 class GroupsPermissionTest(PermissionsViewTest):