internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Kinto/kinto

Issue:
Disparity between memory and postgresql when filtering with missing values
Once #814 will be merged, there is another minor bug to tackle:

when filtering with `?max_field=10`, the memory backend returns records which don't have `field` defined, and postgresql does not.



Discussion:
Let's follow what Postgresql does on that one.

As a side note, [this test](https://github.com/Kinto/kinto/blob/54572a2d9aa4a56fdf49cf463324dd2dee8260a3/kinto/core/storage/testing.py#L341) doesn't fail for the memory backend because there's a `min` constraint that "fixes" the issue.


Resolving patch:
diff --git a/kinto/core/storage/memory.py b/kinto/core/storage/memory.py
--- a/kinto/core/storage/memory.py
+++ b/kinto/core/storage/memory.py
@@ -329,13 +329,16 @@ def apply_filters(records, filters):
             if f.operator in (COMPARISON.IN, COMPARISON.EXCLUDE):
                 right, left = left, right
             else:
-                # Python3 cannot compare None to other value.
                 if left is None:
-                    if f.operator in (COMPARISON.GT, COMPARISON.MIN):
+                    right_is_number = (
+                        isinstance(right, (int, float)) and
+                        right not in (True, False))
+                    if right_is_number:
+                        # Python3 cannot compare None to a number.
                         matches = False
                         continue
-                    elif f.operator in (COMPARISON.LT, COMPARISON.MAX):
-                        continue  # matches = matches and True
+                    else:
+                        left = ''  # To mimic what we do for postgresql.
             matches = matches and operators[f.operator](left, right)
         if matches:
             yield record


Test patch:
diff --git a/kinto/core/storage/testing.py b/kinto/core/storage/testing.py
--- a/kinto/core/storage/testing.py
+++ b/kinto/core/storage/testing.py
@@ -343,8 +343,7 @@ def test_get_all_can_filter_with_numeric_values(self):
             self.create_record({'code': l})
 
         sorting = [Sort('code', 1)]
-        filters = [Filter('code', 0, utils.COMPARISON.MIN),
-                   Filter('code', 10, utils.COMPARISON.MAX)]
+        filters = [Filter('code', 10, utils.COMPARISON.MAX)]
         records, _ = self.storage.get_all(sorting=sorting, filters=filters,
                                           **self.storage_kw)
         self.assertEqual(records[0]['code'], 1)
@@ -352,6 +351,13 @@ def test_get_all_can_filter_with_numeric_values(self):
         self.assertEqual(records[2]['code'], 10)
         self.assertEqual(len(records), 3)
 
+        filters = [Filter('code', 10, utils.COMPARISON.LT)]
+        records, _ = self.storage.get_all(sorting=sorting, filters=filters,
+                                          **self.storage_kw)
+        self.assertEqual(records[0]['code'], 1)
+        self.assertEqual(records[1]['code'], 6)
+        self.assertEqual(len(records), 2)
+
     def test_get_all_can_filter_with_numeric_strings(self):
         for l in ["0566199093", "0781566199"]:
             self.create_record({'phone': l})
Repository: Kinto/kinto

Issue:
StatsD backend keys contain twice backend type
![capture d ecran de 2016-10-10 10-26-46](https://cloud.githubusercontent.com/assets/546692/19230297/216fae70-8ed4-11e6-90f6-b73fd694a9d2.png)

`kinto.storage.storage.get` instead of `kinto.storage.get` or `kinto.storage.call.get`



Discussion:
@leplatrem I'd like to work on this. How should I go about it?

Great!

As you can see here: https://github.com/Kinto/kinto/blob/4.3.1/kinto/core/statsd.py#L19-L28, we already use the classname (here `storage`). Since we specify the `prefix` during initialization: https://github.com/Kinto/kinto/blob/4.3.1/kinto/core/initialization.py#L247-L249, the key ends up being `storage.storage`

I believe we'd want to keep the current approach, just that the prefix could just be `backend` so that we have `kinto.backend.storage.get` (and `kinto.backend.cache.get` etc. for example)

Should I just be changing the prefixes in [initialization.py](https://github.com/Kinto/kinto/blob/4.3.1/kinto/core/initialization.py#L247-L249)? Did that, but running `make tests` gave 3 errors all referring to the 3 (cache, storage, `permission).

@d1ndra yes It is possible that you need to change the three tests you are talking about.


Resolving patch:
diff --git a/kinto/core/initialization.py b/kinto/core/initialization.py
--- a/kinto/core/initialization.py
+++ b/kinto/core/initialization.py
@@ -244,9 +244,9 @@ def setup_statsd(config):
 
         config.registry.statsd = client
 
-        client.watch_execution_time(config.registry.cache, prefix='cache')
-        client.watch_execution_time(config.registry.storage, prefix='storage')
-        client.watch_execution_time(config.registry.permission, prefix='permission')
+        client.watch_execution_time(config.registry.cache, prefix='backend')
+        client.watch_execution_time(config.registry.storage, prefix='backend')
+        client.watch_execution_time(config.registry.permission, prefix='backend')
 
         # Commit so that configured policy can be queried.
         config.commit()


Test patch:
diff --git a/tests/core/test_initialization.py b/tests/core/test_initialization.py
--- a/tests/core/test_initialization.py
+++ b/tests/core/test_initialization.py
@@ -302,15 +302,15 @@ def test_statsd_is_expose_in_the_registry_if_url_is_set(self):
 
     def test_statsd_is_set_on_cache(self):
         c = initialization.setup_statsd(self.config)
-        c.watch_execution_time.assert_any_call({}, prefix='cache')
+        c.watch_execution_time.assert_any_call({}, prefix='backend')
 
     def test_statsd_is_set_on_storage(self):
         c = initialization.setup_statsd(self.config)
-        c.watch_execution_time.assert_any_call({}, prefix='storage')
+        c.watch_execution_time.assert_any_call({}, prefix='backend')
 
     def test_statsd_is_set_on_permission(self):
         c = initialization.setup_statsd(self.config)
-        c.watch_execution_time.assert_any_call({}, prefix='permission')
+        c.watch_execution_time.assert_any_call({}, prefix='backend')
 
     def test_statsd_is_set_on_authentication(self):
         c = initialization.setup_statsd(self.config)
Repository: Kinto/kinto

Issue:
Add user principals to hello view
Along `user` info, show the list of user principals (for groups notably)



Discussion:
I would like to work on this one. 

Should the list include the user id or the  `system.Authenticated` and `system.Everyone` default principals?

\o/

Yes, everything returned by `request.effective_principals` is just fine :)


Resolving patch:
diff --git a/kinto/core/views/hello.py b/kinto/core/views/hello.py
--- a/kinto/core/views/hello.py
+++ b/kinto/core/views/hello.py
@@ -32,6 +32,12 @@ def get_hello(request):
     # (Note: this will call authenticated_userid() with multiauth+groupfinder)
     if Authenticated in request.effective_principals:
         data['user'] = request.get_user_info()
+        principals = request.effective_principals + [request.prefixed_userid]
+        prefix, user_id = request.prefixed_userid.split(':', 1)
+        # Remove unprefixed user id on effective_principals to avoid conflicts.
+        if user_id in principals:
+            principals.remove(user_id)
+        data['user']['principals'] = principals
 
     # Application can register and expose arbitrary capabilities.
     data['capabilities'] = request.registry.api_capabilities


Test patch:
diff --git a/tests/test_views_hello.py b/tests/test_views_hello.py
--- a/tests/test_views_hello.py
+++ b/tests/test_views_hello.py
@@ -2,7 +2,8 @@
 
 from kinto.core.testing import unittest
 
-from .support import BaseWebTest
+from .support import (BaseWebTest, MINIMALIST_BUCKET,
+                      MINIMALIST_GROUP)
 
 
 class HelloViewTest(BaseWebTest, unittest.TestCase):
@@ -23,6 +24,17 @@ def test_returns_user_id_if_authenticated(self):
         response = self.app.get('/', headers=self.headers)
         self.assertEqual(response.json['user']['id'], self.principal)
 
+    def test_returns_user_principals_if_authenticated(self):
+        group_url = '/buckets/beers/groups/users'
+        group = MINIMALIST_GROUP.copy()
+        group['data']['members'].append(self.principal)
+        self.app.put_json('/buckets/beers', MINIMALIST_BUCKET, headers=self.headers)
+        self.app.put_json(group_url, group, headers=self.headers)
+        response = self.app.get('/', headers=self.headers).json['user']['principals']
+        principals = ('system.Everyone', 'system.Authenticated',
+                      group_url, self.principal)
+        self.assertEqual(sorted(response), sorted(principals))
+
     def test_capability_is_exposed_if_setting_is_set(self):
         settings = {'experimental_collection_schema_validation': True}
         app = self.make_app(settings=settings)
Repository: Kinto/kinto

Issue:
Forbidden is numbered 401 in the docs
Both Forbidden and Unauthorized responses are listed as 401 in the docs,
I think Forbidden should be 403, right?

For example:
http://kinto.readthedocs.io/en/stable/api/1.x/buckets.html#http-status-codes
Forbidden is numbered 401 in the docs
Both Forbidden and Unauthorized responses are listed as 401 in the docs,
I think Forbidden should be 403, right?

For example:
http://kinto.readthedocs.io/en/stable/api/1.x/buckets.html#http-status-codes


Discussion:
The error response is matching 403

```
$ echo '{}' | http PUT localhost:8888/v1/buckets/b1 --auth aaa:aaa
HTTP/1.1 403 Forbidden
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 95
Content-Type: application/json; charset=UTF-8
Date: Fri, 11 Nov 2016 00:51:39 GMT
Server: waitress

{
    "code": 403,
    "errno": 121,
    "error": "Forbidden",
    "message": "This user cannot access this resource."
}
```

The error response is matching 403

```
$ echo '{}' | http PUT localhost:8888/v1/buckets/b1 --auth aaa:aaa
HTTP/1.1 403 Forbidden
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 95
Content-Type: application/json; charset=UTF-8
Date: Fri, 11 Nov 2016 00:51:39 GMT
Server: waitress

{
    "code": 403,
    "errno": 121,
    "error": "Forbidden",
    "message": "This user cannot access this resource."
}
```


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -137,7 +137,7 @@
 .. |status-304| replace:: ``304 Not Modified``
 .. |status-400| replace:: ``400 Bad Request``
 .. |status-401| replace:: ``401 Unauthorized``
-.. |status-403| replace:: ``401 Forbidden``
+.. |status-403| replace:: ``403 Forbidden``
 .. |status-404| replace:: ``404 Not Found``
 .. |status-405| replace:: ``405 Method Not Allowed``
 .. |status-406| replace:: ``406 Not Acceptable``


Test patch:
Repository: Kinto/kinto

Issue:
'set' object has no attribute extends
```kinto\/views\/permissions.py\", line 107, in get_records\n    perms_by_object_uri.setdefault(subobj_uri, []).extend(resource_perms)\nAttributeError: 'set' object has no attribute 'extend'"```




Discussion:

Resolving patch:
diff --git a/kinto/views/permissions.py b/kinto/views/permissions.py
--- a/kinto/views/permissions.py
+++ b/kinto/views/permissions.py
@@ -96,7 +96,7 @@ def get_records(self, filters=None, sorting=None, pagination_rules=None,
                     resource_perms = from_settings[res]
                     # Bucket is always fetched.
                     if res == 'bucket':
-                        perms_by_object_uri.setdefault(bucket_uri, []).extend(resource_perms)
+                        perms_by_object_uri.setdefault(bucket_uri, set()).update(resource_perms)
                         continue
                     # Fetch bucket collections and groups.
                     # XXX: wrong approach: query in a loop!
@@ -104,7 +104,7 @@ def get_records(self, filters=None, sorting=None, pagination_rules=None,
                                                           collection_id=res)
                     for subobject in every_subobjects:
                         subobj_uri = bucket_uri + '/{0}s/{1}'.format(res, subobject['id'])
-                        perms_by_object_uri.setdefault(subobj_uri, []).extend(resource_perms)
+                        perms_by_object_uri.setdefault(subobj_uri, set()).update(resource_perms)
 
         entries = []
         for object_uri, perms in perms_by_object_uri.items():


Test patch:
diff --git a/tests/test_views_permissions.py b/tests/test_views_permissions.py
--- a/tests/test_views_permissions.py
+++ b/tests/test_views_permissions.py
@@ -175,3 +175,23 @@ def test_record_create_taken_into_account(self):
         collections = [e for e in resp.json['data'] if e['resource_name'] == 'collection']
         self.assertEqual(collections[0]['id'], 'barley')
         self.assertIn('record:create', collections[0]['permissions'])
+
+    def test_settings_permissions_are_merged_with_perms_backend(self):
+        self.app.patch_json('/buckets/beers',
+                            {'permissions': {'collection:create': [self.admin_principal]}},
+                            headers=self.headers)
+        self.app.patch_json('/buckets/beers/collections/barley',
+                            {'permissions': {'read': [self.admin_principal]}},
+                            headers=self.headers)
+
+        resp = self.app.get('/permissions', headers=self.admin_headers)
+
+        buckets = [e for e in resp.json['data'] if e['resource_name'] == 'bucket']
+        self.assertEqual(buckets[0]['id'], 'beers')
+        self.assertIn('group:create', buckets[0]['permissions'])
+        self.assertIn('collection:create', buckets[0]['permissions'])
+
+        collections = [e for e in resp.json['data'] if e['resource_name'] == 'collection']
+        self.assertEqual(collections[0]['id'], 'barley')
+        self.assertIn('record:create', collections[0]['permissions'])
+        self.assertIn('read', collections[0]['permissions'])
Repository: Kinto/kinto

Issue:
make docs recipe is failing
`make docs` recipe is failing due to an invalid reference on `docs/core/resource.rst`. It produces just a warning, but `sphinx-build` is set with `-W` flag to treat it as an error. 

```
> make docs 
.venv/bin/sphinx-build -a -W -n -b html -d docs/_build/doctrees docs docs/_build/html
Running Sphinx v1.4.8
loading pickled environment... not yet created
loading intersphinx inventory from https://colander.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://pyramid.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://cornice.readthedocs.io/en/latest/objects.inv...
building [mo]: all of 0 po files
building [html]: all source files
updating environment: 73 added, 0 changed, 0 removed
reading sources... [100%] tutorials/write-plugin                                
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] tutorials/write-plugin                                 

Warning, treated as error:
kinto/docs/core/resource.rst:30: WARNING: py:class reference target not found: ResourceSchema

Makefile:130: recipe for target 'docs' failed
```


Discussion:
Oh, interesting!

That may be related to the PYTHONPATH, which wouldn't contain the appropriate path to the source code. We do some black magic here:
https://github.com/Kinto/kinto/blob/master/docs/conf.py#L33

Maybe we should improve that...

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -97,26 +97,28 @@
 autodoc_member_order = 'bysource'
 # Enable nitpicky mode - which ensures that all references in the docs
 # resolve.
+# See: http://stackoverflow.com/a/30624034/186202
 nitpicky = True
 nitpick_ignore = [
-    ('py:obj', 'bool'),
     ('py:obj', 'Exception'),
-    ('py:obj', 'int'),
-    ('py:obj', 'str'),
+    ('py:obj', 'bool'),
     ('py:obj', 'dict'),
+    ('py:obj', 'float'),
+    ('py:obj', 'int'),
     ('py:obj', 'list'),
+    ('py:obj', 'str'),
     ('py:obj', 'tuple'),
-    ('py:obj', 'float'),
     ('py:obj', 'cornice.Service'),
     # Member autodoc fails with those:
     # kinto.core.resource.schema
     ('py:class', 'Integer'),
     ('py:class', 'String'),
     # kinto.core.resource
-    ('py:class', 'ViewSet'),
-    ('py:class', 'ShareableViewSet'),
     ('py:class', 'Model'),
+    ('py:class', 'ResourceSchema'),
     ('py:class', 'ShareableModel'),
+    ('py:class', 'ShareableViewSet'),
+    ('py:class', 'ViewSet'),
 ]
 
 


Test patch:
Repository: Kinto/kinto

Issue:
Don't use /tmp but os.gettempdir
```
>> Issue: [B108:hardcoded_tmp_directory] Probable insecure usage of temp file/directory.
   Severity: Medium   Confidence: Medium
   Location: kinto/core/__init__.py:66
```
Don't use /tmp but os.gettempdir
```
>> Issue: [B108:hardcoded_tmp_directory] Probable insecure usage of temp file/directory.
   Severity: Medium   Confidence: Medium
   Location: kinto/core/__init__.py:66
```


Discussion:

Resolving patch:
diff --git a/kinto/core/__init__.py b/kinto/core/__init__.py
--- a/kinto/core/__init__.py
+++ b/kinto/core/__init__.py
@@ -1,6 +1,7 @@
 """Main entry point
 """
 import pkg_resources
+import tempfile
 
 from cornice import Service as CorniceService
 from pyramid.settings import aslist
@@ -62,7 +63,7 @@
     'permission_backend': '',
     'permission_url': '',
     'permission_pool_size': 25,
-    'profiler_dir': '/tmp',
+    'profiler_dir': tempfile.gettempdir(),
     'profiler_enabled': False,
     'project_docs': '',
     'project_name': '',


Test patch:
Repository: Kinto/kinto

Issue:
Heartbeat leaves tombstones

Heartbeat leaves tombstones



Discussion:

Resolving patch:
diff --git a/kinto/core/storage/__init__.py b/kinto/core/storage/__init__.py
--- a/kinto/core/storage/__init__.py
+++ b/kinto/core/storage/__init__.py
@@ -261,7 +261,8 @@ def ping(request):
             else:
                 if random.SystemRandom().random() < _HEARTBEAT_DELETE_RATE:
                     backend.delete_all(_HEARTBEAT_COLLECTION_ID,
-                                       _HEART_PARENT_ID, auth=auth)
+                                       _HEART_PARENT_ID, with_deleted=False,
+                                       auth=auth)
                 else:
                     backend.create(_HEARTBEAT_COLLECTION_ID, _HEART_PARENT_ID,
                                    _HEARTBEAT_RECORD, auth=auth)


Test patch:
diff --git a/kinto/core/storage/testing.py b/kinto/core/storage/testing.py
--- a/kinto/core/storage/testing.py
+++ b/kinto/core/storage/testing.py
@@ -142,6 +142,19 @@ def test_ping_logs_error_if_unavailable(self):
 
         self.assertTrue(exc_handler.called)
 
+    def test_ping_leaves_no_tombstone(self):
+        request = DummyRequest()
+        request.headers['Authorization'] = 'Basic bWF0OjI='
+        ping = heartbeat(self.storage)
+        with mock.patch('kinto.core.storage.random.SystemRandom.random', return_value=0.7):
+            ping(request)
+        with mock.patch('kinto.core.storage.random.SystemRandom.random', return_value=0.5):
+            ping(request)
+        records, count = self.storage.get_all(parent_id='__heartbeat__',
+                                              collection_id='__heartbeat__',
+                                              include_deleted=True)
+        self.assertEqual(len(records), 0)
+
     def test_create_adds_the_record_id(self):
         record = self.create_record()
         self.assertIsNotNone(record['id'])
Repository: Kinto/kinto

Issue:
Add a Total-Records header to delete on plural endpoints
For consistency with GET, now that we support pagination on DELETE.


Discussion:

Resolving patch:
diff --git a/kinto/core/resource/__init__.py b/kinto/core/resource/__init__.py
--- a/kinto/core/resource/__init__.py
+++ b/kinto/core/resource/__init__.py
@@ -386,6 +386,9 @@ def collection_delete(self):
         else:
             self._add_timestamp_header(self.request.response)
 
+        headers = self.request.response.headers
+        headers['Total-Records'] = encode_header('%s' % total_records)
+
         action = len(deleted) > 0 and ACTIONS.DELETE or ACTIONS.READ
         return self.postprocess(deleted, action=action, old=records)
 


Test patch:
diff --git a/tests/core/resource/test_model.py b/tests/core/resource/test_model.py
--- a/tests/core/resource/test_model.py
+++ b/tests/core/resource/test_model.py
@@ -8,12 +8,6 @@ def setUp(self):
         super(ModelTest, self).setUp()
         self.record = self.model.create_record({'field': 'value'})
 
-    def test_list_gives_number_of_results_in_headers(self):
-        self.resource.collection_get()
-        headers = self.last_response.headers
-        count = headers['Total-Records']
-        self.assertEquals(int(count), 1)
-
     def test_list_returns_all_records_in_data(self):
         result = self.resource.collection_get()
         records = result['data']
diff --git a/tests/core/resource/test_pagination.py b/tests/core/resource/test_pagination.py
--- a/tests/core/resource/test_pagination.py
+++ b/tests/core/resource/test_pagination.py
@@ -58,6 +58,13 @@ def test_forced_limit_has_precedence_over_provided_limit(self):
             result = self.resource.collection_get()
             self.assertEqual(len(result['data']), 5)
 
+    def test_return_total_records_in_headers(self):
+        self.resource.request.GET = {'_limit': '5'}
+        self.resource.collection_get()
+        headers = self.last_response.headers
+        count = headers['Total-Records']
+        self.assertEquals(int(count), 20)
+
     def test_return_next_page_url_is_given_in_headers(self):
         self.resource.request.GET = {'_limit': '10'}
         self.resource.collection_get()
@@ -216,6 +223,13 @@ def test_paginated_delete(self):
         results2_ids = [r['id'] for r in results2['data']]
         self.assertEqual(expected_ids, results1_ids + results2_ids)
 
+    def test_return_total_records_in_headers_matching_deletable(self):
+        self.resource.request.GET = {'_limit': '5'}
+        self.resource.collection_delete()
+        headers = self.last_response.headers
+        count = headers['Total-Records']
+        self.assertEquals(int(count), 20)
+
 
 class BuildPaginationTokenTest(BaseTest):
     def setUp(self):