internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Kinto/kinto

Issue:
500 instead of 503 when storage backend fails with default bucket
Currently we do not catch a potential backend error when creating the implicit objects. And we use a reference to `bucket` and `collection` in the `finally` close...

```
  File "/home/travis/build/Kinto/kinto/kinto/views/buckets.py", line 155, in default_bucket
    create_collection(request, bucket_id)
  File "/home/travis/build/Kinto/kinto/kinto/views/buckets.py", line 123, in create_collection
    already_created[collection_uri] = collection
UnboundLocalError: local variable 'collection' referenced before assignment;
```



Discussion:

Resolving patch:
diff --git a/kinto/views/buckets.py b/kinto/views/buckets.py
--- a/kinto/views/buckets.py
+++ b/kinto/views/buckets.py
@@ -83,8 +83,7 @@ def create_bucket(request, bucket_id):
         bucket = resource.model.create_record({'id': bucket_id})
     except storage_exceptions.UnicityError as e:
         bucket = e.record
-    finally:
-        already_created[bucket_id] = bucket
+    already_created[bucket_id] = bucket
 
 
 def create_collection(request, bucket_id):
@@ -120,8 +119,7 @@ def create_collection(request, bucket_id):
         collection = resource.model.create_record({'id': collection_id})
     except storage_exceptions.UnicityError as e:
         collection = e.record
-    finally:
-        already_created[collection_uri] = collection
+    already_created[collection_uri] = collection
     request.matchdict = backup
 
 


Test patch:
diff --git a/kinto/tests/test_default_bucket.py b/kinto/tests/test_default_bucket.py
--- a/kinto/tests/test_default_bucket.py
+++ b/kinto/tests/test_default_bucket.py
@@ -5,6 +5,7 @@
 from pyramid.httpexceptions import HTTPBadRequest
 
 from cliquet.errors import ERRORS, http_error
+from cliquet.storage import exceptions as storage_exceptions
 from cliquet.tests.support import FormattedErrorMixin
 from cliquet.utils import hmac_digest
 
@@ -196,3 +197,22 @@ def get_app_settings(self, extras=None):
 
     def test_implicit_creation_is_rejected(self):
         self.app.get('/buckets/default', headers=self.headers, status=405)
+
+
+class BackendErrorTest(BaseWebTest, unittest.TestCase):
+    def setUp(self):
+        super(BackendErrorTest, self).setUp()
+        self.patcher = mock.patch.object(
+            self.storage, 'create',
+            side_effect=storage_exceptions.BackendError())
+        self.addCleanup(self.patcher.stop)
+
+    def test_implicit_bucket_creation_raises_503_if_backend_fails(self):
+        self.patcher.start()
+        self.app.get('/buckets/default', headers=self.headers, status=503)
+
+    def test_implicit_collection_creation_raises_503_if_backend_fails(self):
+        self.app.get('/buckets/default', headers=self.headers)
+        self.patcher.start()
+        self.app.get('/buckets/default/collections/articles',
+                     headers=self.headers, status=503)
Repository: Kinto/kinto

Issue:
cliquet.storage.exceptions.BackendError: ConnectionError: Error 8 connecting to localhos:6379
The config wizard has typos:

https://github.com/Kinto/kinto/blob/master/kinto/config/__init__.py#L39
https://github.com/Kinto/kinto/blob/master/kinto/config/__init__.py#L41
https://github.com/Kinto/kinto/blob/master/kinto/config/__init__.py#L43

I am not sure how, but maybe we could have a way to test that the produced default config works



Discussion:
Also, these lines are not necessary:
https://github.com/Kinto/kinto/blob/master/kinto/config/__init__.py#L48

And this: https://github.com/Kinto/kinto/blob/master/kinto/config/__init__.py#L55-L61
could be simplified to `**values`

by the way, I could not find where init() can be used from the shell. I had to use a __main__ section to run it. Is this somewhere or we should add it?

@tarekziade you should be able to use:

```
from kinto.config import init
init("config/kinto.ini")
```

It's even simpler:

``` bash
.venv/bin/kinto --ini config/kinto.ini init
```

There's a bug though when this file does not exists, for some reason "make serve" does not re-create it, I;ll open another bug

see https://github.com/Kinto/kinto/issues/290


Resolving patch:
diff --git a/kinto/config/__init__.py b/kinto/config/__init__.py
--- a/kinto/config/__init__.py
+++ b/kinto/config/__init__.py
@@ -36,11 +36,11 @@ def init(config_file):
     elif backend == '2':
         # Redis configuration
         values['storage_backend'] = "cliquet.storage.redis"
-        values['storage_url'] = "redis://localhos:6379/1"
+        values['storage_url'] = "redis://localhost:6379/1"
         values['cache_backend'] = "cliquet.cache.redis"
-        values['cache_url'] = "redis://localhos:6379/2"
+        values['cache_url'] = "redis://localhost:6379/2"
         values['permission_backend'] = "cliquet.permission.redis"
-        values['permission_url'] = "redis://localhos:6379/3"
+        values['permission_url'] = "redis://localhost:6379/3"
 
     else:
         # Memory configuration / default backend
@@ -51,11 +51,4 @@ def init(config_file):
         values['permission_backend'] = "cliquet.permission.memory"
         values['permission_url'] = ""
 
-    render_template("kinto.tpl", config_file,
-                    secret=values['secret'],
-                    storage_backend=values['storage_backend'],
-                    storage_url=values['storage_url'],
-                    cache_backend=values['cache_backend'],
-                    cache_url=values['cache_url'],
-                    permission_backend=values['permission_backend'],
-                    permission_url=values['permission_url'])
+    render_template("kinto.tpl", config_file, **values)


Test patch:
Repository: Kinto/kinto

Issue:
kinto init fails if config folder does not exist
```
kinto init 
Which backend to use? (1 - postgresql, 2 - redis, default - memory) 
Traceback (most recent call last):
  File "/var/www/kinto.leplat.re/venv/bin/kinto", line 9, in <module>
    load_entry_point('kinto==1.9.0', 'console_scripts', 'kinto')()
  File "/var/www/kinto.leplat.re/venv/local/lib/python2.7/site-packages/kinto/__main__.py", line 44, in main
    init(config_file)
  File "/var/www/kinto.leplat.re/venv/local/lib/python2.7/site-packages/kinto/config/__init__.py", line 61, in init
    permission_url=values['permission_url'])
  File "/var/www/kinto.leplat.re/venv/local/lib/python2.7/site-packages/kinto/config/__init__.py", line 15, in render_template
    with codecs.open(destination, 'w+', encoding='utf-8') as output:
  File "/var/www/kinto.leplat.re/venv/lib/python2.7/codecs.py", line 881, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: 'config/kinto.ini'
```



Discussion:

Resolving patch:
diff --git a/kinto/config/__init__.py b/kinto/config/__init__.py
--- a/kinto/config/__init__.py
+++ b/kinto/config/__init__.py
@@ -8,6 +8,9 @@
 def render_template(template, destination, **kwargs):
     template = os.path.join(HERE, template)
 
+    folder = os.path.dirname(destination)
+    os.makedirs(folder)
+
     with codecs.open(template, 'r', encoding='utf-8') as f:
         raw_template = f.read()
         rendered = raw_template.format(**kwargs)


Test patch:
Repository: Kinto/kinto

Issue:
Implicit collection creation doesn't validate the ID format
```
$ http GET http://localhost:8888/v1/buckets/default/collections/見/records --auth natim:
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Content-Length, Expires, Alert, Retry-After, Last-Modified, Total-Records, ETag, Pragma, Cache-Control, Backoff, Next-Page
Cache-Control: no-cache
Content-Length: 11
Content-Type: application/json; charset=UTF-8
Date: Thu, 05 Nov 2015 14:19:38 GMT
Etag: "1446733178498"
Last-Modified: Thu, 05 Nov 2015 14:19:38 GMT
Server: waitress
Total-Records: 0

{
    "data": []
}
```



Discussion:

Resolving patch:
diff --git a/kinto/views/buckets.py b/kinto/views/buckets.py
--- a/kinto/views/buckets.py
+++ b/kinto/views/buckets.py
@@ -4,6 +4,7 @@
 from pyramid.view import view_config
 
 from cliquet import resource
+from cliquet.errors import raise_invalid
 from cliquet.utils import build_request, reapply_cors
 from cliquet.storage import exceptions as storage_exceptions
 
@@ -112,6 +113,12 @@ def create_collection(request, bucket_id):
                              id=collection_id,
                              **request.matchdict)
     resource = Collection(request, context)
+    if not resource.model.id_generator.match(collection_id):
+        error_details = {
+            'location': 'path',
+            'description': "Invalid collection_id id"
+        }
+        raise_invalid(request, **error_details)
     try:
         collection = resource.model.create_record({'id': collection_id})
     except storage_exceptions.UnicityError as e:


Test patch:
diff --git a/kinto/tests/test_default_bucket.py b/kinto/tests/test_default_bucket.py
--- a/kinto/tests/test_default_bucket.py
+++ b/kinto/tests/test_default_bucket.py
@@ -159,6 +159,14 @@ def test_parent_collection_is_taken_from_the_one_checked_in_batch(self):
             self.app.post_json('/batch', batch, headers=self.headers)
             self.assertEqual(patched.call_count, 0)
 
+    def test_collection_id_is_validated(self):
+        collection_url = '/buckets/default/collections/__files__/records'
+        self.app.get(collection_url, headers=self.headers, status=400)
+
+    def test_collection_id_does_not_support_unicode(self):
+        collection_url = '/buckets/default/collections/%E8%A6%8B/records'
+        self.app.get(collection_url, headers=self.headers, status=400)
+
     def test_405_is_a_valid_formatted_error(self):
         response = self.app.post(self.collection_url,
                                  headers=self.headers, status=405)
Repository: Kinto/kinto

Issue:
kinto init should install postgresql dependencies



Discussion:

Resolving patch:
diff --git a/kinto/__main__.py b/kinto/__main__.py
--- a/kinto/__main__.py
+++ b/kinto/__main__.py
@@ -2,6 +2,8 @@
 import argparse
 import os
 import sys
+
+import pip
 from six.moves import input
 from cliquet.scripts import cliquet
 from pyramid.scripts import pserve
@@ -65,6 +67,13 @@ def main(args=None):
 
         init(config_file, backend)
 
+        # Install postgresql libraries if necessary
+        if backend == "postgresql":
+            try:
+                import psycopg2  # NOQA
+            except ImportError:
+                pip.main(['install', "cliquet[postgresql]"])
+
     elif args['which'] == 'migrate':
         env = bootstrap(config_file)
         cliquet.init_schema(env)


Test patch:
Repository: Kinto/kinto

Issue:
Do not require cliquet master branch in dev
As discussed with @Natim @almet 



Discussion:
Even more clever than I planned to do :+1: 

PPPerfect!


Resolving patch:
diff --git a/kinto/config/__init__.py b/kinto/config/__init__.py
--- a/kinto/config/__init__.py
+++ b/kinto/config/__init__.py
@@ -1,6 +1,8 @@
 import os
-import binascii
 import codecs
+
+from cliquet import utils as cliquet_utils
+
 from kinto import logger
 
 HERE = os.path.abspath(os.path.dirname(__file__))
@@ -24,7 +26,8 @@ def render_template(template, destination, **kwargs):
 
 def init(config_file, backend):
     values = {}
-    values['secret'] = binascii.b2a_hex(os.urandom(32))
+
+    values['secret'] = cliquet_utils.random_bytes_hex(32)
 
     values['storage_backend'] = "cliquet.storage.%s" % backend
     values['cache_backend'] = "cliquet.cache.%s" % backend


Test patch:
diff --git a/kinto/tests/test_config.py b/kinto/tests/test_config.py
--- a/kinto/tests/test_config.py
+++ b/kinto/tests/test_config.py
@@ -4,6 +4,8 @@
 import tempfile
 import unittest
 
+import six
+
 from kinto.config import render_template, init
 
 
@@ -45,6 +47,12 @@ def test_create_destination_directory(self):
 
         self.assertTrue(os.path.exists(dest))
 
+    @mock.patch('kinto.config.render_template')
+    def test_hmac_secret_is_text(self, mocked_render_template):
+        init('kinto.ini', 'postgresql')
+        args, kwargs = list(mocked_render_template.call_args)
+        self.assertEquals(type(kwargs['secret']), six.text_type)
+
     @mock.patch('kinto.config.render_template')
     def test_init_postgresql_values(self, mocked_render_template):
         init('kinto.ini', 'postgresql')
Repository: Kinto/kinto

Issue:
kinto start takes too much time.
It can sometimes take more than 2 seconds, as reported by @n1k0 on JS clients integration tests.

I investigated a bit and found out, that on my machine, loading the entrypoint takes more than 1 second already:

```
$ time python -c "from pkg_resources import load_entry_point; load_entry_point('kinto', 'console_scripts', 'kinto')"
python -c   0,96s user 0,16s system 99% cpu 1,132 total
```

In comparison, `pserve` takes 200msec: 

```
$ time python -c "from pkg_resources import load_entry_point; load_entry_point('pyramid', 'console_scripts', 'pcreate')"
python -c   0,18s user 0,09s system 98% cpu 0,272 total
```

I realized that moving `import requests` from `cliquet.initialization` imports [PR](https://github.com/mozilla-services/cliquet/pull/674), and remove `import pip` from `kinto.__main__` I could reduce by half:

```
$ time python -c "from pkg_resources import load_entry_point; load_entry_point('kinto', 'console_scripts', 'kinto')"
python -c   0,36s user 0,18s system 98% cpu 0,543 total
```

I knew this was not going to speed up the `kinto start` command too much. I tracked down and noticed the `__main__:main` was executed twice because of `--reload` argument.



Discussion:
:+1: (please) 


Resolving patch:
diff --git a/kinto/__main__.py b/kinto/__main__.py
--- a/kinto/__main__.py
+++ b/kinto/__main__.py
@@ -45,6 +45,11 @@ def main(args=None):
     parser_migrate.set_defaults(which='migrate')
 
     parser_start = subparsers.add_parser('start')
+    parser_start.add_argument('--reload',
+                              action='store_true',
+                              help='Restart when code or config changes',
+                              required=False,
+                              default=False)
     parser_start.set_defaults(which='start')
 
     args = vars(parser.parse_args())
@@ -83,7 +88,9 @@ def main(args=None):
         cliquet.init_schema(env)
 
     elif args['which'] == 'start':
-        pserve_argv = ['pserve', config_file, '--reload']
+        pserve_argv = ['pserve', config_file]
+        if args['reload']:
+            pserve_argv.append('--reload')
         pserve.main(pserve_argv)
 
 


Test patch:
Repository: Kinto/kinto

Issue:
flush_endpoint capability added if flush_enabled (fixes #496)



Discussion:

Resolving patch:
diff --git a/kinto/__init__.py b/kinto/__init__.py
--- a/kinto/__init__.py
+++ b/kinto/__init__.py
@@ -67,7 +67,16 @@ def main(global_config, config=None, **settings):
     # Scan Kinto views.
     kwargs = {}
     flush_enabled = asbool(settings.get('flush_endpoint_enabled'))
-    if not flush_enabled:
+
+    if flush_enabled:
+        config.add_api_capability(
+            "flush_endpoint",
+            description="The __flush__ endpoint can be used to remove all "
+                        "data from all backends.",
+            url="http://kinto.readthedocs.org/en/latest/configuration/"
+                "settings.html#activating-the-flush-endpoint"
+        )
+    else:
         kwargs['ignore'] = 'kinto.views.flush'
     config.scan("kinto.views", **kwargs)
 


Test patch:
diff --git a/kinto/tests/test_views_hello.py b/kinto/tests/test_views_hello.py
--- a/kinto/tests/test_views_hello.py
+++ b/kinto/tests/test_views_hello.py
@@ -49,3 +49,26 @@ def test_capability_is_exposed_if_setting_is_not_set(self):
         resp = app.get('/')
         capabilities = resp.json['capabilities']
         self.assertNotIn('schema', capabilities)
+
+    def test_flush_capability_if_enabled(self):
+        settings = self.get_app_settings()
+        settings['flush_endpoint_enabled'] = True
+        app = self._get_test_app(settings=settings)
+        resp = app.get('/')
+        capabilities = resp.json['capabilities']
+        self.assertIn('flush_endpoint', capabilities)
+        expected = {
+            "description": "The __flush__ endpoint can be used to remove "
+                           "all data from all backends.",
+            "url": "http://kinto.readthedocs.org/en/latest/configuration/"
+                   "settings.html#activating-the-flush-endpoint"
+        }
+        self.assertEqual(expected, capabilities['flush_endpoint'])
+
+    def test_flush_capability_if_not_enabled(self):
+        settings = self.get_app_settings()
+        settings['flush_endpoint_enabled'] = False
+        app = self._get_test_app(settings=settings)
+        resp = app.get('/')
+        capabilities = resp.json['capabilities']
+        self.assertNotIn('flush_endpoint', capabilities)
Repository: Kinto/kinto

Issue:
Explain in the doc how to disable default_buckets and bucket creation from authenticated users.
From a discussion with @chrismbeckett on IRC:

1) Anyone can register for my application and get an auth token that allows them to login to Kinto. With personal buckets, someone could figure out how to start storing anything they want. I am concerned I may need to _turn off_ personal buckets since they may be a security risk.

2) Same for buckets. I am probably going to lock down creating buckets so it is done by my auth server when an account is created.



Discussion:

Resolving patch:
diff --git a/kinto/__init__.py b/kinto/__init__.py
--- a/kinto/__init__.py
+++ b/kinto/__init__.py
@@ -45,11 +45,6 @@ def main(global_config, config=None, **settings):
 
     settings = config.get_settings()
 
-    # In Kinto API 1.x, a default bucket is available.
-    # Force its inclusion if not specified in settings.
-    if 'kinto.plugins.default_bucket' not in settings['includes']:
-        config.include('kinto.plugins.default_bucket')
-
     # Retro-compatibility with first Kinto clients.
     config.registry.public_settings.add('cliquet.batch_max_requests')
 


Test patch:
diff --git a/kinto/tests/support.py b/kinto/tests/support.py
--- a/kinto/tests/support.py
+++ b/kinto/tests/support.py
@@ -48,6 +48,7 @@ def get_app_settings(self, additional_settings=None):
         settings['storage_backend'] = 'cliquet.storage.memory'
         settings['permission_backend'] = 'cliquet.permission.memory'
         settings['userid_hmac_secret'] = "this is not a secret"
+        settings['includes'] = "kinto.plugins.default_bucket"
 
         if additional_settings is not None:
             settings.update(additional_settings)
diff --git a/kinto/tests/test_views_disable_default.py b/kinto/tests/test_views_disable_default.py
new file mode 100644
--- /dev/null
+++ b/kinto/tests/test_views_disable_default.py
@@ -0,0 +1,16 @@
+from .support import (BaseWebTest, unittest)
+
+
+class DisableDefaultBucketViewTest(BaseWebTest, unittest.TestCase):
+
+    test_url = '/buckets/default'
+
+    def test_returns_403_if_excluded_in_configuration(self):
+        extra = {'includes': ''}
+        app = self._get_test_app(settings=extra)
+        app.get(self.test_url, headers=self.headers, status=403)
+
+    def test_returns_200_if_included_in_configuration(self):
+        extra = {'includes': 'kinto.plugins.default_bucket'}
+        app = self._get_test_app(settings=extra)
+        app.get(self.test_url, headers=self.headers, status=200)
Repository: Kinto/kinto

Issue:
POST with If-None-Match: * and provided id in body always return 412
Detected using kinto-client v0.4.0 https://github.com/Kinto/kinto-client/blob/v0.4.0/src/requests.js#L188-L205

See https://github.com/mozilla-services/cliquet/issues/673



Discussion:
Note: kinto-client v0.4.1 has been released with an appropriate client-side fix.
- npm package: https://www.npmjs.com/package/kinto-client
- release notes: https://github.com/Kinto/kinto-client/releases/tag/v0.4.1

So I believe we should close this issue!

Well, kinto-client now performs PUT requests only for creations because of this limitation, though there are stil legit use cases for allowing POST methinks. This should be reopened, but I'm not 100% sure so pinging @leplatrem here. 

The fix was merged in cliquet https://github.com/mozilla-services/cliquet/pull/675


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -18,16 +18,16 @@ def read_file(filename):
 
 REQUIREMENTS = [
     'waitress',
-    'cliquet>=3,<4',
+    'cliquet>=3.1,<4',
     'jsonschema',
 ]
 
 POSTGRESQL_REQUIREMENTS = REQUIREMENTS + [
-    'cliquet[postgresql]>=3,<4'
+    'cliquet[postgresql]>=3.1,<4'
 ]
 
 MONITORING_REQUIREMENTS = REQUIREMENTS + [
-    'cliquet[monitoring]>=3,<4'
+    'cliquet[monitoring]>=3.1,<4'
 ]
 
 FXA_REQUIREMENTS = REQUIREMENTS + [


Test patch:
Repository: Kinto/kinto

Issue:
Metadata on Groups
Similarily to how you can store extra properties (metadata) on a collection, it would be useful to be able to do this with groups.

In my applications, almost everything is dynamic. Users can create groups on the fly, rename them, etc., so I tend to use generated ID's for everything. It would be nice to be able to set a title and description on groups for UI presentation.

Right now I have to create a collection for storing group metadata separately from the actual group.



Discussion:
See an example of implementation (for buckets): https://github.com/Kinto/kinto/pull/462/files


Resolving patch:
diff --git a/kinto/__init__.py b/kinto/__init__.py
--- a/kinto/__init__.py
+++ b/kinto/__init__.py
@@ -12,7 +12,7 @@
 __version__ = pkg_resources.get_distribution(__package__).version
 
 # Implemented HTTP API Version
-HTTP_API_VERSION = '1.4'
+HTTP_API_VERSION = '1.5'
 
 # Main kinto logger
 logger = logging.getLogger(__name__)
diff --git a/kinto/views/groups.py b/kinto/views/groups.py
--- a/kinto/views/groups.py
+++ b/kinto/views/groups.py
@@ -11,6 +11,9 @@ class GroupSchema(resource.ResourceSchema):
     members = colander.SchemaNode(colander.Sequence(),
                                   colander.SchemaNode(colander.String()))
 
+    class Options:
+        preserve_unknown = True
+
 
 @resource.register(name='group',
                    collection_path='/buckets/{{bucket_id}}/groups',


Test patch:
diff --git a/kinto/tests/test_views_groups.py b/kinto/tests/test_views_groups.py
--- a/kinto/tests/test_views_groups.py
+++ b/kinto/tests/test_views_groups.py
@@ -39,6 +39,17 @@ def test_groups_name_should_be_simple(self):
                           headers=self.headers,
                           status=400)
 
+    def test_groups_can_have_arbitrary_attributes(self):
+        group = MINIMALIST_GROUP.copy()
+        mailinglist = "kinto@mozilla.com"
+        group['data']['mailinglist'] = mailinglist
+        resp = self.app.put_json('/buckets/beers/groups/moderator',
+                                 group,
+                                 headers=self.headers)
+        data = resp.json['data']
+        self.assertIn('mailinglist', data)
+        self.assertEqual(data['mailinglist'], mailinglist)
+
     def test_groups_should_reject_unaccepted_request_content_type(self):
         headers = self.headers.copy()
         headers['Content-Type'] = 'text/plain'
Repository: Kinto/kinto

Issue:
Remove structlog initialization hack when new version is released
Once hynek/structlog#72 is released, we can revert #633.



Discussion:

Resolving patch:
diff --git a/kinto/core/initialization.py b/kinto/core/initialization.py
--- a/kinto/core/initialization.py
+++ b/kinto/core/initialization.py
@@ -331,15 +331,6 @@ def setup_logging(config):
             renderer,
         ])
 
-    # Hack to work around https://github.com/hynek/structlog/issues/71.
-    #
-    # This clears a magic field in the logger so that it will regenerate
-    # the right kind of logger on its next use.
-    #
-    # Otherwise, the logging_factory field of the logger will be
-    # outdated and you may get strange behavior.
-    logger._logger = None
-
     def on_new_request(event):
         request = event.request
         # Save the time the request was received by the server.
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -28,7 +28,7 @@ def read_file(filename):
     'redis',  # Default backend
     'requests',
     'six',
-    'structlog',
+    'structlog >= 16.1.0',
     'enum34',
     'waitress',
 ]


Test patch: