internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
[CircleCI] - Test fails - SSL connection has been closed unexpectedly
Sometimes CircleCI test fails, the cause is that SSL connection has been closed unexpectedly.



Discussion:

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -284,6 +284,8 @@ def update_config():
     except sqlalchemy.exc.InternalError:
         # The database is not initialised.  Travis hits this
         pass
-    # if an extension or our code does not finish
-    # transaction properly db cli commands can fail
+
+    # Close current session and open database connections to ensure a clean
+    # clean environment even if an error occurs later on
     model.Session.remove()
+    model.Session.bind.dispose()
diff --git a/ckan/lib/jobs.py b/ckan/lib/jobs.py
--- a/ckan/lib/jobs.py
+++ b/ckan/lib/jobs.py
@@ -29,6 +29,7 @@
 from ckan.lib.redis import connect_to_redis
 from ckan.common import config
 from ckan.config.environment import load_environment
+from ckan.model import meta
 
 
 log = logging.getLogger(__name__)
@@ -256,3 +257,18 @@ def main_work_horse(self, job, queue):
         # after forking.
         load_environment(config[u'global_conf'], config)
         return super(Worker, self).main_work_horse(job, queue)
+
+    def perform_job(self, *args, **kwargs):
+        result = super(Worker, self).perform_job(*args, **kwargs)
+        # rq.Worker.main_work_horse does a hard exit via os._exit directly
+        # after its call to perform_job returns. Hence here is the correct
+        # location to clean up.
+        try:
+            meta.Session.remove()
+        except Exception:
+            log.exception(u'Error while closing database session')
+        try:
+            meta.engine.dispose()
+        except Exception:
+            log.exception(u'Error while disposing database engine')
+        return result


Test patch:
Repository: ckan/ckan

Issue:
Ckan creates ErrorHandler route before calling plugins before_map method, making it impossible to customize error handling.
Ckan documentation states [here](http://docs.ckan.org/en/latest/extensions/plugin-interfaces.html?highlight=before_map#ckan.plugins.interfaces.IRoutes.before_map) that IRoutes.before_map gets called before the routes map is generated. However, in routing.make_map  code, the error handler route is generated before calling the IRoute methods :

    # The ErrorController route (handles 404/500 error pages); it should
    # likely stay at the top, ensuring it can always be resolved.
    map.connect('/error/{action}', controller='error', ckan_core=True)
    map.connect('/error/{action}/{id}', controller='error', ckan_core=True)

    map.connect('*url', controller='home', action='cors_options',
                conditions=OPTIONS, ckan_core=True)

    # CUSTOM ROUTES HERE
    for plugin in p.PluginImplementations(p.IRoutes):
        map = plugin.before_map(map)

This prevents connecting error actions to a customized error controller.


Discussion:
That sounds reasonable. Do you want to submit a PR that moves the routes below the extension point?
Sure, I will make the changes and I will do the PR.

Resolving patch:
diff --git a/ckan/config/routing.py b/ckan/config/routing.py
--- a/ckan/config/routing.py
+++ b/ckan/config/routing.py
@@ -93,6 +93,10 @@ def make_map():
     map.minimization = False
     map.explicit = True
 
+    # CUSTOM ROUTES HERE
+    for plugin in p.PluginImplementations(p.IRoutes):
+        map = plugin.before_map(map)
+
     # The ErrorController route (handles 404/500 error pages); it should
     # likely stay at the top, ensuring it can always be resolved.
     map.connect('/error/{action}', controller='error', ckan_core=True)
@@ -101,10 +105,6 @@ def make_map():
     map.connect('*url', controller='home', action='cors_options',
                 conditions=OPTIONS, ckan_core=True)
 
-    # CUSTOM ROUTES HERE
-    for plugin in p.PluginImplementations(p.IRoutes):
-        map = plugin.before_map(map)
-
     # Mark all routes added from extensions on the `before_map` extension point
     # as non-core
     for route in map.matchlist:


Test patch:
Repository: ckan/ckan

Issue:
Initializing test db is not idempotent
Running nosetests deletes the `migrate_version` table, which means that subsequent runs of `paster db init -c test-core.ini` [don't work](https://gist.github.com/davidread/af650b9227e5ed54d891dbeeb0e2c254). This is an issue when you are writing provision scripts, because the easiest thing is for these commands to be idempotent so they can be run repeatedly without harmful effect or error. So it is helpful if our tests don't delete the `migrate_version` table.

How to reproduce:
```
dropdb ckan_test
createdb ckan_test -O ckan_default
paster --plugin=ckan db init -c test-core.ini
nosetests --ckan --with-pylons=test-core.ini ckan/tests/controllers/test_admin.py:TestTrashView.test_trash_purge_deleted_datasets  # or any test
# You can see the table is gone with this:
sudo -u postgres psql ckan_test -c 'select * from migrate_version;'
# Now run init again, as if you are running the provision script again
paster --plugin=ckan db init -c test-core.ini
# That didn't work - see the error here: https://gist.github.com/davidread/af650b9227e5ed54d891dbeeb0e2c254
```

Because of this issue, my ansible play is:
```
  shell: psql ckan_test -c 'select name from package limit 1' | grep row && echo TestDatabaseAlreadyInitialized || /usr/lib/ckan/bin/paster --plugin=ckan db init -c /usr/lib/ckan/src/ckan/test-core.ini
  sudo: true
  sudo_user: postgres
  register: cmd_result
  changed_when: cmd_result.stdout.find('TestDatabaseAlreadyInitialized') == -1
```
whereas with this fix it can simply be:
```
  command: /usr/lib/ckan/bin/paster --plugin=ckan db init -c /usr/lib/ckan/src/ckan/test-core.ini
```


Discussion:

Resolving patch:
diff --git a/ckan/model/__init__.py b/ckan/model/__init__.py
--- a/ckan/model/__init__.py
+++ b/ckan/model/__init__.py
@@ -255,6 +255,8 @@ def delete_all(self):
         else:
             tables = reversed(self.metadata.sorted_tables)
         for table in tables:
+            if table.name == 'migrate_version':
+                continue
             connection.execute('delete from "%s"' % table.name)
         self.session.commit()
         log.info('Database table data deleted')


Test patch:
Repository: ckan/ckan

Issue:
Reset link not including site_url
### CKAN Version if known (or site URL)

master
### Please describe the expected behaviour

The reset email should include the complete link to reset a users password. e.g  http://mydomain.com/user/reset/<id>?key=<somekey>
### Please describe the actual behaviour

The email only includes the subpath. e.g  /user/reset/<id>?key=<somekey>
### What steps can be taken to reproduce the issue?

Try to send a password recovery email.

I think the fix lies in https://github.com/ckan/ckan/blob/master/ckan/lib/mailer.py#L147 Shouldn't it use `ckan.site_url` instead of just `site_url`



Discussion:
The code should probably just pass `qualified=True` to `url_for` instead of manually adding the host.


Resolving patch:
diff --git a/ckan/lib/mailer.py b/ckan/lib/mailer.py
--- a/ckan/lib/mailer.py
+++ b/ckan/lib/mailer.py
@@ -144,11 +144,11 @@ def get_invite_body(user, group_dict=None, role=None):
 
 
 def get_reset_link(user):
-    return urljoin(config.get('site_url'),
-                   h.url_for(controller='user',
-                             action='perform_reset',
-                             id=user.id,
-                             key=user.reset_key))
+    return h.url_for(controller='user',
+                     action='perform_reset',
+                     id=user.id,
+                     key=user.reset_key,
+                     qualified=True)
 
 
 def send_reset_link(user):


Test patch:
Repository: ckan/ckan

Issue:
Minimum setuptools version
CKAN requires a recent version of setuptools ever since some of CKAN's dependencies were upgraded. However there's nothing in the install instructions to sort this out. The default setuptools for our recommended distro, Ubuntu Trusty 14.04, is 2.2, which is too old.

Specifically:
* repoze.who needs setuptools 11.3 or later https://github.com/repoze/repoze.who/issues/26
* html5lib needs setuptools 18.5 or later https://github.com/ckan/ckan/issues/3232



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -6,15 +6,31 @@
     del os.link
 
 try:
-    from setuptools import setup, find_packages
+    from setuptools import (setup, find_packages,
+                            __version__ as setuptools_version)
 except ImportError:
     from ez_setup import use_setuptools
     use_setuptools()
-    from setuptools import setup, find_packages
+    from setuptools import (setup, find_packages,
+                            __version__ as setuptools_version)
 
 from ckan import (__version__, __description__, __long_description__,
                   __license__)
 
+MIN_SETUPTOOLS_VERSION = 20.4
+assert setuptools_version >= str(MIN_SETUPTOOLS_VERSION) and \
+    int(setuptools_version.split('.')[0]) >= int(MIN_SETUPTOOLS_VERSION),\
+    ('setuptools version error'
+     '\nYou need a newer version of setuptools.\n'
+     'You have {current}, you need at least {minimum}'
+     '\nInstall the recommended version:\n'
+     '    pip install -r requirement-setuptools.txt\n'
+     'and then try again to install ckan into your python environment.'.format(
+         current=setuptools_version,
+         minimum=MIN_SETUPTOOLS_VERSION
+         ))
+
+
 entry_points = {
     'nose.plugins.0.10': [
         'main = ckan.ckan_nose_plugin:CkanNose',


Test patch:
Repository: ckan/ckan

Issue:
Ubuntu 16.04 Xenial compatibility
16.04 has been out a while, so it would be good to tweak the ckan install instructions to be compatible with it.


Discussion:

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -52,7 +52,7 @@
 .. |storage_path| replace:: |storage_parent_dir|/default
 .. |reload_apache| replace:: sudo service apache2 reload
 .. |restart_apache| replace:: sudo service apache2 restart
-.. |restart_solr| replace:: sudo service jetty restart
+.. |restart_solr| replace:: sudo service jetty8 restart
 .. |solr| replace:: Solr
 .. |restructuredtext| replace:: reStructuredText
 .. |nginx| replace:: Nginx


Test patch:
Repository: ckan/ckan

Issue:
Groups should support multilingual schema fields
Group templates are missing translation support for multilingual fields. E.g. templates should call get_translated for displayed group fields. I'll provide suggestions with a PR in a minute.


Discussion:

Resolving patch:
diff --git a/ckan/controllers/group.py b/ckan/controllers/group.py
--- a/ckan/controllers/group.py
+++ b/ckan/controllers/group.py
@@ -186,6 +186,7 @@ def index(self):
             'type': group_type or 'group',
             'limit': items_per_page,
             'offset': items_per_page * (page - 1),
+            'include_extras': True
         }
         page_results = self._action('group_list')(context,
                                                   data_dict_page_results)


Test patch:
Repository: ckan/ckan

Issue:
Exception when providing a wrong sort param on group/org page
Providing a sort param like `sort=title xx desc` or `sort=title xx` causes an exception due to an uncaught `ValidationError`


Discussion:

Resolving patch:
diff --git a/ckan/controllers/group.py b/ckan/controllers/group.py
--- a/ckan/controllers/group.py
+++ b/ckan/controllers/group.py
@@ -170,14 +170,24 @@ def index(self):
             context['user_id'] = c.userobj.id
             context['user_is_admin'] = c.userobj.sysadmin
 
-        data_dict_global_results = {
-            'all_fields': False,
-            'q': q,
-            'sort': sort_by,
-            'type': group_type or 'group',
-        }
-        global_results = self._action('group_list')(context,
-                                                    data_dict_global_results)
+        try:
+            data_dict_global_results = {
+                'all_fields': False,
+                'q': q,
+                'sort': sort_by,
+                'type': group_type or 'group',
+            }
+            global_results = self._action('group_list')(
+                context, data_dict_global_results)
+        except ValidationError as e:
+            if e.error_dict and e.error_dict.get('message'):
+                msg = e.error_dict['message']
+            else:
+                msg = str(e)
+            h.flash_error(msg)
+            c.page = h.Page([], 0)
+            return render(self._index_template(group_type),
+                          extra_vars={'group_type': group_type})
 
         data_dict_page_results = {
             'all_fields': True,


Test patch:
diff --git a/ckan/tests/controllers/test_group.py b/ckan/tests/controllers/test_group.py
--- a/ckan/tests/controllers/test_group.py
+++ b/ckan/tests/controllers/test_group.py
@@ -53,6 +53,20 @@ def test_page_thru_list_of_groups_preserves_sort_order(self):
         assert groups[-1]['title'] not in response2
         assert groups[0]['title'] in response2
 
+    def test_invalid_sort_param_does_not_crash(self):
+        app = self._get_test_app()
+        group_url = url_for(controller='group',
+                            action='index',
+                            sort='title desc nope')
+
+        app.get(url=group_url)
+
+        group_url = url_for(controller='group',
+                            action='index',
+                            sort='title nope desc nope')
+
+        app.get(url=group_url)
+
 
 def _get_group_new_page(app):
     user = factories.User()
Repository: ckan/ckan

Issue:
Deprecate notify_after_commit
Because it doesn't work.

notify_after_commit (from interface IDomainObjectModification) never gets called because session._object_cache is blank for this sort of SessionExtension event, so CKAN don't know what object has been committed.

I see there are no tests for it, and no-one (on Github) has used it.


Discussion:

Resolving patch:
diff --git a/ckan/model/modification.py b/ckan/model/modification.py
--- a/ckan/model/modification.py
+++ b/ckan/model/modification.py
@@ -24,22 +24,11 @@ class DomainObjectModificationExtension(plugins.SingletonPlugin):
 
     plugins.implements(plugins.ISession, inherit=True)
 
-    def notify_observers(self, func):
-        """
-        Call func(observer) for all registered observers.
-
-        :param func: Any callable, which will be called for each observer
-        :returns: EXT_CONTINUE if no errors encountered, otherwise EXT_STOP
-        """
-        for observer in plugins.PluginImplementations(
-                plugins.IDomainObjectModification):
-            func(observer)
-
     def before_commit(self, session):
         self.notify_observers(session, self.notify)
 
     def after_commit(self, session):
-        self.notify_observers(session, self.notify_after_commit)
+        pass
 
     def notify_observers(self, session, method):
         session.flush()
@@ -95,18 +84,3 @@ def notify(self, entity, operation):
                 log.exception(ex)
                 # Don't reraise other exceptions since they are generally of
                 # secondary importance so shouldn't disrupt the commit.
-
-    def notify_after_commit(self, entity, operation):
-        for observer in plugins.PluginImplementations(
-                plugins.IDomainObjectModification):
-            try:
-                observer.notify_after_commit(entity, operation)
-            except SearchIndexError, search_error:
-                log.exception(search_error)
-                # Reraise, since it's pretty crucial to ckan if it can't index
-                # a dataset
-                raise
-            except Exception, ex:
-                log.exception(ex)
-                # Don't reraise other exceptions since they are generally of
-                # secondary importance so shouldn't disrupt the commit.
diff --git a/ckan/plugins/interfaces.py b/ckan/plugins/interfaces.py
--- a/ckan/plugins/interfaces.py
+++ b/ckan/plugins/interfaces.py
@@ -239,7 +239,10 @@ def notify(self, entity, operation):
 
     def notify_after_commit(self, entity, operation):
         u'''
-        Send a notification after entity modification.
+        ** DEPRECATED **
+
+        Supposed to send a notification after entity modification, but it
+        doesn't work.
 
         :param entity: instance of module.Package.
         :param operation: 'new', 'changed' or 'deleted'.


Test patch:
Repository: ckan/ckan

Issue:
In API docs "package_create" lists "owner_org" as optional
In API docs, the action `package_create` lists the parameter `owner_org` as optional which is not correct.

The parameter `owner_org` can be optional only if `ckan.auth.create_unowned_dataset` is set to `true` in the config.

I suggest to remove the "(optional)" text and add to the description of the parameter that it can be optional only if `ckan.auth.create_unowned_dataset` is set to `true` in the config.


Discussion:

Resolving patch:
diff --git a/ckan/logic/action/create.py b/ckan/logic/action/create.py
--- a/ckan/logic/action/create.py
+++ b/ckan/logic/action/create.py
@@ -120,7 +120,8 @@ def package_create(context, data_dict):
     :param owner_org: the id of the dataset's owning organization, see
         :py:func:`~ckan.logic.action.get.organization_list` or
         :py:func:`~ckan.logic.action.get.organization_list_for_user` for
-        available values (optional)
+        available values. This parameter can be made optional if the config
+        option :ref:`ckan.auth.create_unowned_dataset` is set to ``True``.
     :type owner_org: string
 
     :returns: the newly created dataset (unless 'return_id_only' is set to True


Test patch:
Repository: ckan/ckan

Issue:
Default logo image not properly saved
### CKAN Version if known (or site URL)
2.7 and 2.8.0a

By default, CKAN uses a default logo image. This can be changed through the UI in the sysadmin config page or in configuration settings.

If you go to the sysadmin config page, and if you don't change the logo image and just click Update, the logo image will break. The reason is because its been given wrong path to the image.

### What steps can be taken to reproduce the issue? 
Go to sysadmin config page. If you previously changed the default logo image, click the Reset button to bring the default settings. After that click on Update and the logo image will break.

I isolated the issue and will provide a fix.


Discussion:

Resolving patch:
diff --git a/ckan/logic/action/update.py b/ckan/logic/action/update.py
--- a/ckan/logic/action/update.py
+++ b/ckan/logic/action/update.py
@@ -7,6 +7,7 @@
 import time
 import json
 import mimetypes
+import os
 
 from ckan.common import config
 import paste.deploy.converters as converters
@@ -1273,8 +1274,11 @@ def config_option_update(context, data_dict):
     for key, value in data.iteritems():
 
         # Set full Logo url
-        if key =='ckan.site_logo' and value and not value.startswith('http'):
-            value = h.url_for_static('uploads/admin/{0}'.format(value))
+        if key == 'ckan.site_logo' and value and not value.startswith('http')\
+                and not value.startswith('/'):
+            image_path = 'uploads/admin/'
+
+            value = h.url_for_static('{0}{1}'.format(image_path, value))
 
         # Save value in database
         model.set_system_info(key, value)


Test patch:
Repository: ckan/ckan

Issue:
radio helper missing
### CKAN Version if known (or site URL)
CKAN 2.6.2

### Please describe the expected behaviour
The radio button helper is removed from helpers.py and it's still used in revisions_table.html

### Please describe the actual behaviour
As I am informed history revision will be removed, but we should be taken into consideration that some extensions may still use it. The radio button helper should be displayed and not produce an error.

### What steps can be taken to reproduce the issue?



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -2342,6 +2342,15 @@ def mail_to(email_address, name):
     return html
 
 
+@core_helper
+def radio(selected, id, checked):
+    if checked:
+        return literal((u'<input checked="checked" id="%s_%s" name="%s" \
+            value="%s" type="radio">') % (selected, id, selected, id))
+    return literal(('<input id="%s_%s" name="%s" \
+        value="%s" type="radio">') % (selected, id, selected, id))
+
+
 core_helper(flash, name='flash')
 core_helper(localised_number)
 core_helper(localised_SI_number)
@@ -2362,6 +2371,7 @@ def mail_to(email_address, name):
 core_helper(urlencode)
 core_helper(clean_html, name='clean_html')
 
+
 def load_plugin_helpers():
     """
     (Re)loads the list of helpers provided by plugins.


Test patch:
Repository: ckan/ckan

Issue:
Frontent build command does not work on master
Running `paster front-end-build` on master returns:

```
Traceback (most recent call last):
  File "/home/adria/dev/pyenvs/ckan/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/home/adria/dev/pyenvs/ckan/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/home/adria/dev/pyenvs/ckan/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/home/adria/dev/pyenvs/ckan/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/home/adria/dev/pyenvs/ckan/src/ckan/ckan/lib/cli.py", line 2120, in command
    cmd.command()
  File "/home/adria/dev/pyenvs/ckan/src/ckan/ckan/lib/cli.py", line 2025, in command
    self._load_config()
  File "/home/adria/dev/pyenvs/ckan/src/ckan/ckan/lib/cli.py", line 310, in _load_config
    self.site_user = load_config(self.options.config, load_site_user)
AttributeError: 'LessCommand' object has no attribute 'options'
```

Looks like the culprit is this [line](https://github.com/ckan/ckan/pull/3547/files#diff-530e98d83857b6bd57795518ad4c2dbdR2027), part of the Bootstrap 3 PR.

@klikstermkd can you have a look?



Discussion:
This works on 2.7 so there's no rush
@amercader Yes I will look into it.

Resolving patch:
diff --git a/ckan/lib/cli.py b/ckan/lib/cli.py
--- a/ckan/lib/cli.py
+++ b/ckan/lib/cli.py
@@ -2117,6 +2117,7 @@ def command(self):
 
         # Less css
         cmd = LessCommand('less')
+        cmd.options = self.options
         cmd.command()
 
         # js translation strings


Test patch: