internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Organization list crashes when called with a limit parameter
**CKAN version**
2.9.1

**Describe the bug**
When querying/listing organizations or groups with an added limit parameter there is a server error.

**Steps to reproduce**
Calling `curl` on the `organization_list` with the limit field set should recreate it.
E.g.
```
curl http://localhost:5000/api/3/action/organization_list\?limit\=31
{"help": "http://localhost:5000/api/3/action/help_show?name=organization_list", "error": {"__type": "Internal Server Error", "message": "Internal Server Error"}, "success": false}% 
```

**Expected behavior**
The server should return a list of organizations/groups.



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -350,11 +350,18 @@ def _group_or_org_list(context, data_dict, is_org=False):
 
     if all_fields:
         # all_fields is really computationally expensive, so need a tight limit
-        max_limit = config.get(
-            'ckan.group_and_organization_list_all_fields_max', 25)
+        try:
+            max_limit = int(config.get(
+                'ckan.group_and_organization_list_all_fields_max', 25))
+        except ValueError:
+            max_limit = 25
     else:
-        max_limit = config.get('ckan.group_and_organization_list_max', 1000)
-    if limit is None or limit > max_limit:
+        try:
+            max_limit = int(config.get('ckan.group_and_organization_list_max', 1000))
+        except ValueError:
+            max_limit = 1000
+
+    if limit is None or int(limit) > max_limit:
         limit = max_limit
 
     # order_by deprecated in ckan 1.8


Test patch:
diff --git a/ckan/tests/logic/action/test_get.py b/ckan/tests/logic/action/test_get.py
--- a/ckan/tests/logic/action/test_get.py
+++ b/ckan/tests/logic/action/test_get.py
@@ -460,6 +460,15 @@ def test_group_list_limit_and_offset(self):
         assert len(group_list) == 1
         assert group_list[0] == group2["name"]
 
+    def test_group_list_limit_as_string(self):
+
+        group1 = factories.Group(name='aa')
+        group2 = factories.Group(name='bb')
+
+        group_list = helpers.call_action("group_list", limit="1")
+
+        assert len(group_list) == 1
+
     def test_group_list_wrong_limit(self):
 
         with pytest.raises(logic.ValidationError):
Repository: ckan/ckan

Issue:
Can't add directories from plugins if partially string matches existing values
**CKAN version**
1.7 or so

**Describe the bug**
If you add a served directory that partially string matches existing values it will be ignored

**Steps to reproduce**
```
if old_ckan:
    add_template_directory(config, 'templates-for-old-ckans')
add_template_directory(config, 'templates')
```

**Expected behavior**
when old_ckan is True I expect both template directories to be in the template search path

**Additional details**
only the `/path/to/my/extension/templates-for-old-ckans` appears in the template search path


Discussion:

Resolving patch:
diff --git a/ckan/plugins/toolkit.py b/ckan/plugins/toolkit.py
--- a/ckan/plugins/toolkit.py
+++ b/ckan/plugins/toolkit.py
@@ -404,7 +404,7 @@ def _add_served_directory(cls, config, relative_path, config_var):
 
         this_dir = os.path.dirname(filename)
         absolute_path = os.path.join(this_dir, relative_path)
-        if absolute_path not in config.get(config_var, ''):
+        if absolute_path not in config.get(config_var, '').split(','):
             if config.get(config_var):
                 config[config_var] += ',' + absolute_path
             else:


Test patch:
Repository: ckan/ckan

Issue:
migrate_package_activity fails under Python 3
**CKAN version**
2.9 with Python 3.6

**Describe the bug**
The migrate_package_activity script fails when run under Python 3 

Initially producing a module not found error for paste.script
After installing pastescript via pip the error becomes an attribute error


**Steps to reproduce**
Clean CKAN 2.9 install on Python 3
Use pg_restore to load database from an earlier version (2.8 in my case)
Run ckan db upgrade
Attempt to run migrate_package_activity

**Expected behavior**
Package activity should be migrated and script exit gracefully

**Additional details**
Initial Error:
```
Loading config
Traceback (most recent call last):
  File "ckan/ckan/migration/migrate_package_activity.py", line 290, in <module>
    load_config(args.config)
  File "ckan/ckan/migration/migrate_package_activity.py", line 279, in load_config
    from ckan.lib.cli import CkanCommand
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 9, in <module>
    import paste.script
ModuleNotFoundError: No module named 'paste.script'
```

After pip install pastescript
```
Traceback (most recent call last):
  File "ckan/ckan/migration/migrate_package_activity.py", line 275, in <module>
    from ckan.lib.cli import load_config
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 212, in <module>
    class CkanCommand(paste.script.command.Command):
AttributeError: module 'paste.script' has no attribute 'command'
```






Discussion:

Resolving patch:
diff --git a/ckan/migration/migrate_package_activity.py b/ckan/migration/migrate_package_activity.py
--- a/ckan/migration/migrate_package_activity.py
+++ b/ckan/migration/migrate_package_activity.py
@@ -273,8 +273,11 @@ def print_errors(errors):
                         u'dataset - specify its name')
     args = parser.parse_args()
     assert args.config, u'You must supply a --config'
+    print(u'Loading config')
     try:
-        from ckan.lib.cli import load_config
+        from ckan.cli import load_config
+        from ckan.config.middleware import make_app
+        make_app(load_config(args.config))
     except ImportError:
         # for CKAN 2.6 and earlier
         def load_config(config):
@@ -287,9 +290,7 @@ class Options(object):
             cmd.options.config = config
             cmd._load_config()
             return
-
-    print(u'Loading config')
-    load_config(args.config)
+        load_config(args.config)
     if not args.dataset:
         migrate_all_datasets()
         wipe_activity_detail(delete_activity_detail=args.delete)


Test patch:
Repository: ckan/ckan

Issue:
sqlalchemy setting "pool_pre_ping"
**CKAN version**

Introduced in commit https://github.com/ckan/ckan/commit/ae9eb1360028a77b35506716eed2768349827806 which is related at least to 2.9.0, 2.9.1, 2.9.2. 

**Describe the bug**

This is the ckan code:
```python
    # Enable pessimistic disconnect handling (added in SQLAlchemy 1.2)
    # to eliminate database errors due to stale pooled connections
    config.setdefault('pool_pre_ping', True)

    # Initialize SQLAlchemy
    engine = sqlalchemy.engine_from_config(config)
```

This is the sqlalchemy function:
```python
    def engine_from_config(configuration, prefix="sqlalchemy.", **kwargs):
```

I guess the `pool_pre_ping` config is not taken into account by sqlalchemy since it's missing the prefix.



Discussion:
Sounds good, want to submit a PR @etj 

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -286,7 +286,7 @@ def update_config():
 
     # Enable pessimistic disconnect handling (added in SQLAlchemy 1.2)
     # to eliminate database errors due to stale pooled connections
-    config.setdefault('pool_pre_ping', True)
+    config.setdefault('sqlalchemy.pool_pre_ping', True)
 
     # Initialize SQLAlchemy
     engine = sqlalchemy.engine_from_config(config)


Test patch:
Repository: ckan/ckan

Issue:
DB INIT error when using usernames with non-alphanumeric chars
**CKAN version**

Master 713716b0689bf848865f05acd328ded29f041c5a (2.9.something)

**Describe the bug**

DB usernames with non alphanum chars will terminate in error the command `ckan db init`.

**Steps to reproduce**

Create a pg user with for instance a "`@`" in it, then update `ckan.ini` with the proper URL, 
e.g.: user `ckan@ckan`:

    sqlalchemy.url = postgresql://ckan@ckan:ckan@localhost/ckan_at

Running `ckan run` will not raise connection error (it won't find table, eventually). 

Running `ckan db init` will terminate with the error:

    invalid interpolation syntax in 'postgresql://ckan%40ckan:ckan@localhost/ckan_at' at position 17

**Expected behavior**

The command should connect to the db without errors.

**Additional details**

At https://github.com/ckan/ckan/blob/713716b0689bf848865f05acd328ded29f041c5a/ckan/model/__init__.py#L268-L270


`self.metadata.bind.url` contains `postgresql://ckan%40ckan:ckan@localhost/ckan_at`

We could replace the line with 
```python
        alembic_config.set_main_option(
            "sqlalchemy.url", config.get("sqlalchemy.url")
        )
```




Discussion:

Resolving patch:
diff --git a/ckan/model/__init__.py b/ckan/model/__init__.py
--- a/ckan/model/__init__.py
+++ b/ckan/model/__init__.py
@@ -266,7 +266,7 @@ def setup_migration_version_control(self):
         self.reset_alembic_output()
         alembic_config = AlembicConfig(self._alembic_ini)
         alembic_config.set_main_option(
-            "sqlalchemy.url", str(self.metadata.bind.url)
+            "sqlalchemy.url", config.get("sqlalchemy.url")
         )
         try:
             sqlalchemy_migrate_version = self.metadata.bind.execute(


Test patch:
Repository: ckan/ckan

Issue:
Password reset doesn't work well with external homepage hosting
**CKAN version**

2.8, 2.9

**Describe the bug**

After requesting or performing a password reset, there is a hardcoded redirect to the homepage. This works so long as CKAN is using its own homepage, but if the site owner chooses to host the homepage externally on a CDN, then it will be unable to display or consume the flash messages.

It would be good to have an option to make the target of this redirect configurable for sites that need it.

Will open a pull request soon with this functionality.



Discussion:

Resolving patch:
diff --git a/ckan/views/user.py b/ckan/views/user.py
--- a/ckan/views/user.py
+++ b/ckan/views/user.py
@@ -661,14 +661,18 @@ def post(self):
                 h.flash_error(_(u'Error sending the email. Try again later '
                                 'or contact an administrator for help'))
                 log.exception(e)
-                return h.redirect_to(u'home.index')
+                return h.redirect_to(config.get(
+                    u'ckan.user_reset_landing_page',
+                    u'home.index'))
 
         # always tell the user it succeeded, because otherwise we reveal
         # which accounts exist or not
         h.flash_success(
             _(u'A reset link has been emailed to you '
               '(unless the account specified does not exist)'))
-        return h.redirect_to(u'home.index')
+        return h.redirect_to(config.get(
+            u'ckan.user_reset_landing_page',
+            u'home.index'))
 
     def get(self):
         self._prepare()
@@ -734,7 +738,9 @@ def post(self, id):
             mailer.create_reset_key(context[u'user_obj'])
 
             h.flash_success(_(u'Your password has been reset.'))
-            return h.redirect_to(u'home.index')
+            return h.redirect_to(config.get(
+                u'ckan.user_reset_landing_page',
+                u'home.index'))
         except logic.NotAuthorized:
             h.flash_error(_(u'Unauthorized to edit user %s') % id)
         except logic.NotFound:


Test patch:
Repository: ckan/ckan

Issue:
UTF-8 not supported in resource_formats.json
**CKAN version**

CKAN 2.9.2
Ubuntu 18.04
Python 3.6

**Describe the bug**
When trying to upload a resource I get a message "Internal server error".

The log file reveals an error that occurs when resource_formats.json is loaded:
`ValueError: Invalid JSON syntax in /usr/lib/ckan/default/src/ckan/ckan/config/resource_formats.json: 'ascii' codec can't decode byte 0xc2 in position 4411: ordinal not in range(128)`.

The problem seems to be caused by utf-8 characters in the JSON file.

**Steps to reproduce**
- Add utf-8 characters to resource_formats.json
- Upload resource

**Expected behavior**
The resource is uploaded without error log

**Additional details**
It seems that JSON files are not loaded as utf-8 by default and that you have to specify the encoding (according to https://stackoverflow.com/a/35052042/395879).

The relevant code is probably `ckan/lib/helpers.py` in line 2685:
```python
        with open(format_file_path) as format_file:
            try:
                file_resource_formats = json.loads(format_file.read())
```

Stack trace:

```
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/update.py", line 295, in package_update
    package_plugin, context, data_dict, schema, 'package_update')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/plugins.py", line 306, in plugin_validate
    return toolkit.navl_validate(data_dict, schema, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 273, in validate
    converted_data, errors = _validate(flattened, schema, validators_context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 314, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py", line 224, in convert
    value = converter(converted_data.get(key))
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/validators.py", line 798, in clean_format
    return h.unified_resource_format(format)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 2713, in unified_resource_format
    formats = resource_formats()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 2691, in resource_formats
    (format_file_path, e))
ValueError: Invalid JSON syntax in /usr/lib/ckan/default/src/ckan/ckan/config/resource_formats.json: 'ascii' codec can't decode byte 0xc2 in position 4411: ordinal not in range(128)
```



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -2735,7 +2735,7 @@ def resource_formats():
                 os.path.dirname(os.path.realpath(ckan.config.__file__)),
                 'resource_formats.json'
             )
-        with open(format_file_path) as format_file:
+        with open(format_file_path, encoding='utf-8') as format_file:
             try:
                 file_resource_formats = json.loads(format_file.read())
             except ValueError as e:


Test patch:
Repository: ckan/ckan

Issue:
Authorization error when a non-sysadmin user deletes a group/org
**CKAN version**

\>=2.7

**Describe the bug**
When a non-sysadmin user deletes a group or organization, they get a 403 Not Authorized error. The group/org gets actually deleted.

This is because after deleting the group/org we are calling `group_show` on the deleted entity:

https://github.com/ckan/ckan/blob/170556a97f38d42b94474943ccab4cef19fd9d49/ckan/views/group.py#L1039

AFAICT we are not using this `group_dict` at all so it can be removed.

The tests in place didn't check the status code returned so that's why it slipped unnoticed.





Discussion:

Resolving patch:
diff --git a/ckan/views/group.py b/ckan/views/group.py
--- a/ckan/views/group.py
+++ b/ckan/views/group.py
@@ -1036,18 +1036,13 @@ def post(self, group_type, is_organization, id=None):
                 u'has been deleted') or _(u'Group')
             h.flash_notice(
                 _(u'%s has been deleted.') % _(group_label))
-            group_dict = _action(u'group_show')(context, {u'id': id})
         except NotAuthorized:
             base.abort(403, _(u'Unauthorized to delete group %s') % u'')
         except NotFound:
             base.abort(404, _(u'Group not found'))
         except ValidationError as e:
             h.flash_error(e.error_dict['message'])
-            return h.redirect_to(u'organization.read', id=id)
-
             return h.redirect_to(u'{}.read'.format(group_type), id=id)
-        # TODO: Remove
-        g.group_dict = group_dict
 
         return h.redirect_to(u'{}.index'.format(group_type))
 


Test patch:
diff --git a/ckan/tests/controllers/test_group.py b/ckan/tests/controllers/test_group.py
--- a/ckan/tests/controllers/test_group.py
+++ b/ckan/tests/controllers/test_group.py
@@ -287,6 +287,7 @@ def test_owner_delete(self, app, initial_data):
             data={"delete": ""},
             extra_environ=initial_data["user_env"],
         )
+        assert response.status_code == 200
         group = helpers.call_action(
             "group_show", id=initial_data["group"]["id"]
         )
@@ -300,6 +301,7 @@ def test_sysadmin_delete(self, app, initial_data):
             data={"delete": ""},
             extra_environ=extra_environ,
         )
+        assert response.status_code == 200
         group = helpers.call_action(
             "group_show", id=initial_data["group"]["id"]
         )
Repository: ckan/ckan

Issue:
Datastore: sqlalchemy setting "pool_pre_ping"
**CKAN version**
2.9.x master at 93ef47244b01db097cd65494677296234a4d37d6

**Describe the bug**

The setting "`pool_pre_ping`" for the datastore connection is not working, because of a wrong setting key.  

This is the [code](https://github.com/ckan/ckan/blob/93ef47244b01db097cd65494677296234a4d37d6/ckanext/datastore/backend/postgres.py#L122-L125):
```python
        config.setdefault('pool_pre_ping', True)
        engine = sqlalchemy.engine_from_config(config,
                                               'ckan.datastore.sqlalchemy.',
                                               **extras)
```
Since sqlachemy is told to read config keys with prefix `ckan.datastore.sqlalchemy.`, the setting `pool_pre_ping` will not be taken into consideration.

**Additional details**

This issue is identical to #5932, but related to the datastore engine and not to the main ckan db.



Discussion:

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -119,7 +119,7 @@ def _get_engine_from_url(connection_url):
     engine = _engines.get(connection_url)
     if not engine:
         extras = {'url': connection_url}
-        config.setdefault('pool_pre_ping', True)
+        config.setdefault('ckan.datastore.sqlalchemy.pool_pre_ping', True)
         engine = sqlalchemy.engine_from_config(config,
                                                'ckan.datastore.sqlalchemy.',
                                                **extras)


Test patch:
Repository: ckan/ckan

Issue:
"latest" documentation not being built in Read The Docs

https://readthedocs.org/projects/ckan/builds/13792875/

```
ERROR: Could not find a version that satisfies the requirement freezegun==1.1.0 (from -r dev-requirements.txt (line 8)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.11, 0.1.12, 0.1.13, 0.1.14, 0.1.15, 0.1.16, 0.1.17, 0.1.18, 0.1.19, 0.1.19.1, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 0.3.8, 0.3.9, 0.3.10, 0.3.11, 0.3.12, 0.3.13, 0.3.14, 0.3.15)
ERROR: No matching distribution found for freezegun==1.1.0 (from -r dev-requirements.txt (line 8))
```

Haven't investigated at all but it looks like a py2/py3 requirements issue



Discussion:
Freezegun apparently dropped support for py2 in 1.0.0. Docs probably should be built with py3.

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1632,7 +1632,8 @@ def sanitize_url(url):
         netloc = parsed_url.netloc.encode('idna').decode('ascii')
         if not _PLAUSIBLE_HOST_IDNA.match(netloc):
             return ''
-        # quote with allowed characters from https://www.ietf.org/rfc/rfc3986.txt
+        # quote with allowed characters from
+        # https://www.ietf.org/rfc/rfc3986.txt
         parsed_url = parsed_url._replace(
             scheme=quote(unquote(parsed_url.scheme), '+'),
             path=quote(unquote(parsed_url.path), "/"),


Test patch:
Repository: ckan/ckan

Issue:
Plugin order for translations is reversed
**CKAN version**
2.8, 2.9, master

**Describe the bug**
If the developer has multiple plugins implementing ITranslation interface and has the same translation keys in them, the last plugin wins.

**Steps to reproduce**
Create two plugins with ITranslation interface and the same translation key.
Translations from last plugin will be used.

**Expected behavior**
Translations from the first plugin should be used as the common convention is that the first plugin wins.

**Additional details**
https://github.com/vrk-kpa/ckanext-forcetranslation we made this couple years ago to circumvent this. Simple plugin which allows to choose which plugins translations to use. Related bug in https://github.com/ckan/ckanext-harvest/issues/266 which in essence is caused by the same thing.



Discussion:
like other similar issues this might be fixed with a `reversed()` in places that have `for p in PluginImplementations(ITranslation)`
hi , @Zharktas  i would like to work on this issue.

Resolving patch:
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -238,7 +238,7 @@ def ungettext_alias():
         (_ckan_i18n_dir, u'ckan')
     ] + [
         (p.i18n_directory(), p.i18n_domain())
-        for p in PluginImplementations(ITranslation)
+        for p in reversed(list(PluginImplementations(ITranslation)))
     ]
 
     i18n_dirs, i18n_domains = zip(*pairs)


Test patch: