internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Problems with translations in plugins
(This came up together with #2893, see there for more details)

I found two issues regarding translations in plugins:
1. `ckan.lib.plugins.DefaultTranslation.i18n_directory` relies on the extension package's `__file__` attribute (`ckanext.my_extension.__file__`). However, many extensions are distributed using namespace packages, and namespace packages don't have a `__file__` attribute.
2. It seems that `.mo` compiled translation files are not automatically picked up when an extension is packaged using `setup.py`.

I will prepare a PR for this.



Discussion:

Resolving patch:
diff --git a/ckan/lib/plugins.py b/ckan/lib/plugins.py
--- a/ckan/lib/plugins.py
+++ b/ckan/lib/plugins.py
@@ -537,7 +537,7 @@ def i18n_directory(self):
         i18n/
         '''
         # assume plugin is called ckanext.<myplugin>.<...>.PluginClass
-        extension_module_name = '.'.join(self.__module__.split('.')[0:2])
+        extension_module_name = '.'.join(self.__module__.split('.')[:3])
         module = sys.modules[extension_module_name]
         return os.path.join(os.path.dirname(module.__file__), 'i18n')
 


Test patch:
Repository: ckan/ckan

Issue:
Ckan Favicon with wrong path
### CKAN Version if known (or site URL)

2.5.1, from source
### Please describe the expected behaviour

Load ckan icon
### Please describe the actual behaviour

Can't lada ckan icon
### What steps can be taken to reproduce the issue?

Open dev console on a browser. Quick search on github shows the path is wrong on the source code. it points to 'images/icons/ckan.ico' instead of 'base/images/icons/ckan.ico'



Discussion:

Resolving patch:
diff --git a/ckan/config/environment.py b/ckan/config/environment.py
--- a/ckan/config/environment.py
+++ b/ckan/config/environment.py
@@ -246,7 +246,7 @@ def update_config():
         config['ckan.site_id'] = ckan_host
 
     # ensure that a favicon has been set
-    favicon = config.get('ckan.favicon', '/images/icons/ckan.ico')
+    favicon = config.get('ckan.favicon', '/base/images/ckan.ico')
     config['ckan.favicon'] = favicon
 
     # Init SOLR settings and check if the schema is compatible


Test patch:
Repository: ckan/ckan

Issue:
Strange facet item order
As of CKAN 2.5.2, the search facet items are listed in a strange order: They are first sorted by the number of matching datasets (that's good) and then by their name in _reverse_ alphabetic order, with lower case before upper case (that's strange). Here's a screenshot from [demo.ckan.org](http://demo.ckan.org):

![facet_order](https://cloud.githubusercontent.com/assets/2877717/14456308/83fdbae6-00a4-11e6-8162-ca6e341b901b.png)

I would have expected the following order: First search by count and then by name in alphabetic order, without distinguishing between upper and lower case.

Strange facet item order
As of CKAN 2.5.2, the search facet items are listed in a strange order: They are first sorted by the number of matching datasets (that's good) and then by their name in _reverse_ alphabetic order, with lower case before upper case (that's strange). Here's a screenshot from [demo.ckan.org](http://demo.ckan.org):

![facet_order](https://cloud.githubusercontent.com/assets/2877717/14456308/83fdbae6-00a4-11e6-8162-ca6e341b901b.png)

I would have expected the following order: First search by count and then by name in alphabetic order, without distinguishing between upper and lower case.



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -670,7 +670,8 @@ def get_facet_items_dict(facet, limit=None, exclude_active=False):
             facets.append(dict(active=False, **facet_item))
         elif not exclude_active:
             facets.append(dict(active=True, **facet_item))
-    facets = sorted(facets, key=lambda item: item['count'], reverse=True)
+    # Sort descendingly by count and ascendingly by case-sensitive display name
+    facets.sort(key=lambda it: (-it['count'], it['display_name'].lower()))
     if c.search_facets_limits and limit is None:
         limit = c.search_facets_limits.get(facet)
     # zero treated as infinite for hysterical raisins


Test patch:
Repository: ckan/ckan

Issue:
Deleted users appear in sysadmin user lists
Sysadmin users that have been deleted still appear in the sysadmin list at `/ckan-admin` and in the output of a `paster sysadmin list` command in CKAN in 2.4.1 and 2.5.2 without any indication the user has been deleted.

I expect that deleted sysadmin users will not appear in either list or possibly appear with an indication they are deleted.

To reproduce:
1. Create a sysadmin user 'test_sysadmin'
2. Delete 'test_sysadmin' user
3. Open `/ckan-admin` - 'test_sysadmin' is listed
4. Run `paster sysadmin list` - 'test_sysadmin' is listed



Discussion:

Resolving patch:
diff --git a/ckan/controllers/admin.py b/ckan/controllers/admin.py
--- a/ckan/controllers/admin.py
+++ b/ckan/controllers/admin.py
@@ -16,7 +16,8 @@
 
 
 def get_sysadmins():
-    q = model.Session.query(model.User).filter(model.User.sysadmin==True)
+    q = model.Session.query(model.User).filter(model.User.sysadmin == True,
+                                               model.User.state == 'active')
     return q.all()
 
 
diff --git a/ckan/lib/cli.py b/ckan/lib/cli.py
--- a/ckan/lib/cli.py
+++ b/ckan/lib/cli.py
@@ -635,7 +635,8 @@ def command(self):
     def list(self):
         import ckan.model as model
         print 'Sysadmins:'
-        sysadmins = model.Session.query(model.User).filter_by(sysadmin=True)
+        sysadmins = model.Session.query(model.User).filter_by(sysadmin=True,
+                                                              state='active')
         print 'count = %i' % sysadmins.count()
         for sysadmin in sysadmins:
             print '%s name=%s id=%s' % (sysadmin.__class__.__name__,


Test patch:
Repository: ckan/ckan

Issue:
Sqlalchemy warning during tests (expensive non-query).
I'm seeing the following warning when running the tests..

```
SAWarning: The IN-predicate on "group.name" was invoked with an empty sequence. 
This results in a contradiction, which nonetheless can be expensive to evaluate.  
Consider alternative strategies for improved performance.
```

which doesn't seem like a big issue, and may just be test specific.  But it might be worth tracking down to ensure we don't have an unnecessary expensive query that isn't guarded against in the action layer.

Maybe good-for-contribution?



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -1895,9 +1895,10 @@ def package_search(context, data_dict):
     for field_name in ('groups', 'organization'):
         group_names.extend(facets.get(field_name, {}).keys())
 
-    groups = session.query(model.Group.name, model.Group.title) \
-                    .filter(model.Group.name.in_(group_names)) \
+    groups = (session.query(model.Group.name, model.Group.title)
+                    .filter(model.Group.name.in_(group_names))
                     .all()
+              if group_names else [])
     group_titles_by_name = dict(groups)
 
     # Transform facets into a more useful data structure.


Test patch:
Repository: ckan/ckan

Issue:
Migration 021 missing after pip install
### CKAN Version if known (or site URL)

CKAN 2.5.2
### Please describe the expected behaviour

After installation of CKAN, the files 021_postgres_downgrade.sql, 021_postgres_upgrade.sql, 021_postgresql_downgrade.sql, and 021_postgresql_upgrade.sql should be in the migration/versions directory.
### Please describe the actual behaviour

They're not in the directory.
Running paster --plugin=ckan db init fails with a KeyError for migration 021.
### What steps can be taken to reproduce the issue?

```
sisyphus:~ $ virtualenv ./v
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/gbowland/v/bin/python2
Also creating executable in /home/gbowland/v/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
sisyphus:~ $ . ./v/bin/activate
(v) sisyphus:~ $ pip install ckan==2.5.2                                        
Collecting ckan==2.5.2
Installing collected packages: ckan
Successfully installed ckan-2.5.2
(v) sisyphus:~ $ ls v/lib/python2.7/site-packages/ckan/migration/versions/021*
zsh: no matches found: v/lib/python2.7/site-packages/ckan/migration/versions/021*
```

Note that 021 is different in that it has .sql files, that may be the issue. Installing with pip via the git tag (git+git://github.com/ckan/ckan.git@ckan-2.5.2) works fine. If I do `python setup.py bdist` from the tag 'ckan-2.5.2` the resulting tarfile includes the .sql files for 021, so I'm a bit puzzled as to what happened.



Discussion:
nigelb on IRC mentioned that the process for releases is sdist, the SQL files for 021 migration are definitely not being included when an sdist is made.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -177,14 +177,6 @@
     zip_safe=False,
     packages=find_packages(exclude=['ez_setup']),
     namespace_packages=['ckanext', 'ckanext.stats'],
-    include_package_data=True,
-    package_data={'ckan': [
-        'i18n/*/LC_MESSAGES/*.mo',
-        'migration/migrate.cfg',
-        'migration/README',
-        'migration/tests/test_dumps/*',
-        'migration/versions/*',
-    ]},
     message_extractors={
         'ckan': [
             ('**.py', 'python', None),


Test patch:
Repository: ckan/ckan

Issue:
Sanitize offset when listing group activity
This is a very minor problem, apart that it causes `HTTP 500` errors where it should simply ignore the `offset` parameter (or reply with a `HTTP 400`). It happens at https://github.com/ckan/ckan/blob/master/ckan/controllers/group.py#L860 because `offset` is not validated and passed as-is to the action api function. 

We have encountered this type of errors several times in our production site, since crawling robots insist on trying URLs like http://labs.geodata.gov.gr/group/activity/imagery-base-maps-earth-cover/None

The above is tested on CKAN 2.2, but seems to be exactly the same at the master branch.



Discussion:
You are right, we have a couple of these on some actions. The best thing is to add [this decorator](https://github.com/ckan/ckan/blob/master/ckan/logic/action/get.py#L3254) to the [relevant action](https://github.com/ckan/ckan/blob/master/ckan/logic/action/get.py#L2695) and try / except a ValidationError on the group controller.

Do you want to submit a small PR for this?

That's ok, and is roughly what we've done in our fork to overcome it. I think the same problem also shows up in user activity lists (https://github.com/ckan/ckan/blob/master/ckan/controllers/user.py#L550). 

So, i will return with a PR.


Resolving patch:
diff --git a/ckan/controllers/group.py b/ckan/controllers/group.py
--- a/ckan/controllers/group.py
+++ b/ckan/controllers/group.py
@@ -825,10 +825,15 @@ def activity(self, id, offset=0):
         except (NotFound, NotAuthorized):
             abort(404, _('Group not found'))
 
-        # Add the group's activity stream (already rendered to HTML) to the
-        # template context for the group/read.html template to retrieve later.
-        c.group_activity_stream = self._action('group_activity_list_html')(
-            context, {'id': c.group_dict['id'], 'offset': offset})
+        try:
+            # Add the group's activity stream (already rendered to HTML) to the
+            # template context for the group/read.html
+            # template to retrieve later.
+            c.group_activity_stream = self._action('group_activity_list_html')(
+                context, {'id': c.group_dict['id'], 'offset': offset})
+
+        except ValidationError as error:
+            base.abort(400)
 
         return render(self._activity_template(group_type),
                       extra_vars={'group_type': group_type})
diff --git a/ckan/controllers/user.py b/ckan/controllers/user.py
--- a/ckan/controllers/user.py
+++ b/ckan/controllers/user.py
@@ -585,8 +585,11 @@ def activity(self, id, offset=0):
 
         self._setup_template_variables(context, data_dict)
 
-        c.user_activity_stream = get_action('user_activity_list_html')(
-            context, {'id': c.user_dict['id'], 'offset': offset})
+        try:
+            c.user_activity_stream = get_action('user_activity_list_html')(
+                context, {'id': c.user_dict['id'], 'offset': offset})
+        except ValidationError:
+            base.abort(400)
 
         return render('user/activity_stream.html')
 


Test patch:
Repository: ckan/ckan

Issue:
Login fails with 404 when using root_path
On master under Apache 2.2.4 and mod_wsgi with CKAN mounted at /data (`WSGIScriptAlias /data ...` and `ckan.root_path = /data`) on submitting the login form user is redirected to `/user/logged_in` (i.e., without the root_path) and Apache returns a 404.



Discussion:

Resolving patch:
diff --git a/ckan/config/middleware.py b/ckan/config/middleware.py
--- a/ckan/config/middleware.py
+++ b/ckan/config/middleware.py
@@ -88,6 +88,8 @@ def make_pylons_stack(conf, full_stack=True, static_files=True, **app_conf):
     for plugin in PluginImplementations(IMiddleware):
         app = plugin.make_middleware(app, config)
 
+    app = RootPathMiddleware(app, config)
+
     # Routing/Session/Cache Middleware
     app = RoutesMiddleware(app, config['routes.map'])
     # we want to be able to retrieve the routes middleware to be able to update
@@ -200,8 +202,6 @@ def make_pylons_stack(conf, full_stack=True, static_files=True, **app_conf):
     if asbool(config.get('ckan.tracking_enabled', 'false')):
         app = TrackingMiddleware(app, config)
 
-    app = RootPathMiddleware(app, config)
-
     return app
 
 


Test patch:
Repository: ckan/ckan

Issue:
DataStore Map and Explorer not displaying map tiles since 11 July 2016
Direct tile access to MapQuest maps has been discontinued as of 11 July 2016 and the DataStore Map and Explorer previews no longer display map tiles. 

The issue actually lies with recline.js and it has been logged https://github.com/okfn/recline/issues/500  and there is a referenced patch to replace MapQuest with Open Street Map frodrigo/recline@3df0c2a2bb8897124bdbbca715b2be1fd99cb08f

Thought it would be useful to have a record here and request the packaged recline.js should be updated.



Discussion:

Resolving patch:
diff --git a/ckanext/reclineview/plugin.py b/ckanext/reclineview/plugin.py
--- a/ckanext/reclineview/plugin.py
+++ b/ckanext/reclineview/plugin.py
@@ -5,6 +5,7 @@
 from ckan.common import json
 import ckan.plugins as p
 import ckan.plugins.toolkit as toolkit
+from pylons import config
 
 log = getLogger(__name__)
 ignore_empty = p.toolkit.get_validator('ignore_empty')
@@ -12,6 +13,15 @@
 Invalid = p.toolkit.Invalid
 
 
+def get_mapview_config():
+    '''
+    Extracts and returns map view configuration of the reclineview extension.
+    '''
+    namespace = 'ckanext.spatial.common_map.'
+    return dict([(k.replace(namespace, ''), v) for k, v in config.iteritems()
+                 if k.startswith(namespace)])
+
+
 def in_list(list_possible_values):
     '''
     Validator that checks that the input value is one of the given
@@ -77,6 +87,8 @@ class ReclineView(ReclineViewBase):
     This extension views resources using a Recline MultiView.
     '''
 
+    p.implements(p.ITemplateHelpers, inherit=True)
+
     def info(self):
         return {'name': 'recline_view',
                 'title': 'Data Explorer',
@@ -98,6 +110,11 @@ def can_view(self, data_dict):
         else:
             return False
 
+    def get_helpers(self):
+        return {
+            'get_map_config': get_mapview_config
+        }
+
 
 class ReclineGridView(ReclineViewBase):
     '''
@@ -174,6 +191,8 @@ class ReclineMapView(ReclineViewBase):
     This extension views resources using a Recline map.
     '''
 
+    p.implements(p.ITemplateHelpers, inherit=True)
+
     map_field_types = [{'value': 'lat_long',
                         'text': 'Latitude / Longitude fields'},
                        {'value': 'geojson', 'text': 'GeoJSON'}]
@@ -234,3 +253,8 @@ def setup_template_variables(self, context, data_dict):
 
     def form_template(self, context, data_dict):
         return 'recline_map_form.html'
+
+    def get_helpers(self):
+        return {
+            'get_mapview_config': get_mapview_config
+        }


Test patch:
Repository: ckan/ckan

Issue:
Datastore API Demo modal window encourages robots to make needless queries
### CKAN Version if known (or site URL)

2.1 ---> 2.5.2 http://demo.ckan.org/dataset/gold-prices/resource/b9aae52b-b082-4159-b46f-7bb9c158d013
### Please describe the expected behaviour

the CKAN Data API modal window designed to demonstrate the syntax for using the datastore API should not encourage robots to make queries against the database
### Please describe the actual behaviour

the modal window includes hyperlinked text for all resources ...

```
Query example (first 5 results)
http://demo.ckan.org/api/action/datastore_search?resource_id=b9aae52b-b082-4159-b46f-7bb9c158d013&limit=5
Query example (results containing 'jones')
http://demo.ckan.org/api/action/datastore_search?resource_id=b9aae52b-b082-4159-b46f-7bb9c158d013&q=jones
Query example (via SQL statement)
http://demo.ckan.org/api/action/datastore_search_sql?sql=SELECT * from "b9aae52b-b082-4159-b46f-7bb9c158d013" WHERE title LIKE 'jones'
```

These hyperlinks encourage robots - that do not follow robots.txt instructions - to conduct needless queries that result in db errors against all the data resources.
### What steps can be taken to reproduce the issue?

The modal window  [page template ](https://github.com/ckan/ckan/blob/b7204bb307abb1f488b5d409262db40b1e840d59/ckan/templates/ajax_snippets/api_info.html) should be redone to remove the hyperlinks.  



Discussion:

Resolving patch:
diff --git a/ckanext/datastore/plugin.py b/ckanext/datastore/plugin.py
--- a/ckanext/datastore/plugin.py
+++ b/ckanext/datastore/plugin.py
@@ -48,6 +48,7 @@ class DatastoreException(Exception):
 
 class DatastorePlugin(p.SingletonPlugin):
     p.implements(p.IConfigurable, inherit=True)
+    p.implements(p.IConfigurer)
     p.implements(p.IActions)
     p.implements(p.IAuthFunctions)
     p.implements(p.IResourceUrlChange)
@@ -71,6 +72,9 @@ def __new__(cls, *args, **kwargs):
 
         return super(cls, cls).__new__(cls, *args, **kwargs)
 
+    def update_config(self, config):
+        p.toolkit.add_template_directory(config, 'templates')
+
     def configure(self, config):
         self.config = config
         # check for ckan.datastore.write_url and ckan.datastore.read_url


Test patch:
Repository: ckan/ckan

Issue:
LanguageError: IOError: [Errno 2] No translation file found for domain: 'ckan'
Hi, 

We are noticing some 500 errors are being generated when  /e or /n are being passed instead of /en

Passing other letters such as /a /b /c will trigger a 404
### CKAN Version if known (or site URL)

Ckan 2.3.X -  2.5.2+
### Please describe the expected behaviour

http://demo.ckan.org/en = 200
http://demo.ckan.org/e = 404
http://demo.ckan.org/n = 404
### Please describe the actual behaviour

http://demo.ckan.org/en = 200
http://demo.ckan.org/e = 500
http://demo.ckan.org/n = 500
### What steps can be taken to reproduce the issue?

pass /e or /n in any ckan instance url and you can trigger a 500 internal server error.

http://demo.ckan.org/en/dataset = 200

http://demo.ckan.org/e/dataset = 500



Discussion:

Resolving patch:
diff --git a/ckan/lib/i18n.py b/ckan/lib/i18n.py
--- a/ckan/lib/i18n.py
+++ b/ckan/lib/i18n.py
@@ -27,8 +27,9 @@ def get_locales_from_config():
     the config AND also the locals available subject to the config. '''
     locales_offered = config.get('ckan.locales_offered', '').split()
     filtered_out = config.get('ckan.locales_filtered_out', '').split()
-    locale_default = config.get('ckan.locale_default', 'en')
+    locale_default = [config.get('ckan.locale_default', 'en')]
     locale_order = config.get('ckan.locale_order', '').split()
+
     known_locales = get_locales()
     all_locales = (set(known_locales) |
                    set(locales_offered) |


Test patch:
Repository: ckan/ckan

Issue:
Create a file filesize function as a part of iUploader interface
### CKAN Version if known (or site URL)


### Please describe the expected behaviour
This PR was merge https://github.com/ckan/ckan/pull/3205 recently, and it works great!
The only problem is that other extensions that override the default upload functionality don't have those alts, and it would be cool to implement a function that would be a part of an interface so that further extensions development will require this function.
### Please describe the actual behaviour


### What steps can be taken to reproduce the issue? 



Discussion:
@gleb-rudenko If I remember correctly, the IUploader interface just returns an Uploader class. In the default uploader that was modified in #3205 the `self.filesize` and `self.mimetype` properties are set on the `__init__()` method, so I guess that any custom implementation could do the same, using whatever logic applies to their backend.

I don't think we need to have a separate function, just [document on the interface](https://github.com/ckan/ckan/blob/master/ckan/plugins/interfaces.py#L1610:L1639) that extensions can set these two properties if they want to support it.

Does this make sense? Do you want to send a quick PR with these changes?

@amercader , sounds good


Resolving patch:
diff --git a/ckan/plugins/interfaces.py b/ckan/plugins/interfaces.py
--- a/ckan/plugins/interfaces.py
+++ b/ckan/plugins/interfaces.py
@@ -1618,6 +1618,12 @@ def get_resource_uploader(self):
         :param resource: resource dict
         :type resource: dictionary
 
+        Optionally, this method can set the following two attributes
+        on the class instance so they are set in the resource object:
+            
+            filesize (int):  Uploaded file filesize.
+            mimetype (str):  Uploaded file mimetype.
+
         ``upload(id, max_size)``
 
         Perform the actual upload.


Test patch:
Repository: ckan/ckan

Issue:
Module 'polib' missing from requirements.txt
### CKAN Version if known (or site URL)
latest (originally reported as 2.6.0)

### Please describe the expected behaviour
I have a copy of contrib/docker/docker-compose.yml, modified to fix #3075.
When I run 'docker-compose up', I expect a running set of containers to be launched.

### Please describe the actual behaviour
When I run 'docker-compose up', this error shows up

```
ckan    |    from ckan.config.environment import load_environment
ckan    |  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 17, in <module>
ckan    |    import ckan.lib.helpers as helpers
ckan    |  File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 33, in <module>
ckan    |    import i18n
ckan    |  File "/usr/lib/ckan/default/src/ckan/ckan/lib/i18n.py", line 52, in <module>
ckan    |    import polib
ckan    | ImportError: No module named polib
```
### What steps can be taken to reproduce the issue? 
run 'docker-compose up'

It looks like you need to recreate the requirements.txt file.


Discussion:

Resolving patch:
diff --git a/ckan/i18n/check_po_files.py b/ckan/i18n/check_po_files.py
--- a/ckan/i18n/check_po_files.py
+++ b/ckan/i18n/check_po_files.py
@@ -6,11 +6,6 @@
     paster check-po-files --help
 
 for usage.
-
-Requires polib <http://pypi.python.org/pypi/polib>:
-
-    pip install polib
-
 '''
 import polib
 import re


Test patch: