internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
datastore_search returns most relevant records last 
### CKAN Version if known (or site URL)
data.govt.nz

### Please describe the expected behaviour
When querying the `datastore_search` API method,  we expected the most relevant results to be listed first. basically `ORDER BY rank DESC`

(relevant means that they closely march the search query)

When querying for "age concern" we expected results with this exact phrase to be first.


### Please describe the actual behaviour

the most relevant results are listed last. `ORDER BY rank `
results containing only "age" or only "concern" are in the first pages of results, and "age concern" is in the last pages.


### What steps can be taken to reproduce the issue? 

http GET `{server}/datastore_search?distinct=true&plain=false&resource_id={resource}&q={keyword}&limit=10`;

e..g 
https://catalogue.data.govt.nz/api/3/action/datastore_search?resource_id=35de6bf8-b254-4025-89f5-da9eb6adf9a0&q=age+concern&limit=10

results for "age" or "concern" are in the first pages.
results for "age concern" are in the last pages





Discussion:
Wow, that's not good. Care to submit a PR? I think we just need a "DESC" after each value here https://github.com/ckan/ckan/blob/master/ckanext/datastore/backend/postgres.py#L474 and a quick test to prevent regressions

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -471,7 +471,10 @@ def _sort(sort, fields_types, rank_columns):
     to order by best text search match
     '''
     if not sort:
-        return rank_columns.values()
+        rank_sorting = []
+        for column in rank_columns.values():
+            rank_sorting.append(u'{0} DESC'.format(column))
+        return rank_sorting
 
     clauses = datastore_helpers.get_list(sort, False)
 


Test patch:
Repository: ckan/ckan

Issue:
Function _resource_preview() in module ckan.controllers.package has been deprecated 
### CKAN Version if known (or site URL)
2.7.2

> Function _resource_preview() in module ckan.controllers.package has been deprecated and will be removed in a later release of ckan. Resource preview is deprecated. Please use the new resource views


### What steps can be taken to reproduce the issue? 
I used the plugin ngsiview and after Clicking on preview dataset. I observe the above message in my logs
So, what changes can i do for it?


Discussion:
Hi @wardi, @torfsen can you please suggest in it?
The only place, CKAN itself uses result of `_resource_preview` is [here](https://github.com/ckan/ckan/blob/master/ckan/controllers/package.py#L1124). In new implementation(Flask blueprints), we are deciding, whether a resource can be previewed depending [on the amount of it's views](https://github.com/ckan/ckan/blob/master/ckan/views/resource.py#L83). If you are developer of extension or going to provide PR into source or just want to make local modifications - just use `bool(len_of_resource_views)` instead of `_resource_preview`. 

Resolving patch:
diff --git a/ckan/controllers/package.py b/ckan/controllers/package.py
--- a/ckan/controllers/package.py
+++ b/ckan/controllers/package.py
@@ -393,14 +393,13 @@ def read(self, id):
 
         # can the resources be previewed?
         for resource in c.pkg_dict['resources']:
-            # Backwards compatibility with preview interface
-            resource['can_be_previewed'] = self._resource_preview(
-                {'resource': resource, 'package': c.pkg_dict})
-
             resource_views = get_action('resource_view_list')(
                 context, {'id': resource['id']})
             resource['has_views'] = len(resource_views) > 0
 
+            # Backwards compatibility with preview interface
+            resource['can_be_previewed'] = bool(len(resource_views))
+
         package_type = c.pkg_dict['type'] or 'dataset'
         self._setup_template_variables(context, {'id': id},
                                        package_type=package_type)
@@ -1116,13 +1115,12 @@ def resource_read(self, id, resource_id):
         c.datastore_api = '%s/api/action' % \
             config.get('ckan.site_url', '').rstrip('/')
 
-        c.resource['can_be_previewed'] = self._resource_preview(
-            {'resource': c.resource, 'package': c.package})
-
         resource_views = get_action('resource_view_list')(
             context, {'id': resource_id})
         c.resource['has_views'] = len(resource_views) > 0
 
+        c.resource['can_be_previewed'] = bool(len(resource_views))
+
         current_resource_view = None
         view_id = request.GET.get('view_id')
         if c.resource['can_be_previewed'] and not view_id:


Test patch:
Repository: ckan/ckan

Issue:
Activity stream actions not overridable
Some calls to action functions in `action/get.py` call the functions directly instead of using `get_action`, meaning they can't be overridden from extensions.


Discussion:
Just noticed that after #4627 this is no longer relevant in master. I'll target 2.8 for this.

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -2698,7 +2698,7 @@ def package_activity_list_html(context, data_dict):
     :rtype: string
 
     '''
-    activity_stream = package_activity_list(context, data_dict)
+    activity_stream = logic.get_action('package_activity_list')(context, data_dict)
     offset = int(data_dict.get('offset', 0))
     extra_vars = {
         'controller': 'package',
@@ -2729,7 +2729,7 @@ def group_activity_list_html(context, data_dict):
     :rtype: string
 
     '''
-    activity_stream = group_activity_list(context, data_dict)
+    activity_stream = logic.get_action('group_activity_list')(context, data_dict)
     offset = int(data_dict.get('offset', 0))
     extra_vars = {
         'controller': 'group',
@@ -2784,8 +2784,8 @@ def recently_changed_packages_activity_list_html(context, data_dict):
     :rtype: string
 
     '''
-    activity_stream = recently_changed_packages_activity_list(
-        context, data_dict)
+    activity_stream = logic.get_action(
+                'recently_changed_packages_activity_list')(context, data_dict)
     offset = int(data_dict.get('offset', 0))
     extra_vars = {
         'controller': 'package',
@@ -3344,7 +3344,7 @@ def dashboard_activity_list_html(context, data_dict):
     :rtype: string
 
     '''
-    activity_stream = dashboard_activity_list(context, data_dict)
+    activity_stream = logic.get_action('dashboard_activity_list')(context, data_dict)
     model = context['model']
     user_id = context['user']
     offset = data_dict.get('offset', 0)


Test patch:
Repository: ckan/ckan

Issue:
Slow group dictization when group has lots of members
### CKAN Version if known (or site URL)
2.8, master

### Please describe the expected behaviour
We have organization which has thousands of members (all accounts are automatically added there).  group_dictize is expected to be fast even is there is lots of members.

### Please describe the actual behaviour
Group_show action, group read and member editing are all slow, some of them timeout before anything is rendered. group_dictize calls user_list_dictize https://github.com/ckan/ckan/blob/master/ckan/lib/dictization/model_dictize.py#L535 which loops over user_dictice.

### What steps can be taken to reproduce the issue? 
Create organization, add thousands of members, try to access organization page.


Discussion:

Resolving patch:
diff --git a/ckan/controllers/group.py b/ckan/controllers/group.py
--- a/ckan/controllers/group.py
+++ b/ckan/controllers/group.py
@@ -220,6 +220,10 @@ def read(self, id, limit=20):
             # Do not query for the group datasets when dictizing, as they will
             # be ignored and get requested on the controller anyway
             data_dict['include_datasets'] = False
+
+            # Do not query group members as they aren't used in the view
+            data_dict['include_users'] = False
+
             c.group_dict = self._action('group_show')(context, data_dict)
             c.group = context['group']
         except (NotFound, NotAuthorized):
diff --git a/ckan/views/group.py b/ckan/views/group.py
--- a/ckan/views/group.py
+++ b/ckan/views/group.py
@@ -454,6 +454,10 @@ def read(group_type, is_organization, id=None, limit=20):
         # Do not query for the group datasets when dictizing, as they will
         # be ignored and get requested on the controller anyway
         data_dict['include_datasets'] = False
+
+        # Do not query group members as they aren't used in the view
+        data_dict['include_users'] = False
+
         group_dict = _action(u'group_show')(context, data_dict)
         group = context['group']
     except (NotFound, NotAuthorized):


Test patch:
Repository: ckan/ckan

Issue:
Remove ofs and pairtree
As mentioned by @davidread in #4681. These are from a prehistoric version of the filestore and definitely not used anymore


Discussion:

Resolving patch:
diff --git a/ckan/lib/app_globals.py b/ckan/lib/app_globals.py
--- a/ckan/lib/app_globals.py
+++ b/ckan/lib/app_globals.py
@@ -47,7 +47,6 @@
     'ckan.template_footer_end': {},
     'ckan.dumps_url': {},
     'ckan.dumps_format': {},
-    'ofs.impl': {'name': 'ofs_impl'},
     'ckan.homepage_style': {'default': '1'},
 
     # split string
diff --git a/ckan/lib/uploader.py b/ckan/lib/uploader.py
--- a/ckan/lib/uploader.py
+++ b/ckan/lib/uploader.py
@@ -79,20 +79,8 @@ def get_storage_path():
     # None means it has not been set. False means not in config.
     if _storage_path is None:
         storage_path = config.get('ckan.storage_path')
-        ofs_impl = config.get('ofs.impl')
-        ofs_storage_dir = config.get('ofs.storage_dir')
         if storage_path:
             _storage_path = storage_path
-        elif ofs_impl == 'pairtree' and ofs_storage_dir:
-            log.warn('''Please use config option ckan.storage_path instead of
-                     ofs.storage_dir''')
-            _storage_path = ofs_storage_dir
-            return _storage_path
-        elif ofs_impl:
-            log.critical('''We only support local file storage form version 2.2
-                         of ckan please specify ckan.storage_path in your
-                         config for your uploads''')
-            _storage_path = False
         else:
             log.critical('''Please specify a ckan.storage_path in your config
                          for your uploads''')


Test patch:
Repository: ckan/ckan

Issue:
After removing the tag the redirect web url is wrong and it causes an error
Hi, thanks for the wonderful CKAN. It makes me set up a open data portal very fast. 
When I search the data by tags, it works very quickly.
![图片](https://user-images.githubusercontent.com/6868237/58615187-05f03880-82ed-11e9-8420-e8b83fe59766.png)

But I am trying to remove the tags like the picture below
![图片](https://user-images.githubusercontent.com/6868237/58615334-641d1b80-82ed-11e9-965e-12e8b968807f.png)

 the redirect web url became` http://10.102.30.154:5000/en/dataset/dataset?tags=%E6%B5%8B%E8%AF%95`. The url has an extra `dataset`, which causes an error in the UI.

![图片](https://user-images.githubusercontent.com/6868237/58615220-1acccc00-82ed-11e9-9d7e-024d229480fc.png)

![图片](https://user-images.githubusercontent.com/6868237/58615241-28825180-82ed-11e9-949e-9ee848366ddd.png)




Discussion:

Resolving patch:
diff --git a/ckan/views/dataset.py b/ckan/views/dataset.py
--- a/ckan/views/dataset.py
+++ b/ckan/views/dataset.py
@@ -84,13 +84,15 @@ def drill_down_url(alternative_url=None, **by):
 
 
 def remove_field(package_type, key, value=None, replace=None):
+    if not package_type or package_type == u'dataset':
+        url = h.url_for(u'dataset.search')
+    else:
+        url = h.url_for(u'{0}_search'.format(package_type))
     return h.remove_url_param(
         key,
         value=value,
         replace=replace,
-        controller=u'dataset',
-        action=u'search',
-        alternative_url=package_type
+        alternative_url=url
     )
 
 


Test patch:
Repository: ckan/ckan

Issue:
More robust auth functions for resource_view_show 
Right now it relies on resource objects being present in the context. You should be able to call the auth function with the same parameters as the action (ie just the resource view `id`). This is not an issue in core but it can be problematic when extending auth from extensions.


Discussion:

Resolving patch:
diff --git a/ckan/logic/auth/get.py b/ckan/logic/auth/get.py
--- a/ckan/logic/auth/get.py
+++ b/ckan/logic/auth/get.py
@@ -163,7 +163,15 @@ def resource_show(context, data_dict):
 
 
 def resource_view_show(context, data_dict):
-    return authz.is_authorized('resource_show', context, data_dict)
+
+    model = context['model']
+
+    resource_view = model.ResourceView.get(data_dict['id'])
+    if not resource_view:
+        raise logic.NotFound(_('Resource view not found, cannot check auth.'))
+    resource = model.Resource.get(resource_view.resource_id)
+
+    return authz.is_authorized('resource_show', context, {'id': resource.id})
 
 
 def resource_view_list(context, data_dict):


Test patch:
diff --git a/ckanext/datastore/tests/test_chained_auth_functions.py b/ckanext/datastore/tests/test_chained_auth_functions.py
--- a/ckanext/datastore/tests/test_chained_auth_functions.py
+++ b/ckanext/datastore/tests/test_chained_auth_functions.py
@@ -96,4 +96,3 @@ def test_user_create_chained_auth(self):
         ctd.CreateTestData.create()
         # check if chained auth fallbacks to built-in user_create
         check_access(u'user_create', {u'user': u'annafan'}, {})
-
Repository: ckan/ckan

Issue:
psycopg2 version depends on old libssl
### CKAN Version if known (or site URL)
2.8.2

### Please describe the expected behaviour
CKAN works with up-to-date libssl

### Please describe the actual behaviour
psycopg segfaults if running environment has libssl 1.1 or newer:
```
#0  0x00007f662989ac1b in ssl3_cleanup_key_block ()
   from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/./libssl-fad88670.so.1.0.2l
#1  0x00007f6629898516 in ssl3_clear ()
   from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/./libssl-fad88670.so.1.0.2l
#2  0x00007f66298a40a9 in tls1_clear ()
   from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/./libssl-fad88670.so.1.0.2l
#3  0x00007f663ecb79d2 in SSL_new () from target:/usr/lib/x86_64-linux-gnu/libssl.so.1.1
#4  0x00007f6629b2f389 in ?? () from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
#5  0x00007f6629b30925 in ?? () from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
#6  0x00007f6629b1a600 in PQconnectPoll ()
   from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
#7  0x00007f6629b1b099 in ?? () from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
#8  0x00007f6629b1de9f in PQconnectdb ()
   from target:/usr/lib/ckan/default/lib/python2.7/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
#9  0x00007f6629d6e4d1 in _conn_sync_connect (self=0x7f662a1baa10) at psycopg/connection_int.c:716
#10 conn_connect (self=self@entry=0x7f662a1baa10, async=<optimized out>) at psycopg/connection_int.c:812
...
```
Notice the two libssl versions. A related psycopg2 issue: psycopg/psycopg2#543

### What steps can be taken to reproduce the issue? 
Install libssl 1.1 on server, restart apache2 and try to open the front page.

The issue can be resolved by updating to `psycopg2==2.8.2`


Discussion:

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -315,7 +315,7 @@ def _cache_types(connection):
             # redo cache types with json now available.
             return _cache_types(connection)
 
-        register_composite('nested', connection.connection, True)
+        register_composite('nested', connection.connection.connection, True)
 
 
 def _pg_version_is_at_least(connection, version):


Test patch:
Repository: ckan/ckan

Issue:
404s for minor recline assets
### CKAN Version if known (or site URL)
latest master

### Bug
When I preview a CSV on the resource page e.g. `/dataset/test/resource/0d534c55-879c-41c3-9e8f-26386b574158`
then it basically works, but there are some 404 errors fetching a few assets:

* /webassets/img/ajaxload-circle.gif
* /webassets/fonts/glyphicons-halflings-regular.woff
* /webassets/fonts/glyphicons-halflings-regular.ttf

There is a mismatch because there is an asset here: /img/ajaxload-circle.gif

I guess this is a side-effect from: https://github.com/ckan/ckan/pull/4614



Discussion:
@smotornyuk I don't suppose you have a moment to look at this?

Resolving patch:
diff --git a/ckan/lib/webassets_tools.py b/ckan/lib/webassets_tools.py
--- a/ckan/lib/webassets_tools.py
+++ b/ckan/lib/webassets_tools.py
@@ -58,7 +58,7 @@ def webassets_init():
     env.debug = config.get(u'debug', False)
     env.url = u'/webassets/'
 
-    env.append_path(base_path, u'/base/')
+    add_public_path(base_path, u'/base/')
 
     logger.debug(u'Base path {0}'.format(base_path))
     create_library(u'vendor', os.path.join(
@@ -159,3 +159,7 @@ def get_webassets_path():
             u'must be specified'
         )
     return webassets_path
+
+
+def add_public_path(path, url):
+    env.append_path(path, url)
diff --git a/ckan/plugins/toolkit.py b/ckan/plugins/toolkit.py
--- a/ckan/plugins/toolkit.py
+++ b/ckan/plugins/toolkit.py
@@ -325,8 +325,19 @@ def _add_public_directory(cls, config, relative_path):
 
         The path is relative to the file calling this function.
 
+        Webassets addition: append directory to webassets load paths
+        in order to correctly rewrite relative css paths and resolve
+        public urls.
+
         '''
-        cls._add_served_directory(config, relative_path, 'extra_public_paths')
+        import ckan.lib.helpers as h
+        from ckan.lib.webassets_tools import add_public_path
+        path = cls._add_served_directory(
+            config,
+            relative_path,
+            'extra_public_paths'
+        )
+        add_public_path(path, h.url_for_static('/'))
 
     @classmethod
     def _add_served_directory(cls, config, relative_path, config_var):
@@ -349,6 +360,7 @@ def _add_served_directory(cls, config, relative_path, config_var):
                 config[config_var] += ',' + absolute_path
             else:
                 config[config_var] = absolute_path
+        return absolute_path
 
     @classmethod
     def _add_resource(cls, path, name):


Test patch:
Repository: ckan/ckan

Issue:
_external parameter kept on Pylons generated URLs
The `url_for` wrapper for Pylons requests correctly handles Flask's `_external` parameter to generate fully qualified URLs, but fails to remove it, resulting in an extraneous param added to the URL, eg: `http://example.com/dataset/test?_external=True`




Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -409,7 +409,9 @@ def _url_for_pylons(*args, **kw):
 
     # We need to provide protocol and host to get full URLs, get them from
     # ckan.site_url
-    if kw.get('qualified', False) or kw.get('_external', False):
+    if kw.pop('_external', None):
+        kw['qualified'] = True
+    if kw.get('qualified'):
         kw['protocol'], kw['host'] = get_site_protocol_and_host()
 
     # The Pylons API routes require a slask on the version number for some


Test patch:
diff --git a/ckan/tests/lib/test_helpers.py b/ckan/tests/lib/test_helpers.py
--- a/ckan/tests/lib/test_helpers.py
+++ b/ckan/tests/lib/test_helpers.py
@@ -314,6 +314,22 @@ def test_url_for_pylons_request_using_flask_url_for(self):
 
         p.unload('test_routing_plugin')
 
+    @helpers.change_config('ckan.site_url', 'http://example.com')
+    def test_url_for_pylons_request_external(self):
+
+        if not p.plugin_loaded('test_routing_plugin'):
+            p.load('test_routing_plugin')
+
+        url = 'http://example.com/from_pylons_extension_before_map'
+        generated_url = h.url_for(
+            controller='ckan.tests.config.test_middleware:MockPylonsController',
+            action='view',
+            _external=True,
+        )
+        eq_(generated_url, url)
+
+        p.unload('test_routing_plugin')
+
 
 class TestHelpersRenderMarkdown(object):
Repository: ckan/ckan

Issue:
Allow to customize the DataProxy URL
This will allow users that really want to use it to host their own instance


Discussion:

Resolving patch:
diff --git a/ckanext/reclineview/plugin.py b/ckanext/reclineview/plugin.py
--- a/ckanext/reclineview/plugin.py
+++ b/ckanext/reclineview/plugin.py
@@ -21,6 +21,14 @@ def get_mapview_config():
                  if k.startswith(namespace)])
 
 
+def get_dataproxy_url():
+    '''
+    Returns the value of the ckan.recline.dataproxy_url config option
+    '''
+    return config.get(
+        'ckan.recline.dataproxy_url', '//jsonpdataproxy.appspot.com')
+
+
 def in_list(list_possible_values):
     '''
     Validator that checks that the input value is one of the given
@@ -83,7 +91,8 @@ def view_template(self, context, data_dict):
 
     def get_helpers(self):
         return {
-            'get_map_config': get_mapview_config
+            'get_map_config': get_mapview_config,
+            'get_dataproxy_url': get_dataproxy_url,
         }
 
 


Test patch: