internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
Recline does not preview datastore anymore
The new plugin does not evaluate `datastore_active`.

<!---
@huboard:{"order":247.0}
-->

Recline does not preview datastore anymore
The new plugin does not evaluate `datastore_active`.

<!---
@huboard:{"order":247.0}
-->



Discussion:

Resolving patch:
diff --git a/ckanext/reclinepreview/plugin.py b/ckanext/reclinepreview/plugin.py
--- a/ckanext/reclinepreview/plugin.py
+++ b/ckanext/reclinepreview/plugin.py
@@ -26,6 +26,9 @@ def update_config(self, config):
         toolkit.add_resource('theme/public', 'ckanext-reclinepreview')
 
     def can_preview(self, data_dict):
+        # if the resource is in the datastore then we can preview it with recline
+        if data_dict['resource'].get('datastore_active'):
+            return True
         format_lower = data_dict['resource']['format'].lower()
         return format_lower in ['csv', 'xls', 'tsv']
 


Test patch:
diff --git a/ckanext/reclinepreview/tests/test_preview.py b/ckanext/reclinepreview/tests/test_preview.py
--- a/ckanext/reclinepreview/tests/test_preview.py
+++ b/ckanext/reclinepreview/tests/test_preview.py
@@ -88,6 +88,22 @@ def test_can_preview(self):
         }
         assert not self.p.can_preview(data_dict)
 
+        data_dict = {
+            'resource': {
+                'format': 'foo',
+                'datastore_active': True
+            }
+        }
+        assert self.p.can_preview(data_dict)
+
+        data_dict = {
+            'resource': {
+                'format': 'foo',
+                'datastore_active': False
+            }
+        }
+        assert not self.p.can_preview(data_dict)
+
 
     def test_js_included(self):
         res_id = self.resource['id']
Repository: ckan/ckan

Issue:
`engine_is_pg` does not recognize engine `postgres`
This pull request also includes minor changes to the commenting style.



Discussion:
@domoritz @seanh I'm afraid I'm getting these messages on latest master:

```
WARNI [ckanext.datastore.plugin] We detected that you do not use a PostgreSQL database. The DataStore will NOT work and datastore tests will be skipped.
```

Looks like this check is not thorough enough:

  def engine_is_pg():
        # Returns true iff the engine is connected to a postgresql database.
        return engine.url.drivername in ['psycopg2', 'postgres']

In my case engine.url.drivername is 'postgresql'. My connection string is `postgresql://ckantest:pass@localhost/ckantest`

Note that there may be other possible url [connection strings](http://docs.sqlalchemy.org/en/rel_0_8/dialects/postgresql.html#dialect-postgresql-psycopg2-connect).

@amercader I'll fix that.

Thanks. I would also drop "Make sure that replication is properly set-up" as it is confusing.


Resolving patch:
diff --git a/ckan/model/meta.py b/ckan/model/meta.py
--- a/ckan/model/meta.py
+++ b/ckan/model/meta.py
@@ -161,4 +161,6 @@ def engine_is_sqlite():
 
 def engine_is_pg():
     # Returns true iff the engine is connected to a postgresql database.
-    return engine.url.drivername in ['psycopg2', 'postgres']
+    # According to http://docs.sqlalchemy.org/en/latest/core/engines.html#postgresql
+    # all Postgres driver names start with `postgresql`
+    return engine.url.drivername.startswith('postgresql')
diff --git a/ckanext/datastore/plugin.py b/ckanext/datastore/plugin.py
--- a/ckanext/datastore/plugin.py
+++ b/ckanext/datastore/plugin.py
@@ -68,8 +68,7 @@ def configure(self, config):
                 self._create_alias_table()
             else:
                 log.warn("We detected that CKAN is running on a read only database. "
-                    "Permission checks and _table_metadata creation are skipped."
-                    "Make sure that replication is properly set-up.")
+                    "Permission checks and the creation of _table_metadata are skipped.")
         else:
             log.warn("We detected that you do not use a PostgreSQL database. "
                     "The DataStore will NOT work and datastore tests will be skipped.")


Test patch:
Repository: ckan/ckan

Issue:
Better check for page parameter on feed controller
Right now we are checking for the "page" parameter to be an integer, but we have to also ensure that is a positive integer, otherwise we get an error from Solr

https://github.com/okfn/ckan/blob/master/ckan/controllers/feed.py



Discussion:
Grabbing this.


Resolving patch:
diff --git a/ckan/controllers/feed.py b/ckan/controllers/feed.py
--- a/ckan/controllers/feed.py
+++ b/ckan/controllers/feed.py
@@ -24,6 +24,7 @@
 
 import webhelpers.feedgenerator
 from pylons import config
+from pylons.i18n import _
 from urllib import urlencode
 
 from ckan import model
@@ -172,7 +173,7 @@ def group(self, id):
                        'user': c.user or c.author}
             group_dict = get_action('group_show')(context, {'id': id})
         except NotFound:
-            abort(404, 'Group not found')
+            abort(404, _('Group not found'))
 
         data_dict, params = self._parse_url_params()
         data_dict['fq'] = 'groups:"%s"' % id
@@ -282,7 +283,9 @@ def custom(self):
         try:
             page = int(request.params.get('page', 1))
         except ValueError:
-            abort(400, ('"page" parameter must be an integer'))
+            abort(400, _('"page" parameter must be a positive integer'))
+        if page < 0:
+            abort(400, _('"page" parameter must be a positive integer'))
 
         limit = ITEMS_LIMIT
         data_dict = {
@@ -434,7 +437,9 @@ def _parse_url_params(self):
         try:
             page = int(request.params.get('page', 1)) or 1
         except ValueError:
-            abort(400, ('"page" parameter must be an integer'))
+            abort(400, _('"page" parameter must be a positive integer'))
+        if page < 0:
+            abort(400, _('"page" parameter must be a positive integer'))
 
         limit = ITEMS_LIMIT
         data_dict = {


Test patch:
diff --git a/ckan/tests/functional/test_group.py b/ckan/tests/functional/test_group.py
--- a/ckan/tests/functional/test_group.py
+++ b/ckan/tests/functional/test_group.py
@@ -74,6 +74,18 @@ def test_atom_feed_page_zero(self):
         assert 'xmlns="http://www.w3.org/2005/Atom"' in res, res
         assert '</feed>' in res, res
 
+    def test_atom_feed_page_negative(self):
+        group_name = 'deletetest'
+        CreateTestData.create_groups([{'name': group_name,
+                                       'packages': []}],
+                                     admin_user_name='testsysadmin')
+
+        offset = url_for(controller='feed', action='group',
+                         id=group_name)
+        offset = offset + '?page=-2'
+        res = self.app.get(offset, expect_errors=True)
+        assert '"page" parameter must be a positive integer' in res, res
+
     def test_children(self):
         if model.engine_is_sqlite() :
             from nose import SkipTest
Repository: ckan/ckan

Issue:
Migration 64 fails if dashboard table exists and contains data
Migration 64 adds a new NOT NULL column to the dashboard table but doesn't specify a default value, so the ALTER will fail if the table already contains data.

https://github.com/okfn/ckan/blob/master/ckan/migration/versions/064_add_email_last_sent_column.py



Discussion:
@seanh 
Do we have a sensible default value we can use here.  If so could you fix this or let me know the default and I can add it.

thanks

@tobes Yeah I plan to fix this just a question of finding the time (so anyone feel free to jump on it if you can). The current time as a `timestamp without time zone` will do as a default I think, but gotta find the postgres constant or method that will give you it.

`localtimestamp` looks like what you want


Resolving patch:
diff --git a/ckan/migration/versions/064_add_email_last_sent_column.py b/ckan/migration/versions/064_add_email_last_sent_column.py
--- a/ckan/migration/versions/064_add_email_last_sent_column.py
+++ b/ckan/migration/versions/064_add_email_last_sent_column.py
@@ -6,5 +6,5 @@ def upgrade(migrate_engine):
     metadata.bind = migrate_engine
     migrate_engine.execute('''
 ALTER TABLE dashboard
-    ADD COLUMN email_last_sent timestamp without time zone NOT NULL;
+    ADD COLUMN email_last_sent timestamp without time zone NOT NULL DEFAULT LOCALTIMESTAMP;
     ''')


Test patch:
Repository: ckan/ckan

Issue:
Multilingual extension's tests are failing on master



Discussion:

Resolving patch:
diff --git a/ckanext/multilingual/plugin.py b/ckanext/multilingual/plugin.py
--- a/ckanext/multilingual/plugin.py
+++ b/ckanext/multilingual/plugin.py
@@ -124,12 +124,11 @@ def before_index(self, search_data):
         for key, value in search_data.iteritems():
             if key in KEYS_TO_IGNORE or key.startswith('title'):
                 continue
-            if isinstance(value, list):
-                all_terms.extend(value)
-            elif value in (None, True, False):
-                continue
-            else:
-                all_terms.append(value)
+            if not isinstance(value, list):
+                value = [value]
+            for item in value:
+                if isinstance(item, basestring):
+                    all_terms.append(item)
 
         field_translations = action_get.term_translation_show(
                           {'model': ckan.model},


Test patch:
Repository: ckan/ckan

Issue:
Postgresql 8.4 error when running paster db init
When running the paster db init command with the CKAN 2.0 beta, there is an error encountered that appears to be related to use of the left() string function in ckan/migration/versions/067_turn_extras_to_strings.py. According to the documentation and my own simple test, this function is not support in Postgresql 8.4. For a stack trace, see: https://gist.github.com/thriuin/5067819.

Is there a new minimum version of Postgresql required -- documentation still says 8.4 which unfortunately is what comes with RedHat Enterprise.



Discussion:
Hi @thriuin,

This is related to https://github.com/okfn/ckan/pull/514. I've added a patch in the comments, at https://github.com/okfn/ckan/pull/514#issuecomment-14255638, that solves this issue. @domoritz created a patch of an issue that you will have down the line.

Could you apply both patches and try to run the tests under Postgres 8.4?

Cheers!

Thanks! I will test the patches on Monday morning when I have access to the RHEL environment again,

Hi @vitorbaptista, 

I have tested and can confirm these patches allow the paster db init to work on Postgres 8.4

Thanks!


Resolving patch:
diff --git a/ckan/migration/versions/067_turn_extras_to_strings.py b/ckan/migration/versions/067_turn_extras_to_strings.py
--- a/ckan/migration/versions/067_turn_extras_to_strings.py
+++ b/ckan/migration/versions/067_turn_extras_to_strings.py
@@ -7,7 +7,7 @@ def upgrade(migrate_engine):
         revision_tables = 'package_extra_revision group_extra_revision'
 
         for table in tables.split():
-            sql = """select id, value from {table} where left(value,1) = '"' """.format(table=table)
+            sql = """select id, value from {table} where substr(value,0,1) = '"' """.format(table=table)
             results = connection.execute(sql)
             for result in results:
                 id, value = result
@@ -16,7 +16,7 @@ def upgrade(migrate_engine):
                                    json.loads(value), id)
 
         for table in revision_tables.split():
-            sql = """select id, revision_id, value from {table} where left(value,1) = '"' """.format(table=table)
+            sql = """select id, revision_id, value from {table} where substr(value,0,1) = '"' """.format(table=table)
 
             results = connection.execute(sql)
             for result in results:


Test patch:
Repository: ckan/ckan

Issue:
Tag pages still use old templates
/tag



Discussion:

Resolving patch:
diff --git a/ckan/controllers/tag.py b/ckan/controllers/tag.py
--- a/ckan/controllers/tag.py
+++ b/ckan/controllers/tag.py
@@ -1,5 +1,5 @@
 from pylons.i18n import _
-from pylons import request, c
+from pylons import request, c, config
 
 import ckan.logic as logic
 import ckan.model as model
@@ -65,4 +65,7 @@ def read(self, id):
         except logic.NotFound:
             base.abort(404, _('Tag not found'))
 
-        return base.render('tag/read.html')
+        if h.asbool(config.get('ckan.legacy_templates', False)):
+            return base.render('tag/read.html')
+        else:
+            h.redirect_to(controller='package', action='search', tags=c.tag.get('name'))


Test patch:
Repository: ckan/ckan

Issue:
Use "Unnamed resource" not URL for display name when resource has no name



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -925,7 +925,6 @@ def dataset_link(package_or_package_dict):
 def resource_display_name(resource_dict):
     name = resource_dict.get('name', None)
     description = resource_dict.get('description', None)
-    url = resource_dict.get('url')
     if name:
         return name
     elif description:
@@ -934,11 +933,8 @@ def resource_display_name(resource_dict):
         if len(description) > max_len:
             description = description[:max_len] + '...'
         return description
-    elif url:
-        return url
     else:
-        noname_string = _('no name')
-        return '[%s] %s' % (noname_string, resource_dict['id'])
+        return _("Unnamed resource")
 
 
 def resource_link(resource_dict, package_id):


Test patch:
Repository: ckan/ckan

Issue:
Unicode filetypes break preview
This pr also improves the descriptions for when no preview is shown. 
- [x] Improve descriptions
- [x] Make it work with unicode
- [x] Remove old libraries
- [x] All strings are translatable



Discussion:
So, is this patch ready for merging?

In my opinion, yes.

Ops... just saw that @johnglover is assigned. It's probably better for him to check it :-)

I've merged this no need to delay things

@domoritz @tobes
This has not been merged properly into the `release-v2.0` branch as can be seen here:
https://github.com/okfn/ckan/blob/release-v2.0/ckan/lib/helpers.py#L1508

This unicode bug currently breaks https://offenedaten.de with umlauts in resource names because it is deployed on the `release-v2.0` branch. Please merge properly.


Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1484,7 +1484,6 @@ def resource_preview(resource, pkg_id):
     data_dict = {'resource': resource, 'package': c.package}
 
     if not resource['url']:
-        log.info('No url for resource {0} defined.'.format(resource['id']))
         return snippet("dataviewer/snippets/no_preview.html",
                        resource_type=format_lower,
                        reason='No valid resource url has been defined.')
@@ -1504,9 +1503,6 @@ def resource_preview(resource, pkg_id):
     elif format_lower in loadable_in_iframe:
         url = resource['url']
     else:
-        log.info(
-            'No preview handler for resource type {0}'.format(format_lower)
-        )
         return snippet("dataviewer/snippets/no_preview.html",
                        resource_type=format_lower)
 


Test patch:
Repository: ckan/ckan

Issue:
resource_update doesn't use IDatasetForm for validation
With an IDatasetForm it is possible to add validation that applies to resources added while creating a dataset, but after creation editing resources has no validation from the IDatasetForm applied, and any values may be entered that satisfy the default resource schema.



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/update.py b/ckan/logic/action/update.py
--- a/ckan/logic/action/update.py
+++ b/ckan/logic/action/update.py
@@ -170,7 +170,6 @@ def resource_update(context, data_dict):
     model = context['model']
     user = context['user']
     id = _get_or_bust(data_dict, "id")
-    schema = context.get('schema') or schema_.default_update_resource_schema()
 
     resource = model.Resource.get(id)
     context["resource"] = resource
@@ -181,6 +180,13 @@ def resource_update(context, data_dict):
 
     _check_access('resource_update', context, data_dict)
 
+    if 'schema' in context:
+        schema = context['schema']
+    else:
+        package_plugin = lib_plugins.lookup_package_plugin(
+            resource.resource_group.package.type)
+        schema = package_plugin.update_package_schema()['resources']
+
     data, errors = _validate(data_dict, schema, context)
     if errors:
         model.Session.rollback()


Test patch:
Repository: ckan/ckan

Issue:
ValueError when showing resources whose "size" is not an int
Example: http://www.publicdata.eu/dataset/pks-register-of-tender-documentation/resource/07554517-5171-4ff8-9ed3-fa5b6431a296

It crashes because the resource's "size" is not an int: http://publicdata.eu/api/3/action/resource_show?id=07554517-5171-4ff8-9ed3-fa5b6431a296

```
File '/usr/lib/ckan/src/ckan/ckan/lib/helpers.py', line 1454 in format_resource_items
  value = formatters.localised_filesize(int(value))
ValueError: invalid literal for int() with base 10: 'None'
```



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1457,7 +1457,12 @@ def format_resource_items(items):
             continue
         # size is treated specially as we want to show in MiB etc
         if key == 'size':
-            value = formatters.localised_filesize(int(value))
+            try:
+                value = formatters.localised_filesize(int(value))
+            except ValueError:
+                # Sometimes values that can't be converted to ints can sneak
+                # into the db. In this case, just leave them as they are.
+                pass
         elif isinstance(value, basestring):
             # check if strings are actually datetime/number etc
             if re.search(reg_ex_datetime, value):


Test patch:
Repository: ckan/ckan

Issue:
recently_changed_packages_activity_stream in helpers.py is broken
We are attempting to put the 'recently changed packages' activity stream on the front page of our website, using code like this:

<pre>
{{ h.recently_changed_packages_activity_stream() }}
</pre>

However, this gives the following error:

<pre>
[Tue Aug 06 15:07:27 2013] [error] File '/usr/lib/ckan/default/lib/python2.7/site-packages/ckanext_qgov_data-SNAPSHOT-py2.7.egg/ckanext/qgov/theme/templates/home/index.html', line 172 in block "home_content"
[Tue Aug 06 15:07:27 2013] [error]   {{ h.recently_changed_packages_activity_stream() }}
[Tue Aug 06 15:07:27 2013] [error] File '/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py', line 1325 in recently_changed_packages_activity_stream
[Tue Aug 06 15:07:27 2013] [error]   context, {})
[Tue Aug 06 15:07:27 2013] [error] File '/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py', line 324 in wrapped
[Tue Aug 06 15:07:27 2013] [error]   return _action(context, data_dict, **kw)
[Tue Aug 06 15:07:27 2013] [error] File '/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py', line 2174 in recently_changed_packages_activity_list_html
[Tue Aug 06 15:07:27 2013] [error]   'id': data_dict['id'],
[Tue Aug 06 15:07:27 2013] [error] KeyError: 'id'
</pre>

This worked correctly in version 1.8, but we are in the process of migrating to 2.0, and are being held up by this issue.



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -2258,7 +2258,6 @@ def recently_changed_packages_activity_list_html(context, data_dict):
     extra_vars = {
         'controller': 'package',
         'action': 'activity',
-        'id': data_dict['id'],
         'offset': offset,
         }
     return activity_streams.activity_list_to_html(context, activity_stream,


Test patch:
Repository: ckan/ckan

Issue:
Add documentation for Nginx configuration with CKAN
We use Nginx and Apache and this needs to be documented.



Discussion:
I'm going to push this to 2.2, to land after the doc reorg has landed.

@nigelbabu Any update on this one?

I'll add this to my list for next week.

@seanh How do you want me to write this documentation? Should I add it as an alternative or should I ask everyone to use Nginx?

I think we should only explicitly mention how to use Nginx in the source deploy instructions (which already mention Nginx but don't explain how to use it)

http://docs.ckan.org/en/latest/deployment.html

The complication is that in the package we have Nginx reverse proxying Apache.

I think is fine to recommend the apache-nginx setup for source installs as well if we are using it for the package install. We can change the port in step 5, explain why are we using nginx and add an extra step 6 for the nginx conf file. Or if you prefer add it as an optional step at the end.


Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -57,6 +57,8 @@
 .. |sqlalchemy| replace:: SQLAlchemy
 .. |javascript| replace:: JavaScript
 .. |apache| replace:: Apache
+.. |nginx_config_file| replace:: /etc/nginx/sites-available/ckan_default
+.. |reload_nginx| replace:: sudo service nginx reload
 
 '''
 
@@ -87,7 +89,7 @@
     href="http://creativecommons.org/licenses/by-sa/3.0/">Creative Commons
     Attribution ShareAlike (Unported) v3.0 License</a>.<br />
     <img src="http://i.creativecommons.org/l/by-sa/3.0/80x15.png" alt="CC License Logo" />
-    <a href="http://opendefinition.org/"><img src="http://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png" border="0" 
+    <a href="http://opendefinition.org/"><img src="http://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png" border="0"
       alt="{{ _('Open Content') }}" /></a>
   '''
 html_show_sphinx = False
@@ -146,7 +148,7 @@
 #'sidebarbgcolor': '#F2F2F2',
 #'sidebartextcolor': 'black',
 #'sidebarlinkcolor': '#355F7C',
-#'headfont': 'Trebuchet MS'    
+#'headfont': 'Trebuchet MS'
 #}
 sys.path.append(os.path.abspath('_themes'))
 html_theme_path = ['_themes']


Test patch: