internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
API: Specifying groups by title for adding datasets doesn't work
the docs say:
groups (list of dictionaries) – the groups to which the dataset belongs (optional), each group dictionary should have one or more of the following keys which identify an existing group: 'id' (the id of the group, string), 'name' (the name of the group, string), **'title' (the title of the group, string)**, to see which groups exist call group_list()

However, specifying the group by title is not foreseen in the code, and this does not work



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/create.py b/ckan/logic/action/create.py
--- a/ckan/logic/action/create.py
+++ b/ckan/logic/action/create.py
@@ -108,9 +108,9 @@ def package_create(context, data_dict):
     :param groups: the groups to which the dataset belongs (optional), each
         group dictionary should have one or more of the following keys which
         identify an existing group:
-        ``'id'`` (the id of the group, string), ``'name'`` (the name of the
-        group, string), ``'title'`` (the title of the group, string), to see
-        which groups exist call :py:func:`~ckan.logic.action.get.group_list`
+        ``'id'`` (the id of the group, string), or ``'name'`` (the name of the
+        group, string),  to see which groups exist
+        call :py:func:`~ckan.logic.action.get.group_list`
     :type groups: list of dictionaries
     :param owner_org: the id of the dataset's owning organization, see
         :py:func:`~ckan.logic.action.get.organization_list` or


Test patch:
Repository: ckan/ckan

Issue:
No auth function for vocabulary_list
I am trying to limit access to vocabulary list by implementing get_auth_functions. But I am getting the following exception when calling the vocabulary_list get api :  "Exception: Action function vocabulary_list did not call its auth function".   I checked logic.auth.get and I don't see any auth functions defined for vocabulary_list.



Discussion:
This raises a 500 on master.ckan.org  - `http://master.ckan.org/api/3/action/vocabulary_list`.  We should probably implement a skeleton auth function that just returns True, you could then override it in your extension. 

If you don't want an auth function, you should be able to decorate your auth function with the following code to avoid this error.

```
import ckan.logic as logic 

@logic.auth_audit_exempt
def vocabulary_list(......
```


Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -2266,6 +2266,8 @@ def vocabulary_list(context, data_dict):
     :rtype: list of dictionaries
 
     '''
+    _check_access('vocabulary_list', context, data_dict)
+
     model = context['model']
     vocabulary_objects = model.Session.query(model.Vocabulary).all()
     return model_dictize.vocabulary_list_dictize(vocabulary_objects, context)
@@ -2280,6 +2282,8 @@ def vocabulary_show(context, data_dict):
     :rtype: dictionary
 
     '''
+    _check_access('vocabulary_show', context, data_dict)
+
     model = context['model']
     vocab_id = data_dict.get('id')
     if not vocab_id:
diff --git a/ckan/logic/auth/get.py b/ckan/logic/auth/get.py
--- a/ckan/logic/auth/get.py
+++ b/ckan/logic/auth/get.py
@@ -66,6 +66,10 @@ def license_list(context, data_dict):
     # Licenses list is visible by default
     return {'success': True}
 
+def vocabulary_list(context, data_dict):
+    # List of all vocabularies are visible by default
+    return {'success': True}
+
 def tag_list(context, data_dict):
     # Tags list is visible by default
     return {'success': True}
@@ -157,6 +161,10 @@ def organization_show(context, data_dict):
     # anyone can see a organization
     return {'success': True}
 
+def vocabulary_show(context, data_dict):
+    # Allow viewing of vocabs by default
+    return {'success': True}
+
 def tag_show(context, data_dict):
     # No authz check in the logic function
     return {'success': True}


Test patch:
diff --git a/ckan/tests/test_coding_standards.py b/ckan/tests/test_coding_standards.py
--- a/ckan/tests/test_coding_standards.py
+++ b/ckan/tests/test_coding_standards.py
@@ -776,8 +776,6 @@ class TestActionAuth(object):
         'get: user_activity_list_html',
         'get: user_followee_count',
         'get: user_follower_count',
-        'get: vocabulary_list',
-        'get: vocabulary_show',
         'update: package_relationship_update_rest',
         'update: task_status_update_many',
         'update: term_translation_update_many',
Repository: ckan/ckan

Issue:
/related gives Internal Server Error
E.g. http://master.ckan.org/related

Looking at the error log, there's a  NameError: name 'request' is not defined in line 38 of related.py.

This seems to have snuck in when #2062 was merged.



Discussion:

Resolving patch:
diff --git a/ckan/controllers/related.py b/ckan/controllers/related.py
--- a/ckan/controllers/related.py
+++ b/ckan/controllers/related.py
@@ -35,7 +35,7 @@ def dashboard(self):
         params_nopage = [(k, v) for k, v in base.request.params.items()
                          if k != 'page']
 
-        page = self._get_page_number(request.params)
+        page = self._get_page_number(base.request.params)
 
         # Update ordering in the context
         related_list = logic.get_action('related_list')(context, data_dict)


Test patch:
Repository: ckan/ckan

Issue:
resource proxy fails when server can't cope with the headers only request
I want to proxy a WMS service:
http://geodienste-hamburg.de/HH_WMS_BWVI_opendata?REQUEST=GetCapabilities&SERVICE=WMS

This link works (GET). But making a HEAD request results in the failure:
400 Invalid method in request

This is, in and of itself, not a CKAN problem. The problem is that the resource proxy gives up if this type of request is not supported:
https://github.com/ckan/ckan/blob/master/ckanext/resourceproxy/controller.py#L46

Is this intended behaviour? The comment on https://github.com/ckan/ckan/blob/master/ckanext/resourceproxy/controller.py#L40 implies (to me) not.



Discussion:
That sounds sensible @mattfullerton do you want to send a patch that does a GET request if the HEAD fails?

will do


Resolving patch:
diff --git a/ckanext/resourceproxy/controller.py b/ckanext/resourceproxy/controller.py
--- a/ckanext/resourceproxy/controller.py
+++ b/ckanext/resourceproxy/controller.py
@@ -40,7 +40,9 @@ def proxy_resource(context, data_dict):
         # first we try a HEAD request which may not be supported
         did_get = False
         r = requests.head(url)
-        if r.status_code == 405:
+        # 405 would be the appropriate response here, but 400 with
+        # the invalid method mentioned in the text is also possible (#2412)
+        if r.status_code in (400, 405):
             r = requests.get(url, stream=True)
             did_get = True
         r.raise_for_status()


Test patch:
Repository: ckan/ckan

Issue:
500 Error on view_form_filters.html with some utf-8 files with BOM 
We get an internal server error when loading the "edit view" page (in our case to delete a view that was created).

Based on the server log, the error is coming from view_form_filters.html, the template/feature used to add predefined filters in the view.

The specific issue seem to be with a special character 

```
 [remote 172.17.42.1:25449] File '/project/ckan/ckan/templates/package/snippets/
 [remote 172.17.42.1:25449]   data-module-template-filter-inputs = "{{ h.
 [remote 172.17.42.1:25449] File '/project/ckan/ckan/lib/helpers.py', line 1806 in 
 [remote 172.17.42.1:25449]   return str(string).replace('\\n', '')
 [remote 172.17.42.1:25449] UnicodeEncodeError: 'ascii' codec can't encode character u'\\ufeff' in position 851: ordinal not in range(128)
```

(Full log available here: http://pastebin.com/uSaTUhjZ)

After playing with the files, it seems that the error appears on some utf-8 encoded files with BOM. I shared 2 files (https://drive.google.com/folderview?id=0B7gAVbY0-eHrflZ6a0dUbVNEVzZKNkJDdFZCN2t0ak9YdWs0UzJHSUR6RUJDOGN3X1lVbm8&usp=sharing), one with BOM that triggers the error, another without BOM that works fine.



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1832,7 +1832,7 @@ def resource_view_full_page(resource_view):
 
 def remove_linebreaks(string):
     '''Remove linebreaks from string to make it usable in JavaScript'''
-    return str(string).replace('\n', '')
+    return unicode(string).replace('\n', '')
 
 
 def list_dict_filter(list_, search_field, output_field, value):


Test patch:
diff --git a/ckan/tests/lib/test_helpers.py b/ckan/tests/lib/test_helpers.py
--- a/ckan/tests/lib/test_helpers.py
+++ b/ckan/tests/lib/test_helpers.py
@@ -91,13 +91,13 @@ def test_remove_linebreaks_removes_linebreaks(self):
         assert result.find('\n') == -1,\
             '"remove_linebreaks" should remove line breaks'
 
-    def test_remove_linebreaks_casts_into_str(self):
-        class StringLike(str):
+    def test_remove_linebreaks_casts_into_unicode(self):
+        class UnicodeLike(unicode):
             pass
 
-        test_string = StringLike('foo')
+        test_string = UnicodeLike('foo')
         result = h.remove_linebreaks(test_string)
 
-        strType = ''.__class__
+        strType = u''.__class__
         assert result.__class__ == strType,\
-            '"remove_linebreaks" casts into str()'
+            '"remove_linebreaks" casts into unicode()'
Repository: ckan/ckan

Issue:
Include the main.debug.css
Hi, I'm new to CKAN in my organization and turned debug to true for development and encountered the `AttributeError: 'module' object has no attribute 'css/main.debug.css'` error. It took me a while to figure out that I had to compile the less to get it.

Wouldn't it be easier to include this file so that debug mode automatically works without needing to change anything else?



Discussion:
I don't see any difference between the main.css and the main.debug.css that I generate from the less files. Should there be a difference? If not, why do we have two versions at all?

if it is the same then let's kill it

@Laurent we agreed at the dev meeting that we're better off without the .debug versions. Do you have time to contribute a PR that removes this?

Sure!


Resolving patch:
diff --git a/ckan/lib/app_globals.py b/ckan/lib/app_globals.py
--- a/ckan/lib/app_globals.py
+++ b/ckan/lib/app_globals.py
@@ -73,13 +73,9 @@
 _CONFIG_CACHE = {}
 
 def set_main_css(css_file):
-    ''' Sets the main_css using debug css if needed.  The css_file
-    must be of the form file.css '''
+    ''' Sets the main_css.  The css_file must be of the form file.css '''
     assert css_file.endswith('.css')
-    if config.get('debug') and css_file == '/base/css/main.css':
-        new_css = '/base/css/main.debug.css'
-    else:
-        new_css = css_file
+    new_css = css_file
     # FIXME we should check the css file exists
     app_globals.main_css = str(new_css)
 


Test patch:
Repository: ckan/ckan

Issue:
Datasets not shown in the featured organization snippet and on the Organizations page
After updating to 2.4.0, I've noticed that the datasets are not shown in the featured organization snippet. Also, when visiting "Organizations" page, organizations are listed but for each one it says 0 datasets. 

I can see that the same happens in the live demo (http://demo.ckan.org/).



Discussion:

Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1921,7 +1921,8 @@ def get_group(id):
         context = {'ignore_auth': True,
                    'limits': {'packages': 2},
                    'for_view': True}
-        data_dict = {'id': id}
+        data_dict = {'id': id,
+                     'include_datasets': True}
 
         try:
             out = logic.get_action(get_action)(context, data_dict)


Test patch:
Repository: ckan/ckan

Issue:
Fileupload removes underscores from filenames
There doesn't seem to be any reason to forbid underscores in filenames, especially since munge allows them elsewhere. Looks like someone just forgot to add an underscore to the replace function. Have a PR all ready to go with fix and updated test, just need to know what branch to submit it to ;)



Discussion:

Resolving patch:
diff --git a/ckan/lib/munge.py b/ckan/lib/munge.py
--- a/ckan/lib/munge.py
+++ b/ckan/lib/munge.py
@@ -139,7 +139,7 @@ def munge_filename(filename):
     # clean up
     filename = substitute_ascii_equivalents(filename)
     filename = filename.lower().strip()
-    filename = re.sub(r'[^a-zA-Z0-9. -]', '', filename).replace(' ', '-')
+    filename = re.sub(r'[^a-zA-Z0-9_. -]', '', filename).replace(' ', '-')
     # resize if needed but keep extension
     name, ext = os.path.splitext(filename)
     # limit overly long extensions


Test patch:
diff --git a/ckan/tests/lib/test_munge.py b/ckan/tests/lib/test_munge.py
--- a/ckan/tests/lib/test_munge.py
+++ b/ckan/tests/lib/test_munge.py
@@ -48,8 +48,9 @@ class TestMungeFilename(object):
         ('random:other%character&', 'randomothercharacter'),
         (u'u with umlaut \xfc', 'u-with-umlaut-u'),
         ('2014-11-10 12:24:05.340603my_image.jpeg',
-         '2014-11-10-122405.340603myimage.jpeg'),
+         '2014-11-10-122405.340603my_image.jpeg'),
         ('file.csv', 'file.csv'),
+        ('underscores_are_awesome', 'underscores_are_awesome'),
         ('f' * 100 + '.csv', 'f' * 96 + '.csv'),
         ('path/to/file.csv', 'file.csv'),
         ('.longextension', '.longextension'),
Repository: ckan/ckan

Issue:
Unicode decode error when creating an extension
I have a weird name. It has a weird letter: _ö_ in it. Trying to create an extension:

```
paster --plugin=ckan create -t ckanext ckanext-example
```

and providing my real name when asked for it ends up with a `UnicodeDecodeError`:

```
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)
```

This could be solved with me changing my name. But that's not going to happen for everyone with weird characters in their names.



Discussion:

Resolving patch:
diff --git a/ckan/pastertemplates/__init__.py b/ckan/pastertemplates/__init__.py
--- a/ckan/pastertemplates/__init__.py
+++ b/ckan/pastertemplates/__init__.py
@@ -51,6 +51,11 @@ class CkanextTemplate(Template):
     def check_vars(self, vars, cmd):
         vars = Template.check_vars(self, vars, cmd)
 
+        # workaround for a paster issue https://github.com/ckan/ckan/issues/2636
+        # this is only used from a short-lived paster command
+        reload(sys)
+        sys.setdefaultencoding('utf-8')
+
         if not vars['project'].startswith('ckanext-'):
             print "\nError: Project name must start with 'ckanext-'"
             sys.exit(1)
@@ -63,7 +68,7 @@ def check_vars(self, vars, cmd):
         keywords = [keyword for keyword in keywords
                     if keyword not in ('ckan', 'CKAN')]
         keywords.insert(0, 'CKAN')
-        vars['keywords'] = ' '.join(keywords)
+        vars['keywords'] = u' '.join(keywords)
 
         # For an extension named ckanext-example we want a plugin class
         # named ExamplePlugin.


Test patch:
Repository: ckan/ckan

Issue:
title munger allows repeated dashes
If you munge title `Flood    data` then those three spaces get converted to 2 dashes: `flood--data`, when it should really just be one `flood-data`.

There are hacks in several harvesters to fix this, but it's better fixed at source in core ckan.



Discussion:

Resolving patch:
diff --git a/ckan/lib/munge.py b/ckan/lib/munge.py
--- a/ckan/lib/munge.py
+++ b/ckan/lib/munge.py
@@ -34,7 +34,7 @@ def munge_title_to_name(name):
     # take out not-allowed characters
     name = re.sub('[^a-zA-Z0-9-_]', '', name).lower()
     # remove doubles
-    name = re.sub('--', '-', name)
+    name = re.sub('-+', '-', name)
     # remove leading or trailing hyphens
     name = name.strip('-')
     # if longer than max_length, keep last word if a year


Test patch:
diff --git a/ckan/tests/lib/test_munge.py b/ckan/tests/lib/test_munge.py
--- a/ckan/tests/lib/test_munge.py
+++ b/ckan/tests/lib/test_munge.py
@@ -104,14 +104,14 @@ class TestMungeTitleToName(object):
     # (original, expected)
     munge_list = [
         ('unchanged', 'unchanged'),
-        ('some spaces  here', 'some-spaces-here'),
+        ('some spaces  here    &here', 'some-spaces-here-here'),
         ('s', 's_'),  # too short
         ('random:other%character&', 'random-othercharacter'),
         (u'u with umlaut \xfc', 'u-with-umlaut-u'),
         ('reallylong' * 12, 'reallylong' * 9 + 'reall'),
         ('reallylong' * 12 + ' - 2012', 'reallylong' * 9 + '-2012'),
         ('10cm - 50cm Near InfraRed (NI) Digital Aerial Photography (AfA142)',
-            '10cm--50cm-near-infrared-ni-digital-aerial-photography-afa142')
+         '10cm-50cm-near-infrared-ni-digital-aerial-photography-afa142')
     ]
 
     def test_munge_title_to_name(self):
@@ -128,7 +128,8 @@ class TestMungeTag:
         ('unchanged', 'unchanged'),
         ('s', 's_'),  # too short
         ('some spaces  here', 'some-spaces--here'),
-        ('random:other%character&', 'randomothercharacter')
+        ('random:other%characters&_.here', 'randomothercharactershere'),
+        ('river-water-dashes', 'river-water-dashes'),
     ]
 
     def test_munge_tag(self):
@@ -137,7 +138,7 @@ def test_munge_tag(self):
             munge = munge_tag(org)
             nose_tools.assert_equal(munge, exp)
 
-    def test_munge_tag_muliple_pass(self):
+    def test_munge_tag_multiple_pass(self):
         '''Munge a list of tags muliple times gives expected results.'''
         for org, exp in self.munge_list:
             first_munge = munge_tag(org)
Repository: ckan/ckan

Issue:
package.metadata_created() should be a field
Currently metadata_created is a property that calls the metadata_created function - which in turn queries PackageRevision to find the earliest one.  This means that modifying it becomes a chore (for instance, when harvesting).

This should be turned into a column on the model (similar to metadata_modified).  Will need a migration.



Discussion:
@wardi says this would help when harvesting.

Maybe it was originally like this and changed - can you look at the history?


Resolving patch:
diff --git a/ckan/migration/versions/082_add_metadata_created.py b/ckan/migration/versions/082_add_metadata_created.py
new file mode 100644
--- /dev/null
+++ b/ckan/migration/versions/082_add_metadata_created.py
@@ -0,0 +1,14 @@
+def upgrade(migrate_engine):
+    migrate_engine.execute('''
+        ALTER TABLE package_revision
+            ADD COLUMN metadata_created timestamp without time zone;
+        ALTER TABLE package
+            ADD COLUMN metadata_created timestamp without time zone;
+
+        UPDATE package SET metadata_created=
+            (SELECT revision_timestamp
+             FROM package_revision
+             WHERE id=package.id
+             ORDER BY revision_timestamp ASC
+             LIMIT 1);
+    ''')
diff --git a/ckan/model/package.py b/ckan/model/package.py
--- a/ckan/model/package.py
+++ b/ckan/model/package.py
@@ -47,6 +47,7 @@
         Column('type', types.UnicodeText, default=u'dataset'),
         Column('owner_org', types.UnicodeText),
         Column('creator_user_id', types.UnicodeText),
+        Column('metadata_created', types.DateTime, default=datetime.datetime.utcnow),
         Column('metadata_modified', types.DateTime, default=datetime.datetime.utcnow),
         Column('private', types.Boolean, default=False),
 )
@@ -481,16 +482,6 @@ def get_groups(self, group_type=None, capacity=None):
             groups = [g[0] for g in groupcaps if g[1] == capacity]
         return groups
 
-    @property
-    def metadata_created(self):
-        import ckan.model as model
-        q = meta.Session.query(model.PackageRevision.revision_timestamp)\
-            .filter(model.PackageRevision.id == self.id)\
-            .order_by(model.PackageRevision.revision_timestamp.asc())
-        ts = q.first()
-        if ts:
-            return ts[0]
-
     @staticmethod
     def get_fields(core_only=False, fields_to_ignore=None):
         '''Returns a list of the properties of a package.


Test patch:
Repository: ckan/ckan

Issue:
Package.get accidentally avoids the sqlalchemy session cache.
SQLAlchemy has a session cache containing objects that it has retrieved recently, which are stored in an identity map (pk->object).  If you call session.query(cls).get(pk) the object will be retrieved from the cache (identity map), if you call session.query(cls).filter() then Sqlalchemy _has_ to query the database again.

In Package.get() the query to find by id (before looking by name) uses filter and so will _always_ bypass the session cache.  Once upon a time it used session.query(cls).get(reference), but this was lost in a change (where eager loading was added, then removed in a later commit).  

For a contrived, and probably sub-optimal example ...

``` python
for pkg in some_query_that_fetches_packages():
    pkg_dict = logic.get_action('package_show')(context, {'id': pkg.id})   
```

In these cases package_show is issuing another query to the database, even though the package object is in the session cache ready for retrieval.  

There is more information on how the Session cache works at http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#is-the-session-a-cache



Discussion:

Resolving patch:
diff --git a/ckan/model/package.py b/ckan/model/package.py
--- a/ckan/model/package.py
+++ b/ckan/model/package.py
@@ -76,8 +76,10 @@ def search_by_name(cls, text_query):
     @classmethod
     def get(cls, reference):
         '''Returns a package object referenced by its id or name.'''
-        query = meta.Session.query(cls).filter(cls.id==reference)
-        pkg = query.first()
+        if not reference:
+            return None
+
+        pkg = meta.Session.query(cls).get(reference)
         if pkg == None:
             pkg = cls.by_name(reference)
         return pkg


Test patch:
Repository: ckan/ckan

Issue:
Opening a deleted dataset or organization logs you out
Steps to reproduce:
1. Create a dataset as user A
2. Remove the dataset
3. Log out
4. Log in as user B (regular user)
5. Open the removed dataset via address bar
6. You are now logged out at the login screen

Redirecting to login screen is OK since the user does not have sufficient privileges to view the removed dataset, but logging them out is, in my opinion, unnecessary.



Discussion:
I think that unauthenticated, or users without permission to edit, should get a 404 if they visit a package where state='deleted'.  The question is really, should we, instead of doing this check in package_show instead do it as part of check_access so that it applies to everything that has state=deleted? 

Confirmed on master. User is logged out :( 
Will investigate a PR to return 404 instead.


Resolving patch:
diff --git a/ckan/controllers/package.py b/ckan/controllers/package.py
--- a/ckan/controllers/package.py
+++ b/ckan/controllers/package.py
@@ -361,10 +361,8 @@ def read(self, id):
         try:
             c.pkg_dict = get_action('package_show')(context, data_dict)
             c.pkg = context['package']
-        except NotFound:
+        except (NotFound, NotAuthorized):
             abort(404, _('Dataset not found'))
-        except NotAuthorized:
-            abort(401, _('Unauthorized to read package %s') % id)
 
         # used by disqus plugin
         c.current_package_id = c.pkg.id


Test patch:
diff --git a/ckan/tests/legacy/functional/test_package.py b/ckan/tests/legacy/functional/test_package.py
--- a/ckan/tests/legacy/functional/test_package.py
+++ b/ckan/tests/legacy/functional/test_package.py
@@ -683,7 +683,7 @@ def teardown_class(self):
 
     def test_read(self):
         offset = url_for(controller='package', action='read', id=self.non_active_name)
-        res = self.app.get(offset, status=[302, 401])
+        res = self.app.get(offset, status=[404])
 
 
     def test_read_as_admin(self):