internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ckan/ckan

Issue:
package_search results don't look like package_show results when using custom schemas
If we used the new validated_data_dict value in solr for the objects returned from package_search then they would match the ones returned from package_show.



Discussion:
@wardi .Happy with this.  Do you want todo it?  Only thing that would be good to add is an extra option for package_search in order to show the original (pre schema). 


Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -761,6 +761,9 @@ def package_show(context, data_dict):
 
     :param id: the id or name of the dataset
     :type id: string
+    :param use_default_schema: use default package schema instead of
+        a custom schema defined with an IDatasetForm plugin (default: False)
+    :type use_default_schema: bool
 
     :rtype: dictionary
 
@@ -778,6 +781,9 @@ def package_show(context, data_dict):
 
     _check_access('package_show', context, data_dict)
 
+    if data_dict.get('use_default_schema', False):
+        context['schema'] = ckan.logic.schema.default_show_package_schema()
+
     package_dict = None
     use_cache = (context.get('use_cache', True)
         and not 'revision_id' in context
@@ -1364,6 +1370,9 @@ def package_search(context, data_dict):
         "count", "display_name" and "name" entries.  The display_name is a
         form of the name that can be used in titles.
     :type search_facets: nested dict of dicts.
+    :param use_default_schema: use default package schema instead of
+        a custom schema defined with an IDatasetForm plugin (default: False)
+    :type use_default_schema: bool
 
     An example result: ::
 
@@ -1429,8 +1438,10 @@ def package_search(context, data_dict):
 
     results = []
     if not abort:
+        data_source = 'data_dict' if data_dict.get('use_default_schema',
+            False) else 'validated_data_dict'
         # return a list of package ids
-        data_dict['fl'] = 'id data_dict'
+        data_dict['fl'] = 'id {0}'.format(data_source)
 
         # If this query hasn't come from a controller that has set this flag
         # then we should remove any mention of capacity from the fq and
@@ -1452,7 +1463,7 @@ def package_search(context, data_dict):
 
         for package in query.results:
             # get the package object
-            package, package_dict = package['id'], package.get('data_dict')
+            package, package_dict = package['id'], package.get(data_source)
             pkg_query = session.query(model.PackageRevision)\
                 .filter(model.PackageRevision.id == package)\
                 .filter(_and_(


Test patch:
Repository: ckan/ckan

Issue:
Switch to default RTD theme.



Discussion:

Resolving patch:
diff --git a/doc/conf.py b/doc/conf.py
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -236,22 +236,11 @@ def write_latest_release_file():
 
 # Options for HTML output
 # -----------------------
-
-#html_theme = 'default'
-#html_theme_options = {
-#"relbarbgcolor": "#777",
-#'sidebarbgcolor': '#F2F2F2',
-#'sidebartextcolor': 'black',
-#'sidebarlinkcolor': '#355F7C',
-#'headfont': 'Trebuchet MS'
-#}
-sys.path.append(os.path.abspath('_themes'))
-html_theme_path = ['_themes']
-html_theme = 'sphinx-theme-okfn'
-html_theme_options = {
-        'logo_icon': 'ckanlogo.png',
-        'show_version': True
-    }
+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+if not on_rtd:
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 html_sidebars = {
     '**':  ['globaltoc.html'],


Test patch:
Repository: ckan/ckan

Issue:
"{User} has been logged in" flash message is pointless
I can see I'm logged in.



Discussion:

Resolving patch:
diff --git a/ckan/controllers/user.py b/ckan/controllers/user.py
--- a/ckan/controllers/user.py
+++ b/ckan/controllers/user.py
@@ -365,8 +365,6 @@ def logged_in(self):
 
             user_dict = get_action('user_show')(context, data_dict)
 
-            h.flash_success(_("%s is now logged in") %
-                            user_dict['display_name'])
             return self.me()
         else:
             err = _('Login failed. Bad username or password.')


Test patch:
diff --git a/ckan/tests/functional/test_user.py b/ckan/tests/functional/test_user.py
--- a/ckan/tests/functional/test_user.py
+++ b/ckan/tests/functional/test_user.py
@@ -168,7 +168,6 @@ def test_login(self):
                res.header('Location').startswith('/dashboard')
         res = res.follow()
         assert_equal(res.status, 200)
-        assert 'testlogin is now logged in' in res.body
         assert 'checkpoint:my-dashboard' in res.body
 
         # check user object created
@@ -372,7 +371,6 @@ def test_register_whilst_logged_in(self):
         while res.status == 302:
             res = res.follow()
         assert_equal(res.status, 200)
-        assert 'User B is now logged in' in res.body, res.body
 
     @search_related
     def test_home_login(self):
Repository: ckan/ckan

Issue:
Alt text for widgets
The widgets for dashboard, settings, etc have no alt text.

They seemingly are inserted with some clever hack using CSS (so in fact the images are visible in Safari or Firefox even if image loading is turned off) but loading the page in a text browser such as Lynx, they don't appear at all, so the user can't see those links.



Discussion:
Unfortunately the images here aren't actually images, they are icons rendered using fonts. As such it isn't possible to give them alt tags.

@rossjones That's lovely, but what are you supposed to do if browsing in text mode? Isn't there an accessibility issue?

Also, the user's gravatar should have `alt=""` (no alt text is needed as the name or username is also linked to the profile page).


Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -774,7 +774,7 @@ def dict_list_reduce(list_, key, unique=True):
 def linked_gravatar(email_hash, size=100, default=None):
     return literal(
         '<a href="https://gravatar.com/" target="_blank" ' +
-        'title="%s">' % _('Update your avatar at gravatar.com') +
+        'title="%s" alt="">' % _('Update your avatar at gravatar.com') +
         '%s</a>' % gravatar(email_hash, size, default)
     )
 


Test patch:
Repository: ckan/ckan

Issue:
Organization image not shown when viewing dataset
When you are viewing a dataset that has an organization the organization's profile image is not shown.

Organization.html is not passed image_display_url when used in package templates.



Discussion:
This is because the `organization` dicts in `package_show` does not have the `image_display_url` added. It's only added by the dictization functions.


Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -1870,6 +1870,15 @@ def unified_resource_format(format):
 def check_config_permission(permission):
     return new_authz.check_config_permission(permission)
 
+
+def get_organization(org=None):
+    if org is None:
+        return {}
+    try:
+        return logic.get_action('organization_show')({}, {'id': org})
+    except (NotFound, ValidationError, NotAuthorized):
+        return {}
+
 # these are the functions that will end up in `h` template helpers
 __allowed_functions__ = [
     # functions defined in ckan.lib.helpers
@@ -1957,6 +1966,7 @@ def check_config_permission(permission):
     'list_dict_filter',
     'new_activities',
     'time_ago_from_timestamp',
+    'get_organization',
     'has_more_facets',
     # imported into ckan.lib.helpers
     'literal',


Test patch:
Repository: ckan/ckan

Issue:
num_followers and package_count not in default_group_schema
When creating a custom group schema, I start from default_group_schema, but it doesn't work on its own - I have to add in the package_count and num_followers fields to it. They are used by the templates.

snippets/organization.html requires the organization.package_count and organization.num_followers. validation is not run on group_show - why not? Shouldn't it run default_group_schema?

Here's how I got my IGroupForm schema to work:

```
    def db_to_form_schema(self):
        from ckan.logic.schema import default_group_schema
        from ckan.lib.navl.validators import ignore_missing
        schema = default_group_schema()
        schema['num_followers'] = [ignore_missing]
        schema['package_count'] = [ignore_missing]
        return schema
```



Discussion:

Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -1055,10 +1055,11 @@ def _group_or_org_show(context, data_dict, is_org=False):
         {'model': model, 'session': model.Session},
         {'id': group_dict['id']})
 
-    if schema:
-        group_dict, errors = lib_plugins.plugin_validate(
-            group_plugin, context, group_dict, schema,
-            'organization_show' if is_org else 'group_show')
+    if schema is None:
+        schema = logic.schema.default_show_group_schema()
+    group_dict, errors = lib_plugins.plugin_validate(
+        group_plugin, context, group_dict, schema,
+        'organization_show' if is_org else 'group_show')
     return group_dict
 
 
diff --git a/ckan/logic/schema.py b/ckan/logic/schema.py
--- a/ckan/logic/schema.py
+++ b/ckan/logic/schema.py
@@ -326,6 +326,22 @@ def default_update_group_schema():
     schema["name"] = [ignore_missing, group_name_validator, unicode]
     return schema
 
+def default_show_group_schema():
+    schema = default_group_schema()
+
+    # make default show schema behave like when run with no validation
+    schema['num_followers'] = []
+    schema['created'] = []
+    schema['display_name'] = []
+    schema['extras'] = {'__extras': [ckan.lib.navl.validators.keep_extras]}
+    schema['package_count'] = []
+    schema['packages'] = {'__extras': [ckan.lib.navl.validators.keep_extras]}
+    schema['revision_id'] = []
+    schema['state'] = []
+    schema['users'] = {'__extras': [ckan.lib.navl.validators.keep_extras]}
+
+    return schema
+
 
 def default_related_schema():
     schema = {


Test patch:
Repository: ckan/ckan

Issue:
organization_list_for_user returns non-"active" organisations
In https://github.com/ckan/ckan/blob/master/ckan/logic/action/get.py at method organization_list_for_user the part

```
q = model.Session.query(model.Member) \
        .filter(model.Member.table_name == 'user') \
        .filter(model.Member.capacity.in_(roles)) \
        .filter(model.Member.table_id == user_id)
```

might be better written as

```
q = model.Session.query(model.Member) \
        .filter(model.Member.table_name == 'user') \
        .filter(model.Member.capacity.in_(roles)) \
        .filter(model.Member.table_id == user_id) \
        .filter(model.Member.state == 'active')
```

because it can return deleted (or other state) members.



Discussion:
Whilst looking at this I originally got confused by the similar `group_list_authz` at https://github.com/ckan/ckan/blob/master/ckan/logic/action/get.py#L427-L486 where it _does_ make sense (you want to know that you can edit deleted orgs).

I'm not sure how much value there is in organization_list_for_user  returning deleted organisations, but then I'm not sure what harm there is either.

Can you elucidate why you think it is a problem?

If you add user to some organization and then remove user from this organization. Then in database the users' state is 'deleted'.

If you now try create dataset with this user (who has deleted state) the organization listing in the dataset creation page shows the deleted organization as selection and if you try to create dataset with that organization, then you get authorization error.

I did not checked other references, but at least in there it produces unwanted result.

Yep, definitely looks like an issue.  Do you fancy submitting a PR for the fix?

Thanks, but you can take the honor.


Resolving patch:
diff --git a/ckan/logic/action/get.py b/ckan/logic/action/get.py
--- a/ckan/logic/action/get.py
+++ b/ckan/logic/action/get.py
@@ -565,7 +565,8 @@ def organization_list_for_user(context, data_dict):
         q = model.Session.query(model.Member) \
             .filter(model.Member.table_name == 'user') \
             .filter(model.Member.capacity.in_(roles)) \
-            .filter(model.Member.table_id == user_id)
+            .filter(model.Member.table_id == user_id) \
+            .filter(model.Member.state == 'active')
 
         group_ids = []
         for row in q.all():


Test patch:
Repository: ckan/ckan

Issue:
Datapusher extension throws an authorization error which it should have caught
If you visit the "DataStore" (from the edit page of the resource), it throws a traceback. Ideally, the datapusher should catch this one through a 403.



Discussion:

Resolving patch:
diff --git a/ckanext/datapusher/plugin.py b/ckanext/datapusher/plugin.py
--- a/ckanext/datapusher/plugin.py
+++ b/ckanext/datapusher/plugin.py
@@ -10,6 +10,8 @@
 import ckan.model as model
 import ckan.plugins.toolkit as toolkit
 
+from ckan.common import _
+
 log = logging.getLogger(__name__)
 _get_or_bust = logic.get_or_bust
 
@@ -61,6 +63,8 @@ def resource_data(self, id, resource_id):
             )
         except logic.NotFound:
             datapusher_status = {}
+        except logic.NotAuthorized:
+            base.abort(401, _('Not authorized to see this page'))
 
         return base.render('package/resource_data.html',
                            extra_vars={'status': datapusher_status})


Test patch:
Repository: ckan/ckan

Issue:
Remove ckan.lib.helpers.get_action()
This function is a bad idea because action functions that raise exceptions crash templates (which cannot catch exceptions). See http://lists.okfn.org/pipermail/ckan-dev/2013-August/005609.html

Instead, if you want to make an action available to a template, wrap each individual action in a template helper function that deals with any exceptions raised appropriately.

(Template helper functions should never raise exceptions.)

If `ckan.lib.helpers.get_action()` has been in a CKAN release already then it should be deprecated for one release, if not we can just delete it right away.



Discussion:
Note you could also call get_action() in your controller and pass the results to the template using extra_vars. Keeps the template simpler, maybe.

Please don't remove this helper. There are some places where this is extremely useful. We should instead recommend not using it.

It's currently only used once in the core templates. Why is it so useful?

I think I've used it in one more place where it's not merged. Precisely because that was the easiest way to fix that bug.


Resolving patch:
diff --git a/ckan/lib/helpers.py b/ckan/lib/helpers.py
--- a/ckan/lib/helpers.py
+++ b/ckan/lib/helpers.py
@@ -737,8 +737,12 @@ def check_access(action, data_dict=None):
     return authorized
 
 
+@maintain.deprecated("helpers.get_action() is deprecated and will be removed "
+                     "in a future version of CKAN. Instead, please use the "
+                     "extra_vars param to render() in your controller to pass "
+                     "results from action functions to your templates.")
 def get_action(action_name, data_dict=None):
-    '''Calls an action function from a template.'''
+    '''Calls an action function from a template. Deprecated in CKAN 2.3.'''
     if data_dict is None:
         data_dict = {}
     return logic.get_action(action_name)({}, data_dict)


Test patch:
Repository: ckan/ckan

Issue:
resource_show authentication function delegates in package_show function incorrectly
The resource_show authentication delegates in the package_show function to check if a user can view a resource. This is correct, but the problem is that the resource_show function uses directly the package_show function defined in the auth file (logic/auth/get.py) instead the one defined in the system (since a developer can change that function within a plugin). 



Discussion:
So you're suggesting that the resource_show auth function should be calling the package_show auth function via check_access, instead of calling it directly? That sounds correct to me. Are you able to send a pull request with this change?


Resolving patch:
diff --git a/ckan/logic/auth/get.py b/ckan/logic/auth/get.py
--- a/ckan/logic/auth/get.py
+++ b/ckan/logic/auth/get.py
@@ -135,7 +135,7 @@ def resource_show(context, data_dict):
         raise logic.NotFound(_('No package found for this resource, cannot check auth.'))
 
     pkg_dict = {'id': pkg.id}
-    authorized = package_show(context, pkg_dict).get('success')
+    authorized = new_authz.is_authorized('package_show', context, pkg_dict).get('success')
 
     if not authorized:
         return {'success': False, 'msg': _('User %s not authorized to read resource %s') % (user, resource.id)}


Test patch:
Repository: ckan/ckan

Issue:
DataStore Upsert operation returns an empty field list when non existing fields are included
When I've tried to upsert an object into a resource and I put a non existing field, API should return the name of the extra field. However, I get the following error:

```
"success": false,
    "error": {
        "fields": [
            "fields \"\" do not exist"
        ],
        "__type": "Validation Error"
    }
```

As can be seen, the field list does not include the non existing fields (basically this list is empty). I expect a message similar to the following one:

```
"success": false,
    "error": {
        "fields": [
            "fields \"test1, test2\" do not exist"
        ],
        "__type": "Validation Error"
    }
```

I've found a simple solution to this bug. Just go to `ckanext/datastore/db.py:633` and replace:

```
', '.join(missing_fields))]
```

by

```
', '.join(non_existing_filed_names))]
```

I've looked in the issue list and I haven't found something similar to this, but if it's already reported, you can close this issue ;) 



Discussion:
@aitormagan I have seen this too and just haven't logged it yet. Thanks for logging!


Resolving patch:
diff --git a/ckanext/datastore/db.py b/ckanext/datastore/db.py
--- a/ckanext/datastore/db.py
+++ b/ckanext/datastore/db.py
@@ -599,7 +599,7 @@ def upsert_data(context, data_dict):
             if non_existing_filed_names:
                 raise ValidationError({
                     'fields': [u'fields "{0}" do not exist'.format(
-                        ', '.join(missing_fields))]
+                        ', '.join(non_existing_filed_names))]
                 })
 
             unique_values = [record[key] for key in unique_keys]


Test patch:
Repository: ckan/ckan

Issue:
Add resource extras to Solr search index
I plan to add resource extra attributes to Solr search index, to make it possible to search datasets by these fields.

As already discussed with @wardi, I will implement this in ckan/lib/search/index.py:PackageSearchIndex. Basic logic would be:

If the field name is not in: 

ckan.model.resource.CORE_RESOURCE_COLUMNS + ['expired_id', 'revision_timestamp', 'expired_timestamp', 'current', 'state']

It is an extra field and should be indexed.
For this to work, I will create a new multivalued dynamic field in the Solr schema, with name 'res_extras_*'.



Discussion:
Hi @rparrapy 

How would you manage same extras in different resources for the same dataset?

Eg:

``` json
{
"name": "my-dataset",
"resources": [
    {"id":"x", "my-extra": "value1"},
    {"id":"y", "my-extra": "value2"},
]
}

```

Hi @amercader 

I would probably concatenate both values with spaces between them (done it before with a non-multivalued field and it worked), like this:

``` python
pkg_dict["res_extras_my-extra"] = "value1 value2"
```

Solr splits the value in words, and it ends up working.

By the way, I just found out about this config setting:
http://ckan.readthedocs.org/en/latest/maintaining/configuration.html#ckan-extra-resource-fields

I tried using it but it didn't index the extra fields I specified (not sure why it is there, then).
Anyway, do you think I should use this config and index only specified resource fields? or stick to the plan and index _all_ extra fields? 

fixed by #1732

@wardi The fix in #1732 will cause an exception if your CKAN is pointing to the previous version of the schema.xml and trying to create / update datasets (eg tests that use CreateTestData will fail. You need to increase the version number on the schema.xml file and change the supported schema versions value in `lib/search/common.py` so CKAN fails on startup and people actually know what is going on.

@wardi ping


Resolving patch:
diff --git a/ckan/lib/search/__init__.py b/ckan/lib/search/__init__.py
--- a/ckan/lib/search/__init__.py
+++ b/ckan/lib/search/__init__.py
@@ -32,7 +32,7 @@ def text_traceback():
 
 SIMPLE_SEARCH = asbool(config.get('ckan.simple_search', False))
 
-SUPPORTED_SCHEMA_VERSIONS = ['2.0']
+SUPPORTED_SCHEMA_VERSIONS = ['2.3']
 
 DEFAULT_OPTIONS = {
     'limit': 20,


Test patch:
Repository: ckan/ckan

Issue:
datastore extension fails tests when using pg 8.4
Appears to be checking for datastore values in a dict when datastore is not active.



Discussion:
Tests pass on master and have been for a while, do you have more details?

https://travis-ci.org/ckan/ckan/jobs/39177131

```
======================================================================
ERROR: ckanext.datastore.tests.test_create.TestDatastoreCreate.test_create_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/ckan/ckan/ckanext/datastore/tests/test_create.py", line 470, in test_create_basic
    assert res_dict['result']['datastore_active'] is False
KeyError: 'datastore_active'
```

This on a branch taken from master and the only changes in cli.py

It's the Python 2.6/PG 8.4 tests

This now works on a second run. Worrying, but not apparently an issue.

This just happened on master, so seems like something we need to investigate.

https://travis-ci.org/ckan/ckan/jobs/39255815

Random test failing debug, yay!

could it be a race? Might this query be failing quietly somehow? https://github.com/ckan/ckan/blob/master/ckanext/datastore/plugin.py#L276-L281 ..or the plugin not loaded?

It should definitely catch the exception rather than failing quietly. Will _table_metadata even exist if datastore is not active?


Resolving patch:
diff --git a/ckanext/datastore/plugin.py b/ckanext/datastore/plugin.py
--- a/ckanext/datastore/plugin.py
+++ b/ckanext/datastore/plugin.py
@@ -273,6 +273,9 @@ def before_show(self, resource_dict):
                 action='dump', resource_id=resource_dict['id'])
 
         connection = None
+
+        resource_dict['datastore_active'] = False
+
         try:
             connection = self.read_engine.connect()
             result = connection.execute(
@@ -281,8 +284,6 @@ def before_show(self, resource_dict):
             ).fetchone()
             if result:
                 resource_dict['datastore_active'] = True
-            else:
-                resource_dict['datastore_active'] = False
         finally:
             if connection:
                 connection.close()


Test patch: