Repository: ckan/ckan

Issue:
Resource creation fails with IDatasetForm extra fields with names sorting before and after 'extra_validation'
Steps to reproduce:
1. Enable an IDatasetForm plugin that has extra fields with names that sort both before and after the string 'extra_validation', such as the one in https://github.com/wardi/ckan/commit/eeb42671620e166581b225dc3cf6ee32581f2fe7
2. Create a dataset (adding resources will still work if your plugin accepts its extra fields in the package_metadata_fields.html as the one above does)
3. Click Edit package, then Add resource
4. Try to save the resource

What happens:

A traceback is issued with an IndexError in ckan.lib.navl.dictization_functions.flatten()

Workaround:

I wrote a patch https://github.com/wardi/ckan/commit/fa9929877573aebc146ef4fde15048d517d4a3a0 that avoids the problem.  This fix is rather inelegant, better would be to execute the 'extra_validation' rule before or after all the other rules.  I leave that decision to the experts.



Discussion:
@wardi could you replace the links  so that they point to the specific file/patch you are referring to.  Also if you could add the traceback in a comment on this issue that would be really helpful

thanks

@tobes links updated and:

```
Module ckan.controllers.package:645 in new_resource          view
>>  get_action('resource_create')(context, data)
Module ckan.logic:324 in wrapped          view
>>  return _action(context, data_dict, **kw)
Module ckan.logic.action.create:251 in resource_create          view
>>  pkg_dict = _get_action('package_update')(context, pkg_dict)
Module ckan.logic:324 in wrapped          view
>>  return _action(context, data_dict, **kw)
Module ckan.logic.action.update:251 in package_update          view
>>  data, errors = _validate(data_dict, schema, context)
Module ckan.lib.navl.dictization_functions:228 in validate          view
>>  converted_data, errors = _validate(flattened, schema, context)
Module ckan.lib.navl.dictization_functions:289 in _validate          view
>>  convert(converter, key, converted_data, errors, context)
Module ckan.lib.navl.dictization_functions:186 in convert          view
>>  converter(key, converted_data, errors, context)
Module ckan.logic.validators:306 in duplicate_extras_key          view
>>  unflattened = unflatten(data)
Module ckan.lib.navl.dictization_functions:393 in unflatten          view
>>  current_pos = current_pos[key]
IndexError: list index out of range
```

From the mailing list http://lists.okfn.org/pipermail/ckan-dev/2013-March/004242.html

Fabian says this applies to another issue as well

```
I was also able to fix my old issue: http://lists.okfn.org/pipermail/ckan-dev/2013-January/003852.html
by adding your fix to the lib.navl.dictization_functions.py validate method:

    data_no_extras = dict(converted_data)
    if ('extras',) in converted_data:
        del data_no_extras[('extras',)]
    converted_data = unflatten(data_no_extras) 

    #converted_data = unflatten(converted_data)
```

@seanh have you any thoughts on this?

Yes, I think I need a drink

:)

The schema is a dict so I don't know how we can make it run the "extra_validation" rule before or after all the others, unless we special-case that rule, If @wardi's fix fixes this problem and doesn't break any of the tests (either the core tests or my IDatasetForm tests) then I would be tempted to take it as-is, on the grounds that `validate()` is best treated as a black box.

I want to merge my pending IDatasetForm pull request before I look at this one

@seanh Assigning this to you just so you are aware of it and in case is related to the IDatasetForm stuff you are working on now. Feel free to ignore it if not.

I think we need to get my other idatasetform pull requests merged first, then we can see if this bug still occurs and if this fix still fixes it

Confirmed this is still happening on master

Confirmed @wardi's commit fixes the problem, and all the tests (including example_idatasetform tests) still pass with this commit. I'm tempted to just take this commit but I'll have a look after lunch and see if there's a more elegant way...


Resolving patch:
diff --git a/ckan/logic/schema.py b/ckan/logic/schema.py
--- a/ckan/logic/schema.py
+++ b/ckan/logic/schema.py
@@ -116,6 +116,7 @@ def default_create_tag_schema():
 
 def default_create_package_schema():
     schema = {
+        '__before': [duplicate_extras_key, ignore],
         'id': [empty],
         'revision_id': [ignore],
         'name': [not_empty, unicode, name_validator, package_name_validator],
@@ -139,7 +140,6 @@ def default_create_package_schema():
         'tags': default_tags_schema(),
         'tag_string': [ignore_missing, tag_string_convert],
         'extras': default_extras_schema(),
-        'extras_validation': [duplicate_extras_key, ignore],
         'save': [ignore],
         'return_to': [ignore],
         'relationships_as_object': default_relationship_schema(),
diff --git a/ckan/logic/validators.py b/ckan/logic/validators.py
--- a/ckan/logic/validators.py
+++ b/ckan/logic/validators.py
@@ -313,7 +313,9 @@ def duplicate_extras_key(key, data, errors, context):
     for extra_key in set(extras_keys):
         extras_keys.remove(extra_key)
     if extras_keys:
-        errors[key].append(_('Duplicate key "%s"') % extras_keys[0])
+        key_ = ('extras_validation',)
+        assert key_ not in errors
+        errors[key_] = [_('Duplicate key "%s"') % extras_keys[0]]
 
 def group_name_validator(key, data, errors, context):
     model = context['model']


Test patch:
diff --git a/ckan/tests/logic/test_action.py b/ckan/tests/logic/test_action.py
--- a/ckan/tests/logic/test_action.py
+++ b/ckan/tests/logic/test_action.py
@@ -1132,6 +1132,40 @@ def test_42_resource_search_accessible_via_get_request(self):
             assert "index" in resource['description'].lower()
             assert "json" in resource['format'].lower()
 
+    def test_package_create_duplicate_extras_error(self):
+        import ckan.tests
+        import paste.fixture
+        import pylons.test
+
+        # Posting a dataset dict to package_create containing two extras dicts
+        # with the same key, should return a Validation Error.
+        app = paste.fixture.TestApp(pylons.test.pylonsapp)
+        error = ckan.tests.call_action_api(app, 'package_create',
+                apikey=self.sysadmin_user.apikey, status=409,
+                name='foobar', extras=[{'key': 'foo', 'value': 'bar'},
+                    {'key': 'foo', 'value': 'gar'}])
+        assert error['__type'] == 'Validation Error'
+        assert error['extras_validation'] == ['Duplicate key "foo"']
+
+    def test_package_update_duplicate_extras_error(self):
+        import ckan.tests
+        import paste.fixture
+        import pylons.test
+
+        # We need to create a package first, so that we can update it.
+        app = paste.fixture.TestApp(pylons.test.pylonsapp)
+        package = ckan.tests.call_action_api(app, 'package_create',
+                apikey=self.sysadmin_user.apikey, name='foobar')
+
+        # Posting a dataset dict to package_update containing two extras dicts
+        # with the same key, should return a Validation Error.
+        package['extras'] = [{'key': 'foo', 'value': 'bar'},
+                    {'key': 'foo', 'value': 'gar'}]
+        error = ckan.tests.call_action_api(app, 'package_update',
+                apikey=self.sysadmin_user.apikey, status=409, **package)
+        assert error['__type'] == 'Validation Error'
+        assert error['extras_validation'] == ['Duplicate key "foo"']
+
 class TestActionTermTranslation(WsgiAppCase):
 
     @classmethod
@@ -1553,5 +1587,3 @@ def test_02_bulk_delete(self):
 
         res = self.app.get('/api/action/package_search?q=*:*')
         assert json.loads(res.body)['result']['count'] == 0
-
-