Repository: ckan/ckan

Issue:
validation error on resource subfields (regression) in 2.9
2.9.4, master

**Describe the bug**
ckanext-scheming demonstrates the failure in its tests https://github.com/ckan/ckanext-scheming/actions/runs/1464096741

**Steps to reproduce**
use ckan 2.9.4 or master branch with a ckanext-scheming dataset schema containing resource subfields

**Expected behavior**
not failing

**Additional details**
This is the traceback of the validation failure, see the form failures in https://github.com/ckan/ckanext-scheming/actions/runs/1464096741 as well
```python
ckanext/scheming/tests/test_validation.py:913: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/site-packages/ckanapi/common.py:51: in action
    return self._ckan.call_action(name, data_dict=kwargs)
/usr/lib/python2.7/site-packages/ckanapi/localckan.py:72: in call_action
    return self._get_action(action)(context, data_dict)
/srv/app/src/ckan/ckan/logic/__init__.py:504: in wrapped
    result = _action(context, data_dict, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

context = {'__auth_audit': [], '__auth_user_obj_checked': True, 'auth_user_obj': <User id=0aa7be9e-0963-44ee-af27-70a04fbf504a n...age_url=None plugin_extras=None>, 'model': <module 'ckan.model' from '/srv/app/src/ckan/ckan/model/__init__.pyc'>, ...}
data_dict = {'name': 'c_sf_1', 'resources': [{'schedule': [{'frequency': '1m', 'impact': 'A'}, {'frequency': '7d', 'impact': 'P'}], 'url': 'http://example.com/data.csv'}], 'type': 'test-subfields'}

    def package_create(context, data_dict):
        '''Create a new dataset (package).
    
        You must be authorized to create new datasets. If you specify any groups
        for the new dataset, you must also be authorized to edit these groups.
    
        Plugins may change the parameters of this function depending on the value
        of the ``type`` parameter, see the
        :py:class:`~ckan.plugins.interfaces.IDatasetForm` plugin interface.
    
        :param name: the name of the new dataset, must be between 2 and 100
            characters long and contain only lowercase alphanumeric characters,
            ``-`` and ``_``, e.g. ``'warandpeace'``
        :type name: string
        :param title: the title of the dataset (optional, default: same as
            ``name``)
        :type title: string
        :param private: If ``True`` creates a private dataset
        :type private: bool
        :param author: the name of the dataset's author (optional)
        :type author: string
        :param author_email: the email address of the dataset's author (optional)
        :type author_email: string
        :param maintainer: the name of the dataset's maintainer (optional)
        :type maintainer: string
        :param maintainer_email: the email address of the dataset's maintainer
            (optional)
        :type maintainer_email: string
        :param license_id: the id of the dataset's license, see
            :py:func:`~ckan.logic.action.get.license_list` for available values
            (optional)
        :type license_id: license id string
        :param notes: a description of the dataset (optional)
        :type notes: string
        :param url: a URL for the dataset's source (optional)
        :type url: string
        :param version: (optional)
        :type version: string, no longer than 100 characters
        :param state: the current state of the dataset, e.g. ``'active'`` or
            ``'deleted'``, only active datasets show up in search results and
            other lists of datasets, this parameter will be ignored if you are not
            authorized to change the state of the dataset (optional, default:
            ``'active'``)
        :type state: string
        :param type: the type of the dataset (optional),
            :py:class:`~ckan.plugins.interfaces.IDatasetForm` plugins
            associate themselves with different dataset types and provide custom
            dataset handling behaviour for these types
        :type type: string
        :param resources: the dataset's resources, see
            :py:func:`resource_create` for the format of resource dictionaries
            (optional)
        :type resources: list of resource dictionaries
        :param tags: the dataset's tags, see :py:func:`tag_create` for the format
            of tag dictionaries (optional)
        :type tags: list of tag dictionaries
        :param extras: the dataset's extras (optional), extras are arbitrary
            (key: value) metadata items that can be added to datasets, each extra
            dictionary should have keys ``'key'`` (a string), ``'value'`` (a
            string)
        :type extras: list of dataset extra dictionaries
        :param relationships_as_object: see :py:func:`package_relationship_create`
            for the format of relationship dictionaries (optional)
        :type relationships_as_object: list of relationship dictionaries
        :param relationships_as_subject: see :py:func:`package_relationship_create`
            for the format of relationship dictionaries (optional)
        :type relationships_as_subject: list of relationship dictionaries
        :param groups: the groups to which the dataset belongs (optional), each
            group dictionary should have one or more of the following keys which
            identify an existing group:
            ``'id'`` (the id of the group, string), or ``'name'`` (the name of the
            group, string),  to see which groups exist
            call :py:func:`~ckan.logic.action.get.group_list`
        :type groups: list of dictionaries
        :param owner_org: the id of the dataset's owning organization, see
            :py:func:`~ckan.logic.action.get.organization_list` or
            :py:func:`~ckan.logic.action.get.organization_list_for_user` for
            available values. This parameter can be made optional if the config
            option :ref:`ckan.auth.create_unowned_dataset` is set to ``True``.
        :type owner_org: string
    
        :returns: the newly created dataset (unless 'return_id_only' is set to True
                  in the context, in which case just the dataset id will
                  be returned)
        :rtype: dictionary
    
        '''
        model = context['model']
        session = context['session']
        user = context['user']
    
        if 'type' not in data_dict:
            package_plugin = lib_plugins.lookup_package_plugin()
            try:
                # use first type as default if user didn't provide type
                package_type = package_plugin.package_types()[0]
            except (AttributeError, IndexError):
                package_type = 'dataset'
                # in case a 'dataset' plugin was registered w/o fallback
                package_plugin = lib_plugins.lookup_package_plugin(package_type)
            data_dict['type'] = package_type
        else:
            package_plugin = lib_plugins.lookup_package_plugin(data_dict['type'])
    
        if 'schema' in context:
            schema = context['schema']
        else:
            schema = package_plugin.create_package_schema()
    
        _check_access('package_create', context, data_dict)
    
        if 'api_version' not in context:
            # check_data_dict() is deprecated. If the package_plugin has a
            # check_data_dict() we'll call it, if it doesn't have the method we'll
            # do nothing.
            check_data_dict = getattr(package_plugin, 'check_data_dict', None)
            if check_data_dict:
                try:
                    check_data_dict(data_dict, schema)
                except TypeError:
                    # Old plugins do not support passing the schema so we need
                    # to ensure they still work
                    package_plugin.check_data_dict(data_dict)
    
        data, errors = lib_plugins.plugin_validate(
            package_plugin, context, data_dict, schema, 'package_create')
        log.debug('package_create validate_errs=%r user=%s package=%s data=%r',
                  errors, context.get('user'),
                  data.get('name'), data_dict)
    
        if errors:
            model.Session.rollback()
>           raise ValidationError(errors)
E           ValidationError: None - {'resources': [{'schedule': [{}, {}]}]}

/srv/app/src/ckan/ckan/logic/action/create.py:182: ValidationError
```


Discussion:
hey @amercader , @wardi , I found what causes the error... 

[https://github.com/ckan/ckan/blob/ace6850ddabb1bc533c6942cc434e806c62236e5/ckan/lib/navl/dictization_functions.py#L290](url)
Here in this function somehow after all iteration are over, in `errors_unflattened` those items `{'citation': [{}], 'contact_address': [{}], 'resources': [{'schedule': [{}, {}, {}]}, {}]}` still lives... then we are calling `_remove_blank_keys()` and inside this func: `citation` and `contact_address`  has no `value` so they are passing the validation: `if not any(value)` but not the `resources` because its `value` is `[{'schedule': [{}, {}, {}]}, {}]}` and is not empty so it didnt pass the validation .. and we are getting the error.

Should i modify `_remove_blank_keys()` and check `if key == 'resources'` and `if not any(value[0]['schedule'])` then remove the `key`?
Or if you have a better solution please let me know.
Thanks

Resolving patch:
diff --git a/ckan/config/declaration/load.py b/ckan/config/declaration/load.py
--- a/ckan/config/declaration/load.py
+++ b/ckan/config/declaration/load.py
@@ -83,9 +83,7 @@ def load_dict(declaration: "Declaration", definition: DeclarationDict):
     if version == 1:
 
         data, errors = validate(definition, config_declaration_v1())
-        if any(
-            options for item in errors["groups"] for options in item["options"]
-        ):
+        if errors:
             raise ValidationError(errors)
         for group in data["groups"]:
             if group["annotation"]:
diff --git a/ckan/lib/navl/dictization_functions.py b/ckan/lib/navl/dictization_functions.py
--- a/ckan/lib/navl/dictization_functions.py
+++ b/ckan/lib/navl/dictization_functions.py
@@ -247,16 +247,14 @@ def convert(converter, key, converted_data, errors, context):
 
 def _remove_blank_keys(schema):
 
-    schema_copy = copy.copy(schema)
-
-    for key, value in schema.items():
+    for key, value in list(schema.items()):
         if isinstance(value[0], dict):
             for item in value:
                 _remove_blank_keys(item)
             if not any(value):
-                schema_copy.pop(key)
+                schema.pop(key)
 
-    return schema_copy
+    return schema
 
 
 def validate(data, schema, context=None):
@@ -297,7 +295,7 @@ def validate(data, schema, context=None):
             if isinstance(value[0], dict):
                 dicts_to_process.extend(value)
 
-    errors_unflattened = _remove_blank_keys(errors_unflattened)
+    _remove_blank_keys(errors_unflattened)
 
     return converted_data, errors_unflattened
 


Test patch: