Package: ckanext-transmute
ckanext-transmuteThe extension helps to validate and converts a dataset based on a specific schema.Working with transmuteckanext-transmuteprovides an actiontsm_transmuteIt helps us to transmute data with the provided convertion scheme. The action doesn't change the original data, but creates a new data dict. There are two mandatory arguments -dataandschema.datais a data dict you have andschemahelps you to validate/change data in it.Example:
We have a data dict:{
            "title": "Test-dataset",
            "email": "test@test.ua",
            "metadata_created": "",
            "metadata_modified": "",
            "metadata_reviewed": "",
            "resources": [
                {
                    "title": "test-res",
                    "extension": "xml",
                    "web": "https://stackoverflow.com/",
                    "sub-resources": [
                        {
                            "title": "sub-res",
                            "extension": "csv",
                            "extra": "should-be-removed",
                        }
                    ],
                },
                {
                    "title": "test-res2",
                    "extension": "csv",
                    "web": "https://stackoverflow.com/",
                },
            ],
        }And we want to achieve this:{
            "name": "test-dataset",
            "email": "test@test.ua",
            "metadata_created": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),
            "metadata_modified": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),
            "metadata_reviewed": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),
            "attachments": [
                {
                    "name": "test-res",
                    "format": "XML",
                    "url": "https://stackoverflow.com/",
                    "sub-resources": [{"name": "SUB-RES", "format": "CSV"}],
                },
                {
                    "name": "test-res2",
                    "format": "CSV",
                    "url": "https://stackoverflow.com/",
                },
            ],
        }Then, our schema must be something like that:{
        "root": "Dataset",
        "types": {
            "Dataset": {
                "fields": {
                    "title": {
                        "validators": [
                            "tsm_string_only",
                            "tsm_to_lowercase",
                            "tsm_name_validator",
                        ],
                        "map": "name",
                    },
                    "resources": {
                        "type": "Resource",
                        "multiple": True,
                        "map": "attachments",
                    },
                    "metadata_created": {
                        "validators": ["tsm_isodate"],
                        "default": "2022-02-03T15:54:26.359453",
                    },
                    "metadata_modified": {
                        "validators": ["tsm_isodate"],
                        "default_from": "metadata_created",
                    },
                    "metadata_reviewed": {
                        "validators": ["tsm_isodate"],
                        "replace_from": "metadata_modified",
                    },
                }
            },
            "Resource": {
                "fields": {
                    "title": {
                        "validators": ["tsm_string_only"],
                        "map": "name",
                    },
                    "extension": {
                        "validators": ["tsm_string_only", "tsm_to_uppercase"],
                        "map": "format",
                    },
                    "web": {
                        "validators": ["tsm_string_only"],
                        "map": "url",
                    },
                    "sub-resources": {
                        "type": "Sub-Resource",
                        "multiple": True,
                    },
                },
            },
            "Sub-Resource": {
                "fields": {
                    "title": {
                        "validators": ["tsm_string_only", "tsm_to_uppercase"],
                        "map": "name",
                    },
                    "extension": {
                        "validators": ["tsm_string_only", "tsm_to_uppercase"],
                        "map": "format",
                    },
                    "extra": {
                        "remove": True,
                    },
                }
            },
        },
    }There is an example of schema with nested types. Therootfield is mandatory, it's must contain a main type name, from which the scheme starts. As you can see,Datasettype containsResourcetype which contansSub-Resource.TransmutatorsThere are few default transmutators you can use in your schema. Of course, you can define a custom transmutator with theITransmuteinterface.tsm_name_validator- Wrapper over CKAN defaultname_validatorvalidatortsm_to_lowercase- Casts string value to a lowercasetsm_to_uppercase- Casts string value to a uppercasetsm_string_only- Validates iffield.valueis stringtsm_isodate- Validates datetime string. Mutates an iso-like string to datetime objecttsm_to_string- Casts afield.valuetostrtsm_get_nested- Allows you to pick up a value from a nested structure. Example:data = "title_translated": [
    {"nested_field": {"en": "en title", "ar": "العنوان ar"}},
]

schema = ...
    "title": {
        "replace_from": "title_translated",
        "validators": [
            ["tsm_get_nested", 0, "nested_field", "en"],
            "tsm_to_uppercase",
        ],
    },
    ...This will take a value for atitlefield fromtitle_translatedfield. Becausetitle_translatedis an array with nested objects, we are using thetsm_get_nestedtransmutator to achieve the value from it.tsm_trim_string- Trim string with max lenght. Example to trimhello worldtohello:data = {"field_name": "hello world}

schema = ...
    "field_name": {
        "validators": [
            ["tsm_trim_string", 5]
        ],
    },
    ...tsm_concat- Trim string with max lenght. Use$selfto point on field value. Example:data = {"id": "dataset-1}

schema = ...
    "package_url": {
        "replace_from": "id",
        "validators": [
            [
                "tsm_concat",
                "https://site.url/dataset/",
                "$self",
            ]
        ],
    },
    ...tsm_unique_only- Preserve only unique values from a list. Works only with lists.The default transmutator must receive at least one mandatory argument -fieldobject. Field contains few properties:field_name,valueandtype.There is a possibility to provide more arguments to a validator like intsm_get_nested. For this use a nested array with first item transmutator and other - arguments to it.Keywordsmap_to(str) - changes thefield.namein result dict.validators(list[str]) - a list of transmutators that will be applied to afield.value. A transmutator could be astringor alistwhere the first item must be transmutator name and others are arbitrary values. Example:...
"validators": [
    ["tsm_get_nested", "nested_field", "en"],
    "tsm_to_uppercase",
,
...There are two transmutators:tsm_get_nestedandtsm_to_uppercase.multiple(bool, default:False) - if the field could have multiple items, e.gresourcesfield in dataset, mark it asmultipleto transmute all the items successively....
"resources": {
    "type": "Resource",
    "multiple": True
},
...remove(bool, default:False) - removes a field from a result dict ifTrue.default(Any) - the default value that will be used if the original field.value evaluates toFalse.default_from(str|list) - acts similar todefaultbut accepts afield.nameof a sibling field from which we want to take its value. Sibling field is a field that located in the sametype. The current implementation doesn't allow to point on fields from othertypes. Could take a string that represents thefield.nameor an array of strings, to use multiple fields. Seeinherit_modekeyword for details....
"metadata_modified": {
    "validators": ["tsm_isodate"],
    "default_from": "metadata_created",
},
...replace_from(str|list) - acts similar todefault_frombut replaces the origin value whenever it's empty or not.inherit_mode(str, default:combine) - defines the mode fordefault_fromandreplace_from. By default we are combining values
from all the fields, but we could just use first non-false value, in case if the field might be empty.value(Any) - a value that will be used for a field. This keyword has the highest priority. Could be used to create a new field with an arbitrary value.update(bool, default:False) - if the original value is mutable (array, object`) - you can update it. You can only update field values of the same types.InstallationTo install ckanext-transmute:Activate your CKAN virtual environment, for example:. /usr/lib/ckan/default/bin/activateClone the source and install it on the virtualenvgit clonehttps://github.com/mutantsan/ckanext-transmute.gitcd ckanext-transmute
pip install -e .
pip install -r requirements.txtAddtransmuteto theckan.pluginssetting in your CKAN
config file (by default the config file is located at/etc/ckan/default/ckan.ini).Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:sudo service apache2 reloadDeveloper installationTo install ckanext-transmute for development, activate your CKAN virtualenv and
do:git clone https://github.com/mutantsan/ckanext-transmute.git
cd ckanext-transmute
python setup.py develop
pip install -r dev-requirements.txtTestsI've used TDD to write this extension, so if you changing something be sure that all the tests are valid. To run the tests, do:pytest --ckan-ini=test.iniLicenseAGPL