Package: ckan-editor-utils
IntroductionThis library assists CKAN editors with doing batch edits and pairs well with a library like pandas.Installationpipinstallckan-editor-utilsTherequestspackage is used for all the underlying API calls.Theboto3AWS SDK package is used for accessing and uploading files from S3.Features and examples>>>importckan_editor_utilsSimple API commandsFor the basic API commands, much of therequestsboilerplate code is done for you.
However, the URL must already have a suffix like/api/action/.>>>url='https://horizon.uat.gsq.digital/api/action/'>>>api_key=os.environ.get('CKAN_API_KEY')>>>dataset_id='my-test-dataset'>>>res_create=ckan_editor_utils.package_create(...url,...api_key,...{...'name':dataset_id,'extra:identifier':dataset_id,...'notes':'my description',...'owner_org':'geological-survey-of-queensland',...})>>>res_create<Response[200]># This requests response can be viewed by using the .text or .json() methods>>>res_create.json(){"help":"https://uat-external.dnrme-qld.links.com.au/api/3/action/help_show?name=package_create","success":true,"result":{...The response text shows the entire package as CKAN has recorded it. It will populate additional items like
the Organisation description automatically.We can usepackage_showto get the metadata for an existing dataset:>>>res_show=ckan_editor_utils.package_show(url,api_key,dataset_id)>>>res_show.json(){'help':'https://uat-external.dnrme-qld.links.com.au/api/3/action/help_show?name=package_show','success':True,'result':{'extra:theme':[]...Always check the HTTP status before interacting with the data payload.
For example, a 409 code will be received if it already exists or if
we did not provide enough information for the type of dataset we want to be created, among other reasons.
If we have the dataset ID wrong, we will get a 404 from CKAN. This is the defaultrequestsresponse:>>>res_missing=ckan_editor_utils.package_show(url,api_key,'missingdataset')>>>res_missing<Response[404]>>>>res_missing.json(){'help':'https://uat-external.dnrme-qld.links.com.au/api/3/action/help_show?name=package_show','success':False,'error':{'message':'Not found','__type':'Not Found Error'}}The next section helps simplify the response using aCKANResponseobject, which is particularly useful when errors occur.More examples of basic API usage can be foundat the GSQ Open Data API GitHub page,
and the official documentation page atdocs.ckan.orgSimplified CKAN ResponsesWhen interacting with the CKAN API, it can be difficult to get a consistent result. Some errors are text not JSON, and
the JSON errors sometimes contain different attributes depending on the context.
Managing the variety of these responses means a lot of extra logic is needed, which clutters up your script.This library offers a newCKANReponseobject that can convertrequestsresponses from CKAN into something
more consistent and manageable. To use it, simply pass it a CKAN response you received when usingrequests.>>>check_res_show=ckan_editor_utils.CKANResponse(res_show)# (response from earlier example)>>>print(check_res_show)Response200OK>>>check_res_show.okTrue>>>check_res_show.result{'extra:theme':[],'license_title':None,'maintainer':None,...A JSON response will always be present in theresultattribute of the CKAN response.
This means you can reliably useresultto capture output and it will always be relevant.
Furthermore the API action made will be logged to stdout/the console, so you can easily track progress.Continuing the 404 example from above, the response can be changed to something easier to manage:>>>cr=ckan_editor_utils.CKANResponse(res_missing)Response404notOK:{"message":"Not found"}>>>cr.result{'message':'Not found'}>>>cr.okFalseThese simplified CKAN responses are included in the managed actions described in the next section.Managed API actionsSome common workflows have been developed and make it easier to do simple actions.RequirementsA sysadmin user API key is requiredThe CKAN instance must have the latestckanext-cloudstorageplugin installedIntroductionThe following managed actions are available via theCKANEditorSessioncontext manager class:put_dataset (create or update)delete_dataset (delete and purge)put_resource_from_s3 (automatically does multipart uploads)Additionally, theCKANEditorSessionwill fix up the provided CKAN URL if it is missing the requiredapi/action/path.withckan_editor_utils.CKANEditorSession(url,api_key)asckaneu:returnckaneu.delete_dataset(dataset_id).resultHere we are able to get theresultattribute without any extra logic or coding because the response object has been simplified.Adding a dataset using put_dataset()As an editor doing bulk changes, you might not be sure if every package already exists before you can safely
callpackage_update(). Instead, you can just callput_dataset(), and the managed session will either create or
update the dataset depending on what it finds.data={'name':'ds000001','extra:identifier':'DS000001','notes':'Some description about this dataset.','owner_org':'my-organisation-lowercase-with-dashes',# include any other required and known fields}withckan_editor_utils.CKANEditorSession(url,api_key)asckaneu:res=ckaneu.put_dataset(data,skip_existing=True)print(res.result)Includingskip_existing=Truemeans if a dataset exists, it will not be modified.
PassingFalsewill update the existing dataset with any attributes you pass in, leaving all others intact.Adding a resource from S3 using put_resource_from_s3()This tool helps you upload a data object located in S3 to CKAN. The following fields are required:resource={'name':'ds000001','resource:name':'My New Resource to Share','resource:description':'Some description about the particular resource.'}The size and format are automatically calculated for you. We useresource:namebecause a common workflow is to load
data from a CSV that includes both the datasetnameand the resourcename, which have the same label in CKAN.You also need ans3_pathvalue to pass in, like so:'s3_path'='s3://mybucket/myprefix/myfile1.zip'Then you can call the function using the same context manager session:withckan_editor_utils.CKANEditorSession(url,api_key)asckaneu:res=ckaneu.put_resource_from_s3(resource,s3_path,skip_existing=True)print(res.result)Includingskip_existing=Truemeans if a resource exists, it will not be modified.
PassingFalsewill update the existing resource with any attributes and data objects you pass in, leaving all others intact.