Package: ckanext-versioned-datastore
ckanext-versioned-datastoreA CKAN extension providing a versioned datastore using MongoDB and ElasticsearchOverviewThis plugin provides a complete replacement for ckan's datastore plugin and therefore shouldn't be used in conjunction with it.
Rather than storing data in PostgreSQL, resource data is stored in MongoDB and then made available to frontend APIs using Elasticsearch.This allows this plugin to:provide full versioning of resource records - records can be updated when new resource data is uploaded without preventing access to the old dataexpose advanced search features using Elasticsearch's extensive feature setachieve fast search response times, particularly when compared to PostgreSQL, due Elasticsearch's search performancestore large resources (millions of rows) and still provide high speed search responsesstore complex data as both MongoDB and Elasticsearch are JSON based, allowing object nesting and arraysThis plugin is built onSplitgill.InstallationPath variables used below:$INSTALL_FOLDER(i.e. where CKAN is installed), e.g./usr/lib/ckan/default$CONFIG_FILE, e.g./etc/ckan/default/development.iniInstalling from PyPIpipinstallckanext-versioned-datastoreInstalling from sourceClone the repository into thesrcfolder:cd$INSTALL_FOLDER/src
gitclonehttps://github.com/NaturalHistoryMuseum/ckanext-versioned-datastore.gitActivate the virtual env:.$INSTALL_FOLDER/bin/activateInstall via pip:pipinstall$INSTALL_FOLDER/src/ckanext-versioned-datastoreInstalling in editable modeInstalling from apyproject.tomlin editable mode (i.e.pip install -e) requiressetuptools>=64; however, CKAN 2.9 requiressetuptools==44.1.0. Seeour CKAN forkfor a version of v2.9 that uses an updated setuptools if this functionality is something you need.Post-install setupAdd 'versioned_datastore' to the list of plugins in your$CONFIG_FILE:ckan.plugins=... versioned_datastoreInstalllesscglobally:npminstall-g"less@~4.1"Other requirementsAt the version ofsplitgillthis plugin uses, you will also need to install:MongoDB 4.xElasticsearch 6.7.x (6.x is probably ok, but untested)This plugin also requires CKAN's job queue, which is included in recent versions of CKAN or can be added to older versions using the ckanext-rq plugin.ConfigurationThere are a number of options that can be specified in your .ini config file.
All configuration options are currently required.[REQUIRED]NameDescriptionExampleckanext.versioned_datastore.elasticsearch_hostsA comma separated list of elasticsearch server hosts1.2.3.4,1.5.4.3,es.mydomain.localckanext.versioned_datastore.elasticsearch_portThe port for to use for the elasticsearch server hosts listed in the elasticsearch_hosts option9200ckanext.versioned_datastore.elasticsearch_index_prefixThe prefix to use for index names in elasticsearch. Each resource in the datastore gets an index and the name of the index is the resource ID with this prefix prepended.nhm-ckanext.versioned_datastore.mongo_hostThe mongo server host10.54.24.10ckanext.versioned_datastore.mongo_portThe port to use to connect to the mongo host27017ckanext.versioned_datastore.mongo_databaseThe name of the mongo database to use to store datastore data innhm[OPTIONAL]NameDescriptionExampleckanext.versioned_datastore.redis_hostThe redis server host. If this is provided slugging is enabled14.1.214.50ckanext.versioned_datastore.redis_portThe port to use to connect to the redis host6379ckanext.versioned_datastore.redis_databaseThe redis database index to use to store datastore multisearch slugs in1ckanext.versioned_datastore.slug_ttlThe amount of time slugs should last for, in days. Default:77ckanext.versioned_datastore.dwc_core_extension_nameThe name of the DwC core extension to use, as defined indwc/writer.py.gbif_occurrenceckanext.versioned_datastore.dwc_extension_namesA comma-separated list of (non-core) DwC extension names, as defined indwc/writer.py.gbif_multimediackanext.versioned_datastore.dwc_org_nameThe organisation name to use in DwC-A metadata. Default: the value ofckanext.doi.publisherorckan.site_titleThe Natural History Museumckanext.versioned_datastore.dwc_org_emailThe contact email to use in DwC-A metadata. Default: the value ofsmtp.mail_fromcontact@yoursite.comckanext.versioned_datastore.dwc_default_licenseThe license to use in DwC-A metadata if the resources have differing licenses or no license is specified. Default:nullhttp://creativecommons.org/publicdomain/zero/1.0/legalcodeUsageA brief tour!The plugin automatically detects resources on upload that can be added to the datastore.
This is accomplished using the resource format.
Currently the accepted formats are:CSV - csv, application/csvTSV - tsvXLS (old excel) - xls, application/vnd.ms-excelXLSX (new excel) - xlsx, application/vnd.openxmlformats-officedocument.spreadsheetml.sheetIf one of these formats is used then an attempt will be made to add the uploaded or URL to the datastore.
Note that only the first sheet in multisheet XLS and XLSX files will be processed.Adding data to the datastore is accomplished in two steps:Ingesting the records into MongoDB. A document is used per unique record ID to store all versions and the documents for a specific resource are stored in a collection named after the resource's ID. For more information on the structure of these documents see theSplitgillrepository for more details.Indexing the documents from MongoDB into Elasticsearch. One indexed is used for all versions of the records and a document in Elasticsearch is created per version of each record. The index is named after the resource's ID with the configured prefix prepended. For more information on the structure of these indexed documents see theSplitgillrepository for more details.The ingesting and indexing is completed in the background using the CKAN's job queue.Once data has been added to the datastore it can be searched using thedatastore_searchor more advanceddatastore_search_rawactions.
Thedatastore_searchaction closely mirrors the default CKAN datastore action of the same name.
Thedatastore_search_rawaction allows users to query the datastore using raw Elasticsearch queries, unlocking the full range of features it provides.ActionsAll of this extension's actions are fully documented inline, including all parameters and results.datastore_createAdds a resource to the versioned datastore (note that this doesn't add any data, it just does setup work. This is different to CKAN's defaultdatastore_createaction).datastore_upsertUpserts data into the datastore for the resource. The data can be provided in the data_dict using the key 'records' or, if data is not specified, the URL on the resource is used.datastore_deleteDeletes the data in the datastore against the given resource ID.datastore_searchSearch a resource's data using a similar API to CKAN's defaultdatastore_searchaction.datastore_get_record_versionsGiven a record id and a resource it appears in, returns the version timestamps available for that record in ascending order.datastore_get_resource_versionsGiven a resource id, returns the version timestamps available for that resource in ascending order along with the number of records modified in the version and the number of records at that version.datastore_autocompleteProvides autocompletion results against a specific field in a specific resource.datastore_reindexTriggers a reindex of the given resource's data.datastore_query_extentReturn the geospatial extent of the results of a given datastore search query.datastore_get_rounded_versionRound the requested version of this query down to the nearest actual version of the resource.datastore_search_rawThis action allows you to search data in a resource using a raw elasticsearch query.datastore_ensure_privacyThis action runs through all resources (or handles a specific resource if a resource id is provided) and makes sure that the privacy set on each resource's package is reflected in the datastore.Commandsvdsinitdb: ensure the tables needed by this plugin exist.ckan-c$CONFIG_FILEinitdbreindex: reindex either a specific resource or all resources.ckan-c$CONFIG_FILEreindex$OPTIONAL_RESOURCE_IDInterfacesIVersionedDatastoreThis is the most general interface.Here is a brief overview of its functions:datastore_modify_data_dict- allows modification of the data dict before it is validated and used to create the search objectdatastore_modify_search- allows modifications to the search before it is made. This is kind of analogous toIDatastore.datastore_searchhowever instead of passing around a query dict, instead an elasticsearch-dslSearchobject is passed arounddatastore_modify_result- allows modifications to the result after the searchdatastore_modify_fields- allows modification of the field definitions before they are returned with the results of a datastore_searchdatastore_modify_index_doc- allows the modification of a resource's data during indexingdatastore_is_read_only_resource- allows implementors to designate certain resources as read onlydatastore_after_indexing- allows implementors to hook onto the completion of an indexing taskSee the interface definition in this plugin for more details about these functions.IVersionedDatastoreQueryThis interface handles hooks and functions specifically relating to search queries.get_query_schemas- allows registering custom query schemasIVersionedDatastoreDownloadsThis interface handles hooks and functions specifically relating to downloads.download_modify_notifier_start_templates- modify the templates used when sending notifications that a download has starteddownload_modify_notifier_end_templates- modify the templates used when sending notifications that a download has endeddownload_modify_notifier_error_templates- modify the templates used when sending notifications that a download has faileddownload_modify_notifier_template_context- modify the context/variables used to populate the notification templatesdownload_derivative_generators- extend or modify the list of derivative generatorsdownload_file_servers- extend or modify the list of file serversdownload_notifiers- extend or modify the list of notifiersdownload_data_transformations- extend or modify the list of data transformationsdownload_modify_manifest- modify the manifest included in the download filedownload_before_run- modify args before any search is run or files generateddownload_after_run- hook notifying that a download has finished (whether failed or completed)TestingThere is a Docker compose configuration available in this repository to make it easier to run tests. The ckan image uses the Dockerfile in thedocker/folder.To run the tests against ckan 2.9.x on Python3:Build the required images:docker-composebuildThen run the tests.
The root of the repository is mounted into the ckan container as a volume by the Docker compose
configuration, so you should only need to rebuild the ckan image if you change the extension's
dependencies.docker-composerunckan