Package: ckanext-dcor-depot
This plugin manages how data are stored in DCOR. There are two types of
files in DCOR:Resources uploaded by users, imported from figshare, or
imported from a data archiveAncillary files that are generated upon resource creation, such as
condensed DC data, preview images (seeckanext-dc_view).This plugin implements:Data storage management. All resources uploaded by a user are moved
to/data/users-HOSTNAME/USERNAME-ORGNAME/PK/ID/PKGNAME_RESID_RESNAMEand symlinks are created in/data/ckan-HOSTNAME/resources/RES/OUR/CEIDvia a background job.
CKAN itself will not notice this. The idea is to have a filesystem overview
about the datasets of each user.A backround job uploads all resources to S3after_resource_create.Import datasets from figshare. Existing datasets from figshare are
downloaded to the/data/depots/figsharedirectory and, upon resource
creation, symlinked there from/data/ckan-HOSTNAME/resources/RES/OUR/CEID(Note that this is an exemption of the data storage management described
above). When running the following command, the “figshare-import” organization
is created and the datasets listed infigshare_dois.txtare added to CKAN:ckan import-figsharePopulate an internal depot from RT-DC data stored in tar archives. This
is part of an effort to have automated imports of RT-DC data from other
sources. The idea is to move experimental data to the DCOR server in
tar archives and DCOR can then populate the internal depot with it.
The location of the internal depot is/data/depots/internal/and it follows a very specific directory structure201X/2019-08/20/2019-08-20_1126_c083de*where the path is generated
from the acquisition date, time, and part of the hash (c083de) of
the original data file. According to this scheme, all files with the
same path stem belong to one dataset:2019-08-20_1126_c083de.sha256sumsa file containing SHA256 sums2019-08-20_1126_c083de_v1.rtdcthe actual measurement2019-08-20_1126_c083de_v1_condensed.rtdcthe condensed dataset2019-08-20_1126_c083de_ad1_m001_bg.pngan ancillary image2019-08-20_1126_c083de_ad2_m002_bg.pnganother ancillary image…ckan depotize-archiveImport datasets from the internal depot. The previous commanddepotize-archivejust populates the depot directory structure.
To make the datasets available in CKAN, this step must be performed:ckan import-internalUpgrade datasets from the internal depot. If you find issues or have to add
new versions of an .rtdc resource, simply create the_v2.rtdcfiles and
run this command:ckan upgrade-internalCLI for symlinking datasets that have failed to symlink before:ckan run-jobs-dcor-depotCLI for appending a resource to a datasetckan append-resource /path/to/file dataset_id --delete-sourcePlease make sure that the necessary file permissions are given in/data.In 2023, it was decided that the huge block storage of DCOR
should be replaced with an S3-compatible object store, because block storage
does not scale well. This partially deprecates some of the commands above
which might be removed or modified to support object storage directly.CLI for migrating data from block storage to an S3-compatible object storage
service. For this, the following configuration keys must be specified in
theckan.inifile:dcor_object_store.access_key_id = ACCESS_KEY_ID
dcor_object_store.secret_access_key = SECRET_ACCESS_KEY
dcor_object_store.endpoint_url = S3_ENDPOINT_URL
dcor_object_store.ssl_verify = true
# The bucket name is by default defined by the circle ID. Resources
# are stored in the "RES/OUR/CEID-SCHEME" in that bucket.
dcor_object_store.bucket_name = circle-{organization_id}Usage:ckan dcor-migrate-resources-to-object-storeInstallationpip install ckanext-dcor_depotAdd this extension to the plugins and defaul_views in ckan.ini:ckan.plugins = [...] dcor_depot
ckan.storage_path=/data/ckan-HOSTNAME
ckanext.dcor_depot.depots_path=/data/depots
ckanext.dcor_depot.users_depot_name=users-HOSTNAMEThis plugin stores resources to/data:mkdir -p /data/depots/users-$(hostname)
chown -R www-data /data/depots/users-$(hostname)TestingIf CKAN/DCOR is installed and setup for testing, this extension can
be tested with pytest:pytest ckanextTesting can also be done via vagrant in a virtualmachine using thedcor-test <https://app.vagrantup.com/paulmueller/boxes/dcor-test/>image.
Make sure thatvagrantandvirtualboxare installed and run the
following commands in the root of this repository:# Setup virtual machine using `Vagrantfile`
vagrant up
# Run the tests
vagrant ssh -- sudo bash /testing/vagrant-run-tests.sh