Package: ckanext-downloadall
ckanext-downloadallThis CKAN extension adds a “Download all” button to datasets. This downloads
a zip file containing all the resource files and a datapackage.json.This zip file is a good way to package data for storing or sending, because:you keep all the data files togetheryou include the documentation (metadata) - avoids the common problem of being
handed some data files and not know anything about it or where to find infothe metadata is machine-readable, so can be used by tools, software and in
automated workflows. For example:validating a series of data releases all meet a standard schemaloading it into a database, using the column types and foreign key
relations specified in the metadataThedatapackage.jsonis aFrictionless Datastandard, also known as a Data Package.Technical notesIf the resource is pushed/xloaded to DataStore then the schema (column types)
is also included in the datapackage.json file.This extension uses a CKAN background job to create the zip every time a
dataset is created or updated (or its data dictionary is changed). This suits
CKANs where all files are uploaded - if the underlying data file changes
without the CKAN URL changing, then the zip will not include the update (until
something else triggers the zip to update).(This extension is inspired byckanext-packagezip, but that is old and relied
on ckanext-archiver and IPipe.)RequirementsDesigned to work with CKAN 2.7+Ideally it is used in conjunction with DataStore andxloader(or datapusher), so that the
Data Dictionary is included as a schema in the datapackage.json, to describe
the column types.InstallationTo install ckanext-downloadall:Activate your CKAN virtual environment, for example:. /usr/lib/ckan/default/bin/activateInstall the ckanext-downloadall Python package into your virtual environment:pip install ckanext-downloadallAdddownloadallto theckan.pluginssetting in your CKAN
config file (by default the config file is located at/etc/ckan/default/production.ini). e.g.ckan.plugins = downloadallRestart the CKAN worker. For example if you’ve deployed it with supervisord:sudo supervisorctl restart ckan-worker:ckan-worker-00Restart CKAN server. For example if you’ve deployed CKAN with Apache on
Ubuntu:sudo service apache2 reload6. Ensure the background job ‘worker’ process is running - seehttps://docs.ckan.org/en/2.8/maintaining/background-tasks.html#running-background-jobsConfig Settings# Include additional fields from the dataset in the datapackage.json (e.g.
# those defined in a ckanext-scheming schema)
# (optional, space separated list).
ckanext.downloadall.dataset_fields_to_add_to_datapackage = district countyCommand-line interfaceThere is a command-line interface:downloadall --helpExamples of use:downloadall update-zip gold-prices
downloadall update-all-zipsTroubleshooting“All resource data” appears as a normal resource, instead of seeing a
“Download All” buttonYou need to enable this extension in the CKAN config and restart the server.
See the Installation section above.ImportError: No module named datapackageThis means you have an older version of ckanapi, which is a dependency of
ckanext-downloadall. Install a newer version.OSError: [Errno 13] Permission denied: ‘/data/ckan/resources/c89’You are trying to update zips from the command-line but running the tasks
synchronously, rather than with the normal worker process. In this case you
need to run it as thewww-datauser e.g.:sudo -u www-data /usr/lib/ckan/default/bin/downloadall -c /etc/ckan/default/production.ini update-all-zips --synchronousDevelopment InstallationTo install ckanext-downloadall for development, activate your CKAN virtualenv
and do:git clone https://github.com/davidread/ckanext-downloadall.git
cd ckanext-downloadall
python setup.py develop
pip install -r dev-requirements.txtRemember to run the worker (in a separate terminal):paster --plugin=ckan jobs worker --config=/etc/ckan/default/development.iniRunning the TestsTo run the tests, do:nosetests --nologcapture --with-pylons=test.iniTo run the tests and produce a coverage report, first make sure you have
coverage installed in your virtualenv (pip install coverage) then run:nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.downloadall --cover-inclusive --cover-erase --cover-testsReleasing a New Version of ckanext-downloadallckanext-downloadall is availabe on PyPI ashttps://pypi.org/project/ckanext-downloadall/.
To publish a new version to PyPI follow these steps:Update the version number in thesetup.pyfile.
SeePEP 440for how to choose version numbers.Update the CHANGELOG.md with details of this release.Make sure you have the latest version of necessary packages:pip install --upgrade setuptools wheel twineCreate a source and binary distributions of the new version:python setup.py sdist bdist_wheel && twine check dist/*Fix any errors you get.Upload the source distribution to PyPI:twine upload dist/*Commit any outstanding changes:git commit -a
git pushTag the new release of the project on GitHub with the version number from
thesetup.pyfile. For example if the version number insetup.pyis
0.0.1 then do:git tag 0.0.1
git push --tags