open-data/ckanext-canada — pull request #92 by wardi (2013-04-05 22:43:34+00:00)
Title: Add portal-update command and factor out worker process management code

portal-update reads from one CKAN instance by the API (with no special authentication) and applies package changes to the local CKAN instance.

* Datasets are created, updated or deleted in batches based on the activity stream
* Unpublished datasets (according to date_published) are treated the same as deleted datasets
* Parallel operation is supported with the -p option, just like the load-datasets command

Creating packages with this command requires the latest canada-v2.0 version of CKAN for https://github.com/open-data/ckan/commit/533b829fc2e06def10e17f30f694fb6e49c698af which allows us to set the id of datasets to match the source dataset exactly.

The worker management code from the load-datasets command has now been factored out. Running multiple worker processes for any long-running jobs is now much easier with the new ckanext.canada.workers.worker_pool() generator. See the documentation in that module for more information.