okfn/ckan — pull request #278 by tobes (2013-01-22 05:29:03+00:00)
Title: Bulk updates of packages

*Essential*
Update public private
Update state to deleted.

*Very nice to have.*
Add remove datasets from groups

*Nice to have*
Add remove tags
Change dataset organization
Properly delete datasets (purge them)

Want speed of about 100 updated datasets per second.

*Idea of methodology.*

Directly update database using bulk inserts/updates.
For each row updated retrieve package_dict from search index modify and then run through reindexing again. This is because retrieving whole package_dict from database is slow. Only do solr commit at end of all updates.

Revisions will also need to be updated. Could potentially cheat here for some bulk operations by updating old revision record and pretending update happened on last revision. Not perfect but would save many writes.