dpla/KriKri — pull request #35 by markbreedlove (2014-12-19 21:47:16+00:00)
Title: Add initial queueing mechanism

Here's a checkpoint pull request. This is a work in progress but it will
be good to get these changes in before too much piles up on this branch.

This changeset adds a general queueing mechanism and a specific implementation of harvest queueing, using Resque.

A Resque control panel has been routed on top of the Krikri application's base path.

The tests in spec/models and the comment in Krikri::SoftwareAgent describe how harvest jobs are put onto the queue and run.

STILL TO DO:
- Have the Activity that persists and records the timeframe of the harvest job
get updated if the job fails. The only way to manage the failure queue is
currently with the Resque web console (e.g. "/krikri/resque") or with the
Rails console. The Krikri application is going to have to do this, too.
- Model the higher-level event that encompasses the harvest job.
For a provider that has multiple sets or collections, a harvest job will
probably be run for each set. How does the greater "harvest event" know when
all of the collections are complete? How does the greater ingestion process
know when all of the harvests are done, and it's time to move on to
enrichment?
- Provide http resources (HTML u.i. and JSON API) to allow interaction with
Krikri to create and manage harvests.