Refactoring work to add in Workflow UUIDs
Fixing some small bugs to get the code to work
Doc spelling fix in lib/galaxy/workflow/run.py.
Some very basic unit tests and docs for workflow modules.
Just a start - I need to keep going with this and test and document all methods.
Couple simple refactors making it slightly cleaner to create new module types.
Also slightly reduces shared logic galaxy.workflow.run.
Merge 'recent jobs' view from PR466/467. Work in progress.
Remove extra p tag from jobs admin.
Fix (old) bug causing Jobs grid to display even when there were no jobs to show in it.  Issue was 'jobs' (and the new recent_jobs) was actually a Query object and not an empty list of results, causing 'if jobs' to evaluate True regardless of whether or not there were jobs.
Clarify wording, explain cutoff.  Add headers to better segment the page and prevent confusion between the two grids.
Merge.
Merged in kellrott/galaxy-farm/workflow-uuid (pull request #471)

Workflow UUIDs
UI, Dataset choice control: add DatasetChoiceModal for choosing single or multiple datasets, add DatasetChoice and MultiDatasetChoice for displaying and activating selection modal from within a page; correct lookup of galaxy_config in dataset-model; add ListPanel for displaying collections of ListViews; add DatasetListPanel for displaying collections of generic datasets; add default no-op localizer (_l) to ui.js until that file is AMD; factor out icon-btn.less from history.less; remove make_popup_menus in default expandable behaviors
(add packed scripts)
UI, Paired collection creator: refactoring of paired HTML and fixes to draggable partition, add brush selection to unpaired
Reduce workflow step state and module population duplication.
Move all operations prior to workflow invocation (and a few other places state and modules are used) relating to populating the non-persistent attributes of `WorkflowStep`s (e.g. module, state, upgrade_messages, and input_connections_by_name) into a single function on a new class.

This should fix a couple of small errors as well - input_connections_by_name was being mis-calculated in a few places (though potentially never used down those roads) and some error handling code in the workflow editor template was expecting step.module to be populated with something (even if None) and it was not always.
Work toward workflow progress refactor.
Implement workflow requests (API only) and a scheduling 'plugin' framework to evaluate them.

The model changes and WorkQueue class are based on modifications to Kyle Ellrott's Galaxy farm work (https://bitbucket.org/kellrott/galaxy-farm/branch/workflow_migrate). I made heavy modifications to the model to enforce referential integrity on parameter to workflow step mappings and made some cosmetic changes various other details. I further filled out WorkQueue (adding ability to 'complete' requests, implementing tests, etc...). The work context (allowing tool's to be executed outside the context of a web thread), workflow module and API modifications, and the workflow scheduling framework are my own contributions.

I like some of this - I remain skeptical about much of it though. I think maintaining queues of generic requests and distributing them across multiple processes, machines, etc... is both a hard problem and one for which many adequete solutions already exist - some of which we are already using (if only optionally) in other parts of Galaxy - so I don't understand why we need/want to so explicitly replicate this in Galaxy and not just defer to an external message queue.

In internal discussions it has been argued that message queues are appropriate for short requests but long term requests are better tracked in the database. I know that intuitively these operations may feel different - but I am unconvienced they are from a technical standpoint. Delaying an action that would have happened in milliseconds to hours later is many order of magnitude larger than the additional delay when going from the hours to days or weeks.

My own preference would be that both this work and Galaxy farm further investigate using message queues before making such substantial changes to the data model. It certainly may prove that this approach is needed - but I personally would like to understand why maintain the work queue in a message queue would not work first.

However, if the Galaxy project is going to add a WorkRequest object to the database and something that looks like the WorkQueue class in the new galaxy.work module and store and distribute workflows that way for the farm project, then I believe that typical workflow requesting and scheduling should make use of it also instead of developing a competing implementation. (Because my interaction pattern with Galaxy has always been if you cannot beat them join them.)

Testing

One can issue a workflow request to the WorkQueue and scheduled via the scheduling framework by running the following test.

./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_workflow_request

Final Note: The workflow scheduling stuff is a bit rough right now - it is just enough to prove to myself that this approach can be taken and the farm-stuff and workflow scheduling stuff can utilize the same models. I wanted to try to spark a broader conversation on this (re-encounrage use of message queues) before putting more effort into this approach.
WIP: Pause module for workflows.