Change 'hda' input name in workflow inputs dict to 'content'...

... 'content' better reflects that this can be either an HDA or an HDCA.
Normalize workflow step parameters earlier...

... this will allow linking to actual steps (referential integrity) if/when we store workflow requests in the database.

Also unit tests.
Normalize workflow input dataset dict earlier...

... this will allow linking to actual steps (referential integrity) if/when we store workflow requests in the database and failing faster.

Also more unit tests.
Move WorkflowConfig so run_request doesn't need to import run.

This is a more logical organization of these modules.
Extract some mixin's out of trans allowing trans-like functionilty outside web threads...

In particular break out mixins defining context-based utilities for interacting with app, an user, and a history. In subsequent changes this proves nessecary and sufficient to create an context for 'execute' tools (create jobs) outside of web threads in response to workflow requests.

Unit tests for some of this.
Implement workflow requests and a scheduling 'plugin' framework to evaluate them (API-only).

The model changes and WorkQueue class are based on modifications to Kyle Ellrott's Galaxy farm work (https://bitbucket.org/kellrott/galaxy-farm/branch/workflow_migrate). I made heavy modifications to the model to enforce referential integrity on parameter to workflow step mappings and made some cosmetic changes various other details. I further filled out WorkQueue (adding ability to 'complete' requests, implementing ests, etc...). The work context (allowing tool's to be executed outside the context of a web thread), workflow module and API modifiations, and the workflow scheduling framework are my own contributions.

I like some of this - I remain somewhat skeptical on much of it though. I think maintaining queues of generic requests and distributing them across multiple processes, machines, etc... is both a hard problem and one for which many adequete solutions already exist - some of which we are already using (if only optionally) in other parts of Galaxy - so I don't understand why we need/want to so explicitly replicate this in Galaxy and not just defer to an external message queue.

In internal discussions it has been argued that message queues are appropriate for short requests but long term requests are better tracked in the database. I know that intuitively these operations may feel different - but I am unconvienced they are and from a technical standpoint spread a delaying a request that would have happened in millisonds to hours later is many order of magnitude large a delay the additional delay when going from the hours to days or weeks.

My own preference would be that both this work and Galaxy farm further investigate using message queues before making such substantial changes to the data model - however if the project is going to add a WorkRequest model to the database and something that looks like the WorkQueue module - then I guess the workflow requesting and scheduling should make use of it instead of developing a competing implementation.

Testing

One can issue a workflow request to the WorkQueue and scheduled via the scheduling framework by running the following test.

./run_tests.sh -with_framework_test_tools -api test/api/test_workflows.py:WorkflowsApiTestCase.test_workflow_request