Fix data table xml definition for tool tarball download.
Synchronize validation of workflows between web and API controllers.
Reduces code duplication and does more correct checking of workflow step replacement parameters. More parameter checking functional tests.
More workflow module commenting.
This time add small comment about config form method.
Unify runtime state handling methods across tool and input modules.
The abstract interface WorkflowModule declared three methods that WorkflowToolModule didn't implement and logic for updating tool state wasn't restricted to the module itself. This introduces a new compute_state method that unifies all of that and includes some documentation.

In addition to simplifing the interface new modules must adhere to, this allows eliminating the switch on module type when injecting trasient state attribute into steps - moving that logic into the module itself. In my opinion this makes WorkflowModule closer to being an abstract interface - allowing components to uniformly reason about workflow modules as a black box (and not dispatch on type attributes).

These changes did require adding add_dummy_datasets to the base class - and while the input modules don't need to utilize the ability to override this - my pause module downstream did so I think it makes sense to place on the interface.
Remove some old commented out client-side stuff.
(Let me know if these were important.)
Refactor logic for tracking workflow progress out of WorkflowInvoker.
Creates a new class WorkflowProgress. This has class achives two objectives:

- I am trying to move specialized per-module-type logic out of WorkflowInvoker and into the modules themselves - making it easier to add new modules and because it is better OOP design (invoker shouldn't be dispatching on .type - the modules should provide a consistent interface for invoking themselves). WorkflowProgress will give something that WorkflowInvoker can pass along to the modules to allow them to find their connections and record their outputs.

- Downstream in workflow scheduling branch I am placing logic in here for rebuilding the in-memory state of the workflow progress when re-evaluating progress.
Unify execute method across tool and input workflow modules.
Previously tool module class did not contain this method (though it was abstract in the base class) and extra logic for dealing with tools was located outside of the module itself. Moving everything into the module (made possible by previously committed WorkflowProgress class and attaching some properties to workflow invocations and sending them in) means that now the worklfow invocation logic can uniformly deal with modules and doesn't need to dispatch on type. This should allow extending the capabilities of the workflow invoker and adding new modules independently.
API functional test for workflow 'replacement_params' and PJA renaming.
Implement workflow requests (API only) and a scheduling 'plugin' framework to evaluate them.

The model changes and WorkQueue class are based on modifications to Kyle Ellrott's Galaxy farm work (https://bitbucket.org/kellrott/galaxy-farm/branch/workflow_migrate). I made heavy modifications to the model to enforce referential integrity on parameter to workflow step mappings and made some cosmetic changes various other details. I further filled out WorkQueue (adding ability to 'complete' requests, implementing tests, etc...). The work context (allowing tool's to be executed outside the context of a web thread), workflow module and API modifications, and the workflow scheduling framework are my own contributions.

I like some of this - I remain skeptical about much of it though. I think maintaining queues of generic requests and distributing them across multiple processes, machines, etc... is both a hard problem and one for which many adequete solutions already exist - some of which we are already using (if only optionally) in other parts of Galaxy - so I don't understand why we need/want to so explicitly replicate this in Galaxy and not just defer to an external message queue.

In internal discussions it has been argued that message queues are appropriate for short requests but long term requests are better tracked in the database. I know that intuitively these operations may feel different - but I am unconvienced they are from a technical standpoint. Delaying an action that would have happened in milliseconds to hours later is many order of magnitude larger than the additional delay when going from the hours to days or weeks.

My own preference would be that both this work and Galaxy farm further investigate using message queues before making such substantial changes to the data model. It certainly may prove that this approach is needed - but I personally would like to understand why maintain the work queue in a message queue would not work first.

However, if the Galaxy project is going to add a WorkRequest object to the database and something that looks like the WorkQueue class in the new galaxy.work module and store and distribute workflows that way for the farm project, then I believe that typical workflow requesting and scheduling should make use of it also instead of developing a competing implementation. (Because my interaction pattern with Galaxy has always been if you cannot beat them join them.)

Testing

One can issue a workflow request to the WorkQueue and scheduled via the scheduling framework by running the following test.

./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_workflow_request

Final Note: The workflow scheduling stuff is a bit rough right now - it is just enough to prove to myself that this approach can be taken and the farm-stuff and workflow scheduling stuff can utilize the same models. I wanted to try to spark a broader conversation on this (re-encounrage use of message queues) before putting more effort into this approach.
Implement pause module for workflows.