enykeev/mistral — pull request #1 by enykeev (2014-03-24 10:37:51+00:00)
Title: Taskflow integration

So, I've been able to recreate some of the core Mistral concepts using TaskFlow as a library and now I'm asking you for input.

Lets take it apart:
- There is no such thing as Actions in TaskFlow, so we have to create one for ourself. For now i've just created the one that prints the line it gets from the task definition.
- Mistral Task is essentially the same as TaskFlow Task. The difference here is instead of relying on data flow for resolving the requirements, Mistral rely on the name of the tasks. Also, the core difference between them is that besides requirements Mistral also have direct, conditional transitions, while TaskFlow have events and reflections (and we have no use for them at the moment).
- Mistral Workbook is the same as TaskFlow Flow. Since we are not using TaskFlow requires\provides mechanic at the moment, there is no reason for us to inherit from one of taskflow.patterns, but we still have to because they are hardcoded in taskflow.utils.flow_utils.Flattener._find_flattener (more about it later).
- The difference between Mistral Engine and TaskFlow Engine is much deeper. The idea of being a service requires the ability to start and stop the execution in an arbitrary moment on external request. Mistral is fulfilling this requirements by providing singleton engine that don't rely on internal state. TaskFlow, at the same time, didn't have such a requirements, so their engine should be instantiated with execution prior to running. Also, TaskFlow ```engine.run()``` will block the thread until it either finishes or fails, so we have to spawn it in another thread. While it solves the problem with blocking, it creates another one: now we need a way to stop the engine. You could probably store the engine in global list, but it would create additional problems with scalability, so we need a way to recreate the one using come kind of easily serializable object that holds the state of current execution. Then we may use this recreated engine to suspend the execution that are already running.
- In Mistral such an object is called Execution, in TaskFlow it called FlowDetails. While the difference between them is minor, the problem lies in a way engine works with another TaskFlow structure called Storage.
- There is no such thing in Mistral, since it does all the changes directly to the persistence db. TaskFlow Storage holds the number of execution-related APIs to store its state and to eventually offload them to the persistent backend (or not, depending on your need of persistence). The problem here is that while it successfully offload the data to backend, it never loads them back, so the engine that is already running would never learn of any external changes in its flow_details. The solution here is to modify its get_flow_state method to get the latest version of the flow_details every time engine would want to check the state.
- Mistral DB is pretty much the same thing as TaskFlow Persistence backend with the difference that Mistral is using oslo.db while TaskFlow is using stevedore drivers with their own implementation (as far as i can tell). I had to create a few additional methods just for convenience, though at some point we would need to merge it with the one Mistral has. I don't think we would be able to use TaskFlow persistence due to additional fields we would need to add (task tags, for example), though i haven't spent too much time thinking about that.

Mistral API and DSL parser stays, but modification is needed. We would have to use TaskFlow states and exceptions. There is whole bunch of helpers in TaskFlow, some of them are useful, some not, we would have to go through them one by one and figure out how to make them more versatile.