cooperative-computing-lab/cctools — pull request #599 by kangkangli (2014-11-10 07:28:50+00:00)
Title: makeflow speedup and improve scalability

To speedup the makeflow and improve the scalability, the key is to reduce the calls of dag_node_ready. That requires us to avoid scanning the "complete" nodes again.

Therefore, we could add another list that maintain all the nodes that are not complete(running state, waiting state, failing state).

We will submit only this list of "incomplete nodes".

The initial list of "incomplete nodes" will be the same as the original list because no node is submitted yet.
Then, after each loop of execution inside function "dag_run", if some node is complete, we could remove that node from the list of "incomplete nodes".

In that case, in the next loop, we do not need to submit that complete node again, avoiding re-scanning complete nodes.

We use the "list" data strucutre to copy the original linked list of dag_nodes. The copied list will function as the "incomplete nodes" list, which is initally the same as the original list.
After some node is complete, we will remove that node from the copied list.

The concrete description of "list" data strucutre is described in dttools/list.h and dttools/list.c