cooperative-computing-lab/cctools — pull request #425 by badi (2014-03-20 00:03:20+00:00)
Title: Add task copy api call to C and Python

I thought I'd open up a discussion on task replication.
Some of my workflows require tasks to be submitted multiple times.
Previously I had my application regenerate tasks, which makes it hard to separate the concerns of creating tasks and running tasks.
To overcome this I've add a WorkQueue api call to perform a copy of a given task.

The following shows how I use this feature to run my MD tasks.

[My WQ loop thus becomes][WQ Loop]:
```python
while not self.empty():
self.wq.replicate()
r = self.wait()
```

[where the replication code is defined in a wrapper class for cctools' WorkQueue][WQ Wrap]
```python
def replicate(self):
"""
Replicate all candidate tasks.
Returns the number of tasks replicated.
"""
count = 0
while self._can_duplicate_tasks():
tag = self._tags.select()
if tag is None: break
task = self._tasks[tag].clone()
self.submit(task)
count += 1
return count
```

[and the call to `wait` cancels redundant tasks][WQ Wait]:
```python
def _cancel(self, task):
"""
Cancels all tasks with the same tag.
Returns the number of tasks canceled.
"""
count = 0
self._tags.discard(task.uuid)
while self.cancel_by_tasktag(task.uuid):
count += 1
return count
```

[WQ Loop]: https://github.com/badi/mdq/blob/e000e7b4d9873cd8128c56a7907a786290df5293/mdq/stream.py#L86-L88

[WQ Wrap]: https://github.com/badi/pwq/blob/d26e9891d55cdd45bf0f151fad8b3d7b29c03d20/pwq/_wq/replication.py#L181-L193

[WQ Wait]: https://github.com/badi/pwq/blob/d26e9891d55cdd45bf0f151fad8b3d7b29c03d20/pwq/_wq/replication.py#L167-L176