vmlaker/mpipe — pull request #1 by frensjan (2014-07-25 16:26:36+00:00)
Title: Adding ability to limit input backlog on an unordered stage

Hi,

I'm using the mpipe library for some I/O intensive tasks followed by some processing and then some more I/O intensive tasks. Some of my pipelines have their bottleneck at the end (i.e. the last stage(s)), however work may be piling up at a rate which causes quite some issues (stressing out some components, using a more than desirable amount of memory, etc.).

I implemented the possibility to limit the maximum size of the input TubeQ pipe on an UnorderedStage. Could you take a look at this feature and it's implementation? I've included a simple example of the idea in test/backlog.py.

I've also added some additions essentially unrelated to the above. 1) a check to prevent stage self-linking and 2) getting all results from a stage via a generator like Stage.results(...)

Thanks,
Frens Jan