grosser/parallel — pull request #106 by amw (2014-08-17 01:40:20+00:00)
Title: Support Queues and Lambdas for producer-consumer workflow without fixed set of items

Hi,

First of all, sorry for such a large pull request that spans multiple smaller and larger features. It was difficult for me to keep everything on separate branches as I was working on making Parallel meet my project requirements.

The big thing is support for queues and lambdas to dynamically generate new items while they are consumed inside `Parallel.map` block. This or something similar has been brought up in multiple issues to date. See #99, #42, #31, #10. It avoids using Marshal.dump when argument is not a queue or lambda. I know you have been concerned about adding too many ifs-and-elses to the codebase, but I hope you find my solution acceptable.

You can see this [chapter](https://github.com/amw/parallel/tree/queues-and-lambdas#producing-data-as-it-is-consumed) of Readme on my branch for a usage overview.

Other additions:

1. `:start` and `:finish` are synchronized on the main thread.

This allows safe usage of shared resources. Even simple debug `puts` calls can compete for terminal and generate unreadable, interleaved logs.

2. `:finish` receives result even from `.each()`.

This allows consuming data on the main thread without holding large map-style array in the memory. I use it along with queues to process gigabytes of data while keeping my processes memory footprint very small.

It does require marshaling result from child processes increasing CPU usage, but throws it away from memory right after :finish completes.

You can read this [chapter](https://github.com/amw/parallel/tree/queues-and-lambdas#consuming-data-from-multiple-processes-without-map) of my Readme for usage example.

3. Progress bar does not conflict with user supplied `:finish` callback.

You can have progress bar and your own :finish callback. This is especially important with point 2. Adding progress bar does not require marshaling results from child processes in `each`.

You will also notice that I have changed keys passed around in `options` hash that are not user-supplied to start with underscore. So you have :_preserve_results instead of :preserve_results.

Thanks a lot for a great library. I found your codebase very well maintained while working on this pull request. Cheers!