jimweirich/rake — pull request #111 by mbtyke (2012-04-17 13:34:53+00:00)
Title: Adds support for -j (max_concurrent_tasks) 

SUMMARY:

Rake can be unusable for builds invoking large numbers of concurrent external processes.

DESCRIPTION:

Rake makes it easy to maximize concurrency in builds with its "multi-task" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multi-tasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can also reach the maximum number of open files (presumably because it's reading stdout for all those processes).

CHANGE SUMMARY:

This change to rake adds support for a "--jobs number (-j)" command-line option that specifies the number of simultaneous tasks to execute.

SOLUTION:

The solution creates a work queue to which task-prerequisites are added and a thread pool to process the blocks that are added. To prevent deadlock, the task that added the pre-requisites processes items on the queue (alongside the thread pool) until its prerequisites have been processed.

To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.