jimweirich/rake — pull request #112 by michaeljbishop (2012-04-18 23:09:01+00:00)
Title: Add support for --jobs <max_number_of_jobs>  (or -j) 

PROBLEM SUMMARY:

Rake can be unusable for builds invoking large numbers of concurrent external processes.

PROBLEM DESCRIPTION:

Rake makes it easy to maximize concurrency in builds with its "multi-task" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multi-tasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can also reach the maximum number of open files (presumably because it's reading stdout for all those processes).

SOLUTION SUMMARY:

This request includes the code to add support for a "--jobs number (-j)" command-line option to specify the number of simultaneous tasks to execute.

SOLUTION:

The solution creates a work queue to which task-prerequisites are added and a thread pool to process the blocks that are added. To prevent deadlock, the task that added the pre-requisites processes items on the queue (alongside the thread pool) until its prerequisites have been processed.

To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.

RUBY VERSION REQUIREMENTS:

The Ruby version requirements remain the same.