jimweirich/rake — pull request #131 by michaeljbishop (2012-10-19 02:02:19+00:00)
Title: Add -j <max_jobs> (solution included)

## PROBLEM SUMMARY (THE CASE FOR -j)

Rake can be unusable for builds invoking large numbers of concurrent external processes.

## PROBLEM DESCRIPTION

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using Rake to build non-ruby projects quite often Rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, Rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

## SOLUTION SUMMARY

This request includes the code to add support for a `--jobs NUMBER (-j)` command-line option to specify the number of simultaneous tasks to execute.

* To maintain backward compatibility, not passing `-j` reverts to the old behavior of unlimited concurrent tasks.

## SOLUTION

Rather than spawning a new thread per prerequisite `MultiTask` now sends its prerequisites to a `Rake::ThreadPool` object. `ThreadPool#future(*args,&block)` has the same semantics as `Thread.new` but the block passed to `future` is added to a queue consumed by a limited number of threads. The returned future can be waited on by issuing `#call` which halts the current thread until the future has completed and returns its value (much like calling `Thread#value`).

### Core Change (in `multi_task.rb`)

threads = @prerequisites.collect { |p|
Thread.new(p) { |r| application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
}
threads.each { |t| t.join }

...becomes...

futures = @prerequisites.collect do |p|
application.thread_pool.future(p) do |r|
application[r, @scope].invoke_with_call_chain(args, invocation_chain)
end
end
futures.each { |f| f.call }

### Details

`ThreadPool#future(*args,&block)` adds the passed-in block to a queue, and returns a future that can be used to realize the result.

This creates a potential problem:

> What if we pass 100 blocks through `ThreadPool#future` and each one is waiting on the value of a single previous future? Wouldn't that starve the pool of threads because rather than dequeuing blocks, they would all be asleep waiting on a single future?

Yes it would. This is solved as the future returned by `ThreadPool#future` knows when a thread is about to go to sleep waiting on it. If the thread is one that belongs to the pool, another is spawned in its place just before it sleeps, raising the maximum in the pool by one. When the sleeping thread is reawakened because its future is finished, the maximum in the thread pool is correspondingly decremented by one. There are always threads available in the pool for processing.

### Accessing the thread pool

The thread pool is owned by the application and is accessible as the public read-only attribute: `Rake.application.thread_pool`. In this way Rakefile authors can create futures for their own use, sharing the pool with the Rake system.

## TESTS

Tests are included for all new functionality and have been tested under ruby 1.9.3 and 1.8.7

## REQUIREMENTS

The Ruby version requirements remain the same. `lib/rake/thread_pool.rb` adds two new requirements: `thread` and `set`