jimweirich/rake — pull request #113 by michaeljbishop (2012-04-27 13:41:03+00:00)
Title: Add -j <max_jobs> and -m (all tasks multitasks) -- solution included

THE CASE FOR -j and -m
----------------------

### PROBLEM SUMMARY

Rake can be unusable for builds invoking large numbers of concurrent external processes.

### PROBLEM DESCRIPTION

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

### SOLUTION SUMMARY

This request includes the code to add support for a `--jobs NUMBER (-j)` command-line option to specify the number of simultaneous tasks to execute.

* To maintain backward compatibility, not passing `-j` reverts to the old behavior of unlimited concurrent tasks.

As a nod to [drake](http://drake.rubyforge.org), a `--multitask (-m)` flag is also included which when supplied, changes tasks into multitasks.

### SOLUTION

Rather than spawning a new thread per prerequisite `MultiTask` now sends its prerequisites to a `WorkerPool` object. `WorkerPool.new(n).execute_blocks` has the same semantics as `Thread.new`...`join` but caps the thread count at `n`.

#### Core Change

threads = @prerequisites.collect { |p|
Thread.new(p) { |r| application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
}
threads.each { |t| t.join }

...becomes...

@@wp ||= WorkerPool.new(application.options.thread_pool_size)

blocks = @prerequisites.collect { |r|
lambda { application[r, @scope].invoke_with_call_chain(args, invocation_chain) }
}
@@wp.execute_blocks blocks

To support `-m`, the `MultiTask` implementation has moved to `Task#invoke_prerequisites_concurrently` and is called from `MultiTask#invoke_prerequisites`. This enables concurrent behavior for `Task` when `-m` is used.

#### Statistics

---LINES-- ----LOC---
old new old new File Name
---------- ---------- ----------
598 605 477 484 lib/rake/application.rb
16 13 11 8 lib/rake/multi_task.rb
327 341 210 222 lib/rake/task.rb
111 80 lib/rake/worker_pool.rb
4264 4393 2696 2792 TOTAL
--------------------------------------
+129 +96 SUMMARY

#### Tests

Tests are included for all new functionality

### REQUIREMENTS

The Ruby version requirements remain the same. `lib/rake/worker_pool.rb` adds two new requirements: `thread` and `set`