massimiliano-mantione/metascript — pull request #46 by drslump (2014-06-19 00:58:50+00:00)
Title: Parallel compilation

# Summary

- Refactor `mjs.js` into separate files
- Accept N input source files
- Compilation module/worker based on EventEmitter interface
- Parallel compilation (`--parallel` switch to enable)

# Parallel compilation

Node doesn't have a real `fork` or any sort of threading support, so to enable parallel execution child processes must be spawned. This operation is costly in time (~200ms) and memory (~10Mb) so parallel compilation is disabled by default.

Compilation is now handled with a pool of workers, when disabled or only one worker is used, no child processes are spawned, thus speed should be similar to the current compiler. When 2 or more workers are needed they are spawned and the parent process just acts as coordinator to collect the result of the compilation on the children and report them back to the user.

The good thing about using processes (instead of threads for instance) is that the compiler service doesn't need any changes. It should provide pretty good performance on Mac/Linux and at least work on Windows (haven't tested myself).

# Garbage collection

It won't come as a surprise that the compiler is heavy on memory allocations. Lots of objects to build the AST and specially when working with macros there is a lot of object copying going around (perhaps this can be optimized?). Even when parallelizing the compilation there is a huge gain to be made by manually forcing the garbage collector after each compilation. To do this we need the `--expose-gc` node flag, so I've added it to the `mjs` launcher.

# Lies, damn lies and benchmarks

To simulate a code base I've make 10 copies of `test/meta-test.mjs`, these are the results on a Macbook i7 2Ghz with 8 cores:

```shell
❫ ls /tmp/*.mjs | time xargs -n 1 ./bin/mjs -v # current compiler
Successfully generated /tmp/01.js in 543 ms.
Successfully generated /tmp/02.js in 602 ms.
Successfully generated /tmp/03.js in 600 ms.
Successfully generated /tmp/04.js in 610 ms.
Successfully generated /tmp/05.js in 529 ms.
Successfully generated /tmp/06.js in 531 ms.
Successfully generated /tmp/07.js in 605 ms.
Successfully generated /tmp/08.js in 530 ms.
Successfully generated /tmp/09.js in 532 ms.
Successfully generated /tmp/10.js in 528 ms.
6.95 real 6.47 user 0.51 sys

❫ ls /tmp/*.mjs | time xargs -n 1 -P8 ./bin/mjs -v # current compiler (shell parallel)
Successfully generated /tmp/06.js in 1.08 s.
Successfully generated /tmp/05.js in 1.09 s.
Successfully generated /tmp/07.js in 1.1 s.
Successfully generated /tmp/03.js in 1.21 s.
Successfully generated /tmp/02.js in 1.23 s.
Successfully generated /tmp/04.js in 1.22 s.
Successfully generated /tmp/01.js in 1.22 s.
Successfully generated /tmp/08.js in 1.23 s.
Successfully generated /tmp/09.js in 673 ms.
Successfully generated /tmp/10.js in 675 ms.
2.26 real 12.03 user 1.05 sys

❫ ./bin/mjs /tmp/*.mjs -v # mutiple files no-parallel
Successfully generated 10 files in 4.38 s.

❫ ./bin/mjs /tmp/*.mjs -v --parallel # Auto -> round(files.length / 3) = 3
Successfully generated 10 files in 2.06 s.

❫ ./bin/mjs /tmp/*.mjs -v --parallel=7 # Forced 7 workers
Successfully generated 10 files in 1.69 s.
```

# Future

If the proposal about metadata and incremental compilation https://github.com/massimiliano-mantione/metascript/issues/42 goes forward it can be used for the parallel compiler too. Here is the plan:

- Build dependency graph (potentially outdated) and store it
- Traverse graph with depth-first-post-order:
- Remove nodes which haven't changed
- Parallel compile each level and store results
- Build dependency graph again based on current metadata
- Traverse graph with depth-first-post-order:
- Remove nodes which deps haven't changed after compiling
- If dependency graph is not empty:
- Parallel compile each level and override stored results
- Report results to the user

The order of compilation for a dependency graph would look like this
```
A H
/\ /
B C J
/|\
D E F
/
G

Compile (G, E, F, C, J), (D, H), B, A
```

In the example above, if all files are given to the compiler but only `D` was changed, the compiler would just be executed for `D`, `B` and `A`.

This scheme should ensure a reliable incremental parallel compilation, allowing users to build a considerably sized project by basically issuing a simple `mjs **/*.mjs` :)