middleman/middleman — pull request #799 by karlfreeman (2013-02-25 17:38:23+00:00)
Title: Parallel building

Quite experimental, however.

Whilst putting together a rebuild of [middleman-sync](https://github.com/karlfreeman/middleman-sync/issues/16) I've had quite a bit of success with [parallel](https://github.com/grosser/parallel). At first I thought I'd see just how much of speed difference it would make if it were utilised in `middleman build` whilst building my small site.

```
middleman build 13.15s user 2.17s system 255% cpu 5.354 total - with
middleman build 7.59s user 0.90s system 99% cpu 8.531 total - without
```

A small gain, however watching the build log it appears to have quite a noticeable effect. Of course some proper benchmarking tests should be done ( pictures below illustrate it 'working' ).

### Problems
* `--clean`ing in parallel ( processes ) is out of the question as we need to keep track of dirty files. However cleaning in parallel ( threads ) could be possible but defeats the purpose of parallel building as we would have to guess or allow the user to specify how many threads to spawn on build ( could be a CLI option `--threads` though )
* Logging goes a little crazy, something we could solve for sure.

### Questions

* Could you try this out on one of your own ( more complex ) middleman apps?
* The tests pass however the order in which the build processes files in parallel would essentially be random. Do other extensions / etc rely on the build to be built in a specific order?
* Would it be worth continuing the idea of executing build in parallel or is it too fraught with possible problems that it should be delayed?

### Pictures
![with-parallel-activity](https://f.cloud.github.com/assets/139095/192872/cea4a97e-7f6f-11e2-8165-28c724f1207f.jpg)

![without-parallel-activity](https://f.cloud.github.com/assets/139095/192874/d3afc5fc-7f6f-11e2-8045-b6c9b26e0aeb.jpg)

![without-parallel](https://f.cloud.github.com/assets/139095/192918/a76c88b2-7f70-11e2-8c9d-03058e985969.jpg)

![with-parallel](https://f.cloud.github.com/assets/139095/192919/ae7a32b2-7f70-11e2-8cc8-9861914bdd6b.jpg)

__pull request more for a discussion then a 'ready to be merged'...__