jenius/roots — pull request #501 by jenius (2014-06-12 20:06:38+00:00)
Title: [WIP] roots workers: parallel compilation architecture

So it's finally here! Yes indeed, the mythical fix to #454 - distributed compilation across all of your computer's cores, making the compiles truly async rather than locking up the main thread on each one, which means that compiles will be (number of cores on your computer) orders of magnitude faster. Whoo!

I've been working on the large variety of architecture changes here for almost a week now, but it has been confirmed to correctly compile large and complex projects, although some tests are still failing due to not really a bug but a way of testing I used previously that I can't use anymore with this method of compilation. This will be fixed up soon.

So -- how it works. When you start up roots, it boots up a child process (worker) for each core your computer has, and loads the entire roots environment into each child process. This is painful and adds a toll of ~2sec to the total compile time, but unfortunately there is no way around this. However, after that, all the files to be compiles are distributed into queues for each of these workers evenly, and they kick off the compilation. After any file finishes being compiled, we look at the queue lengths and redistribute files between them to ensure they are all even. This makes sure that no cores are idling after being given a bunch of quick compiles while another one has some larger compiles queued up still. So all of your cores are working as efficiently as they can be throughout the entire compile process.

In watch mode, the workers only boot once on the initial compile, so every compile after this is very quick. This was the intent, as quick compiles are never more important than when you are actively making changes and watching for results.

For very small simple sites, this actually is very slightly slower. But this is a cost I'm willing to pay in exchange for roots being quite literally orders of magnitude faster at compiling larger and more complex sites.

This PR is not complete, there are still a few bugs to be worked out, comments and logs to be removed, and a few spots to polish up and refactor. So please, no need to add comments saying "remove this comment" or "drop this console.log" -- I'm aware that they are there, and they will be removed before merge. But the bulk of the work is done, and this is functional, which is awesome :tada: