kr/hk — pull request #80 by bgentry (2013-10-08 05:30:29+00:00)
Title: Parallelize diff generation using dynos

This is definitely not ready, and IMO the code is terrible & overcomplicated so far. But it does at least work.

Hoping to iterate on this a bit, so don't worry too much about reviewing it yet. I wouldn't be surprised if I can cut the LoC and complexity significantly.

Mostly I've just been struggling to find the right concurrency patterns, and as-is it's a bad mix of a few different things. Essentially it just needs to:

1. Perform one build at a time
2. Upload the build
3. Register the build
4. Add the needed diffs for that build to a work queue (`dchan` currently)
5. Run a dyno for each diff that needs to be generated, preferably with a reasonable limit on parallelism (don't want 1000 dyno runs at once)
6. Know when all diffs have been generated successfully so it can then mark that release as current

If you have suggestions on that concurrency patterns, I'm all ears! The process after step 3 doesn't feel right yet.

I've tried to make the process idempotent. If a build fails at a particular step, that particular build shouldn't complete, but the others still may. Running the build process again should just figure out what's left to be done and do that.