jondot/sneakers — pull request #85 by jcalvert (2014-12-17 23:36:51+00:00)
Title: Workers from config script

Sorry for the mess in my previous PR. This is what I was meaning to send.

In some cases where processes are managed through something like runit and environment variables through Chef configurations, it can be more convenient to have one master process managing several different worker pools. We may also want to do configuration of Sneakers in one place for a Rails application, but with the number of workers varying with each group, such that simply having two processes with two ```WORKERS``` envs being insufficient. An example might be one regular job queue and then a low volume, long running process queue. We want to logically separate them with different worker classes and queues, but may need many more processes for the regular queue than the low priority one.

By specifying an ENV ```WORKER_GROUP_CONFIG``` as a path to YAML file or by convention ```./config/sneaker_worker_groups.yml``` we can use a file to specify groups like so:

```
foogroup:
classes: FooWorker
workers: 8
slowgroup
classes: SlowWorker
workers: 2
```

Then we can create a single lightweight ruby process to spawn the individual rake worker tasks with the correct settings. One reason for doing it separately from Rake is that if forking from a Rake task, your memory footprint overall will be unnecessarily large.

Once the YAML file is in place, you can simply use ```be ruby -e "require 'sneakers/spawner';Sneakers::Spawner.spawn"``` to spawn workers specified from the config file. The signals ```"TERM", "USR1", "HUP", "USR2"``` are all forwarded to the child processes.