puma/puma — pull request #467 by schneems (2014-02-17 05:27:49+00:00)
Title: Remove non-functioning workers

I found this behavior while writing a puma analog to Unicorn Worker Killer, named conveniently enough [PumaWorkerKiller](https://github.com/schneems/puma_worker_killer).

Currently when a worker is dead puma does not properly recognize that so it will not remove the worker from its @worker list and it will not spawn a replacement. Here is the running output of puma master with a `bundle open` and a few puts inside of `spawn_worker`

```
22:01:42 web.1 | [76322] PumaWorkerKiller: Out of memory. 3 workers consuming total: 273.71484375 mb out of max: 198.0 mb. Sending TERM to # consuming 90.1640625 mb.
22:01:42 web.1 | actual: 3
22:01:42 web.1 | == spawn_workers
22:01:42 web.1 | requested: 3
22:01:42 web.1 | actual: 3
22:01:47 web.1 | == spawn_workers
22:01:47 web.1 | requested: 3
22:01:47 web.1 | actual: 3
```

Where requested is `@options[:workers]` and actual is `@workers.size`. Note that after PumaWorkerKiller sends term to a worker it Puma should see that it is dead and it's actual count drop. Eventually if enough out of memory errors are received and PumaWorkerKiller sends term to all child processes then the program will exit.

```

22:04:43 web.1 | /Users/schneems/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/puma-2.7.1/lib/puma/cluster.rb:89:in `waitpid': No child processes (Errno::ECHILD)
22:04:43 web.1 | from /Users/schneems/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/puma-2.7.1/lib/puma/cluster.rb:89:in `check_workers'
22:04:43 web.1 | from /Users/schneems/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/puma-2.7.1/lib/puma/cluster.rb:314:in `run'
22:04:43 web.1 | from /Users/schneems/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/puma-2.7.1/lib/puma/cli.rb:442:in `run'
22:04:43 web.1 | from /Users/schneems/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/puma-2.7.1/bin/puma:10:in `’
```

With this PR, Puma will now correctly see that a process is no longer alive:

```

22:18:50 web.1 | [76868] PumaWorkerKiller: Out of memory. 3 workers consuming total: 453.921875 mb out of max: 198.0 mb. Sending TERM to # consuming 154.23046875 mb.== spawn_workers
22:18:50 web.1 | [76890] 127.0.0.1 - - [16/Feb/2014 22:18:50] "GET /tab_navigation.js?body=1 HTTP/1.1" 304 - 0.0241
22:18:50 web.1 | [76890] 127.0.0.1 - - [16/Feb/2014 22:18:50] "GET / HTTP/1.1" 304 - 0.1070
22:18:50 web.1 | == spawn_workers
22:18:50 web.1 | requested: 3
22:18:50 web.1 | actual: 2
```

It will then go on to re-spawn the TERM-ed process:

```
22:29:48 web.1 | requested: 3
22:29:48 web.1 | actual: 2
22:29:48 web.1 | [78053] - Worker 2 (pid: 78086) booted, phase: 0
```