cloudfoundry/cf-release — pull request #296 by sykesm (2014-02-20 02:50:23+00:00)
Title: Monitor warden.sock as part of dea health check

Ran into an issue where one of the warden servers was no longer accepting connections from the DEA via the warden socket. [[1]](https://gist.github.com/sykesm/9101882) This caused all staging requests targeted to the sick DEA to fail. Even though the warden server was non-responsive to its DEA client, monit did not restart it as the warden health check http server was still responding.

While there's still no clear explanation of why warden stopped accepting connections, it's clear it can happen so it seems like a good idea to start monitoring.

If the situation needs to be recreated, it can be forced by using `gdb` to attach to a running warden server and closing the socket. The file descriptor for the socket can be found with `lsof`. Once you have the file descriptor, you can connect with gdb and issue `call close(xx)` to stop accepting connections. After a little bit the monit log will contain messages like these:
```
[UTC Feb 20 00:58:19] error : 'warden' failed, cannot open a connection to UNIX[/var/vcap/data/warden/warden.sock]
[UTC Feb 20 00:58:29] error : 'warden' failed, cannot open a connection to UNIX[/var/vcap/data/warden/warden.sock]
[UTC Feb 20 00:58:39] error : 'warden' failed, cannot open a connection to UNIX[/var/vcap/data/warden/warden.sock]
[UTC Feb 20 00:58:49] error : 'warden' failed, cannot open a connection to UNIX[/var/vcap/data/warden/warden.sock]
[UTC Feb 20 00:58:59] error : 'warden' failed, cannot open a connection to UNIX[/var/vcap/data/warden/warden.sock]
[UTC Feb 20 00:58:59] info : 'warden' trying to restart
[UTC Feb 20 00:58:59] info : 'dea_next' stop: /var/vcap/jobs/dea_next/bin/dea_ctl
[UTC Feb 20 00:59:00] info : 'warden' stop: /var/vcap/jobs/dea_next/bin/warden_ctl
[UTC Feb 20 00:59:01] info : 'warden' start: /var/vcap/jobs/dea_next/bin/warden_ctl
[UTC Feb 20 00:59:02] info : 'dea_next' start: /var/vcap/jobs/dea_next/bin/dea_ctl
[UTC Feb 20 00:59:13] info : 'warden' connection succeeded to UNIX[/var/vcap/data/warden/warden.sock]
```

I couldn't see any reason to maintain the http socket or health check but was reluctant to pull it out. If that's desired, I can amend this PR or create a new one.

[1] https://gist.github.com/sykesm/9101882