cityindex/logsearch — pull request #288 by sopel (2013-12-27 16:52:12+00:00)
Title: Reconfigure status check alarms

This implements #285 based on the analysis in https://github.com/cityindex/labs-operations/issues/109 as follows:

* it splits the `StatusCheckFailed` alarm into one for each constituent `StatusCheckFailed_Instance` and `StatusCheckFailed_System` and reconfigures the former 10 minute sensitivity to
* `StatusCheckFailed_Instance` => _Max = 1 for 5 periods of 60 seconds_ to hide the reachability failure implied in the nightly reboot, i.e. sensitivity for these is going to be 5 minutes going forward
* `StatusCheckFailed_System` => _Max = 1 for 1 periods of 60 seconds_ to provide the earliest possible notification for AWS system failures, i.e. sensitivity for these is going to be 1 minute going forward

:information_source: this is mostly a temporary measure to removed/reduce the current notification noise (and increase alarm sensitivity on the side), which might become partially obsolete or require further adjustments by converting one or more of see affected tiers to Auto Scaling groups (see https://github.com/cityindex/logsearch/issues/149#issuecomment-30311478).

@dpb587 - other than the aforementioned topic, this is ready for review/merge as such.