Remove phantom cleanup
Merge pull request #637 from alphagov/remove-phantom-cleanup

Remove phantom cleanup
Add missing logs-elasticsearch-3 to box checks

When logs-elasticsearch-3 was created it was missed off being added to
the list of machines that monitoring-1 should watch. It should be in
this list so I have added it.
We need to support installing oracle-java-7

The logstash package expects oracle's java to be installed so our java
module needs to support all of the bumf around installing that pacakge.
Upgrade logstash to 1.4.2

We were previously running 1.1.* and a lot has changed to logstash in
the interim. The puppet module has also changed significantly so there
is quite a lot in this commit. Below is a link to the logstash changelog
for your perusal:

   https://github.com/elasticsearch/logstash/blob/master/CHANGELOG

Rabbitmq module had to be updated to handle the newer version of rabbit
and to fix some issues with initial provisioning of machines.

I have split up the old monitoring.pp into a directory with the three
main groups of configuration in it (sensu, graphite and logstash). This
was done to aid with clarity as it was getting a little gnarly in that
file having a whole heap of tenuously related configuration.

The new logstash apt package from the repo will expect oracle's version
of java to be installed so I have had to change the java module (in a
previous commit) and configure monitoring to install oracle's version.

We can no longer run two agents with the new puppet module, hopefully
the work on performance and reliability between version 1.1 and 1.4 will
mean that this is no longer needed.

We no longer need to setup a template for elasticsearch as logstash will
now manage this for us and set it up with sensible defaults.

I have installed the kopf plugin for elasticsearch, which is a better
version of the head plugin we are used to using. This has been
instrumental in helping me diagnose issues between logstash and
elasticsearch so I thought I'd include it in this commit.

In the /etc/defaults files we drop in as part of install logstash we set
its user to be root, the module would usually use a logstash user.
This is because logstash will need to bind to port
514 (syslog port) and any port beneath 1024 requires root access to
bind. I tried using setcap to allow logstash to bind to the port without
root access but because the jvm is the executable that it would need to
be applied to I decided not to as I didn't want to allow any jvm process
to bind to these ports.

I have introduced a require on in the elasticsearch_logs checks so that
it doesn't try and drop files in before the /etc/sensu directory has
been created on the server.

The monitoring::graphite class has a new resource ordering to force the
installation of ::graphite before ::nginx. This is because the ::graphite
class will install apache and reconfigure it to run on a custom port but
I had problems where nginx would be setup before apache being
reconfigured. This ended up with errors in the initial provision because
nginx couldn't bing to port 80.

All of the inputs, filters and outputs have been migrated over to files
as the new puppet-logstash module does not support the same set of
resources to setup these files.