Make the Jenkins log pusher a service.

* modules/openstack_project/manifests/logstash.pp: Run the Jenkins log
pusher script as a service. This is the first step in making Logstash
use the TCP inputs instead of pipe inputs.

* modules/openstack_project/files/logstash/jenkins-log-pusher.init: Add
a simple init script for the Jenkins log pusher.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Switch
to TCP input instead of pipe input as the new Jenkins log pusher service
will push log events over TCP.

Change-Id: Id80c710abd5facd71d18afb2b250b2d7d92dec2d
Reviewed-on: https://review.openstack.org/28074
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Make log-pusher.py properly configurable.

* modules/openstack_project/files/logstash/log-pusher.py: Make the log
pusher properly configurable with a yaml configuration. As part of this
change support multiple zmq publisher inputs, multiple file retrievers,
job name filtering, and event tagging (with the filename).

* modules/openstack_project/files/logstash/jenkins-log-pusher.yaml:
Initial config for the log pusher.

* modules/openstack_project/manifests/logstash.pp: Put new log pusher:
config in place.

* modules/openstack_proejct/files/logstash/jenkins-log-pusher.init: Run
the log pusher service with the new config file.

Change-Id: I4c8405b1edfa16bbcc8f998627c6240bef23f302
Reviewed-on: https://review.openstack.org/28113
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Use log-pusher injected tags during logstash grok

* modules/openstack_project/templates/logstash/indexer.conf.erb: The
log-pusher.py script tags events with the filename of the log generating
the event. Use these values instead of different types to differentiate
Jenkins console logs from nova logs and so on.

Note that filters must match all of the values in the tags array (they
are ANDed together). This may mean that as the logstash filter rulesets
grow we will need a set of initial mutates to convert
'logs/screen-n-api.txt' and 'logs/screen-n-cpu.txt' tags to 'nova' to
allow for common grok and parsing filters.

Change-Id: I2769bc05a2e9cc7e8dbc46849e052146b9fee75e
Reviewed-on: https://review.openstack.org/28119
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins