logstash/logstash — pull request #194 by JeremyEinfeld (2012-08-29 00:08:59+00:00)
Title: Correction to GELF forwarding

Our need: we are going to be sending GELF messages to logstash which would then be translated back to GELF messages for Graylog2.

The problem: the simplest config would be as follows:
input {
gelf {
type => "gelf"
}
}
output {
gelf {
type => "gelf"
host => "127.0.0.1"
port => 12202
}
}

Unfortunately, that results in nearly all the GELF data being lost. The ideal is that if you pointed the GELF output at the GELF input, the data would just keep cycling (and would remain consistent). Unfortunately, with each iteration all fields would be prepended with another "_" and none of correct data would be used for the output GELF message.

Some of the problems could have been fixed by specifying certain values, but in the end you would have a GELF message like such at the output (BADDATA is where the data did not originate from the input GELF message):

{
"version": "BADDATA",
"host": "BADDATA",
"short_message": "BADDATA",
"full_message": "Backtrace here\n\nmore stuff",
"timestamp": BADDATA,
"level": BADDATA,
"facility": "BADDATA",
"file": "BADDATA",
"line": BADDATA,
"_user_id": 42,
"_version": "1.0",
"_host": "www1",
"_short_message": "Short message",
"_full_message": "Backtrace here\n\nmore stuff",
"_timestamp": 1291899928.412,
"_level": 1,
"_facility": "payment-backend",
"_file": "/var/www/somefile.rb",
"_line": 356,
"__user_id": 42,
}

What I have changed here is making the GELF input map a few more values from the GELF values (host and file) and the GELF message try to pull other GELF fields from the logstash message object.

I have tried to preserve the old data, so that existing uses of the GELF output are not broken. With the changes I have made, the following config is enough to correctly forward a GELF message:

input {
gelf {
type => "gelf"
}
}
output {
gelf {
type => "gelf"
host => "127.0.0.1"
port => 12202
level => "%{level}"
facility => "%{facility}"
}
}