logstash/logstash — pull request #793 by 20goto10 (2013-11-22 16:25:42+00:00)
Title: Add refiltering option for created events

Currently in logstash an event created by a filter (e.g. "split") is not processed a second time by the filters. This PR adds a "refilter" which will reinsert new events into the chain for further processing. Tags and conditionals in the configuration can be used to ensure that the filter is only applied to the created events.

Example filter configuration -- in this example, my log data is presented in one large JSON array under "message" => "Records", which must be split to be handled correctly. On the second pass, which only filters the created events, the date filter is used to correct the @timestamp.

filter {
if !("splitted" in [tags])
{
json {
source => "message"
}
split {
field => "Records"
reuse_element => false
refilter => true
add_tag => ["splitted"]
}
}
if ("splitted" in [tags])
{
date {
add_tag => ["dated"]
match => ["eventTime", "ISO8601"]
}
}
}