stewi2/flume-ng-ext — pull request #1 by jvorenkamp (2013-06-03 21:55:39+00:00)
Title: DATA-140 Add header to unparsable events and allow them to continue in the flow

Default parsers simply remove the events should an error occur. I think we want to keep them around to avoid any loss of data should a problem occur and all of a sudden all the log messages are unparsable. This change adds a new header to each nginx event: parsable: (true | false).

In the configuration we will add a channel selector off of this value, and send any unparsable events to a new sink (log file):

# channel selector configuration
writer.sources.nginx.selector.type=multiplexing
writer.sources.nginx.selector.header=parsable
writer.sources.nginx.selector.mapping.true=nginx-memory
writer.sources.nginx.selector.mapping.false=nginx-error
writer.sources.nginx.selector.default=nginx-error

# channel for unparsable events
writer.channels.nginx-error.type=memory
writer.channels.nginx-error.capacity=20000
writer.channels.nginx-error.transactionCapacity=10000

# sink error lines to log file
writer.sinks.nginx-file.type=file_roll
writer.sinks.nginx-file.sink.directory=/var/spool/flume/nginx_error

# bind the error sink to the error channel
writer.sinks.nginx-file.channel=nginx-error