cityindex/logsearch-development-flow — pull request #198 by dpb587 (2013-10-08 13:05:12+00:00)
Title: Logstash 1.2 - `datetime` filter

With logstash 1.2, we could simplify our timezone injection code thanks to [support](https://github.com/logstash/logstash/blob/v1.2.1/CHANGELOG#L98) for an explicit `timezone` [field](http://logstash.net/docs/1.2.1/filters/date#timezone) in the `datetime` [filter](http://logstash.net/docs/1.2.1/filters/date). The only drawback is that currently we can support message senders dynamically overriding the timezone, but the added `timezone` property does not seem capable of supporting dynamic field lookups from the message. Additionally, we cannot use `+01:00` and must instead use them in the format of `Europe/London` ([full list](http://joda-time.sourceforge.net/timezones.html)) - which simplifies daylight savings concerns and does seem better in general.

Obviously the official implementation is much simpler and easier to maintain when it comes to upstream breaks. So, if we can *always* assume a timezone for a given log type, this should be a safe change. If a given log type may use multiple timezones and needs `@source_tz`, I don't think we can. Currently only `ci_log4net` observes this custom timezone hack within the cloud instances.

We should re-familiarize ourselves with how our shippers are shipping to determine if this is something we can change. This obviously won't go into effect until we're able to switch over the stack to logstash 1.2 and merge the rest of the `logstash-1.2` branch.