mozilla-services/heka — pull request #332 by benmmurphy (2013-07-17 08:44:47+00:00)
Title: Kibana Support

I think this pull request might be a bit controversial because we have added a new elastic search formatter that is very specific to our use case :) So our use case for heka is to replace an existing use of logstash (http://logstash.net) which is being used with kibana (http://three.kibana.org/).

Part of this pull request is adding another Format option "kibana" to ElasticSearchOutput. In this formatter we rename some fields "Hostname" -> "@source_host", add @ prefix to fields like logstash does and move custom fields into an "@fields" structure like logstash does.

We've also used a different method of outputting json. We've ensured that the field names are properly escaped (we have dynamic fields in some of our logs), that invalid utf-8 in field values is replaced with the unicode substitution character and that json is properly escaped. The strconv.Quote function will produce \xZZ and other weird escapes that are not valid json. Though these escapes might still work with elasticsearch. However, we only use this in the "kibana" formatter and not the other formatters. I think there is a good case that we should be using this in the other formatters.

We've also added an option to use the timestamp in the message when creating the name for the index instead of the current time. I'm not sure if this was a feature or a bug.

Also, when we are formatting messages we convert from unix nanos to a go time structure. However, according to go docs this conversation creates a go time structure with a timezone set to local time. The default time format is also "2006-01-02T15:04:05.000Z" which appears to indicate the format should be in UTC time. However, if you try and format a go time structure which has non-utc time with a format that ends with Z it will still format using the local time. In the new kibana formatter and the 'usetimestamp' option for the index we use .UTC() to force the go time structure to UTC before formatting it. We did it this way because kibana requires UTC times. However, we have left the existing time formatting the same because we are not sure if people are relying on this behaviour.