mozilla-services/heka — pull request #809 by mattcottingham (2014-04-27 21:58:01+00:00)
Title: Support reading of gzipped files with logstreamer

This checks if log files look like they are gzipped (ending in ".gz") and creates a gzip reader if so.

In places where Read is called on the current file descriptor, an io.Reader value is used instead of the fd.

Where files are seeked, bytes are consumed from the gzip reader if one exists. Because we only track the number of bytes read, we can't use this number to seek in a gzipped file (file bytes don't correspond to uncompressed bytes read). There might be a better way.

This could do with more tests, but I'd like to get feedback on the design first.

So far I've tested this with the following config, with and without `/var/hekad/cache/*`:

```
[LogstreamerInput]
log_directory = "/var/log"
file_match = 'syslog\.?(?P\d*).*'
priority = ["^Seq"]

[LogOutput]
message_matcher = "TRUE"
payload_only = true
```

It processes compressed and uncompressed syslogs. The current tests all pass.