fluent/fluent-plugin-scribe — pull request #4 by krobertson (2012-05-30 06:11:30+00:00)
Title: Fix for inefficient memory handling

We've been using the fluent-plugin-scribe in production for a couple of weeks and love it, but today we encountered a problem where some data got backed up, and it then it grew into a bigger issue as we eventually found some issues with the plugin's memory handling.

In fluent, it defaults to a file chunk being 256mb. For us, this is about 700k log messages.

There were two main issues:

1) The plugin has the entire file chunk in memory 3x. Once as it is passed in, it then loops over and msgpacks all the messages, and then it loops over that and creates a LogEntry for each.
2) It attempts to push all the messages to scribe in one call. We found this would typically time out.

I rewrote the plugin to focus on processing in batch. It will default to pushing to scribe every 10k messages, clears the set between batches, and only aggregates the LogEntries rather than msgpack and LogEntries.

As an example of the memory usage before, we were seeing a single thread of the scribe plugin using 1gb resident set size, taking forever to process a single file, and it was also preventing additional data from coming in due to heavily swapping and causing high IO wait (ec2, swap on ephemeral storage).

After the change, with a 1k batch size it was at 36mb RSS after 2 million messages. Upped it to 10k batch size and had it averaging 150mb RSS.

Although a lowered file chunk size would negate the batch size, I don't consider fluent's 256mb as a sane default in this particular use case. I think the batch is a good safety.