josegonzalez/python-beaver — pull request #282 by KaanErturk (2014-12-19 09:02:31+00:00)
Title: Add some improvements to the previous message_batch size checking update

Couple of changes to make sure we don't send more than 256KiB (with some headroom left) in batch:

* Drop the message if it's larger than the maximum size
* Check if total message size "is going to" exceed the maximum size, before it does. This makes sure message size is always under the maximum value
* Don't send an empty batch

Beaver on multiple environments were having issues when some of the log files were rotated; as they were receiving huge amounts of data at once. That needed checking the individual messages and dropping them in that case. This was already implemented in #204 and I regretfully hadn't paid the necessary attention to it, apologies for that. So this fix has some idea and code from the commit in issue #204, along with some more improvements and Beaver log message changes for clarity. To be applied on master branch which already has the commit from #280.

I've been testing this since yesterday (~15 hours) and I can say the fix does its job well as I haven't seen any issues on 4 different environments.