heroku/logplex — pull request #52 by ferd (2013-08-07 19:25:35+00:00)
Title: OOM remediation patches

Logplex has had plenty of OOM errors in productions lately, related to
contention on logging processes, refc binary leaks, and blocking operations
in drains.

This branch contains fixes to attempt to fix these, although without guarantee:

- forcing hibernation on drain reconnection, to force fullsweep GCs and help with
refc leaks.
- forcing hibernation on inactive drains, to save memory, reduce fragmentation,
trigger fullsweep GCs when it won't hurt.
- make `logplex_msg_buffer` use explicit sizes instead of re-counting queue length
dynamically all the time.
- Move to lager for logging instead of io:format/2.

It's possible these commit won't accomplish much more than partial remediation. I
would then recommend maybe trying them live before merging them to master, seeing
what helps and what doesn't.