wavii/darner — pull request #11 by erikfrey (2012-08-24 00:37:30+00:00)
Title: quick compact: compact only evicted range when it reaches 32MB

When darner pops an item, it writes a deletion marker to its journal. Occasionally, it should sweep through the journal and delete the old items and the deletion markers.

The old approach had two problems:

* Old Darner compacted every 1024 pops, which meant the actual compaction range varied widely depending on item size. With really large items going through the queue, compaction happened infrequently, caused unbounded memory bloat, and was expensive (freeze the world) when it did.

* Old Darner swept the whole journal instead of just the evicted range. Sweeping just the known evicted range is much more efficient than trying to compact the whole journal.