Don't spill more blocks than we need to

Also fixes a memory leak
IIUC, I think it should be `!inMemoryBlocks.isEmpty`.