Speed up MemoryPool#remove_for_block