I think I lost a comment by accident.  What did I say?

Another alternative is to allocate these objects on the roxie heap.  That would remove some of the issues of running out of memory.

Your current scheme has a lower overhead if there are fewer duplicates.  However if there are lots (especially out of order) then it would use much more memory.

@richardkchapman Do you have any strong opinions?