Removed the std::set.  Now everything gets stored in a single contiguous block of memory over a gig in size.  This should help deal with memory fragmentation.