Shrink SmallPtrSet. It gets swapped a lot which copies the whole small part.

Testing shows that it's empty in >99% of the cases and I couldn't find a case
where it contained more than 2 elements.