Shrink SmallPtrSet. It gets swapped a lot which copies the whole small part.

Testing shows that it's empty in >99% of the cases and I couldn't find a case
where it contained more than 2 elements.

git-svn-id: http://llvm.org/svn/llvm-project/cfe/trunk@175967 91177308-0d34-0410-b5e6-96231b3b80d8