Const-correct and prevent a copy of a SmallPtrSet.