[PCH] Remove building a SmallPtrSet that is not actually used for anything.