Fix allocations of SmallVector and SmallPtrSet so they are more prone to
being power-of-two sized.