Avoid some unnecessary SmallVector copies.

No functionality change.