Replace some typically large vectors with SmallVector.

This may seem counter-intuitive but the POD-like optimization helps when the
vectors grow into multimegabyte buffers. SmallVector calls realloc which knows
how to twiddle virtual memory bits and avoids large copies.