Build NumPy, SciPy, and matplotlib at the same time.

This avoids them accidentally pulling in a different version than the
one that's pinned.