Only upgrade numpy -- everything else should use the specified versions.