We may need to keep scipy; I think there's currently a scipy dependency somewhere in astroquery