Remove astropy from requirements - we don't actually use it yet.

Although we might in future, since it replaces pyfits.