Manually install astropy

This is needed to work around https://github.com/aplpy/aplpy/issues/181.