Fix rebuilding bug

https://github.com/astropy/astropy/issues/2531