Remove spec file before building

This might fix #750, but I'm not sure because I can't reproduce it.