Remove the entire tempdir we use to host the spec file