Remove the temporary directory whatever happens