Really cleans-up the temp directory if it has been created by a previous worker