Only try to remove the tmp file if we actually created one