Ensure we remove the signature in case it's a retry