Repository: pypa/twine
Issue category: Feature Request

Issue:
Support the "Trusted Publishing" flow directly in `twine`
Last month PyPI added support for "Trusted Publishing": https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/. This uses an credential exchange flow to swap OIDC credentials with a short-lived PyPI API token, removing the need to maintain long-lived credentials/secrets. (More technical details on the exchange are here: https://docs.pypi.org/trusted-publishers/internals/)

We were able to do launch this without modifying `twine` directly because we instead updated https://github.com/pypa/gh-action-pypi-publish to do the token exchange prior to invoking `twine upload`. 

That said, we're actively working on expanding support to other providers for which we don't have "canonical" workflows (such as https://github.com/pypi/warehouse/issues/13551, https://github.com/pypi/warehouse/issues/13575, and https://github.com/pypi/warehouse/issues/13888).

For these providers (as well as for GitHub users who prefer not to use the `pypi-publish` GitHub action for whatever reason) it would be preferable to have `twine` be able to support the OIDC/API token exchange directly.

This would ideally include:

1. detecting whether an "ambient" OIDC token is available at the time `twine upload` is run (this can be handled by https://pypi.org/p/id, so `twine` doesn't need to understand how to do this for N different providers)
2. verifying that the OIDC token is acceptable for use with PyPI (this just requires that the `aud` claim of the token is set to the audience of the index that `twine` is attempting to upload to)
3. exchanging the OIDC token for a PyPI API token (this is essentially the same as what the `pypi-publish` workflow does here: https://github.com/pypa/gh-action-pypi-publish/blob/110f54a3871763056757c3e203635d4c5711439f/oidc-exchange.py)

The alternative to `twine` supporting this directly is that the end user has to perform a fairly manual token exchange themselves directly (https://docs.pypi.org/trusted-publishers/using-a-publisher/#the-manual-way) which would be pretty bad UX overall.

Opening this issue to make sure maintainers are on board with the general idea before any implementation is started, please let me know if you have any thoughts.

(cc @woodruffw)


Hints:
I wonder if it makes sense to make authentication pluggable in some way? I'm pretty sure that twine right now assumes that every repository is authenticated with a username/password over basic auth. Things like API Tokens from PyPI get shoehorned into this by using a dummy username, but it would be nice to support any auth flow that a repository might want to implement.


Yes, right now a Repository expects username/password https://github.com/pypa/twine/blob/0bb428c410b8df64c04dc881ac1db37d932f3066/twine/repository.py#L45 but we could easily refactor that, since no one has picked up on that being a publicly available API. Regardless, we can make an `authentication` parameter that accepts a pluggable interface. I haven't looked at the interfaces PyPI is expecting, but if this is intended to be contributed, then I'm not sure I or Brian need to care too much.
Thanks for the ping! I agree with the approach that @di has outlined.

> Things like API Tokens from PyPI get shoehorned into this by using a dummy username, but it would be nice to support any auth flow that a repository might want to implement.

I agree with this in principle, although that might be a premature optimization/generalization for this particular use case (since the trusted publishing flow ultimately produces a normal looking PyPI API token). But there may be other indices that I'm not aware of that would benefit from pluggable auth methods?
PyPI itself would benefit from not having to use the BasicAuth hack anymore :)

But more generally, most of the cloud providers don't use long lived secrets that are usable directly with twine, and they expect that you run some command to get a short lived token currently or they "fake" a pluggable auth method by making a keyring backend that doesn't actually do keyring stuff, but that does their auth flow.

**[AWS](https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-twine.html)**

```
$ aws codeartifact login --tool twine --domain my_domain --domain-owner 111122223333 --repository my_repo
$ twine upload -r codeartifact dist/*
```

**[Google](https://cloud.google.com/artifact-registry/docs/python/authentication)**

```
# Deps
$ pip install keyring
$ pip install keyrings.google-artifactregistry-auth

# Teach Keyring where your auth file is
export GOOGLE_APPLICATION_CREDENTIALS=KEY-FILE
# OR
gcloud auth activate-service-account --key-file=KEY-FILE

# Upload
twine upload -r $PYTHON-REPO-ID dist/*
```

**[Azure](https://learn.microsoft.com/en-us/azure/devops/artifacts/quickstarts/python-packages?view=azure-devops)**

```
$ pip install artifacts-keyring
$ twine upload --repository-url https://pkgs.dev.azure.com/<org_name>/_packaging/<feed_name>/pypi/upload dist/*
```


So I think in principle there's already a need for this (and honestly the same is true for pip), and currently people are working around the lack of pluggable authentication by treating the keyring library as that pluggable authentication.

Using the keyring library works, but it's kind of an abuse of the keyring API to do that. Keyrings are supposed to support `get`, `set`, and `delete` for passwords, but all these things (Trusted Publishing included) really care about is supporting `get`, and the other operations don't make sense and do nothing.
I've been wondering if it makes sense to have vendor-specific OIDC in twine ever since it was implemented in the action… I think that this might not belong in twine because this is something specific to runtimes where twine is used, not twine itself. Implementing it here would introduce the burden of having to track ways of identifying if something is a GitHub or GitLab Runner, or some AWS env etc., and actively monitoring changes in those. Besides, twine calls are often wrapped via other command runners like tox. Tox screens the environment variables by default, making the auto-detection broken, putting the burden of _knowing upfront_ that they must to pre-configure tox to pass specific environment variables that twine needs (and knowing what those are!) in order for it to work. I imagine other command wrappers may behave similarly.

Keeping it in a GitHub Action, though, eliminates said burdens (and it also allows us to push the best practices to the wider community). I know that for GitLab, it'd also make sense to make some reusable pipeline.

The idea with plugins is interesting, but why would one install such a plugin locally? They'd probably need to add their own env detection to install it conditionally, since it'd only be really needed @ GHA.

I don't have very strong opinions on this, but I wanted to post the above thoughts to see if anybody else thinks that maybe having this inside twine might be not what people really need…
This may help: 

https://github.com/marketplace/actions/pypi-token-mint 

Gets the token and nothing else...
> Keeping it in a GitHub Action, though, eliminates said burdens (and it also allows us to push the best practices to the wider community). I know that for GitLab, it'd also make sense to make some reusable pipeline.

How about people that don't use GitHub actions? Would it be possible for twine to integrate with other CI providers that offer OIDC capability?
> How about people that don't use GitHub actions? Would it be possible for twine to integrate with other CI providers that offer OIDC capability?

Each OIDC provider will have to be special-cased. So if an ecosystem has standard means of implementing this, I probably wouldn't drag that into twine.
OTOH, plugins seems like a reasonable middle ground.
> This may help:
> 
> [github.com/marketplace/actions/pypi-token-mint](https://github.com/marketplace/actions/pypi-token-mint)
> 
> Gets the token and nothing else...

The example there looks harmful security-wise — it puts the build step in a job with OIDC privileges. If you follow the security best practice and build in a separate job, you won't really need poetry for publishing. Hence, no need for the action as pypi-publish has that embedded.
> Would it be possible for twine to integrate with other CI providers that offer OIDC capability?

FWIW, this is the idea behind `id` (https://github.com/di/id) (which is what the current official publish action uses under the hood): it abstracts the OIDC token retrieval operation across any of its supported providers.

In principle, `twine` should be able to depend on `id` and do `id.detect_credential(audience="pypi")` to obtain an OIDC credential in a platform-agnostic manner.
> to obtain an OIDC credential in a platform-agnostic manner.

But the calling code would still have to special-case working with different platforms, right?
> But the calling code would still have to special-case working with different platforms, right?

Yes, good point: at the moment the token exchange needs to be done on a provider-specific endpoint (e.g. `/_/oidc/github/mint-token/` for GitHub). It's possible we could change that in the future (and perform OIDC validation against all known IdPs rather than expecting a match on a specific pre-identified one), however.
> It's possible we could change that in the future (and perform OIDC validation against all known IdPs rather than expecting a match on a specific pre-identified one), however.

That does sound excessive, though.
Yeah, potentially 🙂 -- I'll need to think about it some more, but doing so shouldn't be _too_ heavyweight: all of these OIDC tokens are keyed on a `keyid` anyways, so fanning across multiple different potential verifiers should be on the same order of complexity as a single provider with multiple keys (which we already support).

Either way, that's all hypothetical 🙂 
I am trying to understand here what are the interoperability challenges (sorry, last time I used openID connect was ages ago in a very specific context). Any pointers on that? As a standard it should be able to offer some form of interoperability...

Is it a matter of parametrising the endpoints and/or asking for the user to provide the token (with the audience set to `pypi`) as input?[^1]

I tried to have a look on the links in the announcement post (e.g. https://docs.pypi.org/trusted-publishers/) but they are either too generic or too Github-centric, so I am not sure I understand...

[^1]: I mean: is the interoperability problem only the "automatic detection"?
There's also https://blog.trailofbits.com/2023/05/23/trusted-publishing-a-new-benchmark-for-packaging-security/ / https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/.
Thank you very much @webknjaz, after reading it seems to me that PyPI for the moment is not supporting generic "trusted publishers" with OIDC, but rather "one trusted publisher" which is GitHub. If that is the case, it does seem that concentrating in the GitHub actions is the best. (I might be wrong, but that is what seems to me, specially after going through https://docs.pypi.org/trusted-publishers/using-a-publisher/#the-manual-way).
On top of what @webknjaz said: the primary interop problem with OIDC is that each IdP has a lot of control over the claims that it exposes, so PyPI is responsible for papering over the differences between them in a way that's coherent/useful for end users.

But yeah, GitHub is really the only currently supported IdP (support for others is currently in progress).
> It's possible we could change that in the future (and perform OIDC validation against all known IdPs rather than expecting a match on a specific pre-identified one), however.

I think we could do that (likely based on the `iss` claim alone, have a provider-specific endpoint is unnecessary).
@abravalheri here's some other OIDC integrations scheduled FYI: https://github.com/pypi/warehouse/issues?q=is%3Aopen+label%3Atrusted-publishing+sort%3Aupdated-desc.
Just to post an update here: PyPI now supports 4 different trusted publisher providers (GitHub, GitLab, Google, and ActiveState), and the token minting endpoint is now fully generic (`/_/oidc/github/mint-token/` is now `/_/oidc/mint-token/`). So this should be relatively easy to add to `twine` in a generic fashion 🙂 
I just noticed that the docs for [trusted publishing from Gitlab](https://docs.pypi.org/trusted-publishers/using-a-publisher/#gitlab-cicd) point you to using `curl` & `jq` for operations that could be done more elegantly in Python code. So I might work on this, if no-one else already is.

Specifically, I'm thinking:

- Add an optional dependency on `id`, so you would `pip install twine[oidc]` to use this
- Use with a `--oidc` flag, mutually exclusive with `--username` & `--password`
- When specified, it uses `id` to get the OIDC token for PyPI, makes the `mint-token` request to PyPI to get an API token, and then uploads with this as the 'password'
- Does it need options to customise the token audience & the `mint-token` URL to use, or can this all be based on the repository URL? I see in other docs that the audience can be fetched from `{index}/_/oidc/audience`, but it's not quite clear if that's a detail of Warehouse or a standard. Although I imagine for now this distinction is probably academic.
> I just noticed that the docs for trusted publishing from Gitlab point you to using curl & jq for operations that could be done more elegantly in Python code.

In terms of reference Python code, the `oidc-exchange.py` inside of pypi-publish does the same thing as the shell commands in the docs, so that would be a good starting point!

> So I might work on this, if no-one else already is.

I was planning on working on it in the near future, but you should go ahead! 

> Use with a --oidc flag, mutually exclusive with --username & --password

As a bikeshed: it might be worth naming this `--trusted-publishing` or similar, since OIDC is an implementation detail. But that's also long so I'm curious about ideas for better names.

> Does it need options to customise the token audience & the mint-token URL to use, or can this all be based on the repository URL? I see in other docs that the audience can be fetched from {index}/_/oidc/audience, but it's not quite clear if that's a detail of Warehouse or a standard. Although I imagine for now this distinction is probably academic.

Definitely academic for now 🙂, but yes: the token audience needs to be customized to whatever the index expects. The mint-token URL itself no longer needs to be customized however; PyPI now uses a single URL for all of the different Trusted Publishing providers, and disambiguates them internally.

Patch:
diff --git a/pyproject.toml b/pyproject.toml
index 582ccaa9..e976d56d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -42,6 +42,7 @@ dependencies = [
 	"rfc3986 >= 1.4.0",
 	"rich >= 12.0.0",
 	"packaging",
+	"id",
 ]
 dynamic = ["version"]
 
diff --git a/twine/auth.py b/twine/auth.py
index 59e2e21d..49ab8987 100644
--- a/twine/auth.py
+++ b/twine/auth.py
@@ -1,17 +1,25 @@
 import functools
 import getpass
+import json
 import logging
 from typing import TYPE_CHECKING, Callable, Optional, Type, cast
+from urllib.parse import urlparse
+
+from id import AmbientCredentialError  # type: ignore
+from id import detect_credential
 
 # keyring has an indirect dependency on PyCA cryptography, which has no
 # pre-built wheels for ppc64le and s390x, see #1158.
 if TYPE_CHECKING:
     import keyring
+    from keyring.errors import NoKeyringError
 else:
     try:
         import keyring
+        from keyring.errors import NoKeyringError
     except ModuleNotFoundError:  # pragma: no cover
         keyring = None
+        NoKeyringError = None
 
 from twine import exceptions
 from twine import utils
@@ -28,7 +36,11 @@ def __init__(
 
 
 class Resolver:
-    def __init__(self, config: utils.RepositoryConfig, input: CredentialInput) -> None:
+    def __init__(
+        self,
+        config: utils.RepositoryConfig,
+        input: CredentialInput,
+    ) -> None:
         self.config = config
         self.input = input
 
@@ -57,9 +69,65 @@ def password(self) -> Optional[str]:
             self.input.password,
             self.config,
             key="password",
-            prompt_strategy=self.password_from_keyring_or_prompt,
+            prompt_strategy=self.password_from_keyring_or_trusted_publishing_or_prompt,
         )
 
+    def make_trusted_publishing_token(self) -> Optional[str]:
+        # Trusted publishing (OpenID Connect): get one token from the CI
+        # system, and exchange that for a PyPI token.
+        repository_domain = cast(str, urlparse(self.system).netloc)
+        session = utils.make_requests_session()
+
+        # Indices are expected to support `https://{domain}/_/oidc/audience`,
+        # which tells OIDC exchange clients which audience to use.
+        audience_url = f"https://{repository_domain}/_/oidc/audience"
+        resp = session.get(audience_url, timeout=5)
+        resp.raise_for_status()
+        audience = cast(str, resp.json()["audience"])
+
+        try:
+            oidc_token = detect_credential(audience)
+        except AmbientCredentialError as e:
+            # If we get here, we're on a supported CI platform for trusted
+            # publishing, and we have not been given any token, so we can error.
+            raise exceptions.TrustedPublishingFailure(
+                "Unable to retrieve an OIDC token from the CI platform for "
+                f"trusted publishing {e}"
+            )
+
+        if oidc_token is None:
+            logger.debug("This environment is not supported for trusted publishing")
+            return None  # Fall back to prompting for a token (if possible)
+
+        logger.debug("Got OIDC token for audience %s", audience)
+
+        token_exchange_url = f"https://{repository_domain}/_/oidc/mint-token"
+
+        mint_token_resp = session.post(
+            token_exchange_url,
+            json={"token": oidc_token},
+            timeout=5,  # S113 wants a timeout
+        )
+        try:
+            mint_token_payload = mint_token_resp.json()
+        except json.JSONDecodeError:
+            raise exceptions.TrustedPublishingFailure(
+                "The token-minting request returned invalid JSON"
+            )
+
+        if not mint_token_resp.ok:
+            reasons = "\n".join(
+                f'* `{error["code"]}`: {error["description"]}'
+                for error in mint_token_payload["errors"]
+            )
+            raise exceptions.TrustedPublishingFailure(
+                "The token request failed; the index server gave the following "
+                f"reasons:\n\n{reasons}"
+            )
+
+        logger.debug("Minted upload token for trusted publishing")
+        return cast(str, mint_token_payload["token"])
+
     @property
     def system(self) -> Optional[str]:
         return self.config["repository"]
@@ -90,6 +158,8 @@ def get_password_from_keyring(self) -> Optional[str]:
             username = cast(str, self.username)
             logger.info("Querying keyring for password")
             return cast(str, keyring.get_password(system, username))
+        except NoKeyringError:
+            logger.info("No keyring backend found")
         except Exception as exc:
             logger.warning("Error getting password from keyring", exc_info=exc)
         return None
@@ -102,12 +172,19 @@ def username_from_keyring_or_prompt(self) -> str:
 
         return self.prompt("username", input)
 
-    def password_from_keyring_or_prompt(self) -> str:
+    def password_from_keyring_or_trusted_publishing_or_prompt(self) -> str:
         password = self.get_password_from_keyring()
         if password:
             logger.info("password set from keyring")
             return password
 
+        if self.is_pypi() and self.username == "__token__":
+            logger.debug(
+                "Trying to use trusted publishing (no token was explicitly provided)"
+            )
+            if (token := self.make_trusted_publishing_token()) is not None:
+                return token
+
         # Prompt for API token when required.
         what = "API token" if self.is_pypi() else "password"
 
diff --git a/twine/cli.py b/twine/cli.py
index 6ff60d4d..f44426e3 100644
--- a/twine/cli.py
+++ b/twine/cli.py
@@ -81,6 +81,7 @@ def list_dependencies_and_versions() -> List[Tuple[str, str]]:
         "requests",
         "requests-toolbelt",
         "urllib3",
+        "id",
     ]
     if sys.version_info < (3, 10):
         deps.append("importlib-metadata")
diff --git a/twine/exceptions.py b/twine/exceptions.py
index b87b938b..29b7d8a1 100644
--- a/twine/exceptions.py
+++ b/twine/exceptions.py
@@ -116,6 +116,12 @@ class NonInteractive(TwineException):
     pass
 
 
+class TrustedPublishingFailure(TwineException):
+    """Raised if we expected to use trusted publishing but couldn't."""
+
+    pass
+
+
 class InvalidPyPIUploadURL(TwineException):
     """Repository configuration tries to use PyPI with an incorrect URL.
 
diff --git a/twine/repository.py b/twine/repository.py
index b512bfb3..f04f9dbc 100644
--- a/twine/repository.py
+++ b/twine/repository.py
@@ -12,18 +12,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import Any, Dict, List, Optional, Set, Tuple, cast
+from typing import Any, Dict, List, Optional, Set, Tuple
 
 import requests
 import requests_toolbelt
 import rich.progress
-import urllib3
-from requests import adapters
-from requests_toolbelt.utils import user_agent
 from rich import print
 
-import twine
 from twine import package as package_file
+from twine.utils import make_requests_session
 
 KEYWORDS_TO_NOT_FLATTEN = {"gpg_signature", "attestations", "content"}
 
@@ -47,7 +44,7 @@ def __init__(
     ) -> None:
         self.url = repository_url
 
-        self.session = requests.session()
+        self.session = make_requests_session()
         # requests.Session.auth should be Union[None, Tuple[str, str], ...]
         # But username or password could be None
         # See TODO for utils.RepositoryConfig
@@ -57,35 +54,10 @@ def __init__(
         logger.info(f"username: {username if username else '<empty>'}")
         logger.info(f"password: <{'hidden' if password else 'empty'}>")
 
-        self.session.headers["User-Agent"] = self._make_user_agent_string()
-        for scheme in ("http://", "https://"):
-            self.session.mount(scheme, self._make_adapter_with_retries())
-
         # Working around https://github.com/python/typing/issues/182
         self._releases_json_data: Dict[str, Dict[str, Any]] = {}
         self.disable_progress_bar = disable_progress_bar
 
-    @staticmethod
-    def _make_adapter_with_retries() -> adapters.HTTPAdapter:
-        retry = urllib3.Retry(
-            allowed_methods=["GET"],
-            connect=5,
-            total=10,
-            status_forcelist=[500, 501, 502, 503],
-        )
-
-        return adapters.HTTPAdapter(max_retries=retry)
-
-    @staticmethod
-    def _make_user_agent_string() -> str:
-        user_agent_string = (
-            user_agent.UserAgentBuilder("twine", twine.__version__)
-            .include_implementation()
-            .build()
-        )
-
-        return cast(str, user_agent_string)
-
     def close(self) -> None:
         self.session.close()
 
diff --git a/twine/utils.py b/twine/utils.py
index 10e374e7..2bc56d0b 100644
--- a/twine/utils.py
+++ b/twine/utils.py
@@ -25,7 +25,11 @@
 
 import requests
 import rfc3986
+import urllib3
+from requests.adapters import HTTPAdapter
+from requests_toolbelt.utils import user_agent
 
+import twine
 from twine import exceptions
 
 # Shim for input to allow testing.
@@ -304,6 +308,28 @@ def get_userpass_value(
 get_clientcert = functools.partial(get_userpass_value, key="client_cert")
 
 
+def make_requests_session() -> requests.Session:
+    """Prepare a requests Session with retries & twine's user-agent string."""
+    s = requests.Session()
+
+    retry = urllib3.Retry(
+        allowed_methods=["GET"],
+        connect=5,
+        total=10,
+        status_forcelist=[500, 501, 502, 503],
+    )
+
+    for scheme in ("http://", "https://"):
+        s.mount(scheme, HTTPAdapter(max_retries=retry))
+
+    s.headers["User-Agent"] = (
+        user_agent.UserAgentBuilder("twine", twine.__version__)
+        .include_implementation()
+        .build()
+    )
+    return s
+
+
 class EnvironmentDefault(argparse.Action):
     """Get values from environment variable."""
 


Test patch: