Repository: pypa/twine

Issue:
Automatically refresh short-lived PyPI token in long Trusted Publishing uploads
### Is there an existing issue for this?

- [x] I have searched the existing issues (open and closed), and could not find an existing issue

### What keywords did you use to search existing issues?

trusted publish
oidc
timeout

### Please describe the problem you are attempting to solve with this request

When uploading several large distributions to an index, depending on their size and network conditions, uploads might take more than 15 mins.

In Trusted Publishing flows where the OIDC token is exchanged for a PyPI token that is short-lived (15 mins), `twine upload` will fail between file uploads if the 15 mins are up and there are still files left to upload.

There's at least one case of someone hitting this issue (see [here](https://github.com/pypa/gh-action-pypi-publish/issues/307)). 




### How do you think we should solve this?

A possible solution would be for `twine` to automatically request a new PyPI token between file uploads, if more than 15 minutes have passed since the original token was requested.

### Anything else you'd like to mention?

cc @woodruffw @webknjaz 


Discussion:
Thanks @facutuesca!

For context/cross-referencing: this came up recently at PyCon because of an initial attempt to switch `gh-action-pypi-publish` to using `twine` for the Trusted Publishing exchange directly: https://github.com/pypa/gh-action-pypi-publish/pull/360


I think I see some easy refactoring for this but I'm on a phone so not able to describe it succinctly. When I get to a keyboard I'll write it up better 
*(more context)*

> There's at least one case of someone hitting this issue (see [here](https://github.com/pypa/gh-action-pypi-publish/issues/307)).

That time, it was happening while PyPI had network issues or something like that. So the likelihood of more projects hitting this increases when their uploads are large and PyPI has instability.

> A possible solution would be for `twine` to automatically request a new PyPI token between file uploads, if more than 15 minutes have passed since the original token was requested.

I'd say 15 - 1 = 14 minutes to allow for possible time drift in between envs/servers. This is a standard practice in the GitHub Apps ecosystem nowadays, for example.
So today, we have authentication resolution via [Settings](https://github.com/pypa/twine/blob/4b6c50bf858604df5cab98f15c2c691eb3bf9dfe/twine/settings.py#L128). This, however, only ever gets a single token. So with the current design we'd need to reinitialize this over and over again somehow, despite not having that kind of reach. The thing we do have is the ability to (from [Settings](https://github.com/pypa/twine/blob/4b6c50bf858604df5cab98f15c2c691eb3bf9dfe/twine/settings.py#L331)) create the Repository we're uploading to.

With this in mind, I would argue that we change the following:

* [Repository](https://github.com/pypa/twine/blob/4b6c50bf858604df5cab98f15c2c691eb3bf9dfe/twine/repository.py#L36-L42) should not take `username` and `password` (or if it does, it's mutually exclusive with the new parameter) but instead it takes `auth` (or `auth_helper`) or whatever we want to call it
* [Resolver](https://github.com/pypa/twine/blob/4b6c50bf858604df5cab98f15c2c691eb3bf9dfe/twine/auth.py#L38) should be rethought entirely to have a `resolve` method which returns an object that implements the [Custom Authentication](https://requests.readthedocs.io/en/latest/user/advanced/#custom-authentication) required for requests.
  * If we're doing basic auth, just return [`requests.auth.HTTPBasicAuth`](https://requests.readthedocs.io/en/latest/api/#requests.auth.HTTPBasicAuth) we can either use that directly with static credentials or we can make it dynamic and cache it for some period of time.
  * If we're using trusted publishing, we should return a different class that caches the oidc token and checks the expiry. Personally, I'd start _attempting_ to replace the token earlier than 1 minute prior to expiry, e.g., tokens last 15 minutes, at 5 minutes remaining I'd attempt to replace it. If we can't replace it, we return the non-expired token. On next use, we attempt again, etc.
* The value of `Resolver#resolve` is now passed into `Repository`.

In other words, a lot of what's on `Resolver` today is broken out into other classes, `Resolver` ends up with just a `resolve` method which returns something for Requests to use for auth, and Repository takes that auth instance instead of username/pw.

--

Technically, `twine.repository`, `twine.settings`, etc. were intended to be part of a public API but we never got buy-in from the folks demanding one (because they just wanted to call `twine.upload` without giving any cares as to where that goes) so I feel zero qualms about changing this. We never made it the official public API, so if someone's using it and broken by it, I don't care. We can release it as 7.0 if needed and drop some Python versions too if we haven't done that recently enough.
An early multi-attempt token swap idea is interesting. Although, it'd complicate the initial solution.

For reference, here are some places where I've seen the expiration offset being used:
* https://github.com/python-trio/snekomatic/blob/43e8e6d/snekomatic/gh.py#L89-L93
* https://github.com/PyGithub/PyGithub/blob/d90323f/github/Consts.py#L172-L174

(I think I also saw this recommendation in some GitHub's docs for Apps)
Having worked on similar projects we'll likely still see issues if we're not a little more aggressive in trying to exchange the token
Agreed. It makes sense for the client side. I'm only worried that the PyPI side might start throttling such requests. We'll have to check in with @miketheman to make sure that we won't cause some unexpected behavior because of the token refresh frequency increase. I think he might've mentioned that each of the requests would generate a security event while we were brainstorming the official PyPI-side support of uploads from reusable workflows last week at PyCon US.
> I think he might've mentioned that each of the requests would generate a security event while we were brainstorming the official PyPI-side support of uploads from reusable workflows last week at PyCon US.

I think that was me, unless @miketheman had the same concern 🙂 -- the observation there was that PyPI creates a project-level security event for every token exchange (so that maintainers can audit it later), which in turn means that we need to be a _little_ judicious about not exchanging too frequently/aggressively. Basically I was worried that exchanging for _each_ distribution upload would produce too much noise, versus only exchanging when we get close to the current token's expiry (minus some leeway).

In practice, exchanging with 1-5 minutes of leeway (i.e. up to 5 minutes before the token actually expires) would be perfectly fine!

(In terms of throttling: PyPI doesn't currently throttle the exchange itself at all except for in the first-use "pending" publisher case, which shouldn't cause any issues here.)
In the past I have avoided the complexity of relying on something like threading to parallelize uploads. Would it potentially be a helpful mitigation here too? It will certainly mess up our progress bars but I can imagine that combined with the token refresh logic would help these users regardless 
@woodruffw yeah, that might've been you. My brain is no longer capable of matching things I've heard over the past two weeks with humans :)

@sigmavirus24 yes, I've also been thinking of threading in this context. Regarding the progress bars, it's best to have them disabled in CI anyway (and we do that in the action automatically).
Ah yeah, I was just thinking that we always enable threading and so outside of CI this would be bad but ideally fewer and fewer people are running twine locally and caring about the UI
Yeah, I like the idea of threading as a mitigation here -- it might even be "good enough" to avoid needing a deeper "refresh" mechanism, since multiple uploads can start at the same time and run at their own paces to avoid having to authenticate with PyPI outside of the short-lived credential window. But that's also not especially sound, so take that idea with a grain of salt.

(Re: progress bars: I think `rich`'s progressbar APIs _do_ support updates from multiple threads, so users outside of CI could probably end up with a halfway-reasonable progress bar render here.)
To be fair though, in order of simple to complex (and ease to understand and debug) just refreshing the tokens is significantly better. And either way, I think it's a good refactor to do first.
> there was that PyPI creates a project-level security event for every token exchange (so that maintainers can audit it later), which in turn means that we need to be a _little_ judicious about not exchanging too frequently/aggressively

This can be seen in a project's security history, and is usually a single entry followed by N lines for however many files are part of a given release. The only thought I have right now is that this page isn't particularly fast - due to an [unindexed query](https://github.com/pypi/warehouse/blob/4612b2bd3bdf6f98d7a4e09d6632dde74320e488/warehouse/manage/views/__init__.py#L3277-L3281), unrelated to security history - but something we should be aware of in general. (Aside, I think this query could probably be refactored to use built-in relationship patterns today, but that's not the point of this topic).




Patch:
diff --git a/.coveragerc b/.coveragerc
index 62ada8f6..3ca81830 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -13,6 +13,7 @@ exclude_lines =
 
     # exclude typing.TYPE_CHECKING
     if TYPE_CHECKING:
+    if t.TYPE_CHECKING:
 
 [html]
 show_contexts = True
diff --git a/changelog/1246.feature.rst b/changelog/1246.feature.rst
new file mode 100644
index 00000000..a56da41e
--- /dev/null
+++ b/changelog/1246.feature.rst
@@ -0,0 +1,7 @@
+Automatically refresh short-lived PyPI token in long running Trusted Publishing
+uploads.
+
+In the event that a trusted publishing upload job is taking longer than the
+validity period of a trusted publishing token (15 minutes at the time of this
+writing), *and* we are already 10 minutes into that validity period, we will
+begin to attempt to replace the token on each subsequent request.
diff --git a/twine/__init__.py b/twine/__init__.py
index c6147574..d35db3b6 100644
--- a/twine/__init__.py
+++ b/twine/__init__.py
@@ -39,6 +39,7 @@
     import importlib_metadata
 
 metadata = importlib_metadata.metadata("twine")
+assert metadata is not None  # nosec: B101
 
 
 __title__ = metadata["name"]
diff --git a/twine/auth.py b/twine/auth.py
index 49ab8987..976921b4 100644
--- a/twine/auth.py
+++ b/twine/auth.py
@@ -1,16 +1,20 @@
+import datetime
 import functools
 import getpass
 import json
 import logging
-from typing import TYPE_CHECKING, Callable, Optional, Type, cast
+import time
+import typing as t
+from typing import cast
 from urllib.parse import urlparse
 
+import requests.auth
 from id import AmbientCredentialError  # type: ignore
 from id import detect_credential
 
 # keyring has an indirect dependency on PyCA cryptography, which has no
 # pre-built wheels for ppc64le and s390x, see #1158.
-if TYPE_CHECKING:
+if t.TYPE_CHECKING:
     import keyring
     from keyring.errors import NoKeyringError
 else:
@@ -26,16 +30,68 @@
 
 logger = logging.getLogger(__name__)
 
+TOKEN_USERNAME: t.Final[str] = "__token__"
+#: Tokens expire after 15 minutes, let's start allowing renewal/replacement
+#: after 10 minutes that way if we fail, we may still have time to replace it
+#: before it expires. Thus, if our current time + this threshold is past the
+#: greater or equal to the expiration time, we should start trying to replace
+#: the token.
+TOKEN_RENEWAL_THRESHOLD: t.Final[datetime.timedelta] = datetime.timedelta(
+    minutes=5,
+)
+
 
 class CredentialInput:
     def __init__(
-        self, username: Optional[str] = None, password: Optional[str] = None
+        self, username: t.Optional[str] = None, password: t.Optional[str] = None
     ) -> None:
         self.username = username
         self.password = password
 
 
+class TrustedPublishingTokenRetrievalError(t.TypedDict):
+    code: str
+    description: str
+
+
+class TrustedPublishingToken(t.TypedDict, total=False):
+    message: t.Optional[str]
+    errors: t.Optional[list[TrustedPublishingTokenRetrievalError]]
+    token: t.Optional[str]
+    success: t.Optional[bool]
+    # Depends on https://github.com/pypi/warehouse/issues/18235
+    expires: t.Optional[int]
+
+
+class TrustedPublishingAuthenticator(requests.auth.AuthBase):
+    def __init__(
+        self,
+        resolver: "Resolver",
+    ) -> None:
+        self.resolver = resolver
+
+    def __call__(
+        self, request: "requests.models.PreparedRequest"
+    ) -> "requests.models.PreparedRequest":
+        token = self.resolver.make_trusted_publishing_token()
+        if token is None:
+            raise exceptions.TrustedPublishingFailure(
+                "Expected a trusted publishing token but got None"
+            )
+
+        # Instead of reconstructing Basic Auth headers ourself, let's just
+        # rely on the underlying class to do the right thing.
+        basic_auth = requests.auth.HTTPBasicAuth(
+            username=TOKEN_USERNAME,
+            password=token,
+        )
+        return cast(requests.models.PreparedRequest, basic_auth(request))
+
+
 class Resolver:
+    _tp_token: t.Optional[TrustedPublishingToken] = None
+    _expires: t.Optional[int] = None
+
     def __init__(
         self,
         config: utils.RepositoryConfig,
@@ -44,16 +100,37 @@ def __init__(
         self.config = config
         self.input = input
 
+    @property
+    @functools.lru_cache()
+    def authenticator(self) -> "requests.auth.AuthBase":
+        username = self.username
+        password = self.password
+        if self._tp_token:
+            # If `self.password` ended up getting a Trusted Publishing token,
+            # we've cached it here so we should use that as the authenticator.
+            # We have a custom authenticator so we can repeatedly invoke
+            # `make_trusted_publishing_token` which if the token is 10 minutes
+            # old or more, we should get a new one automatically.
+            return TrustedPublishingAuthenticator(resolver=self)
+        if username and password:
+            return requests.auth.HTTPBasicAuth(
+                username=username,
+                password=password,
+            )
+        raise exceptions.InvalidConfiguration(
+            "could not determine credentials for configured repository"
+        )
+
     @classmethod
-    def choose(cls, interactive: bool) -> Type["Resolver"]:
+    def choose(cls, interactive: bool) -> t.Type["Resolver"]:
         return cls if interactive else Private
 
     @property
     @functools.lru_cache()
-    def username(self) -> Optional[str]:
+    def username(self) -> t.Optional[str]:
         if self.is_pypi() and not self.input.username:
             # Default username.
-            self.input.username = "__token__"
+            self.input.username = TOKEN_USERNAME
 
         return utils.get_userpass_value(
             self.input.username,
@@ -64,7 +141,7 @@ def username(self) -> Optional[str]:
 
     @property
     @functools.lru_cache()
-    def password(self) -> Optional[str]:
+    def password(self) -> t.Optional[str]:
         return utils.get_userpass_value(
             self.input.password,
             self.config,
@@ -72,7 +149,15 @@ def password(self) -> Optional[str]:
             prompt_strategy=self.password_from_keyring_or_trusted_publishing_or_prompt,
         )
 
-    def make_trusted_publishing_token(self) -> Optional[str]:
+    def _has_valid_cached_tp_token(self) -> bool:
+        return self._tp_token is not None and (
+            int(time.time()) + TOKEN_RENEWAL_THRESHOLD.seconds
+            < cast(int, self._tp_token.get("expires", self._expires))
+        )
+
+    def _make_trusted_publishing_token(self) -> t.Optional[TrustedPublishingToken]:
+        if self._has_valid_cached_tp_token():
+            return self._tp_token
         # Trusted publishing (OpenID Connect): get one token from the CI
         # system, and exchange that for a PyPI token.
         repository_domain = cast(str, urlparse(self.system).netloc)
@@ -97,7 +182,13 @@ def make_trusted_publishing_token(self) -> Optional[str]:
 
         if oidc_token is None:
             logger.debug("This environment is not supported for trusted publishing")
-            return None  # Fall back to prompting for a token (if possible)
+            if self._tp_token and int(time.time()) > cast(
+                int, self._tp_token.get("expires", self._expires)
+            ):
+                return None  # Fall back to prompting for a token (if possible)
+            # The cached trusted publishing token may still be valid for a
+            # while longer, let's continue using it instead of prompting
+            return self._tp_token
 
         logger.debug("Got OIDC token for audience %s", audience)
 
@@ -121,18 +212,26 @@ def make_trusted_publishing_token(self) -> Optional[str]:
                 for error in mint_token_payload["errors"]
             )
             raise exceptions.TrustedPublishingFailure(
-                "The token request failed; the index server gave the following "
-                f"reasons:\n\n{reasons}"
+                "The token request failed; the index server gave the following"
+                f" reasons:\n\n{reasons}"
             )
 
         logger.debug("Minted upload token for trusted publishing")
+        self._tp_token = cast(TrustedPublishingToken, mint_token_payload)
+        self._expires = int(time.time()) + 900
+        return self._tp_token
+
+    def make_trusted_publishing_token(self) -> t.Optional[str]:
+        mint_token_payload = self._make_trusted_publishing_token()
+        if not mint_token_payload:
+            return None
         return cast(str, mint_token_payload["token"])
 
     @property
-    def system(self) -> Optional[str]:
+    def system(self) -> t.Optional[str]:
         return self.config["repository"]
 
-    def get_username_from_keyring(self) -> Optional[str]:
+    def get_username_from_keyring(self) -> t.Optional[str]:
         if keyring is None:
             logger.info("keyring module is not available")
             return None
@@ -149,7 +248,7 @@ def get_username_from_keyring(self) -> Optional[str]:
             logger.warning("Error getting username from keyring", exc_info=exc)
         return None
 
-    def get_password_from_keyring(self) -> Optional[str]:
+    def get_password_from_keyring(self) -> t.Optional[str]:
         if keyring is None:
             logger.info("keyring module is not available")
             return None
@@ -178,7 +277,7 @@ def password_from_keyring_or_trusted_publishing_or_prompt(self) -> str:
             logger.info("password set from keyring")
             return password
 
-        if self.is_pypi() and self.username == "__token__":
+        if self.is_pypi() and self.username == TOKEN_USERNAME:
             logger.debug(
                 "Trying to use trusted publishing (no token was explicitly provided)"
             )
@@ -190,7 +289,7 @@ def password_from_keyring_or_trusted_publishing_or_prompt(self) -> str:
 
         return self.prompt(what, getpass.getpass)
 
-    def prompt(self, what: str, how: Callable[..., str]) -> str:
+    def prompt(self, what: str, how: t.Callable[..., str]) -> str:
         return how(f"Enter your {what}: ")
 
     def is_pypi(self) -> bool:
@@ -204,5 +303,5 @@ def is_pypi(self) -> bool:
 
 
 class Private(Resolver):
-    def prompt(self, what: str, how: Optional[Callable[..., str]] = None) -> str:
+    def prompt(self, what: str, how: t.Optional[t.Callable[..., str]] = None) -> str:
         raise exceptions.NonInteractive(f"Credential not found for {what}.")


Test patch:
diff --git a/tests/test_auth.py b/tests/test_auth.py
index 1750f633..6cec8c7e 100644
--- a/tests/test_auth.py
+++ b/tests/test_auth.py
@@ -1,9 +1,13 @@
+import base64
 import getpass
 import logging
 import platform
 import re
+import time
+import typing as t
 
 import pytest
+import requests.auth
 
 from twine import auth
 from twine import exceptions
@@ -299,3 +303,154 @@ def test_warns_for_empty_password(
 )
 def test_keyring_module():
     assert auth.keyring is not None
+
+
+def test_resolver_authenticator_config_authentication(config):
+    config.update(username="username", password="password")
+    res = auth.Resolver(config, auth.CredentialInput())
+    assert isinstance(res.authenticator, requests.auth.HTTPBasicAuth)
+
+
+def test_resolver_authenticator_credential_input_authentication(config):
+    res = auth.Resolver(config, auth.CredentialInput("username", "password"))
+    assert isinstance(res.authenticator, requests.auth.HTTPBasicAuth)
+
+
+def test_resolver_authenticator_trusted_publishing_authentication(config):
+    res = auth.Resolver(
+        config, auth.CredentialInput(username="__token__", password="skip-stdin")
+    )
+    res._tp_token = auth.TrustedPublishingToken(
+        success=True,
+        token="fake-tp-token",
+    )
+    assert isinstance(res.authenticator, auth.TrustedPublishingAuthenticator)
+
+
+class MockResponse:
+    def __init__(self, status_code: int, json: t.Any) -> None:
+        self.status_code = status_code
+        self._json = json
+
+    def json(self, *args, **kwargs) -> t.Any:
+        return self._json
+
+    def raise_for_status(self) -> None:
+        if 400 <= self.status_code:
+            raise requests.exceptions.HTTPError()
+
+    def ok(self) -> bool:
+        return self.status_code == 200
+
+
+class MockSession:
+    def __init__(
+        self,
+        get_response_list: t.List[MockResponse],
+        post_response_list: t.List[MockResponse],
+    ) -> None:
+        self.post_counter = self.get_counter = 0
+        self.get_response_list = get_response_list
+        self.post_response_list = post_response_list
+
+    def get(self, url: str, **kwargs) -> MockResponse:
+        response = self.get_response_list[self.get_counter]
+        self.get_counter += 1
+        return response
+
+    def post(self, url: str, **kwargs) -> MockResponse:
+        response = self.post_response_list[self.post_counter]
+        self.post_counter += 1
+        return response
+
+
+def test_trusted_publish_authenticator_refreshes_token(monkeypatch, config):
+    def make_session():
+        return MockSession(
+            get_response_list=[
+                MockResponse(status_code=200, json={"audience": "fake-aud"})
+            ],
+            post_response_list=[
+                MockResponse(
+                    status_code=200,
+                    json={
+                        "success": True,
+                        "token": "new-token",
+                        "expires": int(time.time()) + 900,
+                    },
+                ),
+            ],
+        )
+
+    def detect_credential(*args, **kwargs) -> str:
+        return "fake-oidc-token"
+
+    config.update({"repository": utils.TEST_REPOSITORY})
+    res = auth.Resolver(config, auth.CredentialInput(username="__token__"))
+    res._tp_token = auth.TrustedPublishingToken(
+        success=True,
+        token="expiring-tp-token",
+    )
+    res._expires = int(time.time()) + 4 * 60
+    monkeypatch.setattr(auth, "detect_credential", detect_credential)
+    monkeypatch.setattr(auth.utils, "make_requests_session", make_session)
+    authenticator = auth.TrustedPublishingAuthenticator(resolver=res)
+    prepped_req = requests.models.PreparedRequest()
+    prepped_req.prepare_headers({})
+    request = authenticator(prepped_req)
+    assert (
+        request.headers["Authorization"]
+        == f"Basic {base64.b64encode(b'__token__:new-token').decode()}"
+    )
+
+
+def test_trusted_publish_authenticator_reuses_token(monkeypatch, config):
+    def make_session():
+        return MockSession(
+            get_response_list=[
+                MockResponse(status_code=200, json={"audience": "fake-aud"})
+            ],
+            post_response_list=[
+                MockResponse(
+                    status_code=200,
+                    json={
+                        "success": True,
+                        "token": "new-token",
+                        "expires": int(time.time()) + 900,
+                    },
+                ),
+            ],
+        )
+
+    def detect_credential(*args, **kwargs) -> str:
+        return "fake-oidc-token"
+
+    config.update({"repository": utils.TEST_REPOSITORY})
+    res = auth.Resolver(config, auth.CredentialInput(username="__token__"))
+    res._tp_token = auth.TrustedPublishingToken(
+        success=True,
+        token="valid-tp-token",
+    )
+    res._expires = int(time.time()) + 900
+    monkeypatch.setattr(auth, "detect_credential", detect_credential)
+    monkeypatch.setattr(auth.utils, "make_requests_session", make_session)
+    authenticator = auth.TrustedPublishingAuthenticator(resolver=res)
+    prepped_req = requests.models.PreparedRequest()
+    prepped_req.prepare_headers({})
+    request = authenticator(prepped_req)
+    assert (
+        request.headers["Authorization"]
+        == f"Basic {base64.b64encode(b'__token__:valid-tp-token').decode()}"
+    )
+
+
+def test_inability_to_make_token_raises_error():
+    class MockResolver:
+        def make_trusted_publishing_token(self) -> None:
+            return None
+
+    authenticator = auth.TrustedPublishingAuthenticator(
+        resolver=MockResolver(),
+    )
+    with pytest.raises(exceptions.TrustedPublishingFailure):
+        authenticator(None)