Repository: theupdateframework/python-tuf
Pull request #2803 (2025-03-03 21:23:36)

build(deps): bump actions/download-artifact from 4.1.8 to 4.1.9 in the action-dependencies group
Bumps the action-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact).

Updates `actions/download-artifact` from 4.1.8 to 4.1.9
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/download-artifact/releases">actions/download-artifact's releases</a>.</em></p>
<blockquote>
<h2>v4.1.9</h2>
<h2>What's Changed</h2>
<ul>
<li>Add workflow file for publishing releases to immutable action package by <a href="https://github.com/Jcambass"><code>@​Jcambass</code></a> in <a href="https://redirect.github.com/actions/download-artifact/pull/354">actions/download-artifact#354</a></li>
<li>docs: small migration fix by <a href="https://github.com/froblesmartin"><code>@​froblesmartin</code></a> in <a href="https://redirect.github.com/actions/download-artifact/pull/370">actions/download-artifact#370</a></li>
<li>Update MIGRATION.md by <a href="https://github.com/andyfeller"><code>@​andyfeller</code></a> in <a href="https://redirect.github.com/actions/download-artifact/pull/372">actions/download-artifact#372</a></li>
<li>Update artifact package to 2.2.2 by <a href="https://github.com/yacaovsnc"><code>@​yacaovsnc</code></a> in <a href="https://redirect.github.com/actions/download-artifact/pull/380">actions/download-artifact#380</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/Jcambass"><code>@​Jcambass</code></a> made their first contribution in <a href="https://redirect.github.com/actions/download-artifact/pull/354">actions/download-artifact#354</a></li>
<li><a href="https://github.com/froblesmartin"><code>@​froblesmartin</code></a> made their first contribution in <a href="https://redirect.github.com/actions/download-artifact/pull/370">actions/download-artifact#370</a></li>
<li><a href="https://github.com/andyfeller"><code>@​andyfeller</code></a> made their first contribution in <a href="https://redirect.github.com/actions/download-artifact/pull/372">actions/download-artifact#372</a></li>
<li><a href="https://github.com/yacaovsnc"><code>@​yacaovsnc</code></a> made their first contribution in <a href="https://redirect.github.com/actions/download-artifact/pull/380">actions/download-artifact#380</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/actions/download-artifact/compare/v4...v4.1.9">https://github.com/actions/download-artifact/compare/v4...v4.1.9</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/actions/download-artifact/commit/cc203385981b70ca67e1cc392babf9cc229d5806"><code>cc20338</code></a> Merge pull request <a href="https://redirect.github.com/actions/download-artifact/issues/380">#380</a> from actions/yacaovsnc/release_4_1_9</li>
<li><a href="https://github.com/actions/download-artifact/commit/1fc0fee191f40422f502da571c0f01ff460afe53"><code>1fc0fee</code></a> Update artifact package to 2.2.2</li>
<li><a href="https://github.com/actions/download-artifact/commit/7fba95161a0924506ed1ae69cdbae8371ee00b3f"><code>7fba951</code></a> Merge pull request <a href="https://redirect.github.com/actions/download-artifact/issues/372">#372</a> from andyfeller/patch-1</li>
<li><a href="https://github.com/actions/download-artifact/commit/f9ceb7763ba1fdfd81b2e2f93aa1f6015ff6b35d"><code>f9ceb77</code></a> Update MIGRATION.md</li>
<li><a href="https://github.com/actions/download-artifact/commit/533298bc57c27f112a2c04a74a04a4d43e2866fd"><code>533298b</code></a> Merge pull request <a href="https://redirect.github.com/actions/download-artifact/issues/370">#370</a> from froblesmartin/patch-1</li>
<li><a href="https://github.com/actions/download-artifact/commit/d06289e120b300840a833b25db66cb8c19f5d274"><code>d06289e</code></a> docs: small migration fix</li>
<li><a href="https://github.com/actions/download-artifact/commit/d0ce8fd1167ed839810201de977912a090ab10a7"><code>d0ce8fd</code></a> Merge pull request <a href="https://redirect.github.com/actions/download-artifact/issues/354">#354</a> from actions/Jcambass-patch-1</li>
<li><a href="https://github.com/actions/download-artifact/commit/1ce0d91ace59dfbf6763107ee5aa8466ebbadf48"><code>1ce0d91</code></a> Add workflow file for publishing releases to immutable action package</li>
<li>See full diff in <a href="https://github.com/actions/download-artifact/compare/fa0a91b85d4f404e444e00e005971372dc801d16...cc203385981b70ca67e1cc392babf9cc229d5806">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/download-artifact&package-manager=github_actions&previous-version=4.1.8&new-version=4.1.9)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>


Patch:
diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml
index da36a6c7..623fae02 100644
--- a/.github/workflows/cd.yml
+++ b/.github/workflows/cd.yml
@@ -54,7 +54,7 @@ jobs:
       release_id: ${{ steps.gh-release.outputs.result }}
     steps:
       - name: Fetch build artifacts
-        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
+        uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
         with:
           name: build-artifacts
 
@@ -96,7 +96,7 @@ jobs:
       id-token: write # to authenticate as Trusted Publisher to pypi.org
     steps:
       - name: Fetch build artifacts
-        uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
+        uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
         with:
           name: build-artifacts
 
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index a1203eb9..d3ccf453 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -1,5 +1,21 @@
 # Changelog
 
+## Unreleased
+
+### Changed
+
+This release is API compatible but contains a major internal change in the HTTP handling.
+
+* ngclient: urllib3 is used as the HTTP library by default instead of requests (#2762,
+  #2773, #2789)
+  * This removes dependencies on `requests`, `idna`, `charset-normalizer` and `certifi`
+  * The deprecated RequestsFetcher implementation is available but requires selecting
+    the fetcher at Updater initialization and explicitly depending on requests
+* ngclient: TLS certificate source was changed. Certificates now come from operating
+  system certificate store instead of `certifi` (#2762)
+* Test infrastucture has improved and should now be more usable externally, e.g. in
+  distro test suites (#2749)
+
 ## v5.1.0
 
 ### Changed
diff --git a/tuf/ngclient/_internal/proxy.py b/tuf/ngclient/_internal/proxy.py
new file mode 100644
index 00000000..b42ea2f4
--- /dev/null
+++ b/tuf/ngclient/_internal/proxy.py
@@ -0,0 +1,101 @@
+# Copyright New York University and the TUF contributors
+# SPDX-License-Identifier: MIT OR Apache-2.0
+
+"""Proxy environment variable handling with Urllib3"""
+
+from __future__ import annotations
+
+from typing import Any
+from urllib.request import getproxies
+
+from urllib3 import BaseHTTPResponse, PoolManager, ProxyManager
+from urllib3.util.url import parse_url
+
+
+# TODO: ProxyEnvironment could implement the whole PoolManager.RequestMethods
+# Mixin: We only need request() so nothing else is currently implemented
+class ProxyEnvironment:
+    """A PoolManager manager for automatic proxy handling based on env variables
+
+    Keeps track of PoolManagers for different proxy urls based on proxy
+    environment variables. Use `get_pool_manager()` or `request()` to access
+    the right manager for a scheme/host.
+
+    Supports '*_proxy' variables, with special handling for 'no_proxy' and
+    'all_proxy'.
+    """
+
+    def __init__(
+        self,
+        **kw_args: Any,  # noqa: ANN401
+    ) -> None:
+        self._pool_managers: dict[str | None, PoolManager] = {}
+        self._kw_args = kw_args
+
+        self._proxies = getproxies()
+        self._all_proxy = self._proxies.pop("all", None)
+        no_proxy = self._proxies.pop("no", None)
+        if no_proxy is None:
+            self._no_proxy_hosts = []
+        else:
+            # split by comma, remove leading periods
+            self._no_proxy_hosts = [
+                h.lstrip(".") for h in no_proxy.replace(" ", "").split(",") if h
+            ]
+
+    def _get_proxy(self, scheme: str | None, host: str | None) -> str | None:
+        """Get a proxy url for scheme and host based on proxy env variables"""
+
+        if host is None:
+            # urllib3 only handles http/https but we can do something reasonable
+            # even for schemes that don't require host (like file)
+            return None
+
+        # does host match any of the "no_proxy" hosts?
+        for no_proxy_host in self._no_proxy_hosts:
+            # wildcard match, exact hostname match, or parent domain match
+            if no_proxy_host in ("*", host) or host.endswith(
+                f".{no_proxy_host}"
+            ):
+                return None
+
+        if scheme in self._proxies:
+            return self._proxies[scheme]
+        if self._all_proxy is not None:
+            return self._all_proxy
+
+        return None
+
+    def get_pool_manager(
+        self, scheme: str | None, host: str | None
+    ) -> PoolManager:
+        """Get a poolmanager for scheme and host.
+
+        Returns a ProxyManager if that is correct based on current proxy env
+        variables, otherwise returns a PoolManager
+        """
+
+        proxy = self._get_proxy(scheme, host)
+        if proxy not in self._pool_managers:
+            if proxy is None:
+                self._pool_managers[proxy] = PoolManager(**self._kw_args)
+            else:
+                self._pool_managers[proxy] = ProxyManager(
+                    proxy,
+                    **self._kw_args,
+                )
+
+        return self._pool_managers[proxy]
+
+    def request(
+        self,
+        method: str,
+        url: str,
+        **request_kw: Any,  # noqa: ANN401
+    ) -> BaseHTTPResponse:
+        """Make a request using a PoolManager chosen based on url and
+        proxy environment variables.
+        """
+        u = parse_url(url)
+        manager = self.get_pool_manager(u.scheme, u.host)
+        return manager.request(method, url, **request_kw)
diff --git a/tuf/ngclient/updater.py b/tuf/ngclient/updater.py
index 5af7cfe4..a40c1fca 100644
--- a/tuf/ngclient/updater.py
+++ b/tuf/ngclient/updater.py
@@ -36,6 +36,19 @@
 A simple example of using the Updater to implement a Python TUF client that
 downloads target files is available in `examples/client
 <https://github.com/theupdateframework/python-tuf/tree/develop/examples/client>`_.
+
+Notes on how Updater uses HTTP by default:
+  * urllib3 is the HTTP library
+  * Typically all requests are retried by urllib3 three times (in cases where
+    this seems useful)
+  * Operating system certificate store is used for TLS, in other words
+    ``certifi`` is not used as the certificate source
+  * Proxy use can be configured with ``https_proxy`` and other similar
+    environment variables
+
+All of the HTTP decisions can be changed with ``fetcher`` argument:
+Custom ``FetcherInterface`` implementations are possible. The alternative
+``RequestsFetcher`` implementation is also provided (although deprecated).
 """
 
 from __future__ import annotations
diff --git a/tuf/ngclient/urllib3_fetcher.py b/tuf/ngclient/urllib3_fetcher.py
index 5d4bb1cf..88d447bd 100644
--- a/tuf/ngclient/urllib3_fetcher.py
+++ b/tuf/ngclient/urllib3_fetcher.py
@@ -15,6 +15,7 @@
 
 import tuf
 from tuf.api import exceptions
+from tuf.ngclient._internal.proxy import ProxyEnvironment
 from tuf.ngclient.fetcher import FetcherInterface
 
 if TYPE_CHECKING:
@@ -49,7 +50,7 @@ def __init__(
         if app_user_agent is not None:
             ua = f"{app_user_agent} {ua}"
 
-        self._poolManager = urllib3.PoolManager(headers={"User-Agent": ua})
+        self._proxy_env = ProxyEnvironment(headers={"User-Agent": ua})
 
     def _fetch(self, url: str) -> Iterator[bytes]:
         """Fetch the contents of HTTP/HTTPS url from a remote server.
@@ -72,7 +73,7 @@ def _fetch(self, url: str) -> Iterator[bytes]:
         #  - connect timeout (max delay before first byte is received)
         #  - read (gap) timeout (max delay between bytes received)
         try:
-            response = self._poolManager.request(
+            response = self._proxy_env.request(
                 "GET",
                 url,
                 preload_content=False,


Test patch:
diff --git a/tests/test_proxy_environment.py b/tests/test_proxy_environment.py
new file mode 100644
index 00000000..ade7b350
--- /dev/null
+++ b/tests/test_proxy_environment.py
@@ -0,0 +1,217 @@
+# Copyright 2025, the TUF contributors
+# SPDX-License-Identifier: MIT OR Apache-2.0
+
+"""Test ngclient ProxyEnvironment"""
+
+from __future__ import annotations
+
+import sys
+import unittest
+from unittest.mock import Mock, patch
+
+from urllib3 import PoolManager, ProxyManager
+
+from tests import utils
+from tuf.ngclient._internal.proxy import ProxyEnvironment
+
+
+class TestProxyEnvironment(unittest.TestCase):
+    """Test ngclient ProxyEnvironment implementation
+
+    These tests use the ProxyEnvironment.get_pool_manager() endpoint and then
+    look at the ProxyEnvironment._poolmanagers dict keys to decide if the result
+    is correct.
+
+    The test environment is changed via mocking getproxies(): this is a urllib
+    method that returns a dict with the proxy environment variable contents.
+
+    Testing ProxyEnvironment.request() would possibly be better but far more
+    difficult: the current test implementation does not require actually setting up
+    all of the different proxies.
+    """
+
+    def assert_pool_managers(
+        self, env: ProxyEnvironment, expected: list[str | None]
+    ) -> None:
+        # Pool managers have the expected proxy urls
+        self.assertEqual(list(env._pool_managers.keys()), expected)
+
+        # Pool manager types are as expected
+        for proxy_url, pool_manager in env._pool_managers.items():
+            self.assertIsInstance(pool_manager, PoolManager)
+            if proxy_url is not None:
+                self.assertIsInstance(pool_manager, ProxyManager)
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_no_variables(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {}
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There is a single pool manager (no proxies)
+        self.assert_pool_managers(env, [None])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_proxy_set(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "https": "http://localhost:8888",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There are two pool managers: A plain poolmanager and https proxymanager
+        self.assert_pool_managers(env, [None, "http://localhost:8888"])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_proxies_set(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "http": "http://localhost:8888",
+            "https": "http://localhost:9999",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There are two pool managers: A http proxymanager and https proxymanager
+        self.assert_pool_managers(
+            env, ["http://localhost:8888", "http://localhost:9999"]
+        )
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_no_proxy_set(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "http": "http://localhost:8888",
+            "https": "http://localhost:9999",
+            "no": "somesite.com, example.com, another.site.com",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+
+        # There is a single pool manager (no proxies)
+        self.assert_pool_managers(env, [None])
+
+        env.get_pool_manager("http", "differentsite.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There are three pool managers: plain poolmanager for no_proxy domains,
+        # http proxymanager and https proxymanager
+        self.assert_pool_managers(
+            env, [None, "http://localhost:8888", "http://localhost:9999"]
+        )
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_no_proxy_subdomain_match(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "https": "http://localhost:9999",
+            "no": "somesite.com, example.com, another.site.com",
+        }
+
+        env = ProxyEnvironment()
+
+        # this should match example.com in no_proxy
+        env.get_pool_manager("https", "subdomain.example.com")
+
+        # There is a single pool manager (no proxies)
+        self.assert_pool_managers(env, [None])
+
+        # this should not match example.com in no_proxy
+        env.get_pool_manager("https", "xexample.com")
+
+        # There are two pool managers: plain poolmanager for no_proxy domains,
+        # and a https proxymanager
+        self.assert_pool_managers(env, [None, "http://localhost:9999"])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_no_proxy_wildcard(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "https": "http://localhost:8888",
+            "no": "*",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "differentsite.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+
+        # There is a single pool manager, no proxies
+        self.assert_pool_managers(env, [None])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_no_proxy_leading_dot(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "https": "http://localhost:8888",
+            "no": ".example.com",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+
+        # There is a single pool manager, no proxies
+        self.assert_pool_managers(env, [None])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_all_proxy_set(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "all": "http://localhost:8888",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There is a single proxy manager
+        self.assert_pool_managers(env, ["http://localhost:8888"])
+
+        # This urllib3 currently only handles http and https but let's test anyway
+        env.get_pool_manager("file", None)
+
+        # proxy manager and a plain pool manager
+        self.assert_pool_managers(env, ["http://localhost:8888", None])
+
+    @patch("tuf.ngclient._internal.proxy.getproxies")
+    def test_all_proxy_and_no_proxy_set(self, mock_getproxies: Mock) -> None:
+        mock_getproxies.return_value = {
+            "all": "http://localhost:8888",
+            "no": "somesite.com, example.com, another.site.com",
+        }
+
+        env = ProxyEnvironment()
+        env.get_pool_manager("http", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "example.com")
+        env.get_pool_manager("https", "subdomain.example.com")
+
+        # There is a single pool manager (no proxies)
+        self.assert_pool_managers(env, [None])
+
+        env.get_pool_manager("http", "differentsite.com")
+        env.get_pool_manager("https", "differentsite.com")
+
+        # There are two pool managers: plain poolmanager for no_proxy domains and
+        # one proxymanager
+        self.assert_pool_managers(env, [None, "http://localhost:8888"])
+
+
+if __name__ == "__main__":
+    utils.configure_test_logging(sys.argv)
+    unittest.main()
diff --git a/tests/test_updater_ng.py b/tests/test_updater_ng.py
index b37003bb..50ef5ee3 100644
--- a/tests/test_updater_ng.py
+++ b/tests/test_updater_ng.py
@@ -331,8 +331,10 @@ def test_non_existing_target_file(self) -> None:
     def test_user_agent(self) -> None:
         # test default
         self.updater.refresh()
-        session = self.updater._fetcher._poolManager
-        ua = session.headers["User-Agent"]
+        poolmgr = self.updater._fetcher._proxy_env.get_pool_manager(
+            "http", "localhost"
+        )
+        ua = poolmgr.headers["User-Agent"]
         self.assertEqual(ua[:11], "python-tuf/")
 
         # test custom UA
@@ -344,8 +346,10 @@ def test_user_agent(self) -> None:
             config=UpdaterConfig(app_user_agent="MyApp/1.2.3"),
         )
         updater.refresh()
-        session = updater._fetcher._poolManager
-        ua = session.headers["User-Agent"]
+        poolmgr = updater._fetcher._proxy_env.get_pool_manager(
+            "http", "localhost"
+        )
+        ua = poolmgr.headers["User-Agent"]
 
         self.assertEqual(ua[:23], "MyApp/1.2.3 python-tuf/")