Repository: streamlink/streamlink

Issue:
AttributeError: module 'urllib3.util.url' has no attribute 'PERCENT_RE'
### Checklist

- [X] This is a bug report and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest stable release

### Description

C:\Program Files\Python310\lib\site-packages\streamlink\plugin\api\http_session.py line 53 typo?

I was trying to run my script with python 3.10.4 in Windows 10 and got that error, but trying to run it with python 3.8.6 is working fine, why?

### Debug log

```text
File "C:\Program Files\Python310\lib\site-packages\streamlink\plugin\api\http_session.py", line 53, in Urllib3UtilUrlPercentReOverride
    _re_percent_encoding: Pattern = urllib3.util.url.PERCENT_RE  # type: ignore[attr-defined]
AttributeError: module 'urllib3.util.url' has no attribute 'PERCENT_RE'. Did you mean: '_PERCENT_RE'?
```



Discussion:
Streamlink applies an override for a faulty url encoding in urllib3, as you can see here:
- https://github.com/streamlink/streamlink/blob/5.4.0/src/streamlink/plugin/api/http_session.py#L41-L62
- https://github.com/streamlink/streamlink/blob/5.4.0/tests/test_api_http_session.py#L12-L33

Apparently, urllib3 has renamed its `PERCENT_RE` constant to `_PERCENT_RE` in its `2.0.0` release:
https://github.com/urllib3/urllib3/blame/2.0.0/src/urllib3/util/url.py#L15

**On the 1.x branch, it's still `PERCENT_RE`:**
https://github.com/urllib3/urllib3/blame/1.26.15/src/urllib3/util/url.py#L17

**Streamlink does only support urllib3 1.x, so if you've installed urllib3 2.x in your python environment, then you're ignoring the dependency version constraints of streamlink and requests:**
- streamlink: https://github.com/streamlink/streamlink/blob/5.4.0/setup.cfg#L49-L50
- requests: https://github.com/psf/requests/blob/v2.30.0/setup.cfg#L10

Not a bug. Fix your Streamlink install / python environment.
ok, thanks
Use a virtual environment if you have other packages in your python environment which depend on urllib3 2.x.
https://streamlink.github.io/install.html#virtual-environment
So apparently `requests` has published version `2.30.0` with a bump to `urllib3` `<3`:
https://github.com/psf/requests/releases/tag/v2.30.0

I unfortunately looked at the wrong package metadata yesterday. The right one is of course here:
https://github.com/psf/requests/blame/v2.30.0/setup.py#L64

They added a changelog just now after tagging the release yesterday already. For some reason, pip didn't pick up the new version for me when I ran `pip install -U --upgrade-strategy=eager -r dev-requirements.txt -r docs-requirements.txt` yesterday after seeing this thread here.

----

This means that Streamlink has to fix the new version incompabilities now.

I've restricted urllib3 to `<2` for now in #5325, but I'm going to bump that to `<3` again with the necessary changes.

This is a bit annoying, because `requests` should've bumped its major version in order to stay compliant with the semantic versioning scheme.

Speaking of semver, urllib3 isn't using this scheme correctly either, with lots of deprecations being added in 2.0.0, while stuff will get removed/changed in a minor release later on:
https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

So what needs to fix now is the name of the `urllib3.utils.url._PATTERN_RE` constant which Streamlink overrides, and maybe some other stuff which I'm going to look at now (TLS stuff / network interface stuff).

Reopening for visibility. Once fixed, there will be a new release, since the current stable release is broken when not limiting `urllib3` to `<2`.
> For some reason, pip didn't pick up the new version for me

They dropped 2.30.0 and re-released it today:
https://github.com/psf/requests/issues/6432#issuecomment-1533483200

Resolving patch:
diff --git a/src/streamlink/plugin/api/http_session.py b/src/streamlink/plugin/api/http_session.py
--- a/src/streamlink/plugin/api/http_session.py
+++ b/src/streamlink/plugin/api/http_session.py
@@ -1,10 +1,11 @@
-import ssl
+import re
 import time
 from typing import Any, Dict, Pattern, Tuple
 
 import requests.adapters
 import urllib3
 from requests import PreparedRequest, Request, Session
+from requests.adapters import HTTPAdapter
 
 from streamlink.exceptions import PluginError
 from streamlink.packages.requests_file import FileAdapter
@@ -12,6 +13,7 @@
 from streamlink.utils.parse import parse_json, parse_xml
 
 
+# urllib3>=2.0.0: enforce_content_length now defaults to True (keep the override for backwards compatibility)
 class _HTTPResponse(urllib3.response.HTTPResponse):
     def __init__(self, *args, **kwargs):
         # Always enforce content length validation!
@@ -50,7 +52,9 @@ def __init__(self, *args, **kwargs):
 # > normalizers should use uppercase hexadecimal digits for all percent-
 # > encodings.
 class Urllib3UtilUrlPercentReOverride:
-    _re_percent_encoding: Pattern = urllib3.util.url.PERCENT_RE  # type: ignore[attr-defined]
+    # urllib3>=2.0.0: _PERCENT_RE, urllib3<2.0.0: PERCENT_RE
+    _re_percent_encoding: Pattern \
+        = getattr(urllib3.util.url, "_PERCENT_RE", getattr(urllib3.util.url, "PERCENT_RE", re.compile(r"%[a-fA-F0-9]{2}")))
 
     # urllib3>=1.25.8
     # https://github.com/urllib3/urllib3/blame/1.25.8/src/urllib3/util/url.py#L219-L227
@@ -59,7 +63,8 @@ def subn(cls, repl: Any, string: str, count: Any = None) -> Tuple[str, int]:
         return string, len(cls._re_percent_encoding.findall(string))
 
 
-urllib3.util.url.PERCENT_RE = Urllib3UtilUrlPercentReOverride  # type: ignore[attr-defined]
+# urllib3>=2.0.0: _PERCENT_RE, urllib3<2.0.0: PERCENT_RE
+urllib3.util.url._PERCENT_RE = urllib3.util.url.PERCENT_RE = Urllib3UtilUrlPercentReOverride  # type: ignore[attr-defined]
 
 
 # requests.Request.__init__ keywords, except for "hooks"
@@ -181,12 +186,24 @@ def request(self, method, url, *args, **kwargs):
         return res
 
 
-class TLSSecLevel1Adapter(requests.adapters.HTTPAdapter):
+class TLSNoDHAdapter(HTTPAdapter):
     def init_poolmanager(self, *args, **kwargs):
-        ctx = ssl.create_default_context()
+        ctx = urllib3.util.create_urllib3_context()
+        ctx.load_default_certs()
+        ciphers = ":".join(cipher.get("name") for cipher in ctx.get_ciphers())
+        ciphers += ":!DH"
+        ctx.set_ciphers(ciphers)
+        kwargs["ssl_context"] = ctx
+        return super().init_poolmanager(*args, **kwargs)
+
+
+class TLSSecLevel1Adapter(HTTPAdapter):
+    def init_poolmanager(self, *args, **kwargs):
+        ctx = urllib3.util.create_urllib3_context()
+        ctx.load_default_certs()
         ctx.set_ciphers("DEFAULT:@SECLEVEL=1")
         kwargs["ssl_context"] = ctx
         return super().init_poolmanager(*args, **kwargs)
 
 
-__all__ = ["HTTPSession", "TLSSecLevel1Adapter"]
+__all__ = ["HTTPSession", "TLSNoDHAdapter", "TLSSecLevel1Adapter"]
diff --git a/src/streamlink/session.py b/src/streamlink/session.py
--- a/src/streamlink/session.py
+++ b/src/streamlink/session.py
@@ -6,13 +6,13 @@
 from typing import Any, Callable, ClassVar, Dict, Iterator, Mapping, Optional, Tuple, Type
 
 import urllib3.util.connection as urllib3_util_connection
-import urllib3.util.ssl_ as urllib3_util_ssl
+from requests.adapters import HTTPAdapter
 
 from streamlink import __version__, plugins
 from streamlink.exceptions import NoPluginError, PluginError, StreamlinkDeprecationWarning
 from streamlink.logger import StreamlinkLogger
 from streamlink.options import Options
-from streamlink.plugin.api.http_session import HTTPSession
+from streamlink.plugin.api.http_session import HTTPSession, TLSNoDHAdapter
 from streamlink.plugin.plugin import NO_PRIORITY, NORMAL_PRIORITY, Matcher, Plugin
 from streamlink.utils.l10n import Localization
 from streamlink.utils.module import load_module
@@ -115,14 +115,12 @@ def _set_http_attr(self, key, value):
 
     def _set_http_disable_dh(self, key, value):
         self.set_explicit(key, value)
-        default_ciphers = [
-            item
-            for item in urllib3_util_ssl.DEFAULT_CIPHERS.split(":")  # type: ignore[attr-defined]
-            if item != "!DH"
-        ]
         if value:
-            default_ciphers.append("!DH")
-        urllib3_util_ssl.DEFAULT_CIPHERS = ":".join(default_ciphers)  # type: ignore[attr-defined]
+            adapter = TLSNoDHAdapter()
+        else:
+            adapter = HTTPAdapter()
+
+        self.session.http.mount("https://", adapter)
 
     @staticmethod
     def _factory_set_http_attr_key_equals_value(delimiter: str) -> Callable[["StreamlinkOptions", str, Any], None]:


Test patch:
diff --git a/tests/test_session.py b/tests/test_session.py
--- a/tests/test_session.py
+++ b/tests/test_session.py
@@ -8,10 +8,12 @@
 import pytest
 import requests_mock
 import urllib3
+from requests.adapters import HTTPAdapter
 
 import tests.plugin
 from streamlink.exceptions import NoPluginError, StreamlinkDeprecationWarning
 from streamlink.plugin import HIGH_PRIORITY, LOW_PRIORITY, NO_PRIORITY, NORMAL_PRIORITY, Plugin, pluginmatcher
+from streamlink.plugin.api.http_session import TLSNoDHAdapter
 from streamlink.session import Streamlink
 from streamlink.stream.hls import HLSStream
 from streamlink.stream.http import HTTPStream
@@ -394,19 +396,17 @@ def test_ipv4_ipv6(self, mock_urllib3_util_connection):
         assert session.get_option("ipv6") is False
         assert mock_urllib3_util_connection.allowed_gai_family is _original_allowed_gai_family
 
-    @patch("streamlink.session.urllib3_util_ssl", DEFAULT_CIPHERS="foo:!bar:baz")
-    def test_http_disable_dh(self, mock_urllib3_util_ssl):
+    def test_http_disable_dh(self):
         session = self.subject(load_plugins=False)
-        assert mock_urllib3_util_ssl.DEFAULT_CIPHERS == "foo:!bar:baz"
+        assert isinstance(session.http.adapters["https://"], HTTPAdapter)
+        assert not isinstance(session.http.adapters["https://"], TLSNoDHAdapter)
 
         session.set_option("http-disable-dh", True)
-        assert mock_urllib3_util_ssl.DEFAULT_CIPHERS == "foo:!bar:baz:!DH"
-
-        session.set_option("http-disable-dh", True)
-        assert mock_urllib3_util_ssl.DEFAULT_CIPHERS == "foo:!bar:baz:!DH"
+        assert isinstance(session.http.adapters["https://"], TLSNoDHAdapter)
 
         session.set_option("http-disable-dh", False)
-        assert mock_urllib3_util_ssl.DEFAULT_CIPHERS == "foo:!bar:baz"
+        assert isinstance(session.http.adapters["https://"], HTTPAdapter)
+        assert not isinstance(session.http.adapters["https://"], TLSNoDHAdapter)
 
 
 class TestSessionOptionHttpProxy: