Repository: streamlink/streamlink

Issue:
--http[s]-proxy does not properly handle schemeless inputs with port due to urllib behavior change in python 3.9
## Bug Report

<!-- Replace the space character between the square brackets with an x in order to check the boxes -->
- [X] This is a bug report and I have read the contribution guidelines.
- [X] I am using the latest development version from the master branch.

### Description

<!-- Explain the bug as thoroughly as you can. Don't leave out information which is necessary for us to reproduce and debug this issue. -->

While troubleshooting an issue with a project that consumes Streamlink by way of its CLI, I discovered an apparent bug in the interaction between Streamlink and urllib w/r/t how proxy URL inputs are handled - stemming from a behavior change in urllib in Python 3.9.

If the input for `--http[s]-proxy` is provided in host:port or IP:port notation WITHOUT a scheme (e.g. `--http-proxy 127.0.0.1:8050`), the `update_scheme()` function does not actually add a scheme to it under Python 3.9. When the still-schemeless url is then fed to urllib3 for connection setup, it fails with a `ProxySchemeUnknown` exception.

The behavior here appears to be caused by urllib's `urlparse()` function, rather than Streamlink itself, but the intent of the Python developers seems to be that the behavior change is handled by client applications.

Reference:
[44007: urlparse("host:123", "http") inconsistent between 3.8 and 3.9](https://bugs.python.org/issue44007)
which leads to:
[27657: urlparse fails if the path is numeric](https://bugs.python.org/issue27657) as the source of the behavior change

In the latter bug (specifically here: https://bugs.python.org/issue27657#msg371024) it's noted that the behavior change was apparently intentional and consumers of urllib are expected to handle it.

### Expected / Actual behavior

<!-- What do you expect to happen, and what is actually happening? -->

Expected: A scheme-less host:port or ip:port input to a proxy option (e.g. `--http-proxy 127.0.0.1:8050`) should be reformatted by `update_scheme()` to the appropriate format (e.g. `http://127.0.0.1:8050`) before being used as input to urllib3.

Actual: *In Python 3.9* the value is used as-is, and because there is no http/https scheme on the URL, urllib3 throws a `ProxySchemeUnknown` exception. (`Proxy URL had no scheme, should start with http:// or https://`).

### Reproduction steps / Explicit stream URLs to test

<!-- How can we reproduce this? Please note the exact steps below using the list format supplied. If you need more steps please add them. -->

The observed behavior can be abstracted entirely away from the Streamlink CLI.

In session.py, we have the following (https://github.com/streamlink/streamlink/blob/master/src/streamlink/session.py#L279-L285):

```
        elif key == "http-proxy":
            self.http.proxies["http"] = update_scheme("http://", value)
            if "https" not in self.http.proxies:
                self.http.proxies["https"] = update_scheme("http://", value)

        elif key == "https-proxy":
            self.http.proxies["https"] = update_scheme("https://", value)
```

The `update_scheme()` function should apply the appropriate scheme to the URL, but on *Python 3.9* it does not in the case of a schemeless input with port:

```
Python 3.9.5 (default, May 24 2021, 12:50:35) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from streamlink import utils
>>> utils.update_scheme("http://", "127.0.0.1:8050")
'127.0.0.1:8050'
>>> utils.update_scheme("https://", "127.0.0.1:8050")
'127.0.0.1:8050'
>>> utils.update_scheme("https://", "localhost:8050")
'localhost:8050'
>>> utils.update_scheme("https://", "localhost")
'https://localhost'
```

Note that the only properly-handled input above was the "schemeless hostname without port" case - any schemeless input _with_ port does not get the expected scheme added.

The root of this issue is actually in urllib, as `update_scheme()` calls `urlparse()` from urllib. In Python 3.8, the `urlparse` call results in the following:

```
Python 3.8.10 (default, May 12 2021, 15:46:43) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import parse
>>> parse.urlparse('127.0.0.1:8080')
ParseResult(scheme='', netloc='', path='127.0.0.1:8080', params='', query='', fragment='')
```

Whereas in Python 3.9, the folloiwng occurs:
```
Python 3.9.5 (default, May 12 2021, 15:26:36) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import parse
>>> parse.urlparse('127.0.0.1:8080')
ParseResult(scheme='127.0.0.1', netloc='', path='8080', params='', query='', fragment='')
```

Note in the latter case, urllib returns the IP address in the `scheme` portion of the tuple instead of ... anywhere else (I'm honestly not sure if it should end up in `netloc` or `path`).

### Log output

The consumer of Streamlink which I was troubleshooting (LazyMan) shells out to Streamlink with `--https-proxy 127.0.0.1:<port>` - a schemeless url with port - as seen here: https://github.com/StevensNJD4/LazyMan/blob/master/src/Objects/Streamlink.java#L45

This results in the following output from LM when run on a system with Python 3.9:

```
-----

-----
Streamlink starting...
urlparse was ParseResult(scheme='hlsvariant', netloc='<censored> name_key=bitrate verify=False', params='', query='', fragment='')
got https-proxy 127.0.0.1:8050
urlparse was ParseResult(scheme='127.0.0.1', netloc='', path='8050', params='', query='', fragment='')
output of update_scheme is 127.0.0.1:8050
urlparse was ParseResult(scheme='127.0.0.1', netloc='', path='8050', params='', query='', fragment='')
self.http.proxies['https'] is 127.0.0.1:8050
[cli][debug] OS:         Linux-5.12.7-arch1-1-x86_64-with-glibc2.33
[cli][debug] Python:     3.9.5
[cli][debug] Streamlink: 2.1.1
[cli][debug] Requests(2.25.1), Socks(1.7.1), Websocket(0.59.0)
[cli][debug] Arguments:
[cli][debug]  url=hlsvariant://<censored> name_key=bitrate verify=False
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --hls-segment-threads=4
[cli][debug]  --https-proxy=127.0.0.1:8050
[cli][debug]  --http-header=[('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36')]
[cli][info] Found matching plugin hls for URL hlsvariant://<censored> name_key=bitrate verify=False
urlparse was ParseResult(scheme='<censored>', params='', query='', fragment='')
[plugins.hls][debug] URL=<censored>; params={'name_key': 'bitrate', 'verify': False}
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: 670k (worst), 1000k, 1450k, 2140k, 2900k, 4100k, 6600k (best)
[cli][info] Opening stream: 6600k (hls)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 0; Last Sequence: 1970
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 0; End Sequence: 1970
[stream.hls][debug] Adding segment 0 to queue
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][debug] Adding segment 5 to queue
[stream.hls][debug] Adding segment 6 to queue
[stream.hls][debug] Adding segment 7 to queue
[stream.hls][debug] Adding segment 8 to queue
[stream.hls][debug] Adding segment 9 to queue
[stream.hls][debug] Adding segment 10 to queue
[stream.hls][debug] Adding segment 11 to queue
[stream.hls][debug] Adding segment 12 to queue
[stream.hls][debug] Adding segment 13 to queue
[stream.hls][debug] Adding segment 14 to queue
[stream.hls][debug] Adding segment 15 to queue
[stream.hls][debug] Adding segment 16 to queue
[stream.hls][debug] Adding segment 17 to queue
[stream.hls][debug] Adding segment 18 to queue
[stream.hls][debug] Adding segment 19 to queue
[stream.hls][debug] Adding segment 20 to queue
[stream.hls][debug] Adding segment 21 to queue

[stream.hls][error] Failed to create decryptor: Unable to open URL: https://<url snipped> (Proxy URL had no scheme, should start with http:// or https://)
[stream.segmented][debug] Closing writer thread
[stream.segmented][debug] Closing worker thread
[cli][error] Try 1/1: Could not open stream <HLSStream('<censored>', 'http://<url snipped>')> (No data returned from stream)
error: Could not open stream <HLSStream('<censored>', 'http://<url snipped>')>, tried 1 times, exiting
[cli][info] Closing currently open stream...

```

In the above, the notable output is the `Proxy URL had no scheme [...]` error which maps to a `ProxySchemeUnknown` exception from urllib3.

### Additional comments, etc.


Comments:

Patch:
diff --git a/src/streamlink/utils/url.py b/src/streamlink/utils/url.py
index 7776c648..bc967126 100644
--- a/src/streamlink/utils/url.py
+++ b/src/streamlink/utils/url.py
@@ -1,24 +1,38 @@
+import re
 from collections import OrderedDict
 from urllib.parse import parse_qsl, quote_plus, urlencode, urljoin, urlparse, urlunparse
 
 
-def update_scheme(current, target):
+_re_uri_implicit_scheme = re.compile(r"""^[a-z0-9][a-z0-9.+-]*://""", re.IGNORECASE)
+
+
+def update_scheme(current: str, target: str) -> str:
     """
-    Take the scheme from the current URL and applies it to the
-    target URL if the target URL startswith // or is missing a scheme
+    Take the scheme from the current URL and apply it to the
+    target URL if the target URL starts with // or is missing a scheme
     :param current: current URL
     :param target: target URL
     :return: target URL with the current URLs scheme
     """
     target_p = urlparse(target)
+
+    if (
+        # target URLs with implicit scheme and netloc including a port: ("http://", "foo.bar:1234") -> "http://foo.bar:1234"
+        # urllib.parse.urlparse has incorrect behavior in py<3.9, so we'll have to use a regex here
+        # py>=3.9: urlparse("127.0.0.1:1234") == ParseResult(scheme='127.0.0.1', netloc='', path='1234', ...)
+        # py<3.9 : urlparse("127.0.0.1:1234") == ParseResult(scheme='', netloc='', path='127.0.0.1:1234', ...)
+        not _re_uri_implicit_scheme.search(target) and not target.startswith("//")
+        # target URLs without scheme and netloc: ("http://", "foo.bar/foo") -> "http://foo.bar/foo"
+        or not target_p.scheme and not target_p.netloc
+    ):
+        return f"{urlparse(current).scheme}://{urlunparse(target_p)}"
+
+    # target URLs without scheme but with netloc: ("http://", "//foo.bar/foo") -> "http://foo.bar/foo"
     if not target_p.scheme and target_p.netloc:
-        return "{0}:{1}".format(urlparse(current).scheme,
-                                urlunparse(target_p))
-    elif not target_p.scheme and not target_p.netloc:
-        return "{0}://{1}".format(urlparse(current).scheme,
-                                  urlunparse(target_p))
-    else:
-        return target
+        return f"{urlparse(current).scheme}:{urlunparse(target_p)}"
+
+    # target URLs with scheme
+    return target
 
 
 def url_equal(first, second, ignore_scheme=False, ignore_netloc=False, ignore_path=False, ignore_params=False,


Test patch:
diff --git a/tests/test_utils_url.py b/tests/test_utils_url.py
index 9f283316..f5b725fd 100644
--- a/tests/test_utils_url.py
+++ b/tests/test_utils_url.py
@@ -9,6 +9,9 @@ def test_update_scheme():
     assert update_scheme("http://other.com/bar", "//example.com/foo") == "http://example.com/foo", "should become http"
     assert update_scheme("https://other.com/bar", "http://example.com/foo") == "http://example.com/foo", "should remain http"
     assert update_scheme("https://other.com/bar", "example.com/foo") == "https://example.com/foo", "should become https"
+    assert update_scheme("http://", "127.0.0.1:1234/foo") == "http://127.0.0.1:1234/foo", "implicit scheme with IPv4+port"
+    assert update_scheme("http://", "foo.bar:1234/foo") == "http://foo.bar:1234/foo", "implicit scheme with hostname+port"
+    assert update_scheme("http://", "foo.1+2-bar://baz") == "foo.1+2-bar://baz", "correctly parses all kinds of schemes"
 
 
 def test_url_equal():