Repository: yt-dlp/yt-dlp
Pull request #7704 (2023-07-27 07:50:30)

[networking] Fix handling of unrelated invalid proxy urls from env
Issue:
```
$ UNRELATED_PROXY=/sfsdffgsdfsd/fsdf/sdf/sdf/ yt-dlp -v
[debug] Command-line config: ['-v']
[debug] User config "/home/coletdjnz/.config/yt-dlp.conf": ['--ffmpeg-location', '/opt/yt-dlp-ffmpeg']
Traceback (most recent call last):
  File "/home/coletdjnz/.local/bin/yt-dlp", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/coletdjnz/.local/lib/python3.11/site-packages/yt_dlp/__init__.py", line 1008, in main
    _exit(*variadic(_real_main(argv)))
                    ^^^^^^^^^^^^^^^^
  File "/home/coletdjnz/.local/lib/python3.11/site-packages/yt_dlp/__init__.py", line 962, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
         ^^^^^^^^^^^^^^^^^^^
  File "/home/coletdjnz/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 688, in __init__
    self._request_director = self.build_request_director(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/coletdjnz/.local/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 4086, in build_request_director
    clean_proxies(proxies, headers)
  File "/home/coletdjnz/.local/lib/python3.11/site-packages/yt_dlp/utils/networking.py", line 101, in clean_proxies
    proxy_scheme = urllib.request._parse_proxy(proxy_url)[0]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 773, in _parse_proxy
    raise ValueError("proxy URL with no authority: %r" % proxy)
ValueError: proxy URL with no authority: '/sfsdffgsdfsd/fsdf/sdf/sdf/'
```

This PR makes it so we catch the ValueError in `clean_proxies` and ignore. 

In RequestHandler we catch and re-raise as an UnsupportedRequest if the proxy is actually going to be used.

Marked as high priority and regression as some environments may have `*_PROXY` environment variables set to strings other than a valid url, and were never to be used in yt-dlp. For example, Google Colab has a  `COLAB_LANGUAGE_SERVER_PROXY` env var which is set to `/usr/colab/bin/language_service`. 


<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [X] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [X] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [X] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [X] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [ ] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [X] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))


<!-- Do NOT edit/remove anything below this! -->
</details><details><summary>Copilot Summary</summary>  

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 542488f</samp>

### Summary
🛠️🧪🧹

<!--
1.  🛠️ for modifying the `_check_proxies` function to catch and re-raise the `ValueError` exception as an `UnsupportedRequest` exception.
2.  🧪 for adding and modifying the test cases in `test_networking.py` to increase the coverage and robustness of the proxy handling and validation logic.
3.  🧹 for improving the proxy handling and validation logic by adding a blank line for formatting and modifying the `clean_proxies` function to ignore invalid proxy URLs.
-->
This pull request enhances the proxy handling and validation logic in `yt_dlp`. It adds and modifies some tests in `test_networking.py`, fixes a formatting issue in `networking.py`, and improves the error handling in `common.py`.

> _`yt-dlp` fixes_
> _proxy validation logic_
> _autumn leaves falling_

### Walkthrough
*  Validate and normalize proxy URLs in request handlers and utils ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78R933-R936), [link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1129-R1140), [link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL265-R271), [link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-b6d91e8afdfc47fb2b59e37d22d3621b79d252bc5a8205edc2b80530dcceecc2R93), [link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-b6d91e8afdfc47fb2b59e37d22d3621b79d252bc5a8205edc2b80530dcceecc2L101-R108))
  * Catch and re-raise `ValueError` exceptions from `urllib.request._parse_proxy` as `UnsupportedRequest` exceptions in `RequestHandler._check_proxies` ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL265-R271))
  * Add a test case for `RequestHandler._check_proxies` using invalid proxy URLs and different request handlers ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78R933-R936))
  * Ignore `ValueError` exceptions from `urllib.request._parse_proxy` in `clean_proxies` and skip modifying the proxy URL ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-b6d91e8afdfc47fb2b59e37d22d3621b79d252bc5a8205edc2b80530dcceecc2L101-R108))
  * Add a parameter to the test case for `clean_proxies` using an invalid proxy URL that should be ignored ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1129-R1140))
  * Add a blank line to `yt_dlp/utils/networking.py` for formatting ([link](https://github.com/yt-dlp/yt-dlp/pull/7704/files?diff=unified&w=0#diff-b6d91e8afdfc47fb2b59e37d22d3621b79d252bc5a8205edc2b80530dcceecc2R93))



</details>



Patch:
diff --git a/yt_dlp/extractor/wrestleuniverse.py b/yt_dlp/extractor/wrestleuniverse.py
index 99a8f0120..dd12804db 100644
--- a/yt_dlp/extractor/wrestleuniverse.py
+++ b/yt_dlp/extractor/wrestleuniverse.py
@@ -300,7 +300,7 @@ def _real_extract(self, url):
             info['hls_aes'] = {
                 'key': hls_aes_key,
                 'iv': traverse_obj(video_data, ('hls', 'iv', {decrypt})),
-            },
+            }
         elif traverse_obj(video_data, ('hls', 'encryptType', {int})):
             self.report_warning('HLS AES-128 key was not found in API response')
 
diff --git a/yt_dlp/networking/common.py b/yt_dlp/networking/common.py
index ab26a0628..3164df49b 100644
--- a/yt_dlp/networking/common.py
+++ b/yt_dlp/networking/common.py
@@ -262,9 +262,13 @@ def _check_proxies(self, proxies):
                 # Skip proxy scheme checks
                 continue
 
-            # Scheme-less proxies are not supported
-            if urllib.request._parse_proxy(proxy_url)[0] is None:
-                raise UnsupportedRequest(f'Proxy "{proxy_url}" missing scheme')
+            try:
+                if urllib.request._parse_proxy(proxy_url)[0] is None:
+                    # Scheme-less proxies are not supported
+                    raise UnsupportedRequest(f'Proxy "{proxy_url}" missing scheme')
+            except ValueError as e:
+                # parse_proxy may raise on some invalid proxy urls such as "/a/b/c"
+                raise UnsupportedRequest(f'Invalid proxy url "{proxy_url}": {e}')
 
             scheme = urllib.parse.urlparse(proxy_url).scheme.lower()
             if scheme not in self._SUPPORTED_PROXY_SCHEMES:
diff --git a/yt_dlp/utils/networking.py b/yt_dlp/utils/networking.py
index ac355ddc8..e6515ec8e 100644
--- a/yt_dlp/utils/networking.py
+++ b/yt_dlp/utils/networking.py
@@ -98,7 +98,13 @@ def clean_proxies(proxies: dict, headers: HTTPHeaderDict):
             continue
         if proxy_url is not None:
             # Ensure proxies without a scheme are http.
-            proxy_scheme = urllib.request._parse_proxy(proxy_url)[0]
+            try:
+                proxy_scheme = urllib.request._parse_proxy(proxy_url)[0]
+            except ValueError:
+                # Ignore invalid proxy URLs. Sometimes these may be introduced through environment
+                # variables unrelated to proxy settings - e.g. Colab `COLAB_LANGUAGE_SERVER_PROXY`.
+                # If the proxy is going to be used, the Request Handler proxy validation will handle it.
+                continue
             if proxy_scheme is None:
                 proxies[proxy_key] = 'http://' + remove_start(proxy_url, '//')
 


Test patch:
diff --git a/test/test_networking.py b/test/test_networking.py
index dbe28359b..f0938ab91 100644
--- a/test/test_networking.py
+++ b/test/test_networking.py
@@ -930,10 +930,10 @@ def test_empty_proxy(self, handler):
         run_validation(handler, False, Request('http://', proxies={'http': None}))
         run_validation(handler, False, Request('http://'), proxies={'http': None})
 
-    @pytest.mark.parametrize('proxy_url', ['//example.com', 'example.com', '127.0.0.1'])
+    @pytest.mark.parametrize('proxy_url', ['//example.com', 'example.com', '127.0.0.1', '/a/b/c'])
     @pytest.mark.parametrize('handler', ['Urllib'], indirect=True)
-    def test_missing_proxy_scheme(self, handler, proxy_url):
-        run_validation(handler, UnsupportedRequest, Request('http://', proxies={'http': 'example.com'}))
+    def test_invalid_proxy_url(self, handler, proxy_url):
+        run_validation(handler, UnsupportedRequest, Request('http://', proxies={'http': proxy_url}))
 
     @pytest.mark.parametrize('handler,extensions,fail', [
         (handler_tests[0], extensions, fail)
@@ -1126,9 +1126,11 @@ def test_legacy_server_connect_error(self):
         ('http', '__noproxy__', None),
         ('no', '127.0.0.1,foo.bar', '127.0.0.1,foo.bar'),
         ('https', 'example.com', 'http://example.com'),
+        ('https', '//example.com', 'http://example.com'),
         ('https', 'socks5://example.com', 'socks5h://example.com'),
         ('http', 'socks://example.com', 'socks4://example.com'),
         ('http', 'socks4://example.com', 'socks4://example.com'),
+        ('unrelated', '/bad/proxy', '/bad/proxy'),  # clean_proxies should ignore bad proxies
     ])
     def test_clean_proxy(self, proxy_key, proxy_url, expected):
         # proxies should be cleaned in urlopen()