Repository: yt-dlp/yt-dlp
Pull request #8510 (2023-11-03 19:19:52)

[cleanup] Misc. (for 2023-11 release)
**IMPORTANT**: PRs without the template will be CLOSED

### Description of your *pull request* and other information

<!--

Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible

-->

<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [x] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (bashonly, coletdjnz, dirkf, gamer191, seproDev)

### What is the purpose of your *pull request*?
- [x] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))


<!-- Do NOT edit/remove anything below this! -->
</details><details><summary>Copilot Summary</summary>  

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 31c5391</samp>

### Summary
:sparkles::warning::mute:

<!--
1.  :sparkles: for the new feature that adds support for the requests library
2.  :warning: for the modified test case that expects more warnings from the extractor
3.  :mute: for the lowered logging level of the urllib3 logger to suppress a warning message
-->
Add support for the requests library as an alternative to urllib3, and update the YoutubePlaylistIE extractor and the changelog accordingly. Fix a logging issue with the requests library in `yt_dlp/networking/_requests.py`.

> _`requests` library_
> _new feature for youtube-dl_
> _autumn changelog cut_

### Walkthrough
*  Add support for the requests HTTP library, which enables HTTPS proxies and persistent connections ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-323808d1d1a45be3d06ea051c76192878238cd5c25e74087fbdad00eab2c6a16R101-R106), [link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-ef5ef94777b4b7a8fd1579b2053264fb5068b9e7baf33e95c23d3541a9eab1e4L258-R259), [link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-b7b9f6790de4427214b61939432e667d95b929d07fd918b9da1a36d7996cc506L6690-R6690))
   * Override the changelog entry for this feature in `changelog_override.json` ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-323808d1d1a45be3d06ea051c76192878238cd5c25e74087fbdad00eab2c6a16R101-R106))
   * Implement a wrapper class `RequestsRH` that adapts the requests library to the request handler interface ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-ef5ef94777b4b7a8fd1579b2053264fb5068b9e7baf33e95c23d3541a9eab1e4L258-R259))
      * Suppress a warning message from the urllib3 logger by lowering its level to ERROR ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-ef5ef94777b4b7a8fd1579b2053264fb5068b9e7baf33e95c23d3541a9eab1e4L258-R259))
   * Update the test case for `YoutubePlaylistIE` to expect two more warnings related to unavailable videos ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-b7b9f6790de4427214b61939432e667d95b929d07fd918b9da1a36d7996cc506L6690-R6690))
      * Reflect the new behavior of the extractor, which uses the requests library and handles unavailable videos differently ([link](https://github.com/yt-dlp/yt-dlp/pull/8510/files?diff=unified&w=0#diff-b7b9f6790de4427214b61939432e667d95b929d07fd918b9da1a36d7996cc506L6690-R6690))



</details>



Patch:
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index cbed82173..c4d3e812e 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -40,10 +40,4 @@ ### What is the purpose of your *pull request*?
 - [ ] Core bug fix/improvement
 - [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))
 
-
-<!-- Do NOT edit/remove anything below this! -->
-</details><details><summary>Copilot Summary</summary>  
-
-copilot:all
-
 </details>
diff --git a/README.md b/README.md
index 1b92c64d6..8b92f827b 100644
--- a/README.md
+++ b/README.md
@@ -163,10 +163,10 @@ ### Differences in default behavior
 For ease of use, a few more compat options are available:
 
 * `--compat-options all`: Use all compat options (Do NOT use)
-* `--compat-options youtube-dl`: Same as `--compat-options all,-multistreams,-playlist-match-filter`
-* `--compat-options youtube-dlc`: Same as `--compat-options all,-no-live-chat,-no-youtube-channel-redirect,-playlist-match-filter`
+* `--compat-options youtube-dl`: Same as `--compat-options all,-multistreams,-playlist-match-filter,-manifest-filesize-approx`
+* `--compat-options youtube-dlc`: Same as `--compat-options all,-no-live-chat,-no-youtube-channel-redirect,-playlist-match-filter,-manifest-filesize-approx`
 * `--compat-options 2021`: Same as `--compat-options 2022,no-certifi,filename-sanitization,no-youtube-prefer-utc-upload-date`
-* `--compat-options 2022`: Same as `--compat-options playlist-match-filter,no-external-downloader-progress,prefer-legacy-http-handler`. Use this to enable all future compat options
+* `--compat-options 2022`: Same as `--compat-options playlist-match-filter,no-external-downloader-progress,prefer-legacy-http-handler,manifest-filesize-approx`. Use this to enable all future compat options
 
 
 # INSTALLATION
diff --git a/devscripts/changelog_override.json b/devscripts/changelog_override.json
index fe0c82c66..010820295 100644
--- a/devscripts/changelog_override.json
+++ b/devscripts/changelog_override.json
@@ -98,5 +98,21 @@
         "action": "add",
         "when": "61bdf15fc7400601c3da1aa7a43917310a5bf391",
         "short": "[priority] Security: [[CVE-2023-40581](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40581)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg)\n    - The shell escape function is now using `\"\"` instead of `\\\"`.\n    - `utils.Popen` has been patched to properly quote commands."
+    },
+    {
+        "action": "change",
+        "when": "8a8b54523addf46dfd50ef599761a81bc22362e6",
+        "short": "[rh:requests] Add handler for `requests` HTTP library (#3668)\n\n\tAdds support for HTTPS proxies and persistent connections (keep-alive)",
+        "authors": ["bashonly", "coletdjnz", "Grub4K"]
+    },
+    {
+        "action": "add",
+        "when": "1d03633c5a1621b9f3a756f0a4f9dc61fab3aeaa",
+        "short": "[priority] **The release channels have been adjusted!**\n\t* [`master`](https://github.com/yt-dlp/yt-dlp-master-builds) builds are made after each push, containing the latest fixes (but also possibly bugs). This was previously the `nightly` channel.\n\t* [`nightly`](https://github.com/yt-dlp/yt-dlp-nightly-builds) builds are now made once a day, if there were any changes."
+    },
+    {
+        "action": "add",
+        "when": "f04b5bedad7b281bee9814686bba1762bae092eb",
+        "short": "[priority] Security: [[CVE-2023-46121](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46121)] Patch [Generic Extractor MITM Vulnerability via Arbitrary Proxy Injection](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-3ch3-jhc6-5r8x)\n\t- Disallow smuggling of arbitary `http_headers`; extractors now only use specific headers"
     }
 ]
diff --git a/yt_dlp/extractor/cybrary.py b/yt_dlp/extractor/cybrary.py
index aeffe93b4..c4c78ee1b 100644
--- a/yt_dlp/extractor/cybrary.py
+++ b/yt_dlp/extractor/cybrary.py
@@ -105,7 +105,7 @@ def _real_extract(self, url):
             'chapter': module.get('title'),
             'chapter_id': str_or_none(module.get('id')),
             'title': activity.get('title'),
-            'url': smuggle_url(f'https://player.vimeo.com/video/{vimeo_id}', {'http_headers': {'Referer': 'https://api.cybrary.it'}})
+            'url': smuggle_url(f'https://player.vimeo.com/video/{vimeo_id}', {'referer': 'https://api.cybrary.it'})
         }
 
 
diff --git a/yt_dlp/extractor/duboku.py b/yt_dlp/extractor/duboku.py
index fb0546cae..fc9564cef 100644
--- a/yt_dlp/extractor/duboku.py
+++ b/yt_dlp/extractor/duboku.py
@@ -138,7 +138,7 @@ def _real_extract(self, url):
             # of the video.
             return {
                 '_type': 'url_transparent',
-                'url': smuggle_url(data_url, {'http_headers': headers}),
+                'url': smuggle_url(data_url, {'referer': webpage_url}),
                 'id': video_id,
                 'title': title,
                 'series': series_title,
diff --git a/yt_dlp/extractor/embedly.py b/yt_dlp/extractor/embedly.py
index 458aaa0a0..a424b49df 100644
--- a/yt_dlp/extractor/embedly.py
+++ b/yt_dlp/extractor/embedly.py
@@ -106,4 +106,4 @@ def _real_extract(self, url):
             return self.url_result(src, YoutubeTabIE)
         return self.url_result(smuggle_url(
             urllib.parse.unquote(traverse_obj(qs, ('src', 0), ('url', 0))),
-            {'http_headers': {'Referer': url}}))
+            {'referer': url}))
diff --git a/yt_dlp/extractor/generic.py b/yt_dlp/extractor/generic.py
index ac7cc673f..1503e5146 100644
--- a/yt_dlp/extractor/generic.py
+++ b/yt_dlp/extractor/generic.py
@@ -17,6 +17,7 @@
     determine_protocol,
     dict_get,
     extract_basic_auth,
+    filter_dict,
     format_field,
     int_or_none,
     is_html,
@@ -2435,10 +2436,10 @@ def _real_extract(self, url):
         # to accept raw bytes and being able to download only a chunk.
         # It may probably better to solve this by checking Content-Type for application/octet-stream
         # after a HEAD request, but not sure if we can rely on this.
-        full_response = self._request_webpage(url, video_id, headers={
+        full_response = self._request_webpage(url, video_id, headers=filter_dict({
             'Accept-Encoding': 'identity',
-            **smuggled_data.get('http_headers', {})
-        })
+            'Referer': smuggled_data.get('referer'),
+        }))
         new_url = full_response.url
         url = urllib.parse.urlparse(url)._replace(scheme=urllib.parse.urlparse(new_url).scheme).geturl()
         if new_url != extract_basic_auth(url)[0]:
@@ -2458,7 +2459,7 @@ def _real_extract(self, url):
         m = re.match(r'^(?P<type>audio|video|application(?=/(?:ogg$|(?:vnd\.apple\.|x-)?mpegurl)))/(?P<format_id>[^;\s]+)', content_type)
         if m:
             self.report_detected('direct video link')
-            headers = smuggled_data.get('http_headers', {})
+            headers = filter_dict({'Referer': smuggled_data.get('referer')})
             format_id = str(m.group('format_id'))
             ext = determine_ext(url, default_ext=None) or urlhandle_detect_ext(full_response)
             subtitles = {}
@@ -2710,7 +2711,7 @@ def _extract_embeds(self, url, webpage, *, urlh=None, info_dict={}):
                 'url': smuggle_url(json_ld['url'], {
                     'force_videoid': video_id,
                     'to_generic': True,
-                    'http_headers': {'Referer': url},
+                    'referer': url,
                 }),
             }, json_ld)]
 
diff --git a/yt_dlp/extractor/la7.py b/yt_dlp/extractor/la7.py
index a3cd12b00..f5fd24134 100644
--- a/yt_dlp/extractor/la7.py
+++ b/yt_dlp/extractor/la7.py
@@ -208,9 +208,9 @@ class LA7PodcastIE(LA7PodcastEpisodeIE):  # XXX: Do not subclass from concrete I
         'url': 'https://www.la7.it/propagandalive/podcast',
         'info_dict': {
             'id': 'propagandalive',
-            'title': "Propaganda Live",
+            'title': 'Propaganda Live',
         },
-        'playlist_count_min': 10,
+        'playlist_mincount': 10,
     }]
 
     def _real_extract(self, url):
diff --git a/yt_dlp/extractor/redtube.py b/yt_dlp/extractor/redtube.py
index 49076ccd8..172c31b39 100644
--- a/yt_dlp/extractor/redtube.py
+++ b/yt_dlp/extractor/redtube.py
@@ -39,7 +39,7 @@ class RedTubeIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
         webpage = self._download_webpage(
-            'http://www.redtube.com/%s' % video_id, video_id)
+            f'https://www.redtube.com/{video_id}', video_id)
 
         ERRORS = (
             (('video-deleted-info', '>This video has been removed'), 'has been removed'),
diff --git a/yt_dlp/extractor/slideslive.py b/yt_dlp/extractor/slideslive.py
index 13f3109d7..df2af3b35 100644
--- a/yt_dlp/extractor/slideslive.py
+++ b/yt_dlp/extractor/slideslive.py
@@ -530,7 +530,7 @@ def _real_extract(self, url):
             if service_name == 'vimeo':
                 info['url'] = smuggle_url(
                     f'https://player.vimeo.com/video/{service_id}',
-                    {'http_headers': {'Referer': url}})
+                    {'referer': url})
 
         video_slides = traverse_obj(slides, ('slides', ..., 'video', 'id'))
         if not video_slides:
diff --git a/yt_dlp/extractor/storyfire.py b/yt_dlp/extractor/storyfire.py
index 035747c31..566f77782 100644
--- a/yt_dlp/extractor/storyfire.py
+++ b/yt_dlp/extractor/storyfire.py
@@ -32,9 +32,7 @@ def _parse_video(self, video):
             'description': video.get('description'),
             'url': smuggle_url(
                 'https://player.vimeo.com/video/' + vimeo_id, {
-                    'http_headers': {
-                        'Referer': 'https://storyfire.com/',
-                    }
+                    'referer': 'https://storyfire.com/',
                 }),
             'thumbnail': video.get('storyImage'),
             'view_count': int_or_none(video.get('views')),
diff --git a/yt_dlp/extractor/videoken.py b/yt_dlp/extractor/videoken.py
index 560b41a6d..eaf0cc8ae 100644
--- a/yt_dlp/extractor/videoken.py
+++ b/yt_dlp/extractor/videoken.py
@@ -11,6 +11,7 @@
     ExtractorError,
     InAdvancePagedList,
     int_or_none,
+    remove_start,
     traverse_obj,
     update_url_query,
     url_or_none,
@@ -39,11 +40,11 @@ def _create_slideslive_url(self, video_url, video_id, referer):
         if not video_url and not video_id:
             return
         elif not video_url or 'embed/sign-in' in video_url:
-            video_url = f'https://slideslive.com/embed/{video_id.lstrip("slideslive-")}'
+            video_url = f'https://slideslive.com/embed/{remove_start(video_id, "slideslive-")}'
         if url_or_none(referer):
             return update_url_query(video_url, {
                 'embed_parent_url': referer,
-                'embed_container_origin': f'https://{urllib.parse.urlparse(referer).netloc}',
+                'embed_container_origin': f'https://{urllib.parse.urlparse(referer).hostname}',
             })
         return video_url
 
@@ -57,12 +58,12 @@ def _extract_videos(self, videos, url):
                 video_url = video_id
                 ie_key = 'Youtube'
             else:
-                video_url = traverse_obj(video, 'embed_url', 'embeddableurl')
-                if urllib.parse.urlparse(video_url).netloc == 'slideslive.com':
+                video_url = traverse_obj(video, 'embed_url', 'embeddableurl', expected_type=url_or_none)
+                if not video_url:
+                    continue
+                elif urllib.parse.urlparse(video_url).hostname == 'slideslive.com':
                     ie_key = SlidesLiveIE
                     video_url = self._create_slideslive_url(video_url, video_id, url)
-            if not video_url:
-                continue
             yield self.url_result(video_url, ie_key, video_id)
 
 
@@ -178,7 +179,7 @@ def _real_extract(self, url):
             return self.url_result(
                 self._create_slideslive_url(None, video_id, url), SlidesLiveIE, video_id)
         elif re.match(r'^[\w-]{11}$', video_id):
-            self.url_result(video_id, 'Youtube', video_id)
+            return self.url_result(video_id, 'Youtube', video_id)
         else:
             raise ExtractorError('Unable to extract without VideoKen API response')
 
diff --git a/yt_dlp/extractor/vimeo.py b/yt_dlp/extractor/vimeo.py
index e72fa50fa..e5e8144bb 100644
--- a/yt_dlp/extractor/vimeo.py
+++ b/yt_dlp/extractor/vimeo.py
@@ -37,14 +37,14 @@ class VimeoBaseInfoExtractor(InfoExtractor):
 
     @staticmethod
     def _smuggle_referrer(url, referrer_url):
-        return smuggle_url(url, {'http_headers': {'Referer': referrer_url}})
+        return smuggle_url(url, {'referer': referrer_url})
 
     def _unsmuggle_headers(self, url):
         """@returns (url, smuggled_data, headers)"""
         url, data = unsmuggle_url(url, {})
         headers = self.get_param('http_headers').copy()
-        if 'http_headers' in data:
-            headers.update(data['http_headers'])
+        if 'referer' in data:
+            headers['Referer'] = data['referer']
         return url, data, headers
 
     def _perform_login(self, username, password):
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
index adbac8e95..f6caf0970 100644
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -6687,7 +6687,7 @@ class YoutubePlaylistIE(InfoExtractor):
             'uploader_url': 'https://www.youtube.com/@milan5503',
             'availability': 'public',
         },
-        'expected_warnings': [r'[Uu]navailable videos? (is|are|will be) hidden'],
+        'expected_warnings': [r'[Uu]navailable videos? (is|are|will be) hidden', 'Retrying', 'Giving up'],
     }, {
         'url': 'http://www.youtube.com/embed/_xDOZElKyNU?list=PLsyOSbh5bs16vubvKePAQ1x3PhKavfBIl',
         'playlist_mincount': 455,
diff --git a/yt_dlp/networking/_requests.py b/yt_dlp/networking/_requests.py
index fe3f60b0b..9fb1d75f4 100644
--- a/yt_dlp/networking/_requests.py
+++ b/yt_dlp/networking/_requests.py
@@ -255,7 +255,8 @@ def __init__(self, *args, **kwargs):
         handler.setFormatter(logging.Formatter('requests: %(message)s'))
         handler.addFilter(Urllib3LoggingFilter())
         logger.addHandler(handler)
-        logger.setLevel(logging.WARNING)
+        # TODO: Use a logger filter to suppress pool reuse warning instead
+        logger.setLevel(logging.ERROR)
 
         if self.verbose:
             # Setting this globally is not ideal, but is easier than hacking with urllib3.
diff --git a/yt_dlp/utils/networking.py b/yt_dlp/utils/networking.py
index ba0493cc2..ed0250011 100644
--- a/yt_dlp/utils/networking.py
+++ b/yt_dlp/utils/networking.py
@@ -123,6 +123,7 @@ def clean_headers(headers: HTTPHeaderDict):
     if 'Youtubedl-No-Compression' in headers:  # compat
         del headers['Youtubedl-No-Compression']
         headers['Accept-Encoding'] = 'identity'
+    headers.pop('Ytdl-socks-proxy', None)
 
 
 def remove_dot_segments(path):


Test patch:
diff --git a/test/test_networking.py b/test/test_networking.py
index 689161fb2..4466fc048 100644
--- a/test/test_networking.py
+++ b/test/test_networking.py
@@ -1293,6 +1293,10 @@ def test_clean_header(self):
             assert 'Youtubedl-no-compression' not in rh.headers
             assert rh.headers.get('Accept-Encoding') == 'identity'
 
+        with FakeYDL({'http_headers': {'Ytdl-socks-proxy': 'socks://localhost:1080'}}) as ydl:
+            rh = self.build_handler(ydl)
+            assert 'Ytdl-socks-proxy' not in rh.headers
+
     def test_build_handler_params(self):
         with FakeYDL({
             'http_headers': {'test': 'testtest'},