Repository: yt-dlp/yt-dlp
Pull request #7484 (2023-07-01 18:14:09)

CBCPlayer
**IMPORTANT**: PRs without the template will be CLOSED

### Description of your *pull request* and other information

<!--

Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible

-->

### Main Goal
Add subtitle and additional format support.
### Details of the changes
Currently, the ThePlatformIE just tries to extract the video directly, as well as the JSON information. This sometimes misses certain formats and subtitles: since often the direct video is the maximum quality, it doesn't currently allow downloading lower quality formats in some cases. The information about subtitles also sometimes doesn't exist here.
As a solution to this, the m3u8 playlist file is also analyzed. This can help find additional formats as well as subtitles.
### Related issues that came out of this improvement
For one, there's the issue of changing ThePlatformIE, as it's an IE that other IEs rely on. There is no overly difficult way to systematically ensure that these changes don't break other IE's. I've done my best to make sure that the changes don't affect the functionality, but it's hard to be sure. See #7483 .
Then there the issue of checking m3u8 files. In this PR, it is handled by the ThePlatformIE, but it seems to me like this should be in the parent InfoExtractor class. Mind you, here, I'm doing it by checking the file extension, but for the parent class, it would make sense to check the first line of the m3u8 file. See #7482 .

Fixes #


<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [X] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [X] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [X] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [X] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [X] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))


<!-- Do NOT edit/remove anything below this! -->
</details><details><summary>Copilot Summary</summary>  

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 9f5d60f</samp>

### Summary
🧪🎬🚫

<!--
1.  🧪 - This emoji represents testing or experimentation, and can be used to indicate the improvement of the test cases and format sorting for the `CBCPlayerIE` extractor.
2.  🎬 - This emoji represents movies or video production, and can be used to indicate the added support for subtitles and chapters from CBC videos, as well as the m3u8 formats in `ThePlatformIE` extractor.
3.  🚫 - This emoji represents prohibition or rejection, and can be used to indicate the skipping of some outdated test cases that are no longer relevant or valid.
-->
Improved extraction of CBC videos and added subtitles and chapters support. Added m3u8 and subtitles support for ThePlatform videos. Updated some test cases in `cbc.py` and `theplatform.py`.

> _Sing, O Muse, of the skillful coder who refined_
> _The `CBCPlayerIE` extractor, and with cunning mind_
> _Added subtitles and chapters, as the heroes of old_
> _Carved their glorious deeds on tablets of gold._

### Walkthrough
*  Add `thumbnail`, `chapters`, and `duration` fields to a test case and override format sorting for `CBCPlayerIE` ([link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-f0840207126abfbed9de8bbd0799354c8163c9d8924b24c8eb0bed3378fe6ceeR177-R179), [link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-f0840207126abfbed9de8bbd0799354c8163c9d8924b24c8eb0bed3378fe6ceeR241-R244))
*  Add a new test case for subtitles and chapters extraction for `CBCPlayerIE` ([link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-f0840207126abfbed9de8bbd0799354c8163c9d8924b24c8eb0bed3378fe6ceeL189-R228))
*  Update `skip` messages for outdated or unavailable videos in `CBCPlayerIE` and `ThePlatformIE` tests ([link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-f0840207126abfbed9de8bbd0799354c8163c9d8924b24c8eb0bed3378fe6ceeL164-R164), [link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-665418b61f97d16d95c134f49e50b95c175a7b154685876b26f468916710e43aL167-R168), [link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-665418b61f97d16d95c134f49e50b95c175a7b154685876b26f468916710e43aL176-R178), [link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-665418b61f97d16d95c134f49e50b95c175a7b154685876b26f468916710e43aR196))
*  Download and parse m3u8 file for additional formats and subtitles in `ThePlatformIE._real_extract` ([link](https://github.com/yt-dlp/yt-dlp/pull/7484/files?diff=unified&w=0#diff-665418b61f97d16d95c134f49e50b95c175a7b154685876b26f468916710e43aR303-R323))



</details>


trainman261: Updated some things.
Current status as to requested changes:

1. As mentioned, I fully agree that truncating the query string is, as @bashonly mentioned, blunt and inexact. I haven't found a better method though. I remain open to suggestions.
2. It now sends a HEADRequest and checks to make sure it actually serves an m3u8 file. If it doesn't, it doesn't download and carries on. As a side effect, in that case, it also doesn't show the "downloading m3u8 information" message, which makes more sense as well.
3. It now properly checks the file extension. It also now uses single quotes.

I've also done some regression tests - at least as far as I could without messing with other IEs. As tested, the following tests worked previously:

- AENetworks 
- CBCGem 
- CBCGemLive 
- NBCSportsStream 
- NBCStations 
- NBCSportsVPlayer (only working with CPython 3.8 as well as pypy 3.7, not working with CPython 3.7)

They all still work with the changes in this PR. Mind you, many of these tests are skipped or not very in-depth. Also, this is 6 tests out of a total of 36, as far as I can tell, so not great coverage. I've noticed a number of the failed tests are just because some fields are missing in the test data. So... should I fix these other tests before it gets merged?
pukkandan: > should I fix these other tests before it gets merged?

It'd be great to fix them, but you don't have to.
trainman261: OK, I'll try to see what I can do.
trainman261: Should I maybe make a seperate PR for fixing up the tests on other sites? They are really unrelated to this PR other than that I'm trying to get it fixed to improve regression testing for this PR.
bashonly: It's a core bug. Apply this patch to the run tests, but don't commit it
```diff
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 1a2f42fe9..f4a80ea11 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -572,7 +572,7 @@ class YoutubeDL:
         'width', 'height', 'aspect_ratio', 'resolution', 'dynamic_range', 'tbr', 'abr', 'acodec', 'asr', 'audio_channels',
         'vbr', 'fps', 'vcodec', 'container', 'filesize', 'filesize_approx', 'rows', 'columns',
         'player_url', 'protocol', 'fragment_base_url', 'fragments', 'is_from_start',
-        'preference', 'language', 'language_preference', 'quality', 'source_preference',
+        'preference', 'language', 'language_preference', 'quality', 'source_preference', 'cookies',
         'http_headers', 'stretched_ratio', 'no_resume', 'has_drm', 'extra_param_to_segment_url', 'hls_aes', 'downloader_options',
         'page_url', 'app', 'play_path', 'tc_url', 'flash_version', 'rtmp_live', 'rtmp_conn', 'rtmp_protocol', 'rtmp_real_time'
     }
```
pukkandan: > Should I maybe make a seperate PR for fixing up the tests on other sites? They are really unrelated to this PR other than that I'm trying to get it fixed to improve regression testing for this PR.

Let it be in this. I can split it later if needed.
trainman261: Aha OK, thanks!
pukkandan: > To make things easier on my end, I've pushed the individual fixes for IE tests to different branches.

I replied under the assumption it is easier for you to have everything in one PR. But if the other way is easier for you, I don't mind that either - you can open another PR with all the test fixes if you want to.

That aside, is there a reason you haven't addressed the requested changes for this PR yet?
bashonly: It's possible that #7667 could resolve the subtitles issue on its own, @trainman261 could you test?

Also, could you show the CBC formats extracted using only the patch from #7667 vs using this PR's patch?
trainman261: I just checked out #7667 , they are addressing separate issues.
Result with #7667 (same as master):
`ID        EXT RESOLUTION | FILESIZE   TBR PROTO | VCODEC  ACODEC
-----------------------------------------------------------------
http-5132 mp4 1920x1080  | ~1.67GiB 5132k https | unknown unknown`
With this patch:
`ID                    EXT  RESOLUTION FPS |   FILESIZE   TBR PROTO | VCODEC   VBR ACODEC     MORE INFO
---------------------------------------------------------------------------------------------------------------
program_audio-English m3u8 audio only     |                  m3u8  | audio only       unknown    [eng] English
422                   mp4  320x180     30 | ~140.60MiB  422k m3u8  | avc1.4d400d  422k video only
580                   mp4  320x180     30 | ~193.32MiB  581k m3u8  | avc1.640029  581k video only
910                   mp4  640x360     30 | ~303.17MiB  911k m3u8  | avc1.640029  911k video only
1350                  mp4  864x486     30 | ~449.62MiB 1351k m3u8  | avc1.640029 1351k video only
2120                  mp4  864x486     30 | ~705.92MiB 2121k m3u8  | avc1.640029 2121k video only
5640                  mp4  1920x1080   30 | ~  1.83GiB 5641k m3u8  | avc1.4d4028 5641k video only
http-5132             mp4  1920x1080      | ~  1.67GiB 5132k https | unknown       unknown`
bashonly: you can ignore the failing linter, doesn't have anything to do with this PR
trainman261: OK. Doing a final regression test. On CPython 3.8 everything worked, currently testing CPython 3.7 then PyPy and a final flake8 check.
trainman261: OK, all tests are through, including with f82aa89, no regressions anymore.
As to [https://github.com/yt-dlp/yt-dlp/pull/7484#discussion_r1278355607](url), does my change meet expectations or is some more slimming down required?
Anything else I've missed or is that it for this PR?
bashonly: > As to https://github.com/yt-dlp/yt-dlp/pull/7484#discussion_r1278355607, does my change meet expectations or is some more slimming down required?

imo it's a given that HLS bitrate info is unreliable and comment could be shortened to what was suggested

the rest LGTM

trainman261: OK. Let me know if you need anything else.

Patch:
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 87bca5bbe..666d89b46 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -34,7 +34,7 @@
 from .extractor.openload import PhantomJSwrapper
 from .minicurses import format_text
 from .networking import HEADRequest, Request, RequestDirector
-from .networking.common import _REQUEST_HANDLERS
+from .networking.common import _REQUEST_HANDLERS, _RH_PREFERENCES
 from .networking.exceptions import (
     HTTPError,
     NoSupportingHandlers,
@@ -683,7 +683,7 @@ def process_color_policy(stream):
         self.params['http_headers'] = HTTPHeaderDict(std_headers, self.params.get('http_headers'))
         self._load_cookies(self.params['http_headers'].get('Cookie'))  # compat
         self.params['http_headers'].pop('Cookie', None)
-        self._request_director = self.build_request_director(_REQUEST_HANDLERS.values())
+        self._request_director = self.build_request_director(_REQUEST_HANDLERS.values(), _RH_PREFERENCES)
 
         if auto_init and auto_init != 'no_verbose_header':
             self.print_debug_header()
@@ -4077,7 +4077,7 @@ def urlopen(self, req):
         except HTTPError as e:  # TODO: Remove in a future release
             raise _CompatHTTPError(e) from e
 
-    def build_request_director(self, handlers):
+    def build_request_director(self, handlers, preferences=None):
         logger = _YDLLogger(self)
         headers = self.params['http_headers'].copy()
         proxies = self.proxies.copy()
@@ -4106,6 +4106,7 @@ def build_request_director(self, handlers):
                     },
                 }),
             ))
+        director.preferences.update(preferences or [])
         return director
 
     def encode(self, s):
diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
index 157f5b0c2..a71fbc28b 100644
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -138,7 +138,7 @@ def _extract_firefox_cookies(profile, container, logger):
         containers_path = os.path.join(os.path.dirname(cookie_database_path), 'containers.json')
         if not os.path.isfile(containers_path) or not os.access(containers_path, os.R_OK):
             raise FileNotFoundError(f'could not read containers.json in {search_root}')
-        with open(containers_path) as containers:
+        with open(containers_path, encoding='utf8') as containers:
             identities = json.load(containers).get('identities', [])
         container_id = next((context.get('userContextId') for context in identities if container in (
             context.get('name'),
diff --git a/yt_dlp/extractor/_extractors.py b/yt_dlp/extractor/_extractors.py
index 9d935a7d1..63bb55ea7 100644
--- a/yt_dlp/extractor/_extractors.py
+++ b/yt_dlp/extractor/_extractors.py
@@ -1420,7 +1420,7 @@
     PatreonIE,
     PatreonCampaignIE
 )
-from .pbs import PBSIE
+from .pbs import PBSIE, PBSKidsIE
 from .pearvideo import PearVideoIE
 from .peekvids import PeekVidsIE, PlayVidsIE
 from .peertube import (
@@ -1709,6 +1709,7 @@
     RuvIE,
     RuvSpilaIE
 )
+from .s4c import S4CIE
 from .safari import (
     SafariIE,
     SafariApiIE,
diff --git a/yt_dlp/extractor/cbc.py b/yt_dlp/extractor/cbc.py
index 41e092422..9413281a5 100644
--- a/yt_dlp/extractor/cbc.py
+++ b/yt_dlp/extractor/cbc.py
@@ -161,7 +161,7 @@ class CBCPlayerIE(InfoExtractor):
             'upload_date': '20160210',
             'uploader': 'CBCC-NEW',
         },
-        'skip': 'Geo-restricted to Canada',
+        'skip': 'Geo-restricted to Canada and no longer available',
     }, {
         # Redirected from http://www.cbc.ca/player/AudioMobile/All%20in%20a%20Weekend%20Montreal/ID/2657632011/
         'url': 'http://www.cbc.ca/player/play/2657631896',
@@ -174,6 +174,9 @@ class CBCPlayerIE(InfoExtractor):
             'timestamp': 1425704400,
             'upload_date': '20150307',
             'uploader': 'CBCC-NEW',
+            'thumbnail': 'http://thumbnails.cbc.ca/maven_legacy/thumbnails/sonali-karnick-220.jpg',
+            'chapters': [],
+            'duration': 494.811,
         },
     }, {
         'url': 'http://www.cbc.ca/player/play/2164402062',
@@ -186,6 +189,28 @@ class CBCPlayerIE(InfoExtractor):
             'timestamp': 1320410746,
             'upload_date': '20111104',
             'uploader': 'CBCC-NEW',
+            'thumbnail': 'https://thumbnails.cbc.ca/maven_legacy/thumbnails/277/67/cancer_852x480_2164412612.jpg',
+            'chapters': [],
+            'duration': 186.867,
+        },
+    }, {
+        # Has subtitles
+        # These broadcasts expire after ~1 month, can find new test URL here:
+        # https://www.cbc.ca/player/news/TV%20Shows/The%20National/Latest%20Broadcast
+        'url': 'http://www.cbc.ca/player/play/2249992771553',
+        'md5': '2f2fb675dd4f0f8a5bb7588d1b13bacd',
+        'info_dict': {
+            'id': '2249992771553',
+            'ext': 'mp4',
+            'title': 'The National | Women’s soccer pay, Florida seawater, Swift quake',
+            'description': 'md5:adba28011a56cfa47a080ff198dad27a',
+            'timestamp': 1690596000,
+            'duration': 2716.333,
+            'subtitles': {'eng': [{'ext': 'vtt', 'protocol': 'm3u8_native'}]},
+            'thumbnail': 'https://thumbnails.cbc.ca/maven_legacy/thumbnails/481/326/thumbnail.jpeg',
+            'uploader': 'CBCC-NEW',
+            'chapters': 'count:5',
+            'upload_date': '20230729',
         },
     }]
 
@@ -199,6 +224,7 @@ def _real_extract(self, url):
                     'force_smil_url': True
                 }),
             'id': video_id,
+            '_format_sort_fields': ('res', 'proto')  # Prioritize direct http formats over HLS
         }
 
 
diff --git a/yt_dlp/extractor/facebook.py b/yt_dlp/extractor/facebook.py
index 574f8e8c9..4fd17b574 100644
--- a/yt_dlp/extractor/facebook.py
+++ b/yt_dlp/extractor/facebook.py
@@ -481,7 +481,8 @@ def extract_dash_manifest(video, formats):
             dash_manifest = video.get('dash_manifest')
             if dash_manifest:
                 formats.extend(self._parse_mpd_formats(
-                    compat_etree_fromstring(urllib.parse.unquote_plus(dash_manifest))))
+                    compat_etree_fromstring(urllib.parse.unquote_plus(dash_manifest)),
+                    mpd_url=video.get('dash_manifest_url')))
 
         def process_formats(info):
             # Downloads with browser's User-Agent are rate limited. Working around
diff --git a/yt_dlp/extractor/fox.py b/yt_dlp/extractor/fox.py
index 8fb4ada6b..e00e977bd 100644
--- a/yt_dlp/extractor/fox.py
+++ b/yt_dlp/extractor/fox.py
@@ -20,7 +20,7 @@
 
 
 class FOXIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?fox\.com/watch/(?P<id>[\da-fA-F]+)'
+    _VALID_URL = r'https?://(?:www\.)?fox(?:sports)?\.com/(?:watch|replay)/(?P<id>[\da-fA-F]+)'
     _TESTS = [{
         # clip
         'url': 'https://www.fox.com/watch/4b765a60490325103ea69888fb2bd4e8/',
@@ -50,6 +50,10 @@ class FOXIE(InfoExtractor):
         # sports event, geo-restricted
         'url': 'https://www.fox.com/watch/b057484dade738d1f373b3e46216fa2c/',
         'only_matching': True,
+    }, {
+        # fox sports replay, geo-restricted
+        'url': 'https://www.foxsports.com/replay/561f3e071347a24e5e877abc56b22e89',
+        'only_matching': True,
     }]
     _GEO_BYPASS = False
     _HOME_PAGE_URL = 'https://www.fox.com/'
diff --git a/yt_dlp/extractor/pbs.py b/yt_dlp/extractor/pbs.py
index 5bdf561db..2bb2ea9f1 100644
--- a/yt_dlp/extractor/pbs.py
+++ b/yt_dlp/extractor/pbs.py
@@ -11,6 +11,7 @@
     orderedSet,
     strip_jsonp,
     strip_or_none,
+    traverse_obj,
     unified_strdate,
     url_or_none,
     US_RATINGS,
@@ -696,3 +697,61 @@ def extract_redirect_urls(info):
             'subtitles': subtitles,
             'chapters': chapters,
         }
+
+
+class PBSKidsIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?pbskids\.org/video/[\w-]+/(?P<id>\d+)'
+    _TESTS = [
+        {
+            'url': 'https://pbskids.org/video/molly-of-denali/3030407927',
+            'md5': '1ded20a017cc6b53446238f1804ce4c7',
+            'info_dict': {
+                'id': '3030407927',
+                'title': 'Bird in the Hand/Bye-Bye Birdie',
+                'channel': 'molly-of-denali',
+                'duration': 1540,
+                'ext': 'mp4',
+                'series': 'Molly of Denali',
+                'description': 'md5:d006b2211633685d8ebc8d03b6d5611e',
+                'categories': ['Episode'],
+                'upload_date': '20190718',
+            }
+        },
+        {
+            'url': 'https://pbskids.org/video/plum-landing/2365205059',
+            'md5': '92e5d189851a64ae1d0237a965be71f5',
+            'info_dict': {
+                'id': '2365205059',
+                'title': 'Cooper\'s Favorite Place in Nature',
+                'channel': 'plum-landing',
+                'duration': 67,
+                'ext': 'mp4',
+                'series': 'Plum Landing',
+                'description': 'md5:657e5fc4356a84ead1c061eb280ff05d',
+                'categories': ['Episode'],
+                'upload_date': '20140302',
+            }
+        }
+    ]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        meta = self._search_json(r'window\._PBS_KIDS_DEEPLINK\s*=', webpage, 'video info', video_id)
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(
+            traverse_obj(meta, ('video_obj', 'URI', {url_or_none})), video_id, ext='mp4')
+
+        return {
+            'id': video_id,
+            'formats': formats,
+            'subtitles': subtitles,
+            **traverse_obj(meta, {
+                'categories': ('video_obj', 'video_type', {str}, {lambda x: [x] if x else None}),
+                'channel': ('show_slug', {str}),
+                'description': ('video_obj', 'description', {str}),
+                'duration': ('video_obj', 'duration', {int_or_none}),
+                'series': ('video_obj', 'program_title', {str}),
+                'title': ('video_obj', 'title', {str}),
+                'upload_date': ('video_obj', 'air_date', {unified_strdate}),
+            })
+        }
diff --git a/yt_dlp/extractor/picarto.py b/yt_dlp/extractor/picarto.py
index 36a062def..d415ba28e 100644
--- a/yt_dlp/extractor/picarto.py
+++ b/yt_dlp/extractor/picarto.py
@@ -1,7 +1,10 @@
+import urllib.parse
+
 from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
-    js_to_json,
+    str_or_none,
+    traverse_obj,
 )
 
 
@@ -84,7 +87,7 @@ def _real_extract(self, url):
 
 
 class PicartoVodIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www.)?picarto\.tv/videopopout/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?picarto\.tv/(?:videopopout|\w+/videos)/(?P<id>[^/?#&]+)'
     _TESTS = [{
         'url': 'https://picarto.tv/videopopout/ArtofZod_2017.12.12.00.13.23.flv',
         'md5': '3ab45ba4352c52ee841a28fb73f2d9ca',
@@ -94,6 +97,18 @@ class PicartoVodIE(InfoExtractor):
             'title': 'ArtofZod_2017.12.12.00.13.23.flv',
             'thumbnail': r're:^https?://.*\.jpg'
         },
+        'skip': 'The VOD does not exist',
+    }, {
+        'url': 'https://picarto.tv/ArtofZod/videos/772650',
+        'md5': '00067a0889f1f6869cc512e3e79c521b',
+        'info_dict': {
+            'id': '772650',
+            'ext': 'mp4',
+            'title': 'Art of Zod - Drawing and Painting',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'channel': 'ArtofZod',
+            'age_limit': 18,
+        }
     }, {
         'url': 'https://picarto.tv/videopopout/Plague',
         'only_matching': True,
@@ -102,21 +117,36 @@ class PicartoVodIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, video_id)
-
-        vod_info = self._parse_json(
-            self._search_regex(
-                r'(?s)#vod-player["\']\s*,\s*(\{.+?\})\s*\)', webpage,
-                'vod player'),
-            video_id, transform_source=js_to_json)
+        data = self._download_json(
+            'https://ptvintern.picarto.tv/ptvapi', video_id, query={
+                'query': f'''{{
+  video(id: "{video_id}") {{
+    id
+    title
+    adult
+    file_name
+    video_recording_image_url
+    channel {{
+      name
+    }}
+  }}
+}}'''
+            })['data']['video']
+
+        file_name = data['file_name']
+        netloc = urllib.parse.urlparse(data['video_recording_image_url']).netloc
 
         formats = self._extract_m3u8_formats(
-            vod_info['vod'], video_id, 'mp4', entry_protocol='m3u8_native',
-            m3u8_id='hls')
+            f'https://{netloc}/stream/hls/{file_name}/index.m3u8', video_id, 'mp4', m3u8_id='hls')
 
         return {
             'id': video_id,
-            'title': video_id,
-            'thumbnail': vod_info.get('vodThumb'),
+            **traverse_obj(data, {
+                'id': ('id', {str_or_none}),
+                'title': ('title', {str}),
+                'thumbnail': 'video_recording_image_url',
+                'channel': ('channel', 'name', {str}),
+                'age_limit': ('adult', {lambda x: 18 if x else 0}),
+            }),
             'formats': formats,
         }
diff --git a/yt_dlp/extractor/s4c.py b/yt_dlp/extractor/s4c.py
new file mode 100644
index 000000000..38a905896
--- /dev/null
+++ b/yt_dlp/extractor/s4c.py
@@ -0,0 +1,62 @@
+from .common import InfoExtractor
+from ..utils import traverse_obj
+
+
+class S4CIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?s4c\.cymru/clic/programme/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://www.s4c.cymru/clic/programme/861362209',
+        'info_dict': {
+            'id': '861362209',
+            'ext': 'mp4',
+            'title': 'Y Swn',
+            'description': 'md5:f7681a30e4955b250b3224aa9fe70cf0',
+            'duration': 5340
+        },
+    }, {
+        'url': 'https://www.s4c.cymru/clic/programme/856636948',
+        'info_dict': {
+            'id': '856636948',
+            'ext': 'mp4',
+            'title': 'Am Dro',
+            'duration': 2880,
+            'description': 'md5:100d8686fc9a632a0cb2db52a3433ffe',
+        },
+    }]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        details = self._download_json(
+            f'https://www.s4c.cymru/df/full_prog_details?lang=e&programme_id={video_id}',
+            video_id, fatal=False)
+
+        filename = self._download_json(
+            'https://player-api.s4c-cdn.co.uk/player-configuration/prod', video_id, query={
+                'programme_id': video_id,
+                'signed': '0',
+                'lang': 'en',
+                'mode': 'od',
+                'appId': 'clic',
+                'streamName': '',
+            }, note='Downloading player config JSON')['filename']
+        m3u8_url = self._download_json(
+            'https://player-api.s4c-cdn.co.uk/streaming-urls/prod', video_id, query={
+                'mode': 'od',
+                'application': 'clic',
+                'region': 'WW',
+                'extra': 'false',
+                'thirdParty': 'false',
+                'filename': filename,
+            }, note='Downloading streaming urls JSON')['hls']
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(m3u8_url, video_id, 'mp4', m3u8_id='hls')
+
+        return {
+            'id': video_id,
+            'formats': formats,
+            'subtitles': subtitles,
+            **traverse_obj(details, ('full_prog_details', 0, {
+                'title': (('programme_title', 'series_title'), {str}),
+                'description': ('full_billing', {str.strip}),
+                'duration': ('duration', {lambda x: int(x) * 60}),
+            }), get_all=False),
+        }
diff --git a/yt_dlp/extractor/scrippsnetworks.py b/yt_dlp/extractor/scrippsnetworks.py
index c3cee6e4a..adfd7e5f2 100644
--- a/yt_dlp/extractor/scrippsnetworks.py
+++ b/yt_dlp/extractor/scrippsnetworks.py
@@ -115,6 +115,7 @@ class ScrippsNetworksIE(InfoExtractor):
             'uploader': 'SCNI-SCND',
         },
         'add_ie': ['ThePlatform'],
+        'expected_warnings': ['No HLS formats found'],
     }, {
         'url': 'https://www.diynetwork.com/videos/diy-barnwood-tablet-stand-0265790',
         'only_matching': True,
diff --git a/yt_dlp/extractor/theplatform.py b/yt_dlp/extractor/theplatform.py
index 8307b912d..99caeb5f9 100644
--- a/yt_dlp/extractor/theplatform.py
+++ b/yt_dlp/extractor/theplatform.py
@@ -19,7 +19,11 @@
     xpath_with_ns,
     mimetype2ext,
     find_xpath_attr,
+    traverse_obj,
+    update_url,
+    urlhandle_detect_ext,
 )
+from ..networking import HEADRequest
 
 default_ns = 'http://www.w3.org/2005/SMIL21/Language'
 _x = lambda p: xpath_with_ns(p, {'smil': default_ns})
@@ -162,7 +166,8 @@ class ThePlatformIE(ThePlatformBaseIE, AdobePassIE):
         'params': {
             # rtmp download
             'skip_download': True,
-        }
+        },
+        'skip': '404 Not Found',
     }, {
         'url': 'https://player.theplatform.com/p/D6x-PC/pulse_preview/embed/select/media/yMBg9E8KFxZD',
         'info_dict': {
@@ -171,7 +176,8 @@ class ThePlatformIE(ThePlatformBaseIE, AdobePassIE):
             'description': 'md5:644ad9188d655b742f942bf2e06b002d',
             'title': 'HIGHLIGHTS: USA bag first ever series Cup win',
             'uploader': 'EGSM',
-        }
+        },
+        'skip': '404 Not Found',
     }, {
         'url': 'http://player.theplatform.com/p/NnzsPC/widget/select/media/4Y0TlYUr_ZT7',
         'only_matching': True,
@@ -189,6 +195,7 @@ class ThePlatformIE(ThePlatformBaseIE, AdobePassIE):
             'upload_date': '20150701',
             'uploader': 'NBCU-NEWS',
         },
+        'skip': '404 Not Found',
     }, {
         # From http://www.nbc.com/the-blacklist/video/sir-crispin-crandall/2928790?onid=137781#vc137781=1
         # geo-restricted (US), HLS encrypted with AES-128
@@ -295,6 +302,17 @@ def _real_extract(self, url):
 
         formats, subtitles = self._extract_theplatform_smil(smil_url, video_id)
 
+        # With some sites, manifest URL must be forced to extract HLS formats
+        if not traverse_obj(formats, lambda _, v: v['format_id'].startswith('hls')):
+            m3u8_url = update_url(url, query='mbr=true&manifest=m3u', fragment=None)
+            urlh = self._request_webpage(
+                HEADRequest(m3u8_url), video_id, 'Checking for HLS formats', 'No HLS formats found', fatal=False)
+            if urlh and urlhandle_detect_ext(urlh) == 'm3u8':
+                m3u8_fmts, m3u8_subs = self._extract_m3u8_formats_and_subtitles(
+                    m3u8_url, video_id, m3u8_id='hls', fatal=False)
+                formats.extend(m3u8_fmts)
+                self._merge_subtitles(m3u8_subs, target=subtitles)
+
         ret = self._extract_theplatform_metadata(path, video_id)
         combined_subtitles = self._merge_subtitles(ret.get('subtitles', {}), subtitles)
         ret.update({
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
index 1e16631b1..023d8fd8c 100644
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -496,16 +496,10 @@ def _initialize_consent(self):
         cookies = self._get_cookies('https://www.youtube.com/')
         if cookies.get('__Secure-3PSID'):
             return
-        consent_id = None
-        consent = cookies.get('CONSENT')
-        if consent:
-            if 'YES' in consent.value:
-                return
-            consent_id = self._search_regex(
-                r'PENDING\+(\d+)', consent.value, 'consent', default=None)
-        if not consent_id:
-            consent_id = random.randint(100, 999)
-        self._set_cookie('.youtube.com', 'CONSENT', 'YES+cb.20210328-17-p0.en+FX+%s' % consent_id)
+        socs = cookies.get('SOCS')
+        if socs and not socs.value.startswith('CAA'):  # not consented
+            return
+        self._set_cookie('.youtube.com', 'SOCS', 'CAI', secure=True)  # accept all (required for mixes)
 
     def _initialize_pref(self):
         cookies = self._get_cookies('https://www.youtube.com/')
diff --git a/yt_dlp/networking/common.py b/yt_dlp/networking/common.py
index 8fba8c1c5..584c7bb4d 100644
--- a/yt_dlp/networking/common.py
+++ b/yt_dlp/networking/common.py
@@ -31,8 +31,19 @@
 )
 from ..utils.networking import HTTPHeaderDict, normalize_url
 
-if typing.TYPE_CHECKING:
-    RequestData = bytes | Iterable[bytes] | typing.IO | None
+
+def register_preference(*handlers: type[RequestHandler]):
+    assert all(issubclass(handler, RequestHandler) for handler in handlers)
+
+    def outer(preference: Preference):
+        @functools.wraps(preference)
+        def inner(handler, *args, **kwargs):
+            if not handlers or isinstance(handler, handlers):
+                return preference(handler, *args, **kwargs)
+            return 0
+        _RH_PREFERENCES.add(inner)
+        return inner
+    return outer
 
 
 class RequestDirector:
@@ -40,12 +51,17 @@ class RequestDirector:
 
     Helper class that, when given a request, forward it to a RequestHandler that supports it.
 
+    Preference functions in the form of func(handler, request) -> int
+    can be registered into the `preferences` set. These are used to sort handlers
+    in order of preference.
+
     @param logger: Logger instance.
     @param verbose: Print debug request information to stdout.
     """
 
     def __init__(self, logger, verbose=False):
         self.handlers: dict[str, RequestHandler] = {}
+        self.preferences: set[Preference] = set()
         self.logger = logger  # TODO(Grub4k): default logger
         self.verbose = verbose
 
@@ -58,6 +74,16 @@ def add_handler(self, handler: RequestHandler):
         assert isinstance(handler, RequestHandler), 'handler must be a RequestHandler'
         self.handlers[handler.RH_KEY] = handler
 
+    def _get_handlers(self, request: Request) -> list[RequestHandler]:
+        """Sorts handlers by preference, given a request"""
+        preferences = {
+            rh: sum(pref(rh, request) for pref in self.preferences)
+            for rh in self.handlers.values()
+        }
+        self._print_verbose('Handler preferences for this request: %s' % ', '.join(
+            f'{rh.RH_NAME}={pref}' for rh, pref in preferences.items()))
+        return sorted(self.handlers.values(), key=preferences.get, reverse=True)
+
     def _print_verbose(self, msg):
         if self.verbose:
             self.logger.stdout(f'director: {msg}')
@@ -73,8 +99,7 @@ def send(self, request: Request) -> Response:
 
         unexpected_errors = []
         unsupported_errors = []
-        # TODO (future): add a per-request preference system
-        for handler in reversed(list(self.handlers.values())):
+        for handler in self._get_handlers(request):
             self._print_verbose(f'Checking if "{handler.RH_NAME}" supports this request.')
             try:
                 handler.validate(request)
@@ -530,3 +555,10 @@ def info(self):
     def getheader(self, name, default=None):
         deprecation_warning('Response.getheader() is deprecated, use Response.get_header', stacklevel=2)
         return self.get_header(name, default)
+
+
+if typing.TYPE_CHECKING:
+    RequestData = bytes | Iterable[bytes] | typing.IO | None
+    Preference = typing.Callable[[RequestHandler, Request], int]
+
+_RH_PREFERENCES: set[Preference] = set()


Test patch:
diff --git a/test/test_networking.py b/test/test_networking.py
index 9c33b0d4c..2622d24da 100644
--- a/test/test_networking.py
+++ b/test/test_networking.py
@@ -1035,17 +1035,17 @@ def test_send(self):
         assert isinstance(director.send(Request('http://')), FakeResponse)
 
     def test_unsupported_handlers(self):
-        director = RequestDirector(logger=FakeLogger())
-        director.add_handler(FakeRH(logger=FakeLogger()))
-
         class SupportedRH(RequestHandler):
             _SUPPORTED_URL_SCHEMES = ['http']
 
             def _send(self, request: Request):
                 return Response(fp=io.BytesIO(b'supported'), headers={}, url=request.url)
 
-        # This handler should by default take preference over FakeRH
+        director = RequestDirector(logger=FakeLogger())
         director.add_handler(SupportedRH(logger=FakeLogger()))
+        director.add_handler(FakeRH(logger=FakeLogger()))
+
+        # First should take preference
         assert director.send(Request('http://')).read() == b'supported'
         assert director.send(Request('any://')).read() == b''
 
@@ -1072,6 +1072,27 @@ def _send(self, request: Request):
         director.add_handler(UnexpectedRH(logger=FakeLogger))
         assert director.send(Request('any://'))
 
+    def test_preference(self):
+        director = RequestDirector(logger=FakeLogger())
+        director.add_handler(FakeRH(logger=FakeLogger()))
+
+        class SomeRH(RequestHandler):
+            _SUPPORTED_URL_SCHEMES = ['http']
+
+            def _send(self, request: Request):
+                return Response(fp=io.BytesIO(b'supported'), headers={}, url=request.url)
+
+        def some_preference(rh, request):
+            return (0 if not isinstance(rh, SomeRH)
+                    else 100 if 'prefer' in request.headers
+                    else -1)
+
+        director.add_handler(SomeRH(logger=FakeLogger()))
+        director.preferences.add(some_preference)
+
+        assert director.send(Request('http://')).read() == b''
+        assert director.send(Request('http://', headers={'prefer': '1'})).read() == b'supported'
+
 
 # XXX: do we want to move this to test_YoutubeDL.py?
 class TestYoutubeDLNetworking: