Repository: yt-dlp/yt-dlp

Issue:
Roosterteeth.com ad filter
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm requesting a site-specific feature
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

_No response_

### Example URLs

https://roosterteeth.com/watch/death-battle-season-10-antman

### Provide a description that is worded well enough to be understood

Some Roosterteeth videos, for example Death Battle season 10, embed ads both at the beginning and in the middle, which I'd like to remove.

Analysis of the HLS M3U files show that the ads are groups of segments, separated from the main video (and each other) by `#EXT-X-DISCONTINUITY` tags.

The existing option `--hls-split-discontinuity` doesn't work well here. While the ads are removed, so is the first half of the video.

I've added a filter of sorts in `downloader/hls.py`, enabled by a new option `--roosterteeth-ad-filter`, that removes these ads on the assumption that each is shorter than 10% of the whole video. My fork is here: https://github.com/Masken3/yt-dlp/tree/roosterteeth_ad_filter

The output below shows a successful run with this option.

I've opened this issue because I'm new to yt-dlp development and would like the maintainers' input on the matter before I open a pull request.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [x] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
> python -m yt_dlp --no-check-certificate -vU --roosterteeth-ad-filter https://roosterteeth.com/watch/death-battle-season-10-antman
[debug] Command-line config: ['--no-check-certificate', '-vU', '--roosterteeth-ad-filter', 'https://roosterteeth.com/watch/death-battle-season-10-antman']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: da85e4757
[debug] Python 3.9.6 (CPython AMD64 64bit) - Windows-10-10.0.17763-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2022-02-14-git-59c647bcf3-essentials_build-www.gyan.dev (setts), ffprobe 2022-02-14-git-59c647bcf3-essentials_build-www.gyan.dev
[debug] Optional libraries: certifi-2021.05.30, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
yt-dlp is up to date (stable@2023.07.06)
[RoosterTeeth] Extracting URL: https://roosterteeth.com/watch/death-battle-season-10-antman
[RoosterTeeth] death-battle-season-10-antman: Downloading video JSON metadata
[RoosterTeeth] death-battle-season-10-antman: Downloading m3u8 information
[RoosterTeeth] death-battle-season-10-antman: Downloading episode JSON metadata
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 50934: Downloading 1 format(s): hls-4606+hls-audio-3-en__Main_
[debug] Invoking hlsnative downloader on "https://ssaimanifest.prod.boltdns.net/us-east-1/playback/once/v1/hls/v4/clear/6203312018001/ddc307e2-0354-47b3-861c-073c2c03cae5/dc628e2a-f1ab-49ff-8f65-700740a06609/28be398c-fdbe-40c4-904a-2e6f0e063ff5/default_video4000_5_1920x1080/media.m3u8?bc_token=NjRiODVmZDVfMTUyNTgxN2RjMTY0Y2UzMjJkNzZmZTA5ZTRmODgzMTgxYTE2OWYwOGI2OTU3YjE1MzQ1YjExOTk0NzJjZjU3YQ%3D%3D"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 123
[download] Destination: Ant-Man VS Atom (Marvel VS DC) [50934].fhls-4606.mp4
[debug] File locking is not supported. Proceeding without locking
[download] 100% of  554.69MiB in 00:03:05 at 2.99MiB/s
[debug] Invoking hlsnative downloader on "https://ssaimanifest.prod.boltdns.net/us-east-1/playback/once/v1/hls/v4/clear/6203312018001/ddc307e2-0354-47b3-861c-073c2c03cae5/dc628e2a-f1ab-49ff-8f65-700740a06609/28be398c-fdbe-40c4-904a-2e6f0e063ff5/4b45bfd7-2729-4aea-b477-f60642c9fd73/media.m3u8?bc_token=NjRiODVmZDVfMTBiNTUzNGNmMTcwNGM3NDM3NDhiOWQzNWJlYTFkYzgyMThkMDI3OGIyMmUxNmQ3YjdkM2IzMDc4NTY0YTJjOA%3D%3D"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 124
[download] Destination: Ant-Man VS Atom (Marvel VS DC) [50934].fhls-audio-3-en__Main_.mp4
[download] 100% of   31.73MiB in 00:02:14 at 242.04KiB/s
[debug] ffmpeg command line: ffprobe -show_streams "file:Ant-Man VS Atom (Marvel VS DC) [50934].fhls-audio-3-en__Main_.mp4"
[Merger] Merging formats into "Ant-Man VS Atom (Marvel VS DC) [50934].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Ant-Man VS Atom (Marvel VS DC) [50934].fhls-4606.mp4" -i "file:Ant-Man VS Atom (Marvel VS DC) [50934].fhls-audio-3-en__Main_.mp4" -c copy -map "0:v:0" -map "1:a:0" "-bsf:a:0" aac_adtstoasc -movflags "+faststart" "file:Ant-Man VS Atom (Marvel VS DC) [50934].temp.mp4"
Deleting original file Ant-Man VS Atom (Marvel VS DC) [50934].fhls-audio-3-en__Main_.mp4 (pass -k to keep)
Deleting original file Ant-Man VS Atom (Marvel VS DC) [50934].fhls-4606.mp4 (pass -k to keep)
```



Discussion:
This design has a few issues which we need to address:
1. We will not add a new switch for it. The functionality should be enabled by the extractor.
2. `length < fragments/10` is a very hacky filter. We need better logic
3. The logic must be in the extractor, not the downloader 

Related #6654 

---

> The existing option `--hls-split-discontinuity` doesn't work well here. While the ads are removed, so is the first half of the video.

It splits the sections into separate formats. Check -F, fetch the ones you want, and concat them with ffmpeg.
no ads here, use the firefox  add-on called the stream detector (set it to ffmpeg)
grab the master.m3u8 file & download.

Resolving patch:
diff --git a/yt_dlp/extractor/roosterteeth.py b/yt_dlp/extractor/roosterteeth.py
--- a/yt_dlp/extractor/roosterteeth.py
+++ b/yt_dlp/extractor/roosterteeth.py
@@ -91,6 +91,15 @@ class RoosterTeethIE(RoosterTeethBaseIE):
             'thumbnail': r're:^https?://.*\.png$',
             'series': 'Million Dollars, But...',
             'episode': 'Million Dollars, But... The Game Announcement',
+            'tags': ['Game Show', 'Sketch'],
+            'season_number': 2,
+            'availability': 'public',
+            'episode_number': 10,
+            'episode_id': '00374575-464e-11e7-a302-065410f210c4',
+            'season': 'Season 2',
+            'season_id': 'ffa27d48-464d-11e7-a302-065410f210c4',
+            'channel_id': '92b6bb21-91d2-4b1b-bf95-3268fa0d9939',
+            'duration': 145,
         },
         'params': {'skip_download': True},
     }, {
@@ -104,6 +113,15 @@ class RoosterTeethIE(RoosterTeethBaseIE):
             'channel_id': '92f780eb-ebfe-4bf5-a3b5-c6ad5460a5f1',
             'thumbnail': r're:^https?://.*\.(png|jpe?g)$',
             'ext': 'mp4',
+            'availability': 'public',
+            'episode_id': 'f8117b13-f068-499e-803e-eec9ea2dec8c',
+            'episode_number': 3,
+            'tags': ['Animation'],
+            'season_id': '4b8f0a9e-12c4-41ed-8caa-fed15a85bab8',
+            'season': 'Season 1',
+            'series': 'RWBY: World of Remnant',
+            'season_number': 1,
+            'duration': 216,
         },
         'params': {'skip_download': True},
     }, {
@@ -133,10 +151,10 @@ def _real_extract(self, url):
 
         try:
             video_data = self._download_json(
-                api_episode_url + '/videos', display_id,
-                'Downloading video JSON metadata')['data'][0]
+                api_episode_url + '/videos', display_id, 'Downloading video JSON metadata',
+                headers={'Client-Type': 'web'})['data'][0]  # web client-type yields ad-free streams
             m3u8_url = video_data['attributes']['url']
-            # XXX: additional URL at video_data['links']['download']
+            # XXX: additional ad-free URL at video_data['links']['download'] but often gives 403 errors
         except ExtractorError as e:
             if isinstance(e.cause, HTTPError) and e.cause.status == 403:
                 if self._parse_json(e.cause.response.read().decode(), display_id).get('access') is False:


Test patch: