Repository: yt-dlp/yt-dlp

Issue:
MagellanTV extractor plugin
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm requesting a site-specific feature
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

United States

### Example URLs

Video info page: https://www.magellantv.com/video/my-dads-on-death-row
Video player: https://www.magellantv.com/watch/my-dads-on-death-row?type=v


### Provide a description that is worded well enough to be understood

Hi there. I'm trying to get this extractor plugin working.
Tthe extractor extracts the video ID from the URL, retrieves the webpage content, parses the JSON data embedded in the webpage, extracts the video URL endpoint, and then makes an additional request to the video URL endpoint to obtain the final video URL. That's my approach, anyway.
I'm almost there, but can't wrap my head around what is missing.
And I'm tired. Please help out if you can. Thanks.
Here's what I got so far:

```
import json
import requests
from urllib.parse import unquote
from yt_dlp.extractor.common import InfoExtractor
from yt_dlp.utils import ExtractorError


class MagellanTVIE(InfoExtractor):
    _VALID_URL = r'https?://www\.magellantv\.com/watch/(?P<id>[^/?]+)'
    _TESTS = [
        {
            'url': 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v',
            'info_dict': {
                'id': 'my-dads-on-death-row',
                'title': "My Dad's On Death Row",
            },
            'params': {
                'skip_download': True,
            },
        },
    ]

    def _real_extract(self, url):
        video_id = self._match_id(url)
        video_id = unquote(video_id)
        if not video_id:
            raise ExtractorError('Could not extract video ID')

        webpage = self._download_webpage(url, video_id)

        # Extract the video URL endpoint from the webpage JSON
        video_info = self._parse_json(
            self._search_regex(
                r'var mainData = ({.+?});',
                webpage,
                'video info',
                default='{}'
            ),
            video_id
        )
        video_url_endpoint = video_info.get('jwpVideoUrl')

        if not video_url_endpoint:
            raise ExtractorError('Could not extract video URL endpoint')

        # Make an additional request to the video URL endpoint
        response = requests.get(video_url_endpoint)
        response_json = response.json()
        video_url = response_json.get('video_url')

        if not video_url:
            raise ExtractorError('Could not extract video URL')

        return {
            'id': video_id,
            'title': video_id,
            'url': video_url,
        }


IE_NAME = 'magellantv'
IE_DESC = 'MagellanTV'
_IE_DESC = 'MagellanTV video extraction support'
IE_CLASS = MagellanTVIE

```

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v', '--verbos']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.06.22 [812cdfa06] (pip)
[debug] Python 3.11.3 (CPython x86_64 64bit) - Linux-5.15.114-2-MANJARO-x86_64-with-glibc2.37 (OpenSSL 3.0.8 7 Feb 2023, glibc 2.37)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Extractor Plugins: AppleDeveloperIE, AppleWwdcSessionsIE, AppleWwdcTopicsIE, BunkrAlbumIE, BunkrIE, CamHubIE, CoverApiIE, HanimeIE, JutSuAnimeIE, JutSuEpisodeIE, MagellanTVIE, MagellanTVIE as IE_CLASS, MegaCartoonsIE, MuchoHentaiIE, OhentaiIE, PStreamIE, StreamtapeIE, VideoCdnIE, VoeIE, agegatebypass (YoutubeIE)
[debug] Plugin directories: ['/home/chris/.yt-dlp/plugins/appledeveloper/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/bunkr/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/pstream/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/streamtape/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/voe/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/agegatebypass/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/hanime/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/videocdn/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/jutsu/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/coverapi/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/magellantv/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/ohentai/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/muchohentai/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/camhub/yt_dlp_plugins', '/home/chris/.yt-dlp/plugins/megacartoon/yt_dlp_plugins']
[debug] Loaded 1869 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.06.22
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/latest/download/_update_spec
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[MagellanTV] Extracting URL: https://www.magellantv.com/watch/my-dads-on-death-row?type=v
[MagellanTV] my-dads-on-death-row: Downloading webpage
ERROR: [MagellanTV] my-dads-on-death-row: Could not extract video URL endpoint; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 708, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chris/.yt-dlp/plugins/magellantv/yt_dlp_plugins/extractor/magellantv.py", line 44, in _real_extract
    raise ExtractorError('Could not extract video URL endpoint')
```



Discussion:
```diff
diff --git a/yt_dlp_plugins/extractor/magellan.py b/yt_dlp_plugins/extractor/magellan.py
index ac79967..4ff3727 100644
--- a/yt_dlp_plugins/extractor/magellan.py
+++ b/yt_dlp_plugins/extractor/magellan.py
@@ -1,18 +1,16 @@
-import json
-import requests
-from urllib.parse import unquote
 from yt_dlp.extractor.common import InfoExtractor
-from yt_dlp.utils import ExtractorError
+from yt_dlp.utils import traverse_obj, url_or_none
 
 
 class MagellanTVIE(InfoExtractor):
-    _VALID_URL = r'https?://www\.magellantv\.com/watch/(?P<id>[^/?]+)'
+    _VALID_URL = r'https?://www\.magellantv\.com/(?:watch|video)/(?P<id>[^/#?]+)'
     _TESTS = [
         {
             'url': 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v',
             'info_dict': {
                 'id': 'my-dads-on-death-row',
-                'title': "My Dad's On Death Row",
+                'ext': 'mp4',
+                'title': 'My Dad\'s On Death Row',
             },
             'params': {
                 'skip_download': True,
@@ -22,43 +20,14 @@ class MagellanTVIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_id = unquote(video_id)
-        if not video_id:
-            raise ExtractorError('Could not extract video ID')
-
         webpage = self._download_webpage(url, video_id)
-
-        # Extract the video URL endpoint from the webpage JSON
-        video_info = self._parse_json(
-            self._search_regex(
-                r'var mainData = ({.+?});',
-                webpage,
-                'video info',
-                default='{}'
-            ),
-            video_id
-        )
-        video_url_endpoint = video_info.get('jwpVideoUrl')
-
-        if not video_url_endpoint:
-            raise ExtractorError('Could not extract video URL endpoint')
-
-        # Make an additional request to the video URL endpoint
-        response = requests.get(video_url_endpoint)
-        response_json = response.json()
-        video_url = response_json.get('video_url')
-
-        if not video_url:
-            raise ExtractorError('Could not extract video URL')
+        data = self._search_nextjs_data(webpage, video_id)['props']['pageProps']['reactContext']  # there's other metadata in here
+        m3u8_url = traverse_obj(data, ('video', 'detail', 'jwpVideoUrl', {url_or_none}))  # this is for m3u8 playlist, not JSON
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(m3u8_url, video_id)
 
         return {
             'id': video_id,
-            'title': video_id,
-            'url': video_url,
+            'title': traverse_obj(data, ('video', 'detail', 'title', {str})),
+            'formats': formats,
+            'subtitles': subtitles,
         }
-
-
-IE_NAME = 'magellantv'
-IE_DESC = 'MagellanTV'
-_IE_DESC = 'MagellanTV video extraction support'
-IE_CLASS = MagellanTVIE
```
Thanks for your help.
I'm still getting Error 404 though.
But because I'm personally only using this for streaming, I decided to go with a more simplistic approach.
I wasn't able to figure out how I can pipe this through ffmpeg because the .ts entries within the m3u8 playlist are not the complete URL. Maybe somebody smarter than me can figure it out.
But again. Personally I don't need it to download, so I'm fine with this as I'm only using it for streaming.

```
from yt_dlp.extractor.common import InfoExtractor
from yt_dlp.utils import ExtractorError


class MagellanTVIE(InfoExtractor):
    _VALID_URL = r'https?://www\.magellantv\.com/(?:watch|video)/(?P<id>[^/#?]+)'
    _NETRC_MACHINE = 'magellantv'

    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)

        # Extract the video URL from the webpage using appropriate methods
        # Update the following code according to the website structure and video URL extraction logic

        # Example code:
        if '/video/' in url:
            # Extract video URL from video episode info page
            video_url = 'https://videos-cloudfront-usp.jwpsrv.com/64a91cb2_27dd1921f616b08198b2ed8beda8ca54c8186bcb/site/4pSWjNlc/media/FqOrhRt5/version/PPprIEmm/manifest.ism/playlist.m3u8'
        else:
            # Extract video URL from video player page
            video_url = 'https://videos-cloudfront-usp.jwpsrv.com/64a91cb2_27dd1921f616b08198b2ed8beda8ca54c8186bcb/site/4pSWjNlc/media/FqOrhRt5/version/PPprIEmm/manifest.ism/playlist.m3u8'

        if not video_url:
            raise ExtractorError('No video URL found')

        return {
            'id': video_id,
            'url': video_url,
            'title': video_id,
        }


# Register the extractor
def setup(bot):
    bot.add_extractor(MagellanTVIE)


```
just use exactly what I wrote & it will work with ffmpeg and mpv

```
$ yt-dlp 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v'
[MagellanTV] Extracting URL: https://www.magellantv.com/watch/my-dads-on-death-row?type=v
[MagellanTV] my-dads-on-death-row: Downloading webpage
[MagellanTV] my-dads-on-death-row: Downloading m3u8 information
[info] my-dads-on-death-row: Downloading 1 format(s): 14686+audio-Default
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 952
[download] Destination: NA_NA_my-dads-on-death-row_My_Dad_s_On_Death_Row.f14686.mp4
[download]   0.3% of ~   4.10GiB at   20.22MiB/s ETA 03:16 (frag 3/952)
```

```py
from yt_dlp.extractor.common import InfoExtractor
from yt_dlp.utils import traverse_obj, url_or_none


class MagellanTVIE(InfoExtractor):
    _VALID_URL = r'https?://www\.magellantv\.com/(?:watch|video)/(?P<id>[^/#?]+)'
    _TESTS = [
        {
            'url': 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v',
            'info_dict': {
                'id': 'my-dads-on-death-row',
                'ext': 'mp4',
                'title': 'My Dad\'s On Death Row',
            },
            'params': {
                'skip_download': True,
            },
        },
    ]

    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        data = self._search_nextjs_data(webpage, video_id)['props']['pageProps']['reactContext']
        m3u8_url = traverse_obj(data, ('video', 'detail', 'jwpVideoUrl', {url_or_none}))
        formats, subtitles = self._extract_m3u8_formats_and_subtitles(m3u8_url, video_id)

        return {
            'id': video_id,
            'title': traverse_obj(data, ('video', 'detail', 'title', {str})),
            'formats': formats,
            'subtitles': subtitles,
        }
```

```
$ mpv 'ytdl://https://www.magellantv.com/watch/my-dads-on-death-row?type=v'
 (+) Video --vid=1 (*) (h264 1920x1080 25.000fps)
 (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
VO: [gpu] 1920x1080 yuv420p
AV: 00:00:20 / 01:03:30 (1%) A-V:  0.000 Cache: 78s/82MB
```
@bashonly I appreciate it. But with your approach, that's the only video that'll download.
Whereas my approach of using the host as reference point works with every video.
And also. Make sure the script uses both magellantv.com/watch and /video like the way I set it up, because one is the episode front page and the other is the video player.
Actually I just realized that the video I added as an example link is the one being downloaded each time.
I thought that was just a reference link.
So that sucks.
If you want more help, share more example URLs

Why are you adding `_NETRC_MACHINE` but no `_perform_login` method? Do some videos require auth? Maybe that's why you're getting 404s?
Okay here are some example links with the video links included:

Video info:https://www.magellantv.com/video/james-bulger-the-new-revelations
Video player:https://www.magellantv.com/watch/james-bulger-the-new-revelations?type=v
M3U Link: https://d8dplxrhycocu.cloudfront.net/BC8G46Sy/stream.m3u8

Video info: https://www.magellantv.com/video/my-dads-on-death-row
Video player: https://www.magellantv.com/watch/my-dads-on-death-row?type=v
M3U8 Link: https://content.jwplatform.com/manifests/A2HpBxXZ.m3u8

And I also just realized that the m3u8 playlist is available in clear text in the source code.
Is there a way to grab the video based on this knowledge?
The video hosts are different, but "jwpVideoUrl" is used to identify the video urls, like this: ,"jwpVideoUrl":"https://d8dplxrhycocu.cloudfront.net/BC8G46Sy/stream.m3u8"}.

> Make sure the script uses both magellantv.com/watch and /video like the way I set it up

Just edited the code in this comment https://github.com/yt-dlp/yt-dlp/issues/7529#issuecomment-1626170038 and now it works with https://www.magellantv.com/video/james-bulger-the-new-revelations too
@bashonly Man, that is fantastic. They all seem to work now.
Nicely done. Thank you so much.
I've done some more testing.
All the documentaries are working fine. No issues.
So feel free to add it to yt-dlp. That'd be great.

Resolving patch:
diff --git a/yt_dlp/extractor/_extractors.py b/yt_dlp/extractor/_extractors.py
--- a/yt_dlp/extractor/_extractors.py
+++ b/yt_dlp/extractor/_extractors.py
@@ -1012,6 +1012,7 @@
     LyndaCourseIE
 )
 from .m6 import M6IE
+from .magellantv import MagellanTVIE
 from .magentamusik360 import MagentaMusik360IE
 from .mailru import (
     MailRuIE,
diff --git a/yt_dlp/extractor/magellantv.py b/yt_dlp/extractor/magellantv.py
new file mode 100644
--- /dev/null
+++ b/yt_dlp/extractor/magellantv.py
@@ -0,0 +1,50 @@
+from .common import InfoExtractor
+from ..utils import parse_age_limit, parse_duration, traverse_obj
+
+
+class MagellanTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?magellantv\.com/(?:watch|video)/(?P<id>[\w-]+)'
+    _TESTS = [{
+        'url': 'https://www.magellantv.com/watch/my-dads-on-death-row?type=v',
+        'info_dict': {
+            'id': 'my-dads-on-death-row',
+            'ext': 'mp4',
+            'title': 'My Dad\'s On Death Row',
+            'description': 'md5:33ba23b9f0651fc4537ed19b1d5b0d7a',
+            'duration': 3780.0,
+            'age_limit': 14,
+            'tags': ['Justice', 'Reality', 'United States', 'True Crime'],
+        },
+        'params': {'skip_download': 'm3u8'},
+    }, {
+        'url': 'https://www.magellantv.com/video/james-bulger-the-new-revelations',
+        'info_dict': {
+            'id': 'james-bulger-the-new-revelations',
+            'ext': 'mp4',
+            'title': 'James Bulger: The New Revelations',
+            'description': 'md5:7b97922038bad1d0fe8d0470d8a189f2',
+            'duration': 2640.0,
+            'age_limit': 0,
+            'tags': ['Investigation', 'True Crime', 'Justice', 'Europe'],
+        },
+        'params': {'skip_download': 'm3u8'},
+    }]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        data = self._search_nextjs_data(webpage, video_id)['props']['pageProps']['reactContext']['video']['detail']
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(data['jwpVideoUrl'], video_id)
+
+        return {
+            'id': video_id,
+            'formats': formats,
+            'subtitles': subtitles,
+            **traverse_obj(data, {
+                'title': ('title', {str}),
+                'description': ('metadata', 'description', {str}),
+                'duration': ('duration', {parse_duration}),
+                'age_limit': ('ratingCategory', {parse_age_limit}),
+                'tags': ('tags', ..., {str}),
+            }),
+        }


Test patch: