Repository: yt-dlp/yt-dlp

Issue:
[unscripted.news] News, media and documentary website
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a new site support request
- [X] I've verified that I'm running yt-dlp version **2022.09.01** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that none of provided URLs [violate any copyrights](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required

### Region

India

### Example URLs

https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor
https://www.unscripted.news/videos/scoopwhoop-townhall-ft-rajkummar-rao-ep-1

### Provide a description that is worded well enough to be understood

Please add support for this site. They seem to have stopped making new content. It'd be nice to archive everything in case they go down in the near future.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (zip)
[debug] Python 3.9.2 (CPython 64bit) - Linux-5.15.32-v8+-aarch64-with-glibc2.31 (glibc 2.31)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-62475-g130d19bf20-static (setts), ffprobe N-62475-g130d19bf20-static
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [generic] Extracting URL: https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor
[generic] you-get-the-politicians-you-deserve-ft-shashi-tharoor: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] you-get-the-politicians-you-deserve-ft-shashi-tharoor: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
ERROR: Unsupported URL: https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1459, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1535, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 670, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/generic.py", line 3078, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor
```



Discussion:
This appears to be a video.js embed. The site has simply disguised the player their javascript that it is not recognized as such.
> This appears to be a video.js embed. The site has simply disguised the player their javascript that it is not recognized as such.

Is there an easy workaround for this?
If this were several years ago, a workaround would be to run PhantomJS and get the HTML as rendered, and send that to the generic extractor. But PhantomJS is very old, so it is being slowly phased out and should not be used in new extractors.

Instead, the way forward is to figure out where in the Javascript the element comes from, and write a simple extractor that can find it.

After all, the idea behind this project is to make it easy to write this sort of code.
For devs: JSON data containing metadata and m3u8 playback links is on the webpage under `__NEXT_DATA__`. `InfoExtractor._search_nextjs_data` can be used to get this. Should be fairly simple to implement.


Resolving patch:
diff --git a/yt_dlp/extractor/_extractors.py b/yt_dlp/extractor/_extractors.py
--- a/yt_dlp/extractor/_extractors.py
+++ b/yt_dlp/extractor/_extractors.py
@@ -1966,6 +1966,7 @@
 from .umg import UMGDeIE
 from .unistra import UnistraIE
 from .unity import UnityIE
+from .unscripted import UnscriptedNewsVideoIE
 from .uol import UOLIE
 from .uplynk import (
     UplynkIE,
diff --git a/yt_dlp/extractor/unscripted.py b/yt_dlp/extractor/unscripted.py
new file mode 100644
--- /dev/null
+++ b/yt_dlp/extractor/unscripted.py
@@ -0,0 +1,53 @@
+from .common import InfoExtractor
+from ..utils import parse_duration, traverse_obj
+
+
+class UnscriptedNewsVideoIE(InfoExtractor):
+    _VALID_URL = r'https?://www\.unscripted\.news/videos/(?P<id>[\w-]+)'
+    _TESTS = [{
+        'url': 'https://www.unscripted.news/videos/a-day-at-the-farmers-protest',
+        'info_dict': {
+            'id': '60c0a55cd1e99b1079918a57',
+            'display_id': 'a-day-at-the-farmers-protest',
+            'ext': 'mp4',
+            'title': 'A Day at the Farmers\' Protest',
+            'description': 'md5:4b3df22747a03e8f14f746dd72190384',
+            'thumbnail': 'https://s3.unscripted.news/anj2/60c0a55cd1e99b1079918a57/5f199a65-c803-4a5c-8fce-2077359c3b72.jpg',
+            'duration': 2251.0,
+            'series': 'Ground Reports',
+        }
+    }, {
+        'url': 'https://www.unscripted.news/videos/you-get-the-politicians-you-deserve-ft-shashi-tharoor',
+        'info_dict': {
+            'id': '5fb3afbf18ac817d341a74d8',
+            'display_id': 'you-get-the-politicians-you-deserve-ft-shashi-tharoor',
+            'ext': 'mp4',
+            'cast': ['Avalok Langer', 'Ashwin Mehta'],
+            'thumbnail': 'https://s3.unscripted.news/anj2/5fb3afbf18ac817d341a74d8/82bd7942-4f20-4cd8-98ae-83f9e814f998.jpg',
+            'description': 'md5:1e91b069238a705ca3a40f87e6f1182c',
+            'duration': 1046.0,
+            'series': 'Dumb Questions Only',
+            'title': 'You Get The Politicians You Deserve! ft. Shashi Tharoor',
+        }
+    }]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+        nextjs_data = self._search_nextjs_data(webpage, display_id)['props']['pageProps']['dataLocal']
+
+        # TODO: get subtitle from srt key
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(nextjs_data['alt_content'], display_id)
+
+        return {
+            'id': nextjs_data['_id'],
+            'display_id': display_id,
+            'title': nextjs_data.get('title') or self._og_search_title(webpage),
+            'description': nextjs_data.get('sh_heading') or self._og_search_description(webpage),
+            'formats': formats,
+            'subtitles': subtitles,
+            'thumbnail': self._og_search_thumbnail(webpage),
+            'duration': parse_duration(nextjs_data.get('duration')),
+            'series': traverse_obj(nextjs_data, ('show', 'topic')),
+            'cast': traverse_obj(nextjs_data, ('cast_crew', ..., 'displayname')),
+        }


Test patch: