Repository: yt-dlp/yt-dlp
Pull request #7280 (2023-06-10 21:40:57)

[extractor/zee5] Fix extraction of new content
A recent site API change requires that the `country` parameter in the API request's query matches the user's `current_country` value in their auth token. This seems to be enforced with all newly released content/episodes.

Previously, the extractor hardcoded `IN` as the API `country` param. This patch decodes the `_USER_TOKEN`, checks if it's expired, extracts the `_USER_COUNTRY`, and applies that value to the API request query.

<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [x] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))


<!-- Do NOT edit/remove anything below this! -->
</details><details><summary>Copilot Summary</summary>  

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at e2461b7</samp>

### Summary
🔒🛠️🌐

<!--
1.  🔒 - This emoji represents security, as the changes involve using a more secure device ID generation method, validating and decoding the user token with a cryptographic algorithm, and checking the user token expiration time.
2.  🛠️ - This emoji represents reliability, as the changes fix some potential issues with the previous extractor, such as device ID collisions, user token tampering, and user token expiration errors.
3.  🌐 - This emoji represents compatibility, as the changes ensure that the extractor can handle different user countries and regions, by extracting the country from the user token and passing it to the detail API URL.
-->
Update Zee5 extractor to use more robust methods for device ID, user token, and country extraction. Fix detail API URL to include user country.

> _`Zee5` extractor_
> _enhanced with `uuid`, `jwt`_
> _autumn of bugs ends_

### Walkthrough
* Replace device ID generation with `uuid` and check user token expiration and country with `jwt_decode_hs256` and `time` ([link](https://github.com/yt-dlp/yt-dlp/pull/7280/files?diff=unified&w=0#diff-a558cbee737fe323389fe0bc41baf7a31165d4728e651f427aeb56d43aa3b131L2-R3), [link](https://github.com/yt-dlp/yt-dlp/pull/7280/files?diff=unified&w=0#diff-a558cbee737fe323389fe0bc41baf7a31165d4728e651f427aeb56d43aa3b131L10-R13), [link](https://github.com/yt-dlp/yt-dlp/pull/7280/files?diff=unified&w=0#diff-a558cbee737fe323389fe0bc41baf7a31165d4728e651f427aeb56d43aa3b131L97-R104), [link](https://github.com/yt-dlp/yt-dlp/pull/7280/files?diff=unified&w=0#diff-a558cbee737fe323389fe0bc41baf7a31165d4728e651f427aeb56d43aa3b131L121-R132))
* Change detail API URL to use query parameters and pass user country as a parameter ([link](https://github.com/yt-dlp/yt-dlp/pull/7280/files?diff=unified&w=0#diff-a558cbee737fe323389fe0bc41baf7a31165d4728e651f427aeb56d43aa3b131L140-R153))



</details>



Patch:
diff --git a/yt_dlp/extractor/dumpert.py b/yt_dlp/extractor/dumpert.py
old mode 100644
new mode 100755
index 010c2d092..0cf84263c
--- a/yt_dlp/extractor/dumpert.py
+++ b/yt_dlp/extractor/dumpert.py
@@ -1,12 +1,17 @@
 from .common import InfoExtractor
 from ..utils import (
+    determine_ext,
     int_or_none,
     qualities,
 )
 
 
 class DumpertIE(InfoExtractor):
-    _VALID_URL = r'(?P<protocol>https?)://(?:(?:www|legacy)\.)?dumpert\.nl/(?:mediabase|embed|item)/(?P<id>[0-9]+[/_][0-9a-zA-Z]+)'
+    _VALID_URL = r'''(?x)
+        (?P<protocol>https?)://(?:(?:www|legacy)\.)?dumpert\.nl(?:
+            /(?:mediabase|embed|item)/|
+            (?:/toppers|/latest|/?)\?selectedId=
+        )(?P<id>[0-9]+[/_][0-9a-zA-Z]+)'''
     _TESTS = [{
         'url': 'https://www.dumpert.nl/item/6646981_951bc60f',
         'md5': '1b9318d7d5054e7dcb9dc7654f21d643',
@@ -16,6 +21,9 @@ class DumpertIE(InfoExtractor):
             'title': 'Ik heb nieuws voor je',
             'description': 'Niet schrikken hoor',
             'thumbnail': r're:^https?://.*\.jpg$',
+            'duration': 9,
+            'view_count': int,
+            'like_count': int,
         }
     }, {
         'url': 'https://www.dumpert.nl/embed/6675421_dc440fe7',
@@ -26,6 +34,28 @@ class DumpertIE(InfoExtractor):
     }, {
         'url': 'http://legacy.dumpert.nl/embed/6675421/dc440fe7',
         'only_matching': True,
+    }, {
+        'url': 'https://www.dumpert.nl/item/100031688_b317a185',
+        'info_dict': {
+            'id': '100031688/b317a185',
+            'ext': 'mp4',
+            'title': 'Epic schijnbeweging',
+            'description': '<p>Die zag je niet eh</p>',
+            'thumbnail': r're:^https?://.*\.(?:jpg|png)$',
+            'duration': 12,
+            'view_count': int,
+            'like_count': int,
+        },
+        'params': {'skip_download': 'm3u8'}
+    }, {
+        'url': 'https://www.dumpert.nl/toppers?selectedId=100031688_b317a185',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.dumpert.nl/latest?selectedId=100031688_b317a185',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.dumpert.nl/?selectedId=100031688_b317a185',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -36,18 +66,23 @@ def _real_extract(self, url):
         title = item['title']
         media = next(m for m in item['media'] if m.get('mediatype') == 'VIDEO')
 
-        quality = qualities(['flv', 'mobile', 'tablet', '720p'])
+        quality = qualities(['flv', 'mobile', 'tablet', '720p', '1080p'])
         formats = []
         for variant in media.get('variants', []):
             uri = variant.get('uri')
             if not uri:
                 continue
             version = variant.get('version')
-            formats.append({
-                'url': uri,
-                'format_id': version,
-                'quality': quality(version),
-            })
+            preference = quality(version)
+            if determine_ext(uri) == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    uri, video_id, 'mp4', m3u8_id=version, quality=preference))
+            else:
+                formats.append({
+                    'url': uri,
+                    'format_id': version,
+                    'quality': preference,
+                })
 
         thumbnails = []
         stills = item.get('stills') or {}
diff --git a/yt_dlp/extractor/odnoklassniki.py b/yt_dlp/extractor/odnoklassniki.py
index 4b73eed37..0d0ad0bb8 100644
--- a/yt_dlp/extractor/odnoklassniki.py
+++ b/yt_dlp/extractor/odnoklassniki.py
@@ -1,3 +1,5 @@
+import urllib.parse
+
 from .common import InfoExtractor
 from ..compat import (
     compat_etree_fromstring,
@@ -7,6 +9,7 @@
 )
 from ..utils import (
     ExtractorError,
+    HEADRequest,
     float_or_none,
     int_or_none,
     qualities,
@@ -15,6 +18,7 @@
     unescapeHTML,
     unified_strdate,
     unsmuggle_url,
+    url_or_none,
     urlencode_postdata,
 )
 
@@ -41,7 +45,7 @@ class OdnoklassnikiIE(InfoExtractor):
             'ext': 'mp4',
             'timestamp': 1545580896,
             'view_count': int,
-            'thumbnail': 'https://coub-attachments.akamaized.net/coub_storage/coub/simple/cw_image/c5ac87553bd/608e806a1239c210ab692/1545580913_00026.jpg',
+            'thumbnail': r're:^https?://.*\.jpg$',
             'title': 'Народная забава',
             'uploader': 'Nevata',
             'upload_date': '20181223',
@@ -65,13 +69,14 @@ class OdnoklassnikiIE(InfoExtractor):
             'title': str,
             'uploader': str,
         },
+        'skip': 'vk extractor error',
     }, {
-        # metadata in JSON
+        # metadata in JSON, webm_dash with Firefox UA
         'url': 'http://ok.ru/video/20079905452',
-        'md5': '5d2b64756e2af296e3b383a0bc02a6aa',
+        'md5': '8f477d8931c531374a3e36daec617b2c',
         'info_dict': {
             'id': '20079905452',
-            'ext': 'mp4',
+            'ext': 'webm',
             'title': 'Культура меняет нас (прекрасный ролик!))',
             'thumbnail': str,
             'duration': 100,
@@ -81,10 +86,14 @@ class OdnoklassnikiIE(InfoExtractor):
             'like_count': int,
             'age_limit': 0,
         },
+        'params': {
+            'format': 'bv[ext=webm]',
+            'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0'},
+        },
     }, {
         # metadataUrl
         'url': 'http://ok.ru/video/63567059965189-0?fromTime=5',
-        'md5': 'f8c951122516af72e6e6ffdd3c41103b',
+        'md5': '2bae2f58eefe1b3d26f3926c4a64d2f3',
         'info_dict': {
             'id': '63567059965189-0',
             'ext': 'mp4',
@@ -98,10 +107,11 @@ class OdnoklassnikiIE(InfoExtractor):
             'age_limit': 0,
             'start_time': 5,
         },
+        'params': {'skip_download': 'm3u8'},
     }, {
         # YouTube embed (metadataUrl, provider == USER_YOUTUBE)
         'url': 'https://ok.ru/video/3952212382174',
-        'md5': '91749d0bd20763a28d083fa335bbd37a',
+        'md5': '5fb5f83ce16cb212d6bf887282b5da53',
         'info_dict': {
             'id': '5axVgHHDBvU',
             'ext': 'mp4',
@@ -116,7 +126,7 @@ class OdnoklassnikiIE(InfoExtractor):
             'live_status': 'not_live',
             'view_count': int,
             'thumbnail': 'https://i.mycdn.me/i?r=AEHujHvw2RjEbemUCNEorZbxYpb_p_9AcN2FmGik64Krkcmz37YtlY093oAM5-HIEAt7Zi9s0CiBOSDmbngC-I-k&fn=external_8',
-            'uploader_url': 'http://www.youtube.com/user/MrKewlkid94',
+            'uploader_url': 'https://www.youtube.com/@MrKewlkid94',
             'channel_follower_count': int,
             'tags': ['youtube-dl', 'youtube playlists', 'download videos', 'download audio'],
             'channel_id': 'UCVGtvURtEURYHtJFUegdSug',
@@ -145,7 +155,6 @@ class OdnoklassnikiIE(InfoExtractor):
         },
         'skip': 'Video has not been found',
     }, {
-        # TODO: HTTP Error 400: Bad Request, it only works if there's no cookies when downloading
         'note': 'Only available in mobile webpage',
         'url': 'https://m.ok.ru/video/2361249957145',
         'info_dict': {
@@ -153,8 +162,8 @@ class OdnoklassnikiIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Быковское крещение',
             'duration': 3038.181,
+            'thumbnail': r're:^https?://i\.mycdn\.me/videoPreview\?.+',
         },
-        'skip': 'HTTP Error 400',
     }, {
         'note': 'subtitles',
         'url': 'https://ok.ru/video/4249587550747',
@@ -226,6 +235,14 @@ class OdnoklassnikiIE(InfoExtractor):
         'skip': 'Site no longer embeds',
     }]
 
+    def _clear_cookies(self, cdn_url):
+        # Direct http downloads will fail if CDN cookies are set
+        # so we need to reset them after each format extraction
+        if self._get_cookies('https://notarealsubdomain.mycdn.me/'):
+            self.cookiejar.clear(domain='.mycdn.me')
+        if self._get_cookies(cdn_url):
+            self.cookiejar.clear(domain=urllib.parse.urlparse(cdn_url).hostname)
+
     @classmethod
     def _extract_embed_urls(cls, url, webpage):
         for x in super()._extract_embed_urls(url, webpage):
@@ -364,14 +381,22 @@ def _extract_desktop(self, url):
         formats = [{
             'url': f['url'],
             'ext': 'mp4',
-            'format_id': f['name'],
-        } for f in metadata['videos']]
+            'format_id': f.get('name'),
+        } for f in traverse_obj(metadata, ('videos', lambda _, v: url_or_none(v['url'])))]
 
-        m3u8_url = metadata.get('hlsManifestUrl')
+        m3u8_url = traverse_obj(metadata, 'hlsManifestUrl', 'ondemandHls')
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', 'm3u8_native',
                 m3u8_id='hls', fatal=False))
+            self._clear_cookies(m3u8_url)
+
+        for mpd_id, mpd_key in [('dash', 'ondemandDash'), ('webm', 'metadataWebmUrl')]:
+            mpd_url = metadata.get(mpd_key)
+            if mpd_url:
+                formats.extend(self._extract_mpd_formats(
+                    mpd_url, video_id, mpd_id=mpd_id, fatal=False))
+                self._clear_cookies(mpd_url)
 
         dash_manifest = metadata.get('metadataEmbedded')
         if dash_manifest:
@@ -390,6 +415,7 @@ def _extract_desktop(self, url):
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+            self._clear_cookies(m3u8_url)
         rtmp_url = metadata.get('rtmpUrl')
         if rtmp_url:
             formats.append({
@@ -423,6 +449,10 @@ def _extract_mobile(self, url):
             r'data-video="(.+?)"', webpage, 'json data')
         json_data = self._parse_json(unescapeHTML(json_data), video_id) or {}
 
+        redirect_url = self._request_webpage(HEADRequest(
+            json_data['videoSrc']), video_id, 'Requesting download URL').geturl()
+        self._clear_cookies(redirect_url)
+
         return {
             'id': video_id,
             'title': json_data.get('videoName'),
@@ -430,7 +460,7 @@ def _extract_mobile(self, url):
             'thumbnail': json_data.get('videoPosterSrc'),
             'formats': [{
                 'format_id': 'mobile',
-                'url': json_data.get('videoSrc'),
+                'url': redirect_url,
                 'ext': 'mp4',
             }]
         }
diff --git a/yt_dlp/extractor/zee5.py b/yt_dlp/extractor/zee5.py
index a64eb9ed0..b4734cc8f 100644
--- a/yt_dlp/extractor/zee5.py
+++ b/yt_dlp/extractor/zee5.py
@@ -1,14 +1,16 @@
 import json
-import random
-import string
+import time
+import uuid
 
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
     ExtractorError,
     int_or_none,
+    jwt_decode_hs256,
     parse_age_limit,
     str_or_none,
+    try_call,
     try_get,
     unified_strdate,
     unified_timestamp,
@@ -94,12 +96,12 @@ class Zee5IE(InfoExtractor):
         'url': 'https://www.zee5.com/music-videos/details/adhento-gaani-vunnapaatuga-jersey-nani-shraddha-srinath/0-0-56973',
         'only_matching': True
     }]
-    _DETAIL_API_URL = 'https://spapi.zee5.com/singlePlayback/getDetails/secure?content_id={}&device_id={}&platform_name=desktop_web&country=IN&check_parental_control=false'
-    _DEVICE_ID = ''.join(random.choices(string.ascii_letters + string.digits, k=20)).ljust(32, '0')
+    _DEVICE_ID = str(uuid.uuid4())
     _USER_TOKEN = None
     _LOGIN_HINT = 'Use "--username <mobile_number>" to login using otp or "--username token" and "--password <user_token>" to login using user token.'
     _NETRC_MACHINE = 'zee5'
     _GEO_COUNTRIES = ['IN']
+    _USER_COUNTRY = None
 
     def _perform_login(self, username, password):
         if len(username) == 10 and username.isdigit() and self._USER_TOKEN is None:
@@ -118,11 +120,16 @@ def _perform_login(self, username, password):
             self._USER_TOKEN = otp_verify_json.get('token')
             if not self._USER_TOKEN:
                 raise ExtractorError(otp_request_json['message'], expected=True)
-        elif username.lower() == 'token' and len(password) > 1198:
+        elif username.lower() == 'token' and try_call(lambda: jwt_decode_hs256(password)):
             self._USER_TOKEN = password
         else:
             raise ExtractorError(self._LOGIN_HINT, expected=True)
 
+        token = jwt_decode_hs256(self._USER_TOKEN)
+        if token.get('exp', 0) <= int(time.time()):
+            raise ExtractorError('User token has expired', expected=True)
+        self._USER_COUNTRY = token.get('current_country')
+
     def _real_extract(self, url):
         video_id, display_id = self._match_valid_url(url).group('id', 'display_id')
         access_token_request = self._download_json(
@@ -137,8 +144,13 @@ def _real_extract(self, url):
             data['X-Z5-Guest-Token'] = self._DEVICE_ID
 
         json_data = self._download_json(
-            self._DETAIL_API_URL.format(video_id, self._DEVICE_ID),
-            video_id, headers={'content-type': 'application/json'}, data=json.dumps(data).encode('utf-8'))
+            'https://spapi.zee5.com/singlePlayback/getDetails/secure', video_id, query={
+                'content_id': video_id,
+                'device_id': self._DEVICE_ID,
+                'platform_name': 'desktop_web',
+                'country': self._USER_COUNTRY or self.get_param('geo_bypass_country') or 'IN',
+                'check_parental_control': False,
+            }, headers={'content-type': 'application/json'}, data=json.dumps(data).encode('utf-8'))
         asset_data = json_data['assetDetails']
         show_data = json_data.get('showDetails', {})
         if 'premium' in asset_data['business_type']:
diff --git a/yt_dlp/jsinterp.py b/yt_dlp/jsinterp.py
index 7c7940efd..d6d555733 100644
--- a/yt_dlp/jsinterp.py
+++ b/yt_dlp/jsinterp.py
@@ -44,7 +44,7 @@ def wrapped(a, b):
 
 
 def _js_div(a, b):
-    if JS_Undefined in (a, b) or not (a and b):
+    if JS_Undefined in (a, b) or not (a or b):
         return float('nan')
     return (a or 0) / b if b else float('inf')
 


Test patch:
diff --git a/test/test_jsinterp.py b/test/test_jsinterp.py
index 4d44e6efe..b01477e6f 100644
--- a/test/test_jsinterp.py
+++ b/test/test_jsinterp.py
@@ -28,6 +28,13 @@ def test_basic(self):
     def test_calc(self):
         self._test('function f(a){return 2*a+1;}', 7, args=[3])
 
+    def test_div(self):
+        jsi = JSInterpreter('function f(a, b){return a / b;}')
+        self.assertTrue(math.isnan(jsi.call_function('f', 0, 0)))
+        self.assertTrue(math.isnan(jsi.call_function('f', JS_Undefined, 1)))
+        self.assertTrue(math.isinf(jsi.call_function('f', 2, 0)))
+        self.assertEqual(jsi.call_function('f', 0, 3), 0)
+
     def test_empty_return(self):
         self._test('function f(){return; y()}', None)
 
diff --git a/test/test_youtube_signature.py b/test/test_youtube_signature.py
index 13120d97f..01f09de88 100644
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@@ -150,6 +150,10 @@
         'https://www.youtube.com/s/player/cfa9e7cb/player_ias.vflset/en_US/base.js',
         'aCi3iElgd2kq0bxVbQ', 'QX1y8jGb2IbZ0w',
     ),
+    (
+        'https://www.youtube.com/s/player/8c7583ff/player_ias.vflset/en_US/base.js',
+        '1wWCVpRR96eAmMI87L', 'KSkWAVv1ZQxC3A',
+    ),
 ]