Repository: yt-dlp/yt-dlp
Pull request #12715 (2025-03-23 22:24:01)

[utils] `js_to_json`: Make function less fatal
<!--
    **IMPORTANT**: PRs without the template will be CLOSED
    
    Due to the high volume of pull requests, it may be a while before your PR is reviewed.
    Please try to keep your pull request focused on a single bugfix or new feature.
    Pull requests with a vast scope and/or very large diff will take much longer to review.
    It is recommended for new contributors to stick to smaller pull requests, so you can receive much more immediate feedback as you familiarize yourself with the codebase.

    PLEASE AVOID FORCE-PUSHING after opening a PR, as it makes reviewing more difficult.
-->

### Description of your *pull request* and other information

`_search_json` together with `transform_source=js_to_json` can currently be fatal. What happens is that the regex extracts part of the html, then `js_to_json` runs and crashes on part of the html. By suppressing the error the html doesn't get transformed. This is not an issue since `LenientJSONDecoder` just discards the additional html data.
From what I can tell it is actually intended that `_search_json` extracts part of the html data. With regex's you can only really either match the first closing `}` or the last `}`. So the function matches the last and then any additional data gets discarded by `LenientJSONDecoder`.


<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--
    # PLEASE FOLLOW THE GUIDE BELOW

    - You will be asked some questions, please read them **carefully** and answer honestly
    - Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
    - Use *Preview* tab to see what your *pull request* will actually look like
-->

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check those that apply and remove the others:
- [x] I am the original author of the code in this PR, and I am willing to release it under [Unlicense](http://unlicense.org/)


### What is the purpose of your *pull request*? Check those that apply and remove the others:
- [x] Core bug fix/improvement

</details>



Patch:
diff --git a/yt_dlp/extractor/_extractors.py b/yt_dlp/extractor/_extractors.py
index c56ec9df6..eb914d2eb 100644
--- a/yt_dlp/extractor/_extractors.py
+++ b/yt_dlp/extractor/_extractors.py
@@ -2392,6 +2392,12 @@
     VoxMediaIE,
     VoxMediaVolumeIE,
 )
+from .vrsquare import (
+    VrSquareChannelIE,
+    VrSquareIE,
+    VrSquareSearchIE,
+    VrSquareSectionIE,
+)
 from .vrt import (
     VRTIE,
     DagelijkseKostIE,
diff --git a/yt_dlp/extractor/vrsquare.py b/yt_dlp/extractor/vrsquare.py
new file mode 100644
index 000000000..9e8740b42
--- /dev/null
+++ b/yt_dlp/extractor/vrsquare.py
@@ -0,0 +1,185 @@
+import itertools
+
+from .common import InfoExtractor
+from ..networking.exceptions import HTTPError
+from ..utils import (
+    ExtractorError,
+    clean_html,
+    extract_attributes,
+    parse_duration,
+    parse_qs,
+)
+from ..utils.traversal import (
+    find_element,
+    find_elements,
+    traverse_obj,
+)
+
+
+class VrSquareIE(InfoExtractor):
+    IE_NAME = 'vrsquare'
+    IE_DESC = 'VR SQUARE'
+
+    _BASE_URL = 'https://livr.jp'
+    _VALID_URL = r'https?://livr\.jp/contents/(?P<id>[\w-]+)'
+    _TESTS = [{
+        'url': 'https://livr.jp/contents/P470896661',
+        'info_dict': {
+            'id': 'P470896661',
+            'ext': 'mp4',
+            'title': 'そこ曲がったら、櫻坂？ ７年間お疲れ様！菅井友香の卒業を祝う会！前半 2022年11月6日放送分',
+            'description': 'md5:523726dc835aa8014dfe1e2b38d36cd1',
+            'duration': 1515.0,
+            'tags': 'count:2',
+            'thumbnail': r're:https?://media\.livr\.jp/vod/img/.+\.jpg',
+        },
+    }, {
+        'url': 'https://livr.jp/contents/P589523973',
+        'info_dict': {
+            'id': 'P589523973',
+            'ext': 'mp4',
+            'title': '薄闇に仰ぐ しだれ桜の妖艶',
+            'description': 'md5:a042f517b2cbb4ed6746707afec4d306',
+            'duration': 1084.0,
+            'tags': list,
+            'thumbnail': r're:https?://media\.livr\.jp/vod/img/.+\.jpg',
+        },
+        'skip': 'Paid video',
+    }, {
+        'url': 'https://livr.jp/contents/P316939908',
+        'info_dict': {
+            'id': 'P316939908',
+            'ext': 'mp4',
+            'title': '2024年5月16日（木） 「今日は誰に恋をする？」公演 小栗有以 生誕祭',
+            'description': 'md5:2110bdcf947f28bd7d06ec420e51b619',
+            'duration': 8559.0,
+            'tags': list,
+            'thumbnail': r're:https?://media\.livr\.jp/vod/img/.+\.jpg',
+        },
+        'skip': 'Premium channel subscribers only',
+    }, {
+        # Accessible only in the VR SQUARE app
+        'url': 'https://livr.jp/contents/P126481458',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        status = self._download_json(
+            f'{self._BASE_URL}/webApi/contentsStatus/{video_id}',
+            video_id, 'Checking contents status', fatal=False)
+        if traverse_obj(status, 'result_code') == '40407':
+            self.raise_login_required('Unable to access this video')
+
+        try:
+            web_api = self._download_json(
+                f'{self._BASE_URL}/webApi/play/url/{video_id}', video_id)
+        except ExtractorError as e:
+            if isinstance(e.cause, HTTPError) and e.cause.status == 500:
+                raise ExtractorError('VR SQUARE app-only videos are not supported', expected=True)
+            raise
+
+        return {
+            'id': video_id,
+            'title': self._html_search_meta(['og:title', 'twitter:title'], webpage),
+            'description': self._html_search_meta('description', webpage),
+            'formats': self._extract_m3u8_formats(traverse_obj(web_api, (
+                'urls', ..., 'url', any)), video_id, 'mp4', fatal=False),
+            'thumbnail': self._html_search_meta('og:image', webpage),
+            **traverse_obj(webpage, {
+                'duration': ({find_element(cls='layout-product-data-time')}, {parse_duration}),
+                'tags': ({find_elements(cls='search-tag')}, ..., {clean_html}),
+            }),
+        }
+
+
+class VrSquarePlaylistBaseIE(InfoExtractor):
+    _BASE_URL = 'https://livr.jp'
+
+    def _fetch_vids(self, source, keys=()):
+        for url_path in traverse_obj(source, (
+            *keys, {find_elements(cls='video', html=True)}, ...,
+            {extract_attributes}, 'data-url', {str}, filter),
+        ):
+            yield self.url_result(
+                f'{self._BASE_URL}/contents/{url_path.removeprefix("/contents/")}', VrSquareIE)
+
+    def _entries(self, path, display_id, query=None):
+        for page in itertools.count(1):
+            ajax = self._download_json(
+                f'{self._BASE_URL}{path}', display_id,
+                f'Downloading playlist JSON page {page}',
+                query={'p': page, **(query or {})})
+            yield from self._fetch_vids(ajax, ('contents_render_list', ...))
+            if not traverse_obj(ajax, (('has_next', 'hasNext'), {bool}, any)):
+                break
+
+
+class VrSquareChannelIE(VrSquarePlaylistBaseIE):
+    IE_NAME = 'vrsquare:channel'
+
+    _VALID_URL = r'https?://livr\.jp/channel/(?P<id>\w+)'
+    _TESTS = [{
+        'url': 'https://livr.jp/channel/H372648599',
+        'info_dict': {
+            'id': 'H372648599',
+            'title': 'AKB48＋チャンネル',
+        },
+        'playlist_mincount': 502,
+    }]
+
+    def _real_extract(self, url):
+        playlist_id = self._match_id(url)
+        webpage = self._download_webpage(url, playlist_id)
+
+        return self.playlist_result(
+            self._entries(f'/ajax/channel/{playlist_id}', playlist_id),
+            playlist_id, self._html_search_meta('og:title', webpage))
+
+
+class VrSquareSearchIE(VrSquarePlaylistBaseIE):
+    IE_NAME = 'vrsquare:search'
+
+    _VALID_URL = r'https?://livr\.jp/web-search/?\?(?:[^#]+&)?w=[^#]+'
+    _TESTS = [{
+        'url': 'https://livr.jp/web-search?w=%23%E5%B0%8F%E6%A0%97%E6%9C%89%E4%BB%A5',
+        'info_dict': {
+            'id': '#小栗有以',
+        },
+        'playlist_mincount': 60,
+    }]
+
+    def _real_extract(self, url):
+        search_query = parse_qs(url)['w'][0]
+
+        return self.playlist_result(
+            self._entries('/ajax/web-search', search_query, {'w': search_query}), search_query)
+
+
+class VrSquareSectionIE(VrSquarePlaylistBaseIE):
+    IE_NAME = 'vrsquare:section'
+
+    _VALID_URL = r'https?://livr\.jp/(?:category|headline)/(?P<id>\w+)'
+    _TESTS = [{
+        'url': 'https://livr.jp/category/C133936275',
+        'info_dict': {
+            'id': 'C133936275',
+            'title': 'そこ曲がったら、櫻坂？VR',
+        },
+        'playlist_mincount': 308,
+    }, {
+        'url': 'https://livr.jp/headline/A296449604',
+        'info_dict': {
+            'id': 'A296449604',
+            'title': 'AKB48 アフターVR',
+        },
+        'playlist_mincount': 22,
+    }]
+
+    def _real_extract(self, url):
+        playlist_id = self._match_id(url)
+        webpage = self._download_webpage(url, playlist_id)
+
+        return self.playlist_result(
+            self._fetch_vids(webpage), playlist_id, self._html_search_meta('og:title', webpage))
diff --git a/yt_dlp/extractor/youtube/_video.py b/yt_dlp/extractor/youtube/_video.py
index c773ba2f1..b8cc72ab1 100644
--- a/yt_dlp/extractor/youtube/_video.py
+++ b/yt_dlp/extractor/youtube/_video.py
@@ -2176,10 +2176,13 @@ def _extract_player_js_global_var(self, jscode):
         """Returns tuple of strings: variable assignment code, variable name, variable value code"""
         return self._search_regex(
             r'''(?x)
-                \'use\s+strict\';\s*
+                (?P<q1>["\'])use\s+strict(?P=q1);\s*
                 (?P<code>
                     var\s+(?P<name>[a-zA-Z0-9_$]+)\s*=\s*
-                    (?P<value>"(?:[^"\\]|\\.)+"\.split\("[^"]+"\))
+                    (?P<value>
+                        (?P<q2>["\'])(?:(?!(?P=q2)).|\\.)+(?P=q2)
+                        \.split\((?P<q3>["\'])(?:(?!(?P=q3)).)+(?P=q3)\)
+                    )
                 )[;,]
             ''', jscode, 'global variable', group=('code', 'name', 'value'), default=(None, None, None))
 
@@ -2187,7 +2190,7 @@ def _fixup_n_function_code(self, argnames, code, full_code):
         global_var, varname, _ = self._extract_player_js_global_var(full_code)
         if global_var:
             self.write_debug(f'Prepending n function code with global array variable "{varname}"')
-            code = global_var + ', ' + code
+            code = global_var + '; ' + code
         else:
             self.write_debug('No global array variable found in player JS')
         return argnames, re.sub(
@@ -2196,7 +2199,7 @@ def _fixup_n_function_code(self, argnames, code, full_code):
 
     def _extract_n_function_code(self, video_id, player_url):
         player_id = self._extract_player_info(player_url)
-        func_code = self.cache.load('youtube-nsig', player_id, min_ver='2025.03.21')
+        func_code = self.cache.load('youtube-nsig', player_id, min_ver='2025.03.24')
         jscode = func_code or self._load_player(video_id, player_url)
         jsi = JSInterpreter(jscode)
 
diff --git a/yt_dlp/utils/_utils.py b/yt_dlp/utils/_utils.py
index 0140acaa3..24525560e 100644
--- a/yt_dlp/utils/_utils.py
+++ b/yt_dlp/utils/_utils.py
@@ -2767,7 +2767,8 @@ def process_escape(match):
     def template_substitute(match):
         evaluated = js_to_json(match.group(1), vars, strict=strict)
         if evaluated[0] == '"':
-            return json.loads(evaluated)
+            with contextlib.suppress(json.JSONDecodeError):
+                return json.loads(evaluated)
         return evaluated
 
     def fix_kv(m):


Test patch:
diff --git a/test/test_utils.py b/test/test_utils.py
index 42dc7f937..e60ceed8f 100644
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -1260,6 +1260,7 @@ def test_js_to_json_edgecases(self):
     def test_js_to_json_malformed(self):
         self.assertEqual(js_to_json('42a1'), '42"a1"')
         self.assertEqual(js_to_json('42a-1'), '42"a"-1')
+        self.assertEqual(js_to_json('{a: `${e("")}`}'), '{"a": "\\"e\\"(\\"\\")"}')
 
     def test_js_to_json_template_literal(self):
         self.assertEqual(js_to_json('`Hello ${name}`', {'name': '"world"'}), '"Hello world"')
diff --git a/test/test_youtube_signature.py b/test/test_youtube_signature.py
index 45dc9113b..453caacd6 100644
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@@ -83,6 +83,11 @@
         '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
         'AAOAOq0QJ8wRAIgXmPlOPSBkkUs1bYFYlJCfe29xx8j7vgpDL0QwbdV06sCIEzpWqMGkFR20CFOS21Tp-7vj_EMu-m37KtXJoOy1',
     ),
+    (
+        'https://www.youtube.com/s/player/363db69b/player_ias.vflset/en_US/base.js',
+        '2aq0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpzEICs69VdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+        '0aqSyOoJXtK73m-uME_jv7-pT15gOFC02RFkGMqWpz2ICs6EVdbwQ0LDp1v7j8xx92efCJlYFYb1sUkkBSPOlPmXgIARw8JQ0qOAOAA',
+    ),
 ]
 
 _NSIG_TESTS = [
@@ -234,6 +239,10 @@
         'https://www.youtube.com/s/player/643afba4/tv-player-ias.vflset/tv-player-ias.js',
         'ir9-V6cdbCiyKxhr', '2PL7ZDYAALMfmA',
     ),
+    (
+        'https://www.youtube.com/s/player/363db69b/player_ias.vflset/en_US/base.js',
+        'eWYu5d5YeY_4LyEDc', 'XJQqf-N7Xra3gg',
+    ),
 ]