Repository: yt-dlp/yt-dlp
Pull request #6623 (2023-03-24 20:35:48)

[utils] `js_to_json`: Implement template strings
**IMPORTANT**: PRs without the template will be CLOSED

### Description of your *pull request* and other information

<!--

Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible

-->

Rel: https://github.com/yt-dlp/yt-dlp/pull/6505#discussion_r1132854567

Note that I have not really tested against edge cases and only in isolation. More tests are probably required. However I currently lack a lot of time and so just wanted to share this. Feel free to use.

Additionally the current implementation of `template_substitute` works only okayish (`undefined`/`void *` => `null` instead of `undefined`). Generally we would have to provide the real string value of each object which seems out of scope for `js_to_json`.

<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [ ] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [x] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))

</details>



Patch:
diff --git a/yt_dlp/extractor/common.py b/yt_dlp/extractor/common.py
index 2091df7fa..838899052 100644
--- a/yt_dlp/extractor/common.py
+++ b/yt_dlp/extractor/common.py
@@ -2983,6 +2983,8 @@ def _parse_ism_formats_and_subtitles(self, ism_doc, ism_url, ism_id=None):
                         'protocol': 'ism',
                         'fragments': fragments,
                         'has_drm': ism_doc.find('Protection') is not None,
+                        'language': stream_language,
+                        'audio_channels': int_or_none(track.get('Channels')),
                         '_download_params': {
                             'stream_type': stream_type,
                             'duration': duration,
@@ -3513,8 +3515,8 @@ def _RETURN_TYPE(cls):
     @classmethod
     def is_single_video(cls, url):
         """Returns whether the URL is of a single video, None if unknown"""
-        assert cls.suitable(url), 'The URL must be suitable for the extractor'
-        return {'video': True, 'playlist': False}.get(cls._RETURN_TYPE)
+        if cls.suitable(url):
+            return {'video': True, 'playlist': False}.get(cls._RETURN_TYPE)
 
     @classmethod
     def is_suitable(cls, age_limit):
diff --git a/yt_dlp/utils.py b/yt_dlp/utils.py
index 8c2c5593c..40533c2cb 100644
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -3366,7 +3366,7 @@ def strip_jsonp(code):
 
 def js_to_json(code, vars={}, *, strict=False):
     # vars is a dict of var, val pairs to substitute
-    STRING_QUOTES = '\'"'
+    STRING_QUOTES = '\'"`'
     STRING_RE = '|'.join(rf'{q}(?:\\.|[^\\{q}])*{q}' for q in STRING_QUOTES)
     COMMENT_RE = r'/\*(?:(?!\*/).)*?\*/|//[^\n]*\n'
     SKIP_RE = fr'\s*(?:{COMMENT_RE})?\s*'
@@ -3384,6 +3384,12 @@ def process_escape(match):
                 else '' if escape == '\n'
                 else escape)
 
+    def template_substitute(match):
+        evaluated = js_to_json(match.group(1), vars, strict=strict)
+        if evaluated[0] == '"':
+            return json.loads(evaluated)
+        return evaluated
+
     def fix_kv(m):
         v = m.group(0)
         if v in ('true', 'false', 'null'):
@@ -3394,7 +3400,8 @@ def fix_kv(m):
             return ''
 
         if v[0] in STRING_QUOTES:
-            escaped = re.sub(r'(?s)(")|\\(.)', process_escape, v[1:-1])
+            v = re.sub(r'(?s)\${([^}]+)}', template_substitute, v[1:-1]) if v[0] == '`' else v[1:-1]
+            escaped = re.sub(r'(?s)(")|\\(.)', process_escape, v)
             return f'"{escaped}"'
 
         for regex, base in INTEGER_TABLE:


Test patch:
diff --git a/test/test_InfoExtractor.py b/test/test_InfoExtractor.py
index e8d94a6ac..1f60abfd2 100644
--- a/test/test_InfoExtractor.py
+++ b/test/test_InfoExtractor.py
@@ -1406,6 +1406,7 @@ def test_parse_ism_formats(self):
                     'vcodec': 'none',
                     'acodec': 'AACL',
                     'protocol': 'ism',
+                    'audio_channels': 2,
                     '_download_params': {
                         'stream_type': 'audio',
                         'duration': 8880746666,
@@ -1419,9 +1420,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'audio_ext': 'isma',
-                    'video_ext': 'none',
-                    'abr': 128,
                 }, {
                     'format_id': 'video-100',
                     'url': 'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
@@ -1445,9 +1443,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 100,
                 }, {
                     'format_id': 'video-326',
                     'url': 'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
@@ -1471,9 +1466,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 326,
                 }, {
                     'format_id': 'video-698',
                     'url': 'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
@@ -1497,9 +1489,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 698,
                 }, {
                     'format_id': 'video-1493',
                     'url': 'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
@@ -1523,9 +1512,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 1493,
                 }, {
                     'format_id': 'video-4482',
                     'url': 'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
@@ -1549,9 +1535,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 4482,
                 }],
                 {
                     'eng': [
@@ -1575,61 +1558,57 @@ def test_parse_ism_formats(self):
                 'ec-3_test',
                 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
                 [{
-                    'format_id': 'audio_deu_1-224',
+                    'format_id': 'audio_deu-127',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
                     'manifest_url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
                     'ext': 'isma',
-                    'tbr': 224,
+                    'tbr': 127,
                     'asr': 48000,
                     'vcodec': 'none',
-                    'acodec': 'EC-3',
+                    'acodec': 'AACL',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    'audio_channels': 2,
+                    '_download_params': {
                         'stream_type': 'audio',
                         'duration': 370000000,
                         'timescale': 10000000,
                         'width': 0,
                         'height': 0,
-                        'fourcc': 'EC-3',
+                        'fourcc': 'AACL',
                         'language': 'deu',
-                        'codec_private_data': '00063F000000AF87FBA7022DFB42A4D405CD93843BDD0700200F00',
+                        'codec_private_data': '1190',
                         'sampling_rate': 48000,
-                        'channels': 6,
+                        'channels': 2,
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'audio_ext': 'isma',
-                    'video_ext': 'none',
-                    'abr': 224,
                 }, {
-                    'format_id': 'audio_deu-127',
+                    'format_id': 'audio_deu_1-224',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
                     'manifest_url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
                     'ext': 'isma',
-                    'tbr': 127,
+                    'tbr': 224,
                     'asr': 48000,
                     'vcodec': 'none',
-                    'acodec': 'AACL',
+                    'acodec': 'EC-3',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    'audio_channels': 6,
+                    '_download_params': {
                         'stream_type': 'audio',
                         'duration': 370000000,
                         'timescale': 10000000,
                         'width': 0,
                         'height': 0,
-                        'fourcc': 'AACL',
+                        'fourcc': 'EC-3',
                         'language': 'deu',
-                        'codec_private_data': '1190',
+                        'codec_private_data': '00063F000000AF87FBA7022DFB42A4D405CD93843BDD0700200F00',
                         'sampling_rate': 48000,
-                        'channels': 2,
+                        'channels': 6,
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'audio_ext': 'isma',
-                    'video_ext': 'none',
-                    'abr': 127,
                 }, {
                     'format_id': 'video_deu-23',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1641,8 +1620,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1655,9 +1634,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 23,
                 }, {
                     'format_id': 'video_deu-403',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1669,8 +1645,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1683,9 +1659,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 403,
                 }, {
                     'format_id': 'video_deu-680',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1697,8 +1670,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1711,9 +1684,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 680,
                 }, {
                     'format_id': 'video_deu-1253',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1725,8 +1695,9 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'vbr': 1253,
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1739,9 +1710,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 1253,
                 }, {
                     'format_id': 'video_deu-2121',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1753,8 +1721,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1767,9 +1735,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 2121,
                 }, {
                     'format_id': 'video_deu-3275',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1781,8 +1746,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1795,9 +1760,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 3275,
                 }, {
                     'format_id': 'video_deu-5300',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1809,8 +1771,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1823,9 +1785,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 5300,
                 }, {
                     'format_id': 'video_deu-8079',
                     'url': 'https://smstr01.dmm.t-online.de/smooth24/smoothstream_m1/streaming/sony/9221438342941275747/636887760842957027/25_km_h-Trailer-9221571562372022953_deu_20_1300k_HD_H_264_ISMV.ism/Manifest',
@@ -1837,8 +1796,8 @@ def test_parse_ism_formats(self):
                     'vcodec': 'AVC1',
                     'acodec': 'none',
                     'protocol': 'ism',
-                    '_download_params':
-                    {
+                    'language': 'deu',
+                    '_download_params': {
                         'stream_type': 'video',
                         'duration': 370000000,
                         'timescale': 10000000,
@@ -1851,9 +1810,6 @@ def test_parse_ism_formats(self):
                         'bits_per_sample': 16,
                         'nal_unit_length_field': 4
                     },
-                    'video_ext': 'ismv',
-                    'audio_ext': 'none',
-                    'vbr': 8079,
                 }],
                 {},
             ),
diff --git a/test/test_utils.py b/test/test_utils.py
index 3045b6d7e..d4a301583 100644
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -1190,6 +1190,13 @@ def test_js_to_json_malformed(self):
         self.assertEqual(js_to_json('42a1'), '42"a1"')
         self.assertEqual(js_to_json('42a-1'), '42"a"-1')
 
+    def test_js_to_json_template_literal(self):
+        self.assertEqual(js_to_json('`Hello ${name}`', {'name': '"world"'}), '"Hello world"')
+        self.assertEqual(js_to_json('`${name}${name}`', {'name': '"X"'}), '"XX"')
+        self.assertEqual(js_to_json('`${name}${name}`', {'name': '5'}), '"55"')
+        self.assertEqual(js_to_json('`${name}"${name}"`', {'name': '5'}), '"5\\"5\\""')
+        self.assertEqual(js_to_json('`${name}`', {}), '"name"')
+
     def test_extract_attributes(self):
         self.assertEqual(extract_attributes('<e x="y">'), {'x': 'y'})
         self.assertEqual(extract_attributes("<e x='y'>"), {'x': 'y'})