Repository: yt-dlp/yt-dlp

Issue:
[iq.com] JS locations issue
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

i couldn't download anything from iq.com. i dd install phantomjs.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
E:\Tools>yt-dlp.exe --cookies "iq.com_cookies.txt" -vU --playlist-start 1 --playlist-end 2 --output "\output\%%(title)s.%%(ext)s" "https://www.iq.com/album/ailongnhai-the-series-2022-1lg5r01do9l?lang=en_us"  --external-downloader aria2c
[debug] Command-line config: ['--cookies', 'iq.com_cookies.txt', '-vU', '--playlist-start', '1', '--playlist-end', '2', '--output', '\\output\\%%(title)s.%%(ext)s', 'https://www.iq.com/album/ailongnhai-the-series-2022-1lg5r01do9l?lang=en_us', '--external-downloader', 'aria2c']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b64402] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.14393-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg git-2021-08-10-3f92496-ffmpeg-windows-build-helpers (fdk,setts), ffprobe 2021-12-27-git-617452ce2c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] [iq.com:album] Extracting URL: https://www.iq.com/album/ailongnhai-the-series-2022-1lg5r01do9l?lang=en_us
[iq.com:album] 1lg5r01do9l: Downloading webpage
[download] Downloading playlist: AiLongNhai The Series
[iq.com:album] 1lg5r01do9l: Downloading video list episodes 1-10
[iq.com:album] Playlist AiLongNhai The Series: Downloading 2 videos
[download] Downloading video 1 of 2
[debug] [iq.com] Extracting URL: https://www.iq.com/play/ailongnhai-the-series-episode-1-1nuogfc4kc8?lang=en_us
[iq.com] 1nuogfc4kc8: Downloading webpage
[iq.com] 1nuogfc4kc8: Downloading webpack JS
ERROR: [iq.com] 1nuogfc4kc8: Unable to extract JS locations; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 674, in extract
  File "yt_dlp\extractor\iqiyi.py", line 566, in _real_extract
  File "yt_dlp\extractor\iqiyi.py", line 550, in _update_bid_tags
  File "yt_dlp\extractor\iqiyi.py", line 531, in _extract_vms_player_js
  File "yt_dlp\extractor\common.py", line 1228, in _search_regex

[download] Downloading video 2 of 2
[debug] [iq.com] Extracting URL: https://www.iq.com/play/ailongnhai-the-series-episode-2-dlgsueqpjw?lang=en_us
[iq.com] dlgsueqpjw: Downloading webpage
[iq.com] dlgsueqpjw: Downloading webpack JS
ERROR: [iq.com] dlgsueqpjw: Unable to extract JS locations; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 674, in extract
  File "yt_dlp\extractor\iqiyi.py", line 566, in _real_extract
  File "yt_dlp\extractor\iqiyi.py", line 550, in _update_bid_tags
  File "yt_dlp\extractor\iqiyi.py", line 531, in _extract_vms_player_js
  File "yt_dlp\extractor\common.py", line 1228, in _search_regex

[download] Finished downloading playlist: AiLongNhai The Series
```



Discussion:
The target pattern for `JS locations` doesn't exist in the first linked JS and the `vms_request` target doesn't exist in any of the JS chunks. Presumably needs a new tactic.
This fixes the `JS locations` problem, but someone with PhantomJS will need to test if the rest of the extractor works as expected:
```diff
diff --git a/yt_dlp/extractor/iqiyi.py b/yt_dlp/extractor/iqiyi.py
index dbc688fb9..6011a33cc 100644
--- a/yt_dlp/extractor/iqiyi.py
+++ b/yt_dlp/extractor/iqiyi.py
@@ -527,11 +527,13 @@ def _extract_vms_player_js(self, webpage, video_id):
         webpack_js_url = self._proto_relative_url(self._search_regex(
             r'<script src="((?:https?)?//stc.iqiyipic.com/_next/static/chunks/webpack-\w+\.js)"', webpage, 'webpack URL'))
         webpack_js = self._download_webpage(webpack_js_url, video_id, note='Downloading webpack JS', errnote='Unable to download webpack JS')
-        webpack_map1, webpack_map2 = [self._parse_json(js_map, video_id, transform_source=js_to_json) for js_map in self._search_regex(
-            r'\(({[^}]*})\[\w+\][^\)]*\)\s*\+\s*["\']\.["\']\s*\+\s*({[^}]*})\[\w+\]\+["\']\.js', webpack_js, 'JS locations', group=(1, 2))]
-        for module_index in reversed(list(webpack_map2.keys())):
+        webpack_map = self._parse_json(self._search_regex(
+            r'["\']\s*\+\s*(\{\s*(?:\d+\s*:\s*["\'][\da-f]+["\']\s*,?\s*)+})\[\w+\]\+["\']\.js', webpack_js, 'JS locations'),
+            video_id, transform_source=js_to_json)
+
+        for module_index in reversed(list(webpack_map.keys())):
             module_js = self._download_webpage(
-                f'https://stc.iqiyipic.com/_next/static/chunks/{webpack_map1.get(module_index, module_index)}.{webpack_map2[module_index]}.js',
+                f'https://stc.iqiyipic.com/_next/static/chunks/{module_index}.{webpack_map[module_index]}.js',
                 video_id, note=f'Downloading #{module_index} module JS', errnote='Unable to download module JS', fatal=False) or ''
             if 'vms request' in module_js:
                 self.cache.store('iq', 'player_js', module_js)
```

> the `vms_request` target doesn't exist in any of the JS chunks.

'vms request' (no underscore) does
i got the same error.
It looks like the `extracted_bid_tags` regex needed a fix too:
```diff
diff --git a/yt_dlp/extractor/iqiyi.py b/yt_dlp/extractor/iqiyi.py
index dbc688fb9..178f9064e 100644
--- a/yt_dlp/extractor/iqiyi.py
+++ b/yt_dlp/extractor/iqiyi.py
@@ -527,11 +527,13 @@ def _extract_vms_player_js(self, webpage, video_id):
         webpack_js_url = self._proto_relative_url(self._search_regex(
             r'<script src="((?:https?)?//stc.iqiyipic.com/_next/static/chunks/webpack-\w+\.js)"', webpage, 'webpack URL'))
         webpack_js = self._download_webpage(webpack_js_url, video_id, note='Downloading webpack JS', errnote='Unable to download webpack JS')
-        webpack_map1, webpack_map2 = [self._parse_json(js_map, video_id, transform_source=js_to_json) for js_map in self._search_regex(
-            r'\(({[^}]*})\[\w+\][^\)]*\)\s*\+\s*["\']\.["\']\s*\+\s*({[^}]*})\[\w+\]\+["\']\.js', webpack_js, 'JS locations', group=(1, 2))]
-        for module_index in reversed(list(webpack_map2.keys())):
+        webpack_map = self._parse_json(self._search_regex(
+            r'["\']\s*\+\s*(\{\s*(?:\d+\s*:\s*["\'][\da-f]+["\']\s*,?\s*)+})\[\w+\]\+["\']\.js', webpack_js, 'JS locations'),
+            video_id, transform_source=js_to_json)
+
+        for module_index in reversed(list(webpack_map.keys())):
             module_js = self._download_webpage(
-                f'https://stc.iqiyipic.com/_next/static/chunks/{webpack_map1.get(module_index, module_index)}.{webpack_map2[module_index]}.js',
+                f'https://stc.iqiyipic.com/_next/static/chunks/{module_index}.{webpack_map[module_index]}.js',
                 video_id, note=f'Downloading #{module_index} module JS', errnote='Unable to download module JS', fatal=False) or ''
             if 'vms request' in module_js:
                 self.cache.store('iq', 'player_js', module_js)
@@ -545,7 +547,7 @@ def _extract_cmd5x_function(self, webpage, video_id):
     def _update_bid_tags(self, webpage, video_id):
         extracted_bid_tags = self._parse_json(
             self._search_regex(
-                r'arguments\[1\][^,]*,\s*function\s*\([^\)]*\)\s*{\s*"use strict";?\s*var \w=({.+}})\s*,\s*\w\s*=\s*{\s*getNewVd',
+                r'function\s*\([^)]*\)\s*\{\s*"use strict";?\s*var \w\s*=\s*(\{\s*\d+\s*:\s*\{\s*nbid\s*:.+}\s*})\s*,\s*\w\s*=\s*\{\s*getNewVd',
                 self._extract_vms_player_js(webpage, video_id), 'video tags', default=''),
             video_id, transform_source=js_to_json, fatal=False)
         if not extracted_bid_tags:
```
> i mean after i download 2 episode, it's saying [iq.com] 1nuogfc4kc8: No video formats found!. i have to wait for like 10-20mins for next download

@whitedragon92 can you show a verbose log of this?
@whitedragon92 it looks like that is a separate issue. Seems quite similar to #2813

Resolving patch:
diff --git a/yt_dlp/extractor/iqiyi.py b/yt_dlp/extractor/iqiyi.py
--- a/yt_dlp/extractor/iqiyi.py
+++ b/yt_dlp/extractor/iqiyi.py
@@ -527,11 +527,14 @@ def _extract_vms_player_js(self, webpage, video_id):
         webpack_js_url = self._proto_relative_url(self._search_regex(
             r'<script src="((?:https?)?//stc.iqiyipic.com/_next/static/chunks/webpack-\w+\.js)"', webpage, 'webpack URL'))
         webpack_js = self._download_webpage(webpack_js_url, video_id, note='Downloading webpack JS', errnote='Unable to download webpack JS')
-        webpack_map1, webpack_map2 = [self._parse_json(js_map, video_id, transform_source=js_to_json) for js_map in self._search_regex(
-            r'\(({[^}]*})\[\w+\][^\)]*\)\s*\+\s*["\']\.["\']\s*\+\s*({[^}]*})\[\w+\]\+["\']\.js', webpack_js, 'JS locations', group=(1, 2))]
-        for module_index in reversed(list(webpack_map2.keys())):
+        webpack_map = self._search_json(
+            r'["\']\s*\+\s*', webpack_js, 'JS locations', video_id,
+            contains_pattern=r'{\s*(?:\d+\s*:\s*["\'][\da-f]+["\']\s*,?\s*)+}',
+            end_pattern=r'\[\w+\]\+["\']\.js', transform_source=js_to_json)
+
+        for module_index in reversed(webpack_map):
             module_js = self._download_webpage(
-                f'https://stc.iqiyipic.com/_next/static/chunks/{webpack_map1.get(module_index, module_index)}.{webpack_map2[module_index]}.js',
+                f'https://stc.iqiyipic.com/_next/static/chunks/{module_index}.{webpack_map[module_index]}.js',
                 video_id, note=f'Downloading #{module_index} module JS', errnote='Unable to download module JS', fatal=False) or ''
             if 'vms request' in module_js:
                 self.cache.store('iq', 'player_js', module_js)
@@ -543,11 +546,11 @@ def _extract_cmd5x_function(self, webpage, video_id):
                                   self._extract_vms_player_js(webpage, video_id), 'signature function')
 
     def _update_bid_tags(self, webpage, video_id):
-        extracted_bid_tags = self._parse_json(
-            self._search_regex(
-                r'arguments\[1\][^,]*,\s*function\s*\([^\)]*\)\s*{\s*"use strict";?\s*var \w=({.+}})\s*,\s*\w\s*=\s*{\s*getNewVd',
-                self._extract_vms_player_js(webpage, video_id), 'video tags', default=''),
-            video_id, transform_source=js_to_json, fatal=False)
+        extracted_bid_tags = self._search_json(
+            r'function\s*\([^)]*\)\s*\{\s*"use strict";?\s*var \w\s*=\s*',
+            self._extract_vms_player_js(webpage, video_id), 'video tags', video_id,
+            contains_pattern=r'{\s*\d+\s*:\s*\{\s*nbid\s*:.+}\s*}',
+            end_pattern=r'\s*,\s*\w\s*=\s*\{\s*getNewVd', fatal=False, transform_source=js_to_json)
         if not extracted_bid_tags:
             return
         self._BID_TAGS = {


Test patch: