Repository: yt-dlp/yt-dlp

Issue:
[Niconico] How to download Anime danmaku on Niconico?
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm asking a question and **not** reporting a bug or requesting a feature
- [X] I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Please make sure the question is worded well enough to be understood

I am using the following command with my own username and password.
yt-dlp --write-info-json --get-comments --write-sub --all-subs --username <USERNAME> --password <PASSWORD> https://www.nicovideo.jp/watch/so42039588
However, the result comments.json is only several KBs with almost nothing in it. The video itself is correctly downloaded though.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '--write-info-json', '--get-comments', '--write-sub', '--all-subs', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://www.nicovideo.jp/watch/so42039588']
[debug] Portable config "D:\Tools\yt-dlp.conf": ['-N', '8', '--proxy', 'socks5://127.0.0.1:8187/', '-R', '1000', '--file-access-retries', '10', '--fragment-retries', '1000', '--extractor-retries', '10']
[debug] Encodings: locale cp936, fs utf-8, pref cp936, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg git-2020-05-19-74dfc88, ffprobe git-2020-05-19-74dfc88
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {'http': 'socks5://127.0.0.1:8187/', 'https': 'socks5://127.0.0.1:8187/'}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a9421
yt-dlp is up to date (stable@2023.03.04)
[niconico] Acquiring Login session
[niconico] Logging in
[niconico] Extracting URL: https://www.nicovideo.jp/watch/so42039588
[niconico] so42039588: Downloading webpage
[niconico] so42039588: Downloading comments
[info] so42039588: Downloading subtitles: comments
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] so42039588: Downloading 1 format(s): h264_360p-aac_64kbps-http
[info] Writing video subtitles to: ちびゴジラの逆襲 第1話「怪獣島のちびゴジラ」 [so42039588].comments.json
[info] Writing video metadata as JSON to: ちびゴジラの逆襲 第1話「怪獣島のちびゴジラ」 [so42039588].info.json
[debug] Invoking niconico_dmc downloader on "niconico_dmc:so42039588/archive_h264_360p/archive_aac_64kbps"
[niconico_dmc] Downloading from DMC
[niconico] so42039588: Downloading JSON metadata for h264_360p-aac_64kbps-http
[niconico] so42039588: Acquiring permission for downloading video
[niconico_dmc] Heartbeat with 40 second interval ...
[debug] File locking is not supported. Proceeding without locking
[download] Destination: ちびゴジラの逆襲 第1話「怪獣島のちびゴジラ」 [so42039588].mp4
[download] 100% of    8.25MiB in 00:01:04 at 130.84KiB/s
```



Discussion:
These comments are recognized as subtitles
Try `--write-sub --sub-lang comments`
> These comments are recognized as subtitles Try `--write-sub --sub-lang comments`

Still nothing is downloaded
[ちびゴジラの逆襲 第1話「怪獣島のちびゴジラ」 [so42039588].comments.zip](https://github.com/yt-dlp/yt-dlp/files/11199094/1.so42039588.comments.zip)
This is the resulting file.
However, it works for https://www.nicovideo.jp/watch/sm42014388 which is user uploaded.
Looks like these comments are served on a separate endpoint
> Looks like these comments are served on a separate endpoint

Then is this a bug? Should I open a new issue?
Nope, that's fine with this issue 

Resolving patch:
diff --git a/yt_dlp/extractor/niconico.py b/yt_dlp/extractor/niconico.py
--- a/yt_dlp/extractor/niconico.py
+++ b/yt_dlp/extractor/niconico.py
@@ -477,23 +477,32 @@ def _get_subtitles(self, video_id, api_data, session_api_data):
         user_id_str = session_api_data.get('serviceUserId')
 
         thread_ids = traverse_obj(api_data, ('comment', 'threads', lambda _, v: v['isActive']))
-        raw_danmaku = self._extract_all_comments(video_id, thread_ids, user_id_str, comment_user_key)
-        if not raw_danmaku:
+        legacy_danmaku = self._extract_legacy_comments(video_id, thread_ids, user_id_str, comment_user_key) or []
+
+        new_comments = traverse_obj(api_data, ('comment', 'nvComment'))
+        new_danmaku = self._extract_new_comments(
+            new_comments.get('server'), video_id,
+            new_comments.get('params'), new_comments.get('threadKey'))
+
+        if not legacy_danmaku and not new_danmaku:
             self.report_warning(f'Failed to get comments. {bug_reports_message()}')
             return
+
         return {
             'comments': [{
                 'ext': 'json',
-                'data': json.dumps(raw_danmaku),
+                'data': json.dumps(legacy_danmaku + new_danmaku),
             }],
         }
 
-    def _extract_all_comments(self, video_id, threads, user_id, user_key):
+    def _extract_legacy_comments(self, video_id, threads, user_id, user_key):
         auth_data = {
             'user_id': user_id,
             'userkey': user_key,
         } if user_id and user_key else {'user_id': ''}
 
+        api_url = traverse_obj(threads, (..., 'server'), get_all=False)
+
         # Request Start
         post_data = [{'ping': {'content': 'rs:0'}}]
         for i, thread in enumerate(threads):
@@ -532,17 +541,32 @@ def _extract_all_comments(self, video_id, threads, user_id, user_key):
         # Request Final
         post_data.append({'ping': {'content': 'rf:0'}})
 
-        for api_url in self._COMMENT_API_ENDPOINTS:
-            comments = self._download_json(
-                api_url, video_id, data=json.dumps(post_data).encode(), fatal=False,
-                headers={
-                    'Referer': 'https://www.nicovideo.jp/watch/%s' % video_id,
-                    'Origin': 'https://www.nicovideo.jp',
-                    'Content-Type': 'text/plain;charset=UTF-8',
-                },
-                note='Downloading comments', errnote=f'Failed to access endpoint {api_url}')
-            if comments:
-                return comments
+        return self._download_json(
+            f'{api_url}/api.json', video_id, data=json.dumps(post_data).encode(), fatal=False,
+            headers={
+                'Referer': f'https://www.nicovideo.jp/watch/{video_id}',
+                'Origin': 'https://www.nicovideo.jp',
+                'Content-Type': 'text/plain;charset=UTF-8',
+            },
+            note='Downloading comments', errnote=f'Failed to access endpoint {api_url}')
+
+    def _extract_new_comments(self, endpoint, video_id, params, thread_key):
+        comments = self._download_json(
+            f'{endpoint}/v1/threads', video_id, data=json.dumps({
+                'additionals': {},
+                'params': params,
+                'threadKey': thread_key,
+            }).encode(), fatal=False,
+            headers={
+                'Referer': 'https://www.nicovideo.jp/',
+                'Origin': 'https://www.nicovideo.jp',
+                'Content-Type': 'text/plain;charset=UTF-8',
+                'x-client-os-type': 'others',
+                'x-frontend-id': '6',
+                'x-frontend-version': '0',
+            },
+            note='Downloading comments (new)', errnote='Failed to download comments (new)')
+        return traverse_obj(comments, ('data', 'threads', ..., 'comments', ...))
 
 
 class NiconicoPlaylistBaseIE(InfoExtractor):


Test patch: