Repository: yt-dlp/yt-dlp

Issue:
[YouTube] comments are not downloading
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [x] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

RU

### Provide a description that is worded well enough to be understood

- This bug can be fixed by disabling cookies or using cookies from another channel.
- This bug can be reproduced locally.
- Comments section in browser is the same as it was.
- The first occurence of this was between 23 and 27 Feb.

My thoughts:

- This can be somehow related to the ReVanced shadowban.
- ...or some kind of A/B testing.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out UTF-8 (No ANSI), error UTF-8 (No ANSI), screen UTF-8 (No ANSI)
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip) API
[debug] params: {'getcomments': True, 'cookiefile': './cookies_old.txt', 'windowsfilenames': True, 'concurrent_fragment_downloads': 4, 'extract_flat': 'discard_in_playlist', 'fragment_retries': 10, 'retries': 10, 'verbose': True, 'ignoreerrors': 'only_download', 'postprocessors': [{'key': 'FFmpegConcat', 'only_multi_video': True, 'when': 'playlist'}], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.41 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.1.58+-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.0.7, websockets-12.0
[debug] Proxy map: {'colab_language_server': '/usr/colab/bin/language_service'}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=WO7pMcKvC8A
[youtube] WO7pMcKvC8A: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] WO7pMcKvC8A: Downloading ios player API JSON
[youtube] WO7pMcKvC8A: Downloading android player API JSON
[debug] [youtube] Extracting signature function js_95cde7ed_109
[debug] Loading youtube-sigfuncs.js_95cde7ed_109 from cache
[debug] Loading youtube-nsig.95cde7ed from cache
[debug] [youtube] Decrypted nsig qpzHlXS-9qKluqYa => f4q2n6Tx1UdeCA
[youtube] WO7pMcKvC8A: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[youtube] Downloading comment section API JSON
[youtube] Downloading ~53 comments
[youtube] Sorting comments by newest first
[youtube] Downloading comment API JSON page 1 (0/~53)
[youtube] Downloading comment API JSON page 2 (0/~53)
[youtube] Downloading comment API JSON page 3 (0/~53)
[youtube] Extracted 0 comments
[debug] Default format spec: bestvideo*+bestaudio/best
[info] WO7pMcKvC8A: Downloading 1 format(s): 303+251
[download] ATTLAS & Mango - Over The Water [Monstercat Release] [WO7pMcKvC8A].webm has already been downloaded
```



Discussion:
> Comments section in browser is the same as it was.

This doesn't really mean anything if you are running yt-dlp on google colab (a different IP and system than where you are running your browser).

> This bug can be reproduced locally.

Please show a log
This is probably caused by the A/B test that YouTube is doing in the comments, they are switching from `commentRenderer`s to `commentViewModel`s. The `commentViewModel`s barely contain any usable information, instead they have a bunch of keys that reference mutations in the `frameworkUpdates.entityBatchUpdate.mutations` part of the response.

Also @bashonly if the A/B test is the cause of the error then this can definitely be reproduced without an account, you just need to get visitor  data with a visitor ID that has the A/B test.
@bashonly
Entware:
```
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip) API
[debug] params: {'getcomments': True, 'writeinfojson': True, 'cookiefile': './cookies_old.txt', 'windowsfilenames': True, 'concurrent_fragment_downloads': 4, 'extract_flat': 'discard_in_playlist', 'fragment_retries': 10, 'retries': 10, 'verbose': True, 'ignoreerrors': 'only_download', 'postprocessors': [{'key': 'FFmpegConcat', 'only_multi_video': True, 'when': 'playlist'}], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.15 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.4 (CPython mips 32bit) - Linux-4.9-ndm-5-mips-with-glibc2.27 (OpenSSL 3.0.10 1 Aug 2023, glibc 2.27)
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, requests-2.31.0, sqlite3-3.41.2, urllib3-2.0.7, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=WO7pMcKvC8A
[youtube] WO7pMcKvC8A: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] WO7pMcKvC8A: Downloading ios player API JSON
[youtube] WO7pMcKvC8A: Downloading android player API JSON
[debug] [youtube] Extracting signature function js_31eb286a_101
[debug] Loading youtube-sigfuncs.js_31eb286a_101 from cache
[debug] Loading youtube-nsig.31eb286a from cache
[debug] [youtube] Decrypted nsig NBLEs1t0svsZQy57V => 1tbxFdEOnSzCIw
[debug] [youtube] Extracting signature function js_31eb286a_105
[debug] Loading youtube-sigfuncs.js_31eb286a_105 from cache
[youtube] WO7pMcKvC8A: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[youtube] Downloading comment section API JSON
[youtube] Downloading ~53 comments
[youtube] Sorting comments by newest first
[youtube] Downloading comment API JSON page 1 (0/~53)
[youtube] Downloading comment API JSON page 2 (0/~53)
[youtube] Downloading comment API JSON page 3 (0/~53)
[youtube] Extracted 0 comments
[debug] Default format spec: bestvideo*+bestaudio/best
[info] WO7pMcKvC8A: Downloading 1 format(s): 303+251
[info] Writing video metadata as JSON to: ATTLAS & Mango - Over The Water [Monstercat Release] [WO7pMcKvC8A].info.json
[download] ATTLAS & Mango - Over The Water [Monstercat Release] [WO7pMcKvC8A].webm has already been downloaded
```

Windows (PC where comments were correct):
```
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip) API
[debug] params: {'getcomments': True, 'writeinfojson': True, 'cookiefile': './cookies_old.txt', 'windowsfilenames': True, 'concurrent_fragment_downloads': 4, 'extract_flat': 'discard_in_playlist', 'fragment_retries': 10, 'retries': 10, 'verbose': True, 'ignoreerrors': 'only_download', 'postprocessors': [{'key': 'FFmpegConcat', 'only_multi_video': True, 'when': 'playlist'}], 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.70 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.7 (CPython AMD64 64bit) - Windows-10-10.0.22631-SP0 (OpenSSL 3.0.13 30 Jan 2024)
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.41.2, urllib3-2.0.7, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=WO7pMcKvC8A
[youtube] WO7pMcKvC8A: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] WO7pMcKvC8A: Downloading ios player API JSON
[youtube] WO7pMcKvC8A: Downloading android player API JSON
[debug] [youtube] Extracting signature function js_31eb286a_105
[debug] Loading youtube-sigfuncs.js_31eb286a_105 from cache
[debug] Loading youtube-nsig.31eb286a from cache
[debug] [youtube] Decrypted nsig d6RktdRv3wwXTqEEE => 2PbMJnbQMpluTQ
[youtube] WO7pMcKvC8A: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[youtube] Downloading comment section API JSON
[youtube] Downloading ~53 comments
[youtube] Sorting comments by newest first
[youtube] Downloading comment API JSON page 1 (0/~53)
[youtube] Downloading comment API JSON page 2 (0/~53)
[youtube] Downloading comment API JSON page 3 (0/~53)
[youtube] Extracted 0 comments
[debug] Default format spec: bestvideo*+bestaudio/best
[info] WO7pMcKvC8A: Downloading 1 format(s): 303+251
[info] Writing video metadata as JSON to: ATTLAS & Mango - Over The Water [Monstercat Release] [WO7pMcKvC8A].info.json
[download] ATTLAS & Mango - Over The Water [Monstercat Release] [WO7pMcKvC8A].webm has already been downloaded
```
@alekskor1063 Can you add `--write-pages` to the yt-dlp command and post the resulting dump files from a **failed** run? The files may contain some sensitive info if you are passing cookies, so feel free to send it privately

Contact: discord (pukkandan#4207) / email (pukkandan.ytdlp@gmail.com)

cc @coletdjnz 
@pukkandan I sent them through Discord.
> This is probably caused by the A/B test that YouTube is doing in the comments, they are switching from `commentRenderer`s to `commentViewModel`s. The `commentViewModel`s barely contain any usable information, instead they have a bunch of keys that reference mutations in the `frameworkUpdates.entityBatchUpdate.mutations` part of the response.
> 
> Also @bashonly if the A/B test is the cause of the error then this can definitely be reproduced without an account, you just need to get visitor data with a visitor ID that has the A/B test.

Looks like I'm getting nothing but the failures now, so maybe the A/B testing is over for me.
Is there any way to proceed with the commentViewModel? How can I use the keys? From frameworkUpdates.entityBatchUpdate.mutations
> Looks like I'm getting nothing but the failures now, so maybe the A/B testing is over for me. Is there any way to proceed with the commentViewModel? How can I use the keys? From frameworkUpdates.entityBatchUpdate.mutations

There's nothing you can do when using yt-dlp. 

The functions _extract_comment and _comment_entries in youtube.py have to be updated or re-written:
https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/youtube.py#L3310
https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/youtube.py#L3359

This is because - as you pointed out - Youtube has moved the comments payload to path frameworkUpdates.entityBatchUpdate.mutations in the JSON response. 

I'm working on it. Hopefully, within the next 2-3 days, I'll post a pull request or maybe give someone else the code I write up. This issue is affecting me as well. 
> I'm working on it. Hopefully, within the next 2-3 days, I'll post a pull request or maybe give someone else the code I write up. This issue is affecting me as well.

Update: Still working on it. Hopefully will be done in another 2-3 days. A complication with Youtube means making temporary accommodations in the code which will be discarded at a later date (after Youtube has finally made up its mind what model to use in its JSON responses.) @themodfather360 had reported [above ](https://github.com/yt-dlp/yt-dlp/issues/9358#issuecomment-2042399513) that the commentRenderer model is no longer in use in the JSON responses from Youtube. But I've encountered otherwise. Youtube still uses the commentRenderer model for some videos, sometimes even for the same video, alternating between commentViewModel and commentRenderer from one moment to the next. 
There's a (so-far) unmerged pull request for Invidious that adds support for the new comment format:
https://github.com/iv-org/invidious/issues/4566
https://github.com/iv-org/invidious/pull/4576
This issue also links a similar patch for youtube-comment-downloader
```diff
# This patch is public domain (CC0).
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -3307,23 +3307,22 @@ def _extract_heatmap(self, data):
                 'value': ('intensityScoreNormalized', {float_or_none}),
             })) or None
 
-    def _extract_comment(self, comment_renderer, parent=None):
-        comment_id = comment_renderer.get('commentId')
-        if not comment_id:
-            return
+    def _extract_comment(self, view_model, entity, parent=None):
+        entity_payload = entity['payload']['commentEntityPayload']
+        comment_id = entity_payload.get('properties').get('commentId')
 
         info = {
             'id': comment_id,
-            'text': self._get_text(comment_renderer, 'contentText'),
-            'like_count': self._get_count(comment_renderer, 'voteCount'),
-            'author_id': traverse_obj(comment_renderer, ('authorEndpoint', 'browseEndpoint', 'browseId', {self.ucid_or_none})),
-            'author': self._get_text(comment_renderer, 'authorText'),
-            'author_thumbnail': traverse_obj(comment_renderer, ('authorThumbnail', 'thumbnails', -1, 'url', {url_or_none})),
+            'text': self._get_text(entity_payload, ('properties', 'content', 'contetn')),
+            'like_count': self._get_count(entity_payload, ('toolbar', 'likeCountNotliked')),
+            'author_id': traverse_obj(entity_payload, ('author', 'channelId', {self.ucid_or_none})),
+            'author': self._get_text(entity_payload, ('author', 'displayName')),
+            'author_thumbnail': traverse_obj(entity_payload, ('author', 'avatarThumbnailUrl', {url_or_none})),
             'parent': parent or 'root',
         }
 
         # Timestamp is an estimate calculated from the current time and time_text
-        time_text = self._get_text(comment_renderer, 'publishedTimeText') or ''
+        time_text = self._get_text(entity_payload, ('properties', 'publishedTime')) or ''
         timestamp = self._parse_time_text(time_text)
 
         info.update({
@@ -3333,25 +3332,23 @@ def _extract_comment(self, comment_renderer, parent=None):
         })
 
         info['author_url'] = urljoin(
-            'https://www.youtube.com', traverse_obj(comment_renderer, ('authorEndpoint', (
-                ('browseEndpoint', 'canonicalBaseUrl'), ('commandMetadata', 'webCommandMetadata', 'url'))),
+            'https://www.youtube.com', traverse_obj(entity_payload,
+                ('author', 'channelCommand', 'innertubeCommand', 'browseEndpoint', 'canonicalBaseUrl'),
                 expected_type=str, get_all=False))
 
-        author_is_uploader = traverse_obj(comment_renderer, 'authorIsChannelOwner')
+        author_is_uploader = traverse_obj(entity_payload, ('author', 'isCreator'))
         if author_is_uploader is not None:
             info['author_is_uploader'] = author_is_uploader
 
         comment_abr = traverse_obj(
-            comment_renderer, ('actionButtons', 'commentActionButtonsRenderer'), expected_type=dict)
+            entity, ('payload', 'engagementToolbarStateEntityPayload', 'heartState'), expected_type=str)
         if comment_abr is not None:
-            info['is_favorited'] = 'creatorHeart' in comment_abr
+            info['is_favorited'] = comment_abr == 'TOOLBAR_HEART_STATE_HEARTED'
 
-        badges = self._extract_badges([traverse_obj(comment_renderer, 'authorCommentBadge')])
-        if self._has_badge(badges, BadgeType.VERIFIED):
-            info['author_is_verified'] = True
+        info['author_is_verified'] = traverse_obj(entity_payload, ('author', 'isVerified')) == 'true'
 
-        is_pinned = traverse_obj(comment_renderer, 'pinnedCommentBadge')
-        if is_pinned:
+        pinned_text = traverse_obj(view_model, 'pinnedText')
+        if pinned_text:
             info['is_pinned'] = True
 
         return info
@@ -3388,21 +3385,25 @@ def extract_header(contents):
                 break
             return _continuation
 
-        def extract_thread(contents):
+        def extract_thread(contents, entity_payloads):
             if not parent:
                 tracker['current_page_thread'] = 0
             for content in contents:
                 if not parent and tracker['total_parent_comments'] >= max_parents:
                     yield
                 comment_thread_renderer = try_get(content, lambda x: x['commentThreadRenderer'])
-                comment_renderer = get_first(
-                    (comment_thread_renderer, content), [['commentRenderer', ('comment', 'commentRenderer')]],
-                    expected_type=dict, default={})
-
-                comment = self._extract_comment(comment_renderer, parent)
-                if not comment:
+                view_model = traverse_obj(comment_thread_renderer, ('commentViewModel', 'commentViewModel'))
+                if not view_model:
+                  view_model = content.get('commentViewModel')
+                if not view_model:
                     continue
-                comment_id = comment['id']
+                comment_id = view_model['commentId']
+                for entity in entity_payloads:
+                    if traverse_obj(entity, ('payload', 'commentEntityPayload', 'properties', 'commentId')) == comment_id:
+                        entity = entity
+                        break
+
+                comment = self._extract_comment(view_model, entity, parent)
                 if comment.get('is_pinned'):
                     tracker['pinned_comment_ids'].add(comment_id)
                 # Sometimes YouTube may break and give us infinite looping comments.
@@ -3495,7 +3496,7 @@ def extract_thread(contents):
             check_get_keys = None
             if not is_forced_continuation and not (tracker['est_total'] == 0 and tracker['running_total'] == 0):
                 check_get_keys = [[*continuation_items_path, ..., (
-                    'commentsHeaderRenderer' if is_first_continuation else ('commentThreadRenderer', 'commentRenderer'))]]
+                    'commentsHeaderRenderer' if is_first_continuation else ('commentThreadRenderer', 'commentViewModel'))]]
             try:
                 response = self._extract_response(
                     item_id=None, query=continuation,
@@ -3527,7 +3528,7 @@ def extract_thread(contents):
                         break
                     continue
 
-                for entry in extract_thread(continuation_items):
+                for entry in extract_thread(continuation_items, response['frameworkUpdates']['entityBatchUpdate']['mutations']):
                     if not entry:
                         return
                     yield entry
```

This patch may work, but has not been tested enough.

Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -3307,7 +3307,36 @@ def _extract_heatmap(self, data):
                 'value': ('intensityScoreNormalized', {float_or_none}),
             })) or None
 
-    def _extract_comment(self, comment_renderer, parent=None):
+    def _extract_comment(self, entities, parent=None):
+        comment_entity_payload = get_first(entities, ('payload', 'commentEntityPayload', {dict}))
+        if not (comment_id := traverse_obj(comment_entity_payload, ('properties', 'commentId', {str}))):
+            return
+
+        toolbar_entity_payload = get_first(entities, ('payload', 'engagementToolbarStateEntityPayload', {dict}))
+        time_text = traverse_obj(comment_entity_payload, ('properties', 'publishedTime', {str})) or ''
+
+        return {
+            'id': comment_id,
+            'parent': parent or 'root',
+            **traverse_obj(comment_entity_payload, {
+                'text': ('properties', 'content', 'content', {str}),
+                'like_count': ('toolbar', 'likeCountA11y', {parse_count}),
+                'author_id': ('author', 'channelId', {self.ucid_or_none}),
+                'author': ('author', 'displayName', {str}),
+                'author_thumbnail': ('author', 'avatarThumbnailUrl', {url_or_none}),
+                'author_is_uploader': ('author', 'isCreator', {bool}),
+                'author_is_verified': ('author', 'isVerified', {bool}),
+                'author_url': ('author', 'channelCommand', 'innertubeCommand', (
+                    ('browseEndpoint', 'canonicalBaseUrl'), ('commandMetadata', 'webCommandMetadata', 'url')
+                ), {lambda x: urljoin('https://www.youtube.com', x)}),
+            }, get_all=False),
+            'is_favorited': (None if toolbar_entity_payload is None else
+                             toolbar_entity_payload.get('heartState') == 'TOOLBAR_HEART_STATE_HEARTED'),
+            '_time_text': time_text,  # FIXME: non-standard, but we need a way of showing that it is an estimate.
+            'timestamp': self._parse_time_text(time_text),
+        }
+
+    def _extract_comment_old(self, comment_renderer, parent=None):
         comment_id = comment_renderer.get('commentId')
         if not comment_id:
             return
@@ -3388,21 +3417,39 @@ def extract_header(contents):
                 break
             return _continuation
 
-        def extract_thread(contents):
+        def extract_thread(contents, entity_payloads):
             if not parent:
                 tracker['current_page_thread'] = 0
             for content in contents:
                 if not parent and tracker['total_parent_comments'] >= max_parents:
                     yield
                 comment_thread_renderer = try_get(content, lambda x: x['commentThreadRenderer'])
-                comment_renderer = get_first(
-                    (comment_thread_renderer, content), [['commentRenderer', ('comment', 'commentRenderer')]],
-                    expected_type=dict, default={})
 
-                comment = self._extract_comment(comment_renderer, parent)
+                # old comment format
+                if not entity_payloads:
+                    comment_renderer = get_first(
+                        (comment_thread_renderer, content), [['commentRenderer', ('comment', 'commentRenderer')]],
+                        expected_type=dict, default={})
+
+                    comment = self._extract_comment_old(comment_renderer, parent)
+
+                # new comment format
+                else:
+                    view_model = (
+                        traverse_obj(comment_thread_renderer, ('commentViewModel', 'commentViewModel', {dict}))
+                        or traverse_obj(content, ('commentViewModel', {dict})))
+                    comment_keys = traverse_obj(view_model, (('commentKey', 'toolbarStateKey'), {str}))
+                    if not comment_keys:
+                        continue
+                    entities = traverse_obj(entity_payloads, lambda _, v: v['entityKey'] in comment_keys)
+                    comment = self._extract_comment(entities, parent)
+                    if comment:
+                        comment['is_pinned'] = traverse_obj(view_model, ('pinnedText', {str})) is not None
+
                 if not comment:
                     continue
                 comment_id = comment['id']
+
                 if comment.get('is_pinned'):
                     tracker['pinned_comment_ids'].add(comment_id)
                 # Sometimes YouTube may break and give us infinite looping comments.
@@ -3495,7 +3542,7 @@ def extract_thread(contents):
             check_get_keys = None
             if not is_forced_continuation and not (tracker['est_total'] == 0 and tracker['running_total'] == 0):
                 check_get_keys = [[*continuation_items_path, ..., (
-                    'commentsHeaderRenderer' if is_first_continuation else ('commentThreadRenderer', 'commentRenderer'))]]
+                    'commentsHeaderRenderer' if is_first_continuation else ('commentThreadRenderer', 'commentViewModel', 'commentRenderer'))]]
             try:
                 response = self._extract_response(
                     item_id=None, query=continuation,
@@ -3519,6 +3566,7 @@ def extract_thread(contents):
                 raise
             is_forced_continuation = False
             continuation = None
+            mutations = traverse_obj(response, ('frameworkUpdates', 'entityBatchUpdate', 'mutations', ..., {dict}))
             for continuation_items in traverse_obj(response, continuation_items_path, expected_type=list, default=[]):
                 if is_first_continuation:
                     continuation = extract_header(continuation_items)
@@ -3527,7 +3575,7 @@ def extract_thread(contents):
                         break
                     continue
 
-                for entry in extract_thread(continuation_items):
+                for entry in extract_thread(continuation_items, mutations):
                     if not entry:
                         return
                     yield entry


Test patch: