Repository: yt-dlp/yt-dlp

Issue:
Youtube comment count sometimes empty
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

USA

### Provide a description that is worded well enough to be understood

comment_count is always null... tried from various geographics.

`yt-dlp -j https://www.youtube.com/watch\?v=TPKX6K2u8Tc`

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-j', 'https://www.youtube.com/watch?v=TPKX6K2u8Tc', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (pip) API
[debug] Python 3.10.9 (CPython 64bit) - macOS-13.0.1-x86_64-i386-64bit
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.10.04
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.11.11/_update_spec
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=TPKX6K2u8Tc
[youtube] TPKX6K2u8Tc: Downloading webpage
[youtube] TPKX6K2u8Tc: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] TPKX6K2u8Tc: Downloading 1 format(s): 248+251
```

Youtube comment count sometimes empty
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

USA

### Provide a description that is worded well enough to be understood

comment_count is always null... tried from various geographics.

`yt-dlp -j https://www.youtube.com/watch\?v=TPKX6K2u8Tc`

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-j', 'https://www.youtube.com/watch?v=TPKX6K2u8Tc', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (pip) API
[debug] Python 3.10.9 (CPython 64bit) - macOS-13.0.1-x86_64-i386-64bit
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.10.04
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.11.11/_update_spec
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=TPKX6K2u8Tc
[youtube] TPKX6K2u8Tc: Downloading webpage
[youtube] TPKX6K2u8Tc: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] TPKX6K2u8Tc: Downloading 1 format(s): 248+251
```



Discussion:
OP: you should probably update your yt-dlp (and make sure that the statements you are checking off in the issue template are actually true.)

However, I can confirm this is an issue on `2022.11.11` and master:

```
$ yt-dlp -v --ignore-config -O comment_count 'https://www.youtube.com/watch?v=TPKX6K2u8Tc'
[debug] Command-line config: ['-v', '-O', 'comment_count', 'https://www.youtube.com/watch?v=TPKX6K2u8Tc']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b644025b]
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Python 3.10.8 (CPython x86_64 64bit) - Linux-6.0.12-arch1-1-x86_64-with-glibc2.36 (OpenSSL 1.1.1q  5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.12.0, brotlicffi-1.0.9.2, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1736 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=TPKX6K2u8Tc
[youtube] TPKX6K2u8Tc: Downloading webpage
[youtube] TPKX6K2u8Tc: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] TPKX6K2u8Tc: Downloading 1 format(s): 248+251
NA
```

A workaround is to add `--write-comments` to your command, but that can take quite a while if there are a lot of comments:

```
$ yt-dlp -O comment_count 'https://www.youtube.com/watch?v=TPKX6K2u8Tc' --write-comments
2588
```

It looks like the current code expects `int_or_none` for the comment count:

https://github.com/yt-dlp/yt-dlp/blob/c73355510629e3eda5a79d4e2876a35316ca6ed2/yt_dlp/extractor/youtube.py#L4152-L4158

...which is a problem because these are the values in the `initial_data` JSON:

```json
"commentCount":{"simpleText":"2.5K"}
```

```json
"contextualInfo":{"runs":[{"text":"2.5K"}]}
```

@bashonly yes sorry, I tested it also in 11.11 - the log isn't from that version.
So like the channel subscribers which comes in millions, I suggest the comments will arrive 2.5k->2500
Changing `int_or_none` to `parse_count` should fix the issue
Is saw same issue with the view count on other videos
`view_count` is working correctly for me. Show verbose
I will try to repro, currently working as expected
I used `parse_count()` in [this patch](https://github.com/ytdl-org/youtube-dl/issues/31372#issuecomment-1322333799) for both `like_count` and `view_count` extraction: I can't recall whether that was defensive or prompted by actual values that broke `int_or_none()` or `str_to_int()` in both cases.
> Changing `int_or_none` to `parse_count` should fix the issue

`self._get_count(...)` should be used in this case
That certainly covers all the bases.
OP: you should probably update your yt-dlp (and make sure that the statements you are checking off in the issue template are actually true.)

However, I can confirm this is an issue on `2022.11.11` and master:

```
$ yt-dlp -v --ignore-config -O comment_count 'https://www.youtube.com/watch?v=TPKX6K2u8Tc'
[debug] Command-line config: ['-v', '-O', 'comment_count', 'https://www.youtube.com/watch?v=TPKX6K2u8Tc']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b644025b]
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Python 3.10.8 (CPython x86_64 64bit) - Linux-6.0.12-arch1-1-x86_64-with-glibc2.36 (OpenSSL 1.1.1q  5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.12.0, brotlicffi-1.0.9.2, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1736 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=TPKX6K2u8Tc
[youtube] TPKX6K2u8Tc: Downloading webpage
[youtube] TPKX6K2u8Tc: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] TPKX6K2u8Tc: Downloading 1 format(s): 248+251
NA
```

A workaround is to add `--write-comments` to your command, but that can take quite a while if there are a lot of comments:

```
$ yt-dlp -O comment_count 'https://www.youtube.com/watch?v=TPKX6K2u8Tc' --write-comments
2588
```

It looks like the current code expects `int_or_none` for the comment count:

https://github.com/yt-dlp/yt-dlp/blob/c73355510629e3eda5a79d4e2876a35316ca6ed2/yt_dlp/extractor/youtube.py#L4152-L4158

...which is a problem because these are the values in the `initial_data` JSON:

```json
"commentCount":{"simpleText":"2.5K"}
```

```json
"contextualInfo":{"runs":[{"text":"2.5K"}]}
```

@bashonly yes sorry, I tested it also in 11.11 - the log isn't from that version.
So like the channel subscribers which comes in millions, I suggest the comments will arrive 2.5k->2500
Changing `int_or_none` to `parse_count` should fix the issue
Is saw same issue with the view count on other videos
`view_count` is working correctly for me. Show verbose
I will try to repro, currently working as expected
I used `parse_count()` in [this patch](https://github.com/ytdl-org/youtube-dl/issues/31372#issuecomment-1322333799) for both `like_count` and `view_count` extraction: I can't recall whether that was defensive or prompted by actual values that broke `int_or_none()` or `str_to_int()` in both cases.
> Changing `int_or_none` to `parse_count` should fix the issue

`self._get_count(...)` should be used in this case
That certainly covers all the bases.

Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -4265,11 +4265,11 @@ def process_language(container, base_url, lang_code, sub_name, query):
 
         info['comment_count'] = traverse_obj(initial_data, (
             'contents', 'twoColumnWatchNextResults', 'results', 'results', 'contents', ..., 'itemSectionRenderer',
-            'contents', ..., 'commentsEntryPointHeaderRenderer', 'commentCount', 'simpleText'
+            'contents', ..., 'commentsEntryPointHeaderRenderer', 'commentCount'
         ), (
             'engagementPanels', lambda _, v: v['engagementPanelSectionListRenderer']['panelIdentifier'] == 'comment-item-section',
-            'engagementPanelSectionListRenderer', 'header', 'engagementPanelTitleHeaderRenderer', 'contextualInfo', 'runs', ..., 'text'
-        ), expected_type=int_or_none, get_all=False)
+            'engagementPanelSectionListRenderer', 'header', 'engagementPanelTitleHeaderRenderer', 'contextualInfo'
+        ), expected_type=self._get_count, get_all=False)
 
         try:  # This will error if there is no livechat
             initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']


Test patch: