Repository: yt-dlp/yt-dlp

Issue:
[Broken] YouTube comments use account language timestamps, breaking yt-dlp
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.07.07** (I'm using current master, cca80fe)
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

```
~/przechowalnia/archiwum_youtube/testy $ yt-dlp -v --no-download --write-comments --cookies ../_meta/cookies.txt 'https://www.youtube.com/watch?v=-b-BjBQrOLc' 2>&1 | tee comments.log
[debug] Command-line config: ['-v', '--no-download', '--write-comments', '--cookies', '../_meta/cookies.txt', 'https://www.youtube.com/watch?v=-b-BjBQrOLc']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] yt-dlp version 2021.07.07 
[debug] Python version 3.7.3 (CPython 64bit) - Linux-4.19.0-17-amd64-x86_64-with-debian-10.10
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=-b-BjBQrOLc
[youtube] -b-BjBQrOLc: Downloading webpage
[youtube] -b-BjBQrOLc: Downloading android player API JSON
[youtube] -b-BjBQrOLc: Downloading MPD manifest
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Testing thumbnail 37
[debug] Default format spec: bestvideo*+bestaudio/best
[info] -b-BjBQrOLc: Downloading 1 format(s): 299+251
[youtube] Downloading comment section API JSON
[youtube] Downloading ~3 comments
[youtube] Sorting comments by od najnowszych
[youtube] Downloading comment API JSON page 1 (0/3)
ERROR: time data 'now-5dni' does not match format '%Y%m%d'
Traceback (most recent call last):
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1145, in wrapper
    return func(self, *args, **kwargs)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1181, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1222, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2217, in process_video_result
    self.process_info(new_info)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2377, in process_info
    self.post_extract(info_dict)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2872, in post_extract
    actual_post_extract(info_dict or {})
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2864, in actual_post_extract
    extra = post_extractor().items()
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 3020, in <lambda>
    info['__post_extractor'] = lambda: self._extract_comments(ytcfg, video_id, contents, webpage)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 2227, in _extract_comments
    for comment in _real_comment_extract(contents):
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 2219, in _real_comment_extract
    account_syncid=self._extract_account_syncid(ytcfg))
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 2156, in _comment_entries
    for count, entry in enumerate(extract_thread(continuation_items)):
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 2073, in extract_thread
    comment = self._extract_comment(comment_renderer, parent)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 1997, in _extract_comment
    time_text_dt = self.parse_time_text(time_text)
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/extractor/youtube.py", line 1986, in parse_time_text
    return datetime_from_str('now-%s%s' % (time_text_split[0], time_text_split[1]), precision='auto')
  File "/home/grzesiek11/.local/pipx/venvs/yt-dlp/lib/python3.7/site-packages/yt_dlp/utils.py", line 3188, in datetime_from_str
    return datetime_round(datetime.datetime.strptime(date_str, format), precision)
  File "/usr/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.7/_strptime.py", line 359, in _strptime
    (data_string, format))
ValueError: time data 'now-5dni' does not match format '%Y%m%d'
```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

YouTube seems to have rolled out a change to comment timestamps (`n units ago`, for example `11 days ago`), which now are dependent on account (?, not sure, but definitely not geolocation) language (for example `11 dni temu`).

This means yt-dlp will break when downloading comments with cookies.txt from non-English accounts, because it uses `timedate` to calculate timestamps, which only works with English units.

A dirty fix by me, definitely not good for a merge, but if somebody needs to get it working now, here it is. Licensed under The Unlicense. You need to adjust the unit prefixes with ones for your language (these are for Polish):

`yt_dlp/extractor/youtube.py:1979`
```py
    def parse_time_text(time_text):
        """
        Parse the comment time text
        time_text is in the format 'X units ago (edited)'
        """
        time_text_split = time_text.split(' ')

        translations = {
            'd': 'days',
            'm': 'months',
            'l': 'years',
            'm': 'minutes',
            'g': 'hours',
            's': 'seconds'
        }

        for i in range(len(time_text_split)):
            for translation in translations.items():
                if translation[0] in time_text_split[i]:
                    time_text_split[i] = translation[1]

        if len(time_text_split) >= 3:
            print(time_text_split)
            return datetime_from_str('now-%s%s' % (time_text_split[0], time_text_split[1]), precision='auto')
```


Discussion:
Another potential solution could be to look into seeing if there is a way to force the API to return the response in English regardless
> Another potential solution could be to look into seeing if there is a way to force the API to return the response in English regardless

Surely this can be done by setting the `Accept-Language` header?

```
'Accept-Language': 'en-US, en, *'
```


> > Another potential solution could be to look into seeing if there is a way to force the API to return the response in English regardless
> 
> Surely this can be done by setting the `Accept-Language` header?
> 
> ```
> 'Accept-Language': 'en-US, en, *'
> ```


Hmm doesn't seem to be (typical YouTube). Looking at the API requests it is probably dependent on the `hl` parameter in the client context data given. Should be easy enough to overwrite.


Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -2394,7 +2394,11 @@ def _real_comment_extract(contents):
         known_entry_comment_renderers = ('itemSectionRenderer',)
         estimated_total = 0
         max_comments = int_or_none(self._configuration_arg('max_comments', [''])[0]) or float('inf')
-
+        # Force English regardless of account setting to prevent parsing issues
+        # See: https://github.com/yt-dlp/yt-dlp/issues/532
+        ytcfg = copy.deepcopy(ytcfg)
+        traverse_obj(
+            ytcfg, ('INNERTUBE_CONTEXT', 'client'), expected_type=dict, default={})['hl'] = 'en'
         try:
             for comment in _real_comment_extract(contents):
                 if len(comments) >= max_comments:


Test patch: