Repository: yt-dlp/yt-dlp
Pull request #2164 (2021-12-29 22:48:44)

[TikTok] Fix extraction for sigi-based webpages
## Please follow the guide below

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *pull request* (like that [x])
- Use *Preview* tab to see how your *pull request* will actually look like

---

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [x] Bug fix
- [ ] Improvement
- [ ] New extractor
- [ ] New feature

---

### Description of your *pull request* and other information

Explanation of your *pull request* in arbitrary form goes here. Please make sure the description explains the purpose and effect of your *pull request* and is worded well enough to be understood. Provide as much context and examples as possible.
"Actually" fixes https://github.com/yt-dlp/yt-dlp/issues/2133


Patch:
diff --git a/README.md b/README.md
index 324a1565a..3490721b9 100644
--- a/README.md
+++ b/README.md
@@ -1090,7 +1090,7 @@ # OUTPUT TEMPLATE
 
 1. **Default**: A literal default value can be specified for when the field is empty using a `|` separator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s`
 
-1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `B`, `j`, `l`, `q`, `D`, `S` can be used for converting to **B**ytes, **j**son (flag `#` for pretty-printing), a comma separated **l**ist (flag `#` for `\n` newline-separated), a string **q**uoted for the terminal (flag `#` to split a list into different arguments), to add **D**ecimal suffixes (Eg: 10M), and to **S**anitize as filename (flag `#` for restricted), respectively
+1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `B`, `j`, `l`, `q`, `D`, `S` can be used for converting to **B**ytes, **j**son (flag `#` for pretty-printing), a comma separated **l**ist (flag `#` for `\n` newline-separated), a string **q**uoted for the terminal (flag `#` to split a list into different arguments), to add **D**ecimal suffixes (Eg: 10M) (flag `#` to use 1024 as factor), and to **S**anitize as filename (flag `#` for restricted), respectively
 
 1. **Unicode normalization**: The format type `U` can be used for NFC [unicode normalization](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize). The alternate form flag (`#`) changes the normalization to NFD and the conversion flag `+` can be used for NFKC/NFKD compatibility equivalence normalization. Eg: `%(title)+.100U` is NFKC
 
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 1d1429b5f..9cec43680 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -1166,7 +1166,9 @@ def create_key(outer_mobj):
                     'NF%s%s' % ('K' if '+' in flags else '', 'D' if '#' in flags else 'C'),
                     value), str_fmt
             elif fmt[-1] == 'D':  # decimal suffix
-                value, fmt = format_decimal_suffix(value, f'%{fmt[:-1]}f%s' if fmt[:-1] else '%d%s'), 's'
+                num_fmt, fmt = fmt[:-1].replace('#', ''), 's'
+                value = format_decimal_suffix(value, f'%{num_fmt}f%s' if num_fmt else '%d%s',
+                                              factor=1024 if '#' in flags else 1000)
             elif fmt[-1] == 'S':  # filename sanitization
                 value, fmt = filename_sanitizer(initial_field, value, restricted='#' in flags), str_fmt
             elif fmt[-1] == 'c':
diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
index 74e133bc9..d376d1460 100644
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -790,7 +790,7 @@ def _get_linux_keyring_password(browser_keyring_name, keyring, logger):
     # Chromium supports a flag: --password-store=<basic|gnome|kwallet> so the automatic detection
     # will not be sufficient in all cases.
 
-    keyring = _LinuxKeyring[keyring] or _choose_linux_keyring(logger)
+    keyring = _LinuxKeyring[keyring] if keyring else _choose_linux_keyring(logger)
     logger.debug(f'Chosen keyring: {keyring.name}')
 
     if keyring == _LinuxKeyring.KWALLET:
diff --git a/yt_dlp/extractor/tiktok.py b/yt_dlp/extractor/tiktok.py
index 18f1c5630..ac537643a 100644
--- a/yt_dlp/extractor/tiktok.py
+++ b/yt_dlp/extractor/tiktok.py
@@ -220,12 +220,13 @@ def extract_addr(addr, add_meta={}):
 
     def _parse_aweme_video_web(self, aweme_detail, webpage_url):
         video_info = aweme_detail['video']
-        author_info = traverse_obj(aweme_detail, 'author', 'authorInfo', default={})
+        author_info = traverse_obj(aweme_detail, 'authorInfo', 'author', expected_type=dict, default={})
         music_info = aweme_detail.get('music') or {}
         stats_info = aweme_detail.get('stats') or {}
         user_url = self._UPLOADER_URL_FORMAT % (traverse_obj(author_info,
                                                              'secUid', 'id', 'uid', 'uniqueId',
-                                                             expected_type=str_or_none, get_all=False))
+                                                             expected_type=str_or_none, get_all=False)
+                                                or aweme_detail.get('authorSecId'))
 
         formats = []
         play_url = video_info.get('playAddr')
@@ -277,8 +278,8 @@ def _parse_aweme_video_web(self, aweme_detail, webpage_url):
             'comment_count': int_or_none(stats_info.get('commentCount')),
             'timestamp': int_or_none(aweme_detail.get('createTime')),
             'creator': str_or_none(author_info.get('nickname')),
-            'uploader': str_or_none(author_info.get('uniqueId')),
-            'uploader_id': str_or_none(author_info.get('id')),
+            'uploader': str_or_none(author_info.get('uniqueId') or aweme_detail.get('author')),
+            'uploader_id': str_or_none(author_info.get('id') or aweme_detail.get('authorId')),
             'uploader_url': user_url,
             'track': str_or_none(music_info.get('title')),
             'album': str_or_none(music_info.get('album')) or None,
@@ -415,19 +416,26 @@ def _real_extract(self, url):
         # If we only call once, we get a 403 when downlaoding the video.
         self._download_webpage(url, video_id)
         webpage = self._download_webpage(url, video_id, note='Downloading video webpage')
-        json_string = self._search_regex(
-            r'id=\"__NEXT_DATA__\"\s+type=\"application\/json\"\s*[^>]+>\s*(?P<json_string_ld>[^<]+)',
-            webpage, 'json_string', group='json_string_ld')
-        json_data = self._parse_json(json_string, video_id)
-        props_data = try_get(json_data, lambda x: x['props'], expected_type=dict)
-
-        # Chech statusCode for success
-        status = props_data.get('pageProps').get('statusCode')
+        next_json = self._search_regex(
+            r'id=\"__NEXT_DATA__\"\s+type=\"application\/json\"\s*[^>]+>\s*(?P<next_data>[^<]+)',
+            webpage, 'next data', group='next_data', default=None)
+
+        if next_json:
+            next_data = self._parse_json(next_json, video_id)
+            status = traverse_obj(next_data, ('props', 'pageProps', 'statusCode'), expected_type=int) or 0
+            video_data = traverse_obj(next_data, ('props', 'pageProps', 'itemInfo', 'itemStruct'), expected_type=dict)
+        else:
+            sigi_json = self._search_regex(
+                r'>\s*window\[[\'"]SIGI_STATE[\'"]\]\s*=\s*(?P<sigi_state>{.+});',
+                webpage, 'sigi data', group='sigi_state')
+            sigi_data = self._parse_json(sigi_json, video_id)
+            status = traverse_obj(sigi_data, ('VideoPage', 'statusCode'), expected_type=int) or 0
+            video_data = traverse_obj(sigi_data, ('ItemModule', video_id), expected_type=dict)
+
         if status == 0:
-            return self._parse_aweme_video_web(props_data['pageProps']['itemInfo']['itemStruct'], url)
+            return self._parse_aweme_video_web(video_data, url)
         elif status == 10216:
             raise ExtractorError('This video is private', expected=True)
-
         raise ExtractorError('Video not available', video_id=video_id)
 
 
diff --git a/yt_dlp/options.py b/yt_dlp/options.py
index 4f51c37c0..83531497c 100644
--- a/yt_dlp/options.py
+++ b/yt_dlp/options.py
@@ -1572,7 +1572,8 @@ def _dict_from_options_callback(
         '--no-hls-split-discontinuity',
         dest='hls_split_discontinuity', action='store_false',
         help='Do not split HLS playlists to different formats at discontinuities such as ad breaks (default)')
-    _extractor_arg_parser = lambda key, vals='': (key.strip().lower().replace('-', '_'), [val.strip() for val in vals.split(',')])
+    _extractor_arg_parser = lambda key, vals='': (key.strip().lower().replace('-', '_'), [
+        val.replace(r'\,', ',').strip() for val in re.split(r'(?<!\\),', vals)])
     extractor.add_option(
         '--extractor-args',
         metavar='KEY:ARGS', dest='extractor_args', default={}, type='str',
diff --git a/yt_dlp/utils.py b/yt_dlp/utils.py
index 788bf16b7..c22aeb464 100644
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -2116,9 +2116,11 @@ def format_decimal_suffix(num, fmt='%d%s', *, factor=1000):
     if num is None:
         return None
     exponent = 0 if num == 0 else int(math.log(num, factor))
-    suffix = ['', *'KMGTPEZY'][exponent]
+    suffix = ['', *'kMGTPEZY'][exponent]
+    if factor == 1024:
+        suffix = {'k': 'Ki', '': ''}.get(suffix, f'{suffix}i')
     converted = num / (factor ** exponent)
-    return fmt % (converted, f'{suffix}i' if suffix and factor == 1024 else suffix)
+    return fmt % (converted, suffix)
 
 
 def format_bytes(bytes):


Test patch:
diff --git a/test/test_YoutubeDL.py b/test/test_YoutubeDL.py
index 61923513e..d2cc423d6 100644
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@@ -645,6 +645,7 @@ def test_add_extra_info(self):
         'ext': 'mp4',
         'width': None,
         'height': 1080,
+        'filesize': 1024,
         'title1': '$PATH',
         'title2': '%PATH%',
         'title3': 'foo/bar\\test',
@@ -778,8 +779,9 @@ def expect_same_infodict(out):
         test('%(title5)#U', 'a\u0301e\u0301i\u0301 𝐀')
         test('%(title5)+U', 'áéí A')
         test('%(title5)+#U', 'a\u0301e\u0301i\u0301 A')
-        test('%(height)D', '1K')
-        test('%(height)5.2D', ' 1.08K')
+        test('%(height)D', '1k')
+        test('%(filesize)#D', '1Ki')
+        test('%(height)5.2D', ' 1.08k')
         test('%(title4)#S', 'foo_bar_test')
         test('%(title4).10S', ('foo \'bar\' ', 'foo \'bar\'' + ('#' if compat_os_name == 'nt' else ' ')))
         if compat_os_name == 'nt':