Repository: yt-dlp/yt-dlp

Issue:
weibo broken
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I'm running yt-dlp version **2023.10.13** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Albania

### Provide a description that is worded well enough to be understood

weibo broken

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['https://weibo.com/u/7573838980', '-v']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out cp1252 (No VT), error cp1252 (No VT), screen cp1252 (No VT)
[debug] yt-dlp version stable@2023.10.13 [b634ba742] (pip)
[debug] Python 3.12.0 (CPython AMD64 64bit) - Windows-10-10.0.17763-SP0 (OpenSSL 3.0.11 19 Sep 2023)
[debug] exe versions: ffmpeg 2023-06-21-git-1bcb8a7338-full_build-www.gyan.dev (setts), ffprobe 2023-06-21-git-1bcb8a7338-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, sqlite3-3.42.0, websockets-12.0
[debug] Proxy map: {}
[debug] Loaded 1890 extractors
[WeiboUser] Extracting URL: https://weibo.com/u/7573838980
[WeiboUser] 7573838980: Downloading videos page 1
[WeiboUser] 7573838980: Generating first-visit guest request
ERROR: 7573838980: An extractor error has occurred. (caused by KeyError('confidence')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\Python\Lib\site-packages\yt_dlp\extractor\common.py", line 715, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Lib\site-packages\yt_dlp\extractor\weibo.py", line 233, in _real_extract
    first_page = self._fetch_page(uid)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Lib\site-packages\yt_dlp\extractor\weibo.py", line 216, in _fetch_page
    return self._weibo_download_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Lib\site-packages\yt_dlp\extractor\weibo.py", line 47, in _weibo_download_json
    self._update_visitor_cookies(video_id)
  File "C:\Python\Lib\site-packages\yt_dlp\extractor\weibo.py", line 38, in _update_visitor_cookies
    'c': '%03d' % visitor_data['data']['confidence'],
                  ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'confidence'
```



Discussion:
I am not sure of the importance of the `confidence` param or what its value is supposed to be, but it looks like doing this works:

```diff
diff --git a/yt_dlp/extractor/weibo.py b/yt_dlp/extractor/weibo.py
index b0c3052b6..425e679be 100644
--- a/yt_dlp/extractor/weibo.py
+++ b/yt_dlp/extractor/weibo.py
@@ -26,16 +26,16 @@ def _update_visitor_cookies(self, video_id):
             data=urlencode_postdata({
                 'cb': 'gen_callback',
                 'fp': '{"os":"2","browser":"Gecko57,0,0,0","fonts":"undefined","screenInfo":"1440*900*24","plugins":""}',
-            }))
+            }))['data']
 
         self._download_webpage(
             'https://passport.weibo.com/visitor/visitor', video_id,
             note='Running first-visit callback to get guest cookies',
             query={
                 'a': 'incarnate',
-                't': visitor_data['data']['tid'],
+                't': visitor_data['tid'],
                 'w': 2,
-                'c': '%03d' % visitor_data['data']['confidence'],
+                'c': '%03d' % visitor_data.get('confidence', 1),
                 'cb': 'cross_domain',
                 'from': 'weibo',
                 '_rand': random.random(),
```

cc @c-basalt
@bashonly That also works on my end, and it seems that weibo is relaxed about all other query params except for `tid`/`t`.
Though I think it could worth doing a slightly better emulation in case weibo tightens validation in the future.

```diff
diff --git a/yt_dlp/extractor/weibo.py b/yt_dlp/extractor/weibo.py
index b0c3052b6..9a7e3b0d9 100644
--- a/yt_dlp/extractor/weibo.py
+++ b/yt_dlp/extractor/weibo.py
@@ -18,24 +18,30 @@


 class WeiboBaseIE(InfoExtractor):
-    def _update_visitor_cookies(self, video_id):
+    def _update_visitor_cookies(self, visitor_url, video_id):
+        chrome_ver = self._search_regex(
+            r'Chrome/(\d+)', traverse_obj(self._downloader.params, ('http_headers', 'User-Agent', {str})),
+            'user agent version', default='90')
         visitor_data = self._download_json(
             'https://passport.weibo.com/visitor/genvisitor', video_id,
             note='Generating first-visit guest request',
+            headers={'Referer': visitor_url},
             transform_source=strip_jsonp,
             data=urlencode_postdata({
                 'cb': 'gen_callback',
-                'fp': '{"os":"2","browser":"Gecko57,0,0,0","fonts":"undefined","screenInfo":"1440*900*24","plugins":""}',
-            }))
+                'fp': f'{{"os":"1","browser":"Chrome{chrome_ver},0,0,0","fonts":"undefined","screenInfo":"1920*1080*24","plugins":""}}',
+            }))['data']

         self._download_webpage(
             'https://passport.weibo.com/visitor/visitor', video_id,
             note='Running first-visit callback to get guest cookies',
+            headers={'Referer': visitor_url},
             query={
                 'a': 'incarnate',
-                't': visitor_data['data']['tid'],
-                'w': 2,
-                'c': '%03d' % visitor_data['data']['confidence'],
+                't': visitor_data['tid'],
+                'w': 3 if visitor_data.get('new_tid') else 2,
+                'c': '%03d' % visitor_data.get('confidence', 100),
+                'gc': '',
                 'cb': 'cross_domain',
                 'from': 'weibo',
                 '_rand': random.random(),
@@ -44,7 +50,7 @@ def _update_visitor_cookies(self, video_id):
     def _weibo_download_json(self, url, video_id, *args, fatal=True, note='Downloading JSON metadata', **kwargs):
         webpage, urlh = self._download_webpage_handle(url, video_id, *args, fatal=fatal, note=note, **kwargs)
         if urllib.parse.urlparse(urlh.url).netloc == 'passport.weibo.com':
-            self._update_visitor_cookies(video_id)
+            self._update_visitor_cookies(urlh.url, video_id)
             webpage = self._download_webpage(url, video_id, *args, fatal=fatal, note=note, **kwargs)
         return self._parse_json(webpage, video_id, fatal=fatal)
```


Resolving patch:
diff --git a/yt_dlp/extractor/weibo.py b/yt_dlp/extractor/weibo.py
--- a/yt_dlp/extractor/weibo.py
+++ b/yt_dlp/extractor/weibo.py
@@ -1,3 +1,4 @@
+import json
 import random
 import itertools
 import urllib.parse
@@ -18,24 +19,33 @@
 
 
 class WeiboBaseIE(InfoExtractor):
-    def _update_visitor_cookies(self, video_id):
+    def _update_visitor_cookies(self, visitor_url, video_id):
+        headers = {'Referer': visitor_url}
+        chrome_ver = self._search_regex(
+            r'Chrome/(\d+)', self.get_param('http_headers')['User-Agent'], 'user agent version', default='90')
         visitor_data = self._download_json(
             'https://passport.weibo.com/visitor/genvisitor', video_id,
             note='Generating first-visit guest request',
-            transform_source=strip_jsonp,
+            headers=headers, transform_source=strip_jsonp,
             data=urlencode_postdata({
                 'cb': 'gen_callback',
-                'fp': '{"os":"2","browser":"Gecko57,0,0,0","fonts":"undefined","screenInfo":"1440*900*24","plugins":""}',
-            }))
+                'fp': json.dumps({
+                    'os': '1',
+                    'browser': f'Chrome{chrome_ver},0,0,0',
+                    'fonts': 'undefined',
+                    'screenInfo': '1920*1080*24',
+                    'plugins': ''
+                }, separators=(',', ':'))}))['data']
 
         self._download_webpage(
             'https://passport.weibo.com/visitor/visitor', video_id,
             note='Running first-visit callback to get guest cookies',
-            query={
+            headers=headers, query={
                 'a': 'incarnate',
-                't': visitor_data['data']['tid'],
-                'w': 2,
-                'c': '%03d' % visitor_data['data']['confidence'],
+                't': visitor_data['tid'],
+                'w': 3 if visitor_data.get('new_tid') else 2,
+                'c': f'{visitor_data.get("confidence", 100):03d}',
+                'gc': '',
                 'cb': 'cross_domain',
                 'from': 'weibo',
                 '_rand': random.random(),
@@ -44,7 +54,7 @@ def _update_visitor_cookies(self, video_id):
     def _weibo_download_json(self, url, video_id, *args, fatal=True, note='Downloading JSON metadata', **kwargs):
         webpage, urlh = self._download_webpage_handle(url, video_id, *args, fatal=fatal, note=note, **kwargs)
         if urllib.parse.urlparse(urlh.url).netloc == 'passport.weibo.com':
-            self._update_visitor_cookies(video_id)
+            self._update_visitor_cookies(urlh.url, video_id)
             webpage = self._download_webpage(url, video_id, *args, fatal=fatal, note=note, **kwargs)
         return self._parse_json(webpage, video_id, fatal=fatal)
 


Test patch: