Repository: yt-dlp/yt-dlp

Issue:
[bilibili.tv] Incomplete acquisition of subtitle format
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that a **supported** site is broken
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Southeast Asia(Singapore and ...)

### Provide a description that is worded well enough to be understood

![image](https://github.com/yt-dlp/yt-dlp/assets/30733960/4765cb21-9c38-4dca-97c4-ee7f20b43695)
![image](https://github.com/yt-dlp/yt-dlp/assets/30733960/d7d317fc-7d68-424f-8494-fa4fc6457540)
https://api.bilibili.tv/intl/gateway/web/v2/subtitle?s_locale=en_US&platform=web&episode_id=12766944

The website has newly added ass subtitles, and video_subtitle contains subtitles in two formats: ass and srt.yt-dlp. It seems that what you get is under subtitles, which is not comprehensive.
as the picture shows

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
yt-dlp https://www.bilibili.tv/en/play/2079519/12766944 --list-subs --cookies bilibili_dny.txt -vU
[debug] Command-line config: ['https://www.bilibili.tv/en/play/2079519/12766944', '--list-subs', '--cookies', 'bilibili_dny.txt', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython aarch64 64bit) - Linux-5.15.0-1017-oracle-aarch64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2019.11.28, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 91cad9f121c1f6f0a81b747415c46ecba0ff331ed38cc6433040b4ac7b6e15ca
yt-dlp is up to date (stable@2023.03.04)
[BiliIntl] Extracting URL: https://www.bilibili.tv/en/play/2079519/12766944
[BiliIntl] 12766944: Downloading JSON metadata
[BiliIntl] 12766944: Downloading webpage
[BiliIntl] 12766944: Downloading JSON metadata
[BiliIntl] 12766944: Downloading video formats
[BiliIntl] 12766944: Downloading subtitles list
[BiliIntl] 12766944: Downloading subtitles for English
WARNING: [BiliIntl] 12766944: Failed to parse JSON: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
[BiliIntl] 12766944: Downloading subtitles for ภาษาไทย
WARNING: [BiliIntl] 12766944: Failed to parse JSON: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
[BiliIntl] 12766944: Downloading subtitles for Tiếng Việt
[BiliIntl] 12766944: Downloading subtitles for Bahasa Indonesia
WARNING: [BiliIntl] 12766944: Failed to parse JSON: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
[BiliIntl] 12766944: Downloading subtitles for 中文（简体）
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available subtitles for 12766944:
Language Formats
vi       srt
zh-Hans  srt
```



Discussion:

Resolving patch:
diff --git a/yt_dlp/extractor/bilibili.py b/yt_dlp/extractor/bilibili.py
--- a/yt_dlp/extractor/bilibili.py
+++ b/yt_dlp/extractor/bilibili.py
@@ -18,6 +18,7 @@
     OnDemandPagedList,
     bool_or_none,
     clean_html,
+    determine_ext,
     filter_dict,
     float_or_none,
     format_field,
@@ -1658,19 +1659,34 @@ def _get_subtitles(self, *, ep_id=None, aid=None):
                 'aid': aid,
             })) or {}
         subtitles = {}
-        for sub in sub_json.get('subtitles') or []:
-            sub_url = sub.get('url')
-            if not sub_url:
-                continue
-            sub_data = self._download_json(
-                sub_url, ep_id or aid, errnote='Unable to download subtitles', fatal=False,
-                note='Downloading subtitles%s' % f' for {sub["lang"]}' if sub.get('lang') else '')
-            if not sub_data:
-                continue
-            subtitles.setdefault(sub.get('lang_key', 'en'), []).append({
-                'ext': 'srt',
-                'data': self.json2srt(sub_data)
-            })
+        fetched_urls = set()
+        for sub in traverse_obj(sub_json, (('subtitles', 'video_subtitle'), ..., {dict})):
+            for url in traverse_obj(sub, ((None, 'ass', 'srt'), 'url', {url_or_none})):
+                if url in fetched_urls:
+                    continue
+                fetched_urls.add(url)
+                sub_ext = determine_ext(url)
+                sub_lang = sub.get('lang_key') or 'en'
+
+                if sub_ext == 'ass':
+                    subtitles.setdefault(sub_lang, []).append({
+                        'ext': 'ass',
+                        'url': url,
+                    })
+                elif sub_ext == 'json':
+                    sub_data = self._download_json(
+                        url, ep_id or aid, fatal=False,
+                        note=f'Downloading subtitles{format_field(sub, "lang", " for %s")} ({sub_lang})',
+                        errnote='Unable to download subtitles')
+
+                    if sub_data:
+                        subtitles.setdefault(sub_lang, []).append({
+                            'ext': 'srt',
+                            'data': self.json2srt(sub_data),
+                        })
+                else:
+                    self.report_warning('Unexpected subtitle extension', ep_id or aid)
+
         return subtitles
 
     def _get_formats(self, *, ep_id=None, aid=None):


Test patch: