Repository: yt-dlp/yt-dlp

Issue:
[tiktok.com] Failure to obtain m3u8 URL of some live streams
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that a **supported** site is broken
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Ireland

### Provide a description that is worded well enough to be understood

There is an error when downloading some tiktok live streams (e.g. https://www.tiktok.com/@mikey_____7/live or  https://www.tiktok.com/@knkjewelrydesign/live)

> ERROR: [tiktok:live] 7207751234156120874: Failed to obtain m3u8 URL;

This error occurs not on every stream, but very often. And it is consistent with particular streams - after retrying yt-dlp fails to download m3u8 URL. For other streams, successful download is also consistent.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.tiktok.com/@mikey_____7/live']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 5.1.2-essentials_build-www.gyan.dev (setts), ffprobe 5.1.2-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a9421
yt-dlp is up to date (stable@2023.03.04)
[tiktok:live] Extracting URL: https://www.tiktok.com/@mikey_____7/live
[tiktok:live] mikey_____7: Downloading webpage
[tiktok:live] 7207751234156120874: Downloading JSON metadata
ERROR: [tiktok:live] 7207751234156120874: Failed to obtain m3u8 URL; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 694, in extract
  File "yt_dlp\extractor\tiktok.py", line 1020, in _real_extract
  File "yt_dlp\extractor\common.py", line 1919, in _extract_m3u8_formats
  File "yt_dlp\extractor\common.py", line 1937, in _extract_m3u8_formats_and_subtitles
```



Discussion:
The livestreams for which the extractor fails to get the m3u8 URL do not make their m3u8 URLs available from the endpoint that we are using. Here's part of the API response JSON from the OP's URL (when stream was still live):

```json
{
  "LiveRoomInfo": {
    "coverUrl": "https://p16-webcast.tiktokcdn.com/img/720x720/tos-maliva-avt-0068/f7ede8b12f61e3947fd20536469470dd~tplv-obj.jpeg",
    "liveRoomStats": {
      "userCount": 1554
    },
    "liveUrl": "",
    "roomID": "7207751234156120874",
    "status": 2,
    "title": "NYC 🗽 Walk n Talk"
```
There's likely another endpoint that we need to call for these.

@bashonly 
I've seen other applications use the "https://webcast.tiktok.com/webcast/room/info" endpoint.
The api call could look something like this:
`
self._download_json(
            "https://webcast.tiktok.com/webcast/room/info/", room_id, query={
                'aid': "1988",
                'room_id': room_id
            }
        )
`
And it returns a bunch of infos including the following relevant for formats:
```json
{
    "candidate_resolution": [
        "SD1",
        "SD2",
        "HD1",
        "FULL_HD1"
    ],
    "complete_push_urls": [],
    "default_resolution": "ORIGION",
    "extra": {
        "anchor_interact_profile": 0,
        "audience_interact_profile": 0,
        "bframe_enable": false,
        "bitrate_adapt_strategy": 0,
        "bytevc1_enable": false,
        "default_bitrate": 0,
        "deprecated1": false,
        "fps": 0,
        "gop_sec": 0,
        "hardware_encode": false,
        "height": 0,
        "max_bitrate": 0,
        "min_bitrate": 0,
        "roi": false,
        "sw_roi": false,
        "video_profile": 0,
        "width": 0
    },
    "flv_pull_url": {
        "FULL_HD1": "https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_uhd.flv",
        "HD1": "https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_hd.flv",
        "SD1": "https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_ld.flv",
        "SD2": "https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_sd.flv"
    },
    "flv_pull_url_params": {
        "FULL_HD1": "{\"VCodec\":\"h264\"}",
        "HD1": "{\"VCodec\":\"h264\"}",
        "SD1": "{\"VCodec\":\"h264\"}",
        "SD2": "{\"VCodec\":\"h264\"}"
    },
    "hls_pull_url": "https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_or4/index.m3u8",
    "hls_pull_url_map": {},
    "hls_pull_url_params": "{\"VCodec\":\"h264\"}",
    "id": 0,
    "id_str": "xxx",
    "live_core_sdk_data": {
        "pull_data": {
            "options": {
                "default_quality": {
                    "level": 0,
                    "name": "Original",
                    "resolution": "",
                    "sdk_key": "origin",
                    "v_codec": ""
                },
                "qualities": [
                    {
                        "level": 10,
                        "name": "Original",
                        "resolution": "",
                        "sdk_key": "origin",
                        "v_codec": ""
                    },
                    {
                        "level": 5,
                        "name": "1080p",
                        "resolution": "",
                        "sdk_key": "uhd",
                        "v_codec": ""
                    },
                    {
                        "level": 3,
                        "name": "720p",
                        "resolution": "",
                        "sdk_key": "hd",
                        "v_codec": ""
                    },
                    {
                        "level": 2,
                        "name": "540p",
                        "resolution": "",
                        "sdk_key": "sd",
                        "v_codec": ""
                    },
                    {
                        "level": 1,
                        "name": "360p",
                        "resolution": "",
                        "sdk_key": "ld",
                        "v_codec": ""
                    }
                ]
            },
            "stream_data": "{\"common\":{\"session_id\":\"044-20230309113232DB2FEB5401F99C100122\",\"rule_ids\":\"{\\\"ab_version_trace\\\":null,\\\"sched\\\":\\\"{\\\\\\\"result\\\\\\\":{\\\\\\\"hit\\\\\\\":\\\\\\\"default\\\\\\\",\\\\\\\"cdn\\\\\\\":107}}\\\"}\"},\"data\":{\"sd\":{\"main\":{\"flv\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_sd.flv\",\"hls\":\"https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_sd/index.m3u8\",\"cmaf\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_sd/index.mpd\",\"dash\":\"\",\"lls\":\"\",\"tsl\":\"\",\"tile\":\"\",\"sdk_params\":\"{\\\"VCodec\\\":\\\"h264\\\",\\\"gop\\\":0,\\\"resolution\\\":\\\"480P\\\",\\\"vbitrate\\\":800000}\"}},\"ld\":{\"main\":{\"flv\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_ld.flv\",\"hls\":\"https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_ld/index.m3u8\",\"cmaf\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_ld/index.mpd\",\"dash\":\"\",\"lls\":\"\",\"tsl\":\"\",\"tile\":\"\",\"sdk_params\":\"{\\\"VCodec\\\":\\\"h264\\\",\\\"gop\\\":0,\\\"resolution\\\":\\\"480P\\\",\\\"vbitrate\\\":500000}\"}},\"hd\":{\"main\":{\"flv\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_hd.flv\",\"hls\":\"https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_hd/index.m3u8\",\"cmaf\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_hd/index.mpd\",\"dash\":\"\",\"lls\":\"\",\"tsl\":\"\",\"tile\":\"\",\"sdk_params\":\"{\\\"VCodec\\\":\\\"h264\\\",\\\"gop\\\":0,\\\"resolution\\\":\\\"540P\\\",\\\"vbitrate\\\":1000000}\"}},\"uhd\":{\"main\":{\"flv\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_uhd.flv\",\"hls\":\"https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_uhd/index.m3u8\",\"cmaf\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_uhd/index.mpd\",\"dash\":\"\",\"lls\":\"\",\"tsl\":\"\",\"tile\":\"\",\"sdk_params\":\"{\\\"VCodec\\\":\\\"h264\\\",\\\"gop\\\":0,\\\"resolution\\\":\\\"720P\\\",\\\"vbitrate\\\":1000000}\"}},\"origin\":{\"main\":{\"flv\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_or4.flv\",\"hls\":\"https://pull-hls-f16-va01.tiktokcdn.com/stage/stream-xxx_or4/index.m3u8\",\"cmaf\":\"https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_or4/index.mpd\",\"dash\":\"\",\"lls\":\"\",\"tsl\":\"\",\"tile\":\"\",\"sdk_params\":\"{\\\"VCodec\\\":\\\"h264\\\",\\\"gop\\\":4,\\\"resolution\\\":\\\"or4\\\",\\\"vbitrate\\\":1000000}\"}}}}"
        }
    },
    "provider": 0,
    "push_urls": [],
    "resolution_name": {
        "AUTO": "AUTO",
        "FULL_HD1": "1080p",
        "HD1": "720p",
        "ORIGION": "Original",
        "SD1": "360p",
        "SD2": "540p",
        "pm_mt_video_1080p60": "1080p60",
        "pm_mt_video_720p60": "720p60"
    },
    "rtmp_pull_url": "https://pull-f5-va01.tiktokcdn.com/stage/stream-xxx_or4.flv",
    "rtmp_pull_url_params": "{\"VCodec\":\"h264\"}",
    "rtmp_push_url": "",
    "rtmp_push_url_params": "",
    "stream_control_type": 0,
    "stream_delay_ms": 0,
    "vr_type": 0
}
```

I'm not keen on the implementation details of the current extractor but maybe this still helps someone understands it more.
I have an example implementation which adds more formats using this endpoint, but as i said before I have no clue how you guys usually implement these extractors properly.

Resolving patch:
diff --git a/yt_dlp/extractor/tiktok.py b/yt_dlp/extractor/tiktok.py
--- a/yt_dlp/extractor/tiktok.py
+++ b/yt_dlp/extractor/tiktok.py
@@ -1,6 +1,7 @@
 import itertools
 import json
 import random
+import re
 import string
 import time
 
@@ -12,15 +13,18 @@
     LazyList,
     UnsupportedError,
     UserNotLive,
+    format_field,
     get_element_by_id,
     get_first,
     int_or_none,
     join_nonempty,
+    merge_dicts,
     qualities,
     remove_start,
     srt_subtitles_timecode,
     str_or_none,
     traverse_obj,
+    try_call,
     try_get,
     url_or_none,
 )
@@ -563,7 +567,7 @@ def _real_extract(self, url):
             self.report_warning(f'{e}; trying with webpage')
 
         url = self._create_url(user_id, video_id)
-        webpage = self._download_webpage(url, video_id, headers={'User-Agent': 'User-Agent:Mozilla/5.0'})
+        webpage = self._download_webpage(url, video_id, headers={'User-Agent': 'Mozilla/5.0'})
         next_data = self._search_nextjs_data(webpage, video_id, default='{}')
         if next_data:
             status = traverse_obj(next_data, ('props', 'pageProps', 'statusCode'), expected_type=int) or 0
@@ -983,40 +987,173 @@ def _real_extract(self, url):
         return self.url_result(new_url)
 
 
-class TikTokLiveIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?tiktok\.com/@(?P<id>[\w\.-]+)/live'
+class TikTokLiveIE(TikTokBaseIE):
+    _VALID_URL = r'''(?x)https?://(?:
+        (?:www\.)?tiktok\.com/@(?P<uploader>[\w.-]+)/live|
+        m\.tiktok\.com/share/live/(?P<id>\d+)
+    )'''
     IE_NAME = 'tiktok:live'
 
     _TESTS = [{
+        'url': 'https://www.tiktok.com/@weathernewslive/live',
+        'info_dict': {
+            'id': '7210809319192726273',
+            'ext': 'mp4',
+            'title': r're:ウェザーニュースLiVE[\d\s:-]*',
+            'creator': 'ウェザーニュースLiVE',
+            'uploader': 'weathernewslive',
+            'uploader_id': '6621496731283095554',
+            'uploader_url': 'https://www.tiktok.com/@weathernewslive',
+            'live_status': 'is_live',
+            'concurrent_view_count': int,
+        },
+        'params': {'skip_download': 'm3u8'},
+    }, {
+        'url': 'https://www.tiktok.com/@pilarmagenta/live',
+        'info_dict': {
+            'id': '7209423610325322522',
+            'ext': 'mp4',
+            'title': str,
+            'creator': 'Pilarmagenta',
+            'uploader': 'pilarmagenta',
+            'uploader_id': '6624846890674683909',
+            'uploader_url': 'https://www.tiktok.com/@pilarmagenta',
+            'live_status': 'is_live',
+            'concurrent_view_count': int,
+        },
+        'skip': 'Livestream',
+    }, {
+        'url': 'https://m.tiktok.com/share/live/7209423610325322522/?language=en',
+        'only_matching': True,
+    }, {
         'url': 'https://www.tiktok.com/@iris04201/live',
         'only_matching': True,
     }]
 
+    def _call_api(self, url, param, room_id, uploader, key=None):
+        response = traverse_obj(self._download_json(
+            url, room_id, fatal=False, query={
+                'aid': '1988',
+                param: room_id,
+            }), (key, {dict}), default={})
+
+        # status == 2 if live else 4
+        if int_or_none(response.get('status')) == 2:
+            return response
+        # If room_id is obtained via mobile share URL and cannot be refreshed, do not wait for live
+        elif not uploader:
+            raise ExtractorError('This livestream has ended', expected=True)
+        raise UserNotLive(video_id=uploader)
+
     def _real_extract(self, url):
-        uploader = self._match_id(url)
-        webpage = self._download_webpage(url, uploader, headers={'User-Agent': 'User-Agent:Mozilla/5.0'})
-        room_id = self._html_search_regex(r'snssdk\d*://live\?room_id=(\d+)', webpage, 'room ID', default=None)
+        uploader, room_id = self._match_valid_url(url).group('uploader', 'id')
+        webpage = self._download_webpage(
+            url, uploader or room_id, headers={'User-Agent': 'Mozilla/5.0'}, fatal=not room_id)
+
+        if webpage:
+            data = try_call(lambda: self._get_sigi_state(webpage, uploader or room_id))
+            room_id = (traverse_obj(data, ('UserModule', 'users', ..., 'roomId', {str_or_none}), get_all=False)
+                       or self._search_regex(r'snssdk\d*://live\?room_id=(\d+)', webpage, 'room ID', default=None)
+                       or room_id)
+            uploader = uploader or traverse_obj(
+                data, ('LiveRoom', 'liveRoomUserInfo', 'user', 'uniqueId'),
+                ('UserModule', 'users', ..., 'uniqueId'), get_all=False, expected_type=str)
+
         if not room_id:
             raise UserNotLive(video_id=uploader)
-        live_info = traverse_obj(self._download_json(
-            'https://www.tiktok.com/api/live/detail/', room_id, query={
-                'aid': '1988',
-                'roomID': room_id,
-            }), 'LiveRoomInfo', expected_type=dict, default={})
 
-        if 'status' not in live_info:
-            raise ExtractorError('Unexpected response from TikTok API')
-        # status = 2 if live else 4
-        if not int_or_none(live_info['status']) == 2:
-            raise UserNotLive(video_id=uploader)
+        formats = []
+        live_info = self._call_api(
+            'https://webcast.tiktok.com/webcast/room/info', 'room_id', room_id, uploader, key='data')
+
+        get_quality = qualities(('SD1', 'ld', 'SD2', 'sd', 'HD1', 'hd', 'FULL_HD1', 'uhd', 'ORIGION', 'origin'))
+        parse_inner = lambda x: self._parse_json(x, None)
+
+        for quality, stream in traverse_obj(live_info, (
+                'stream_url', 'live_core_sdk_data', 'pull_data', 'stream_data',
+                {parse_inner}, 'data', {dict}), default={}).items():
+
+            sdk_params = traverse_obj(stream, ('main', 'sdk_params', {parse_inner}, {
+                'vcodec': ('VCodec', {str}),
+                'tbr': ('vbitrate', {lambda x: int_or_none(x, 1000)}),
+                'resolution': ('resolution', {lambda x: re.match(r'(?i)\d+x\d+|\d+p', x).group().lower()}),
+            }))
+
+            flv_url = traverse_obj(stream, ('main', 'flv', {url_or_none}))
+            if flv_url:
+                formats.append({
+                    'url': flv_url,
+                    'ext': 'flv',
+                    'format_id': f'flv-{quality}',
+                    'quality': get_quality(quality),
+                    **sdk_params,
+                })
+
+            hls_url = traverse_obj(stream, ('main', 'hls', {url_or_none}))
+            if hls_url:
+                formats.append({
+                    'url': hls_url,
+                    'ext': 'mp4',
+                    'protocol': 'm3u8_native',
+                    'format_id': f'hls-{quality}',
+                    'quality': get_quality(quality),
+                    **sdk_params,
+                })
+
+        def get_vcodec(*keys):
+            return traverse_obj(live_info, (
+                'stream_url', *keys, {parse_inner}, 'VCodec', {str}))
+
+        for stream in ('hls', 'rtmp'):
+            stream_url = traverse_obj(live_info, ('stream_url', f'{stream}_pull_url', {url_or_none}))
+            if stream_url:
+                formats.append({
+                    'url': stream_url,
+                    'ext': 'mp4' if stream == 'hls' else 'flv',
+                    'protocol': 'm3u8_native' if stream == 'hls' else 'https',
+                    'format_id': f'{stream}-pull',
+                    'vcodec': get_vcodec(f'{stream}_pull_url_params'),
+                    'quality': get_quality('ORIGION'),
+                })
+
+        for f_id, f_url in traverse_obj(live_info, ('stream_url', 'flv_pull_url', {dict}), default={}).items():
+            if not url_or_none(f_url):
+                continue
+            formats.append({
+                'url': f_url,
+                'ext': 'flv',
+                'format_id': f'flv-{f_id}'.lower(),
+                'vcodec': get_vcodec('flv_pull_url_params', f_id),
+                'quality': get_quality(f_id),
+            })
+
+        # If uploader is a guest on another's livestream, primary endpoint will not have m3u8 URLs
+        if not traverse_obj(formats, lambda _, v: v['ext'] == 'mp4'):
+            live_info = merge_dicts(live_info, self._call_api(
+                'https://www.tiktok.com/api/live/detail/', 'roomID', room_id, uploader, key='LiveRoomInfo'))
+            if url_or_none(live_info.get('liveUrl')):
+                formats.append({
+                    'url': live_info['liveUrl'],
+                    'ext': 'mp4',
+                    'protocol': 'm3u8_native',
+                    'format_id': 'hls-fallback',
+                    'vcodec': 'h264',
+                    'quality': get_quality('origin'),
+                })
+
+        uploader = uploader or traverse_obj(live_info, ('ownerInfo', 'uniqueId'), ('owner', 'display_id'))
 
         return {
             'id': room_id,
-            'title': live_info.get('title') or self._html_search_meta(['og:title', 'twitter:title'], webpage, default=''),
             'uploader': uploader,
-            'uploader_id': traverse_obj(live_info, ('ownerInfo', 'id')),
-            'creator': traverse_obj(live_info, ('ownerInfo', 'nickname')),
-            'concurrent_view_count': traverse_obj(live_info, ('liveRoomStats', 'userCount'), expected_type=int),
-            'formats': self._extract_m3u8_formats(live_info['liveUrl'], room_id, 'mp4', live=True),
+            'uploader_url': format_field(uploader, None, self._UPLOADER_URL_FORMAT) or None,
             'is_live': True,
+            'formats': formats,
+            '_format_sort_fields': ('quality', 'ext'),
+            **traverse_obj(live_info, {
+                'title': 'title',
+                'uploader_id': (('ownerInfo', 'owner'), 'id', {str_or_none}),
+                'creator': (('ownerInfo', 'owner'), 'nickname'),
+                'concurrent_view_count': (('user_count', ('liveRoomStats', 'userCount')), {int_or_none}),
+            }, get_all=False),
         }


Test patch: