Repository: yt-dlp/yt-dlp

Issue:
[TwitCasting] Some archive downloads only the first hour of it
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [ ] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.06.01**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.06.01
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
WIP
```
<!--
Do not remove the above ```
-->


## Description/Background
For some TwitCasting archive, it sometimes split the archive 1 hour each.
This especially happens when the video is longer than an hour, with high bitrate.

Here's an example video: https://twitcasting.tv/loft_heaven/movie/685979292
And another one: https://twitcasting.tv/nana_fujisawa/movie/686749195 

With current yt-dlp, it can only download the first split(=hour). But there's more big problem:
1. TC allows downloading only one stream simultaneously. Failing to follow this **causes all playlists to expire.**
    - Some `ffmpeg` hack like `ffmpeg -i STREAM1.m3u8 -i STREAM2.m3u8 -filter_complex "blablabla" -map [v] -map [a] outpur.mp4.part` don't work because of this.
    - It seems we have to request the next playlist after we download last segment of previous playlist.
    - (More to say, it should be realtime. Download breaks sometimes)
1. ~~Current youtube-dl/yt-dlp architecture doesn't support such videos.~~ ... but still not able to concat them.
    - I think it's okay to ask users to download these splits manually, but who likes it?
    - They do not have a proper method to download multiple videos separately and combine later.

I'm also trying to work better with these problems, but I want to hear idea from you.

Thanks.



Discussion:
> TC allows downloading only one stream simultaneously.

But then, how does the browser play the video + audio?
> > TC allows downloading only one stream simultaneously.
> 
> But then, how does the browser play the video + audio?

They do not deliver audio and video separately. On browser, when the first split reaches to the end, player replaced by new split and start from zero seconds.
Seems I misunderstood your issue. If the streams are split in time, the general approach is to implement it as a `multi_video`. There is currently no support for concatting the parts together though. However, something like https://github.com/ytdl-org/youtube-dl/pull/28153 can be implemented for it using the concat demuxer of #360 
> It seems we have to request the next playlist after we download last segment of previous playlist.

This is still an issue though 🤔 
> If the streams are split in time, the general approach is to implement it as a multi_video. 

~~Wow, `multi_video` looks good for such purpose. My bad.~~ no, it's not that different from playlists.

It's better if we're able to complete downloading and joining multiple video as a format. This behavior makes newbies happy.

> > It seems we have to request the next playlist after we download last segment of previous playlist.
> 
> This is still an issue though thinking

This means that we must not download two or more streams in parallel. Don't think it too difficult.

> This means that we must not download two or more streams in parallel. Don't think it too difficult.

Again, I seem to have misunderstood. I thought you have to fetch the second stream's URL after the download of first stream is finished. Due to the strict separation of extraction and downloading in youtube-dl, this is difficult to achieve without creating a custom downloader like for niconico


Yes, we probably need new FD. But to confirm, all m3u8 playlists needed to download all splits are all provided in the webpage. We don't have to make a request for each splits every time. (`video_js_data` contains this)
> Yes, we probably need new FD

Why?
> > Yes, we probably need new FD
> 
> Why?

Oh, maybe not. When `multi_video` supports joining after downloading, we don't need it at all. Sorry.
Does #399 close this?
> Does #399 close this?

No. Unrelated PR.

Resolving patch:
diff --git a/yt_dlp/extractor/twitcasting.py b/yt_dlp/extractor/twitcasting.py
--- a/yt_dlp/extractor/twitcasting.py
+++ b/yt_dlp/extractor/twitcasting.py
@@ -8,6 +8,7 @@
 from ..downloader.websocket import has_websockets
 from ..utils import (
     clean_html,
+    ExtractorError,
     float_or_none,
     get_element_by_class,
     get_element_by_id,
@@ -19,12 +20,15 @@
     unified_timestamp,
     urlencode_postdata,
     urljoin,
-    ExtractorError,
 )
 
 
 class TwitCastingIE(InfoExtractor):
     _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<uploader_id>[^/]+)/(?:movie|twplayer)/(?P<id>\d+)'
+    _M3U8_HEADERS = {
+        'Origin': 'https://twitcasting.tv',
+        'Referer': 'https://twitcasting.tv/',
+    }
     _TESTS = [{
         'url': 'https://twitcasting.tv/ivetesangalo/movie/2357609',
         'md5': '745243cad58c4681dc752490f7540d7f',
@@ -61,6 +65,16 @@ class TwitCastingIE(InfoExtractor):
             'skip_download': True,
             'videopassword': 'abc',
         },
+    }, {
+        'note': 'archive is split in 2 parts',
+        'url': 'https://twitcasting.tv/loft_heaven/movie/685979292',
+        'info_dict': {
+            'id': '685979292',
+            'ext': 'mp4',
+            'title': '南波一海のhear_here “ナタリー望月哲さんに聞く編集と「渋谷系狂騒曲」”',
+            'duration': 6964.599334,
+        },
+        'playlist_mincount': 2,
     }]
 
     def _real_extract(self, url):
@@ -79,63 +93,33 @@ def _real_extract(self, url):
         title = (clean_html(get_element_by_id('movietitle', webpage))
                  or self._html_search_meta(['og:title', 'twitter:title'], webpage, fatal=True))
 
-        video_js_data = {}
-        m3u8_url = self._search_regex(
-            r'data-movie-url=(["\'])(?P<url>(?:(?!\1).)+)\1',
-            webpage, 'm3u8 url', group='url', default=None)
-        if not m3u8_url:
-            video_js_data = self._parse_json(self._search_regex(
-                r'data-movie-playlist=(["\'])(?P<url>(?:(?!\1).)+)',
-                webpage, 'movie playlist', group='url', default='[{}]'), video_id)
-            if isinstance(video_js_data, dict):
-                video_js_data = list(video_js_data.values())[0]
-            video_js_data = video_js_data[0]
-            m3u8_url = try_get(video_js_data, lambda x: x['source']['url'])
-
-        stream_server_data = self._download_json(
-            'https://twitcasting.tv/streamserver.php?target=%s&mode=client' % uploader_id, video_id,
-            'Downloading live info', fatal=False)
-
-        is_live = 'data-status="online"' in webpage
+        video_js_data = try_get(
+            webpage,
+            lambda x: self._parse_json(self._search_regex(
+                r'data-movie-playlist=\'([^\']+?)\'',
+                x, 'movie playlist', default=None), video_id)['2'], list)
 
-        if not traverse_obj(stream_server_data, 'llfmp4') and is_live:
-            self.raise_login_required(method='cookies')
-
-        formats = []
-        if is_live and not m3u8_url:
-            m3u8_url = 'https://twitcasting.tv/%s/metastream.m3u8' % uploader_id
-        if is_live and has_websockets and stream_server_data:
-            qq = qualities(['base', 'mobilesource', 'main'])
-            streams = traverse_obj(stream_server_data, ('llfmp4', 'streams')) or {}
-            for mode, ws_url in streams.items():
-                formats.append({
-                    'url': ws_url,
-                    'format_id': 'ws-%s' % mode,
-                    'ext': 'mp4',
-                    'quality': qq(mode),
-                    'protocol': 'websocket_frag',  # TwitCasting simply sends moof atom directly over WS
-                })
-
-        thumbnail = video_js_data.get('thumbnailUrl') or self._og_search_thumbnail(webpage)
+        thumbnail = traverse_obj(video_js_data, (0, 'thumbnailUrl')) or self._og_search_thumbnail(webpage)
         description = clean_html(get_element_by_id(
             'authorcomment', webpage)) or self._html_search_meta(
             ['description', 'og:description', 'twitter:description'], webpage)
-        duration = float_or_none(video_js_data.get(
-            'duration'), 1000) or parse_duration(clean_html(
-                get_element_by_class('tw-player-duration-time', webpage)))
+        duration = (try_get(video_js_data, lambda x: sum(float_or_none(y.get('duration')) for y in x) / 1000)
+                    or parse_duration(clean_html(get_element_by_class('tw-player-duration-time', webpage))))
         view_count = str_to_int(self._search_regex(
-            r'Total\s*:\s*([\d,]+)\s*Views', webpage, 'views', None))
+            (r'Total\s*:\s*([\d,]+)\s*Views', r'総視聴者\s*:\s*([\d,]+)\s*</'), webpage, 'views', None))
         timestamp = unified_timestamp(self._search_regex(
             r'data-toggle="true"[^>]+datetime="([^"]+)"',
             webpage, 'datetime', None))
 
-        if m3u8_url:
-            formats.extend(self._extract_m3u8_formats(
-                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', live=is_live))
-        self._sort_formats(formats)
+        stream_server_data = self._download_json(
+            'https://twitcasting.tv/streamserver.php?target=%s&mode=client' % uploader_id, video_id,
+            'Downloading live info', fatal=False)
 
-        return {
-            'id': video_id,
+        is_live = 'data-status="online"' in webpage
+        if not traverse_obj(stream_server_data, 'llfmp4') and is_live:
+            self.raise_login_required(method='cookies')
+
+        base_dict = {
             'title': title,
             'description': description,
             'thumbnail': thumbnail,
@@ -143,10 +127,73 @@ def _real_extract(self, url):
             'uploader_id': uploader_id,
             'duration': duration,
             'view_count': view_count,
-            'formats': formats,
             'is_live': is_live,
         }
 
+        def find_dmu(x):
+            data_movie_url = self._search_regex(
+                r'data-movie-url=(["\'])(?P<url>(?:(?!\1).)+)\1',
+                x, 'm3u8 url', group='url', default=None)
+            if data_movie_url:
+                return [data_movie_url]
+
+        m3u8_urls = (try_get(webpage, find_dmu, list)
+                     or traverse_obj(video_js_data, (..., 'source', 'url'))
+                     or ([f'https://twitcasting.tv/{uploader_id}/metastream.m3u8'] if is_live else None))
+        if not m3u8_urls:
+            raise ExtractorError('Failed to get m3u8 playlist')
+
+        if is_live:
+            m3u8_url = m3u8_urls[0]
+            formats = self._extract_m3u8_formats(
+                m3u8_url, video_id, ext='mp4', m3u8_id='hls',
+                live=True, headers=self._M3U8_HEADERS)
+
+            formats.extend(self._extract_m3u8_formats(
+                m3u8_url, video_id, ext='mp4', m3u8_id='source',
+                live=True, query={'mode': 'source'},
+                note='Downloading source quality m3u8',
+                headers=self._M3U8_HEADERS, fatal=False))
+
+            if has_websockets:
+                qq = qualities(['base', 'mobilesource', 'main'])
+                streams = traverse_obj(stream_server_data, ('llfmp4', 'streams')) or {}
+                for mode, ws_url in streams.items():
+                    formats.append({
+                        'url': ws_url,
+                        'format_id': 'ws-%s' % mode,
+                        'ext': 'mp4',
+                        'quality': qq(mode),
+                        # TwitCasting simply sends moof atom directly over WS
+                        'protocol': 'websocket_frag',
+                    })
+
+            self._sort_formats(formats)
+
+            infodict = {
+                'formats': formats
+            }
+        else:
+            infodict = {
+                '_type': 'multi_video',
+                'entries': [{
+                    'id': f'{video_id}-{num}',
+                    'url': m3u8_url,
+                    'ext': 'mp4',
+                    # Requesting the manifests here will cause download to fail.
+                    # So use ffmpeg instead. See: https://github.com/yt-dlp/yt-dlp/issues/382
+                    'protocol': 'm3u8',
+                    'http_headers': self._M3U8_HEADERS,
+                    **base_dict,
+                } for (num, m3u8_url) in enumerate(m3u8_urls)],
+            }
+
+        return {
+            'id': video_id,
+            **base_dict,
+            **infodict,
+        }
+
 
 class TwitCastingLiveIE(InfoExtractor):
     _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<id>[^/]+)/?(?:[#?]|$)'


Test patch: