Repository: yt-dlp/yt-dlp

Issue:
Fix Picarto.tv video downloads (not stream)
### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

USA

### Description

When trying to download [this stream recording that is going to be auto-deleted soon](https://picarto.tv/DarkCookie/videos/616575) I found that yt-dlp and JDownloader2 both can't fetch the stream. I found issue #1373 but it's a little old and the error is different, plus I've figured out roughly how to fix this, so I'm opening this new issue to put all the info in one place.

The actual player data is hiding behind a blob.
![image](https://user-images.githubusercontent.com/1005586/156415138-854c9c7b-fcd2-424a-a6b3-ebcd2ab2dc4c.png)

It loads the video info through a Websocket.
![image](https://user-images.githubusercontent.com/1005586/156415351-79d58355-f9d1-40ec-9542-0329f1ddb8fa.png)

Swap wss:// to https:// and you get playlist info, BUT the playlists only contain a reference to ANOTHER playlist.
![image](https://user-images.githubusercontent.com/1005586/156415667-c4cc7ef3-f3bc-4b21-974b-1a648a908c41.png)

This is what's inside the HLS `index.m3u8` file:
![image](https://user-images.githubusercontent.com/1005586/156417119-c8bf19da-6381-4819-99e2-bc44c437edd0.png)

If you use the relative URL in that file, you'll get the ACTUAL `m3u8` playlist for the MPEG-TS segments.
![image](https://user-images.githubusercontent.com/1005586/156417367-b3a375a9-ef52-4728-af4e-9ea761135efd.png)

From there, it's a straightforward "download all segments and concat with ffmpeg" strategy. I'm fetching it with `wget` as I type this, and I've verified the segments are playable.

### Verbose log

```shell
FYI: I rename to youtube-dl.exe so I don't have to change my scripts; it's actually yt-dlp latest master.

X:\>youtube-dl --throttled-rate 20k --match-filter "!is_live & !live" --cookies "Y:\cookies.txt" --add-metadata --write-description --write-info-json --write-thumbnail --write-subs --write-auto-subs --sub-lang en,en_US --convert-subs srt -ciw -o "%(title)s.%(ext)s" -v https://picarto.tv/DarkCookie/videos/616575
[debug] Command-line config: ['--throttled-rate', '20k', '--match-filter', '!is_live & !live', '--cookies', 'Y:\\cookies.txt', '--add-metadata', '--write-description', '--write-info-json', '--write-thumbnail', '--write-subs', '--write-auto-subs', '--sub-lang', 'en,en_US', '--convert-subs', 'srt', '-ciw', '-o', '%(title)s.%(ext)s', '-v', 'https://picarto.tv/DarkCookie/videos/616575']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (win_exe)
[debug] Lazy loading extractors is disabled
[debug] Python version 3.9.0 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 2021-01-12-git-ca21cb1e36-full_build-www.gyan.dev, ffprobe 4.3.1-2020-11-19-full_build-www.gyan.dev
[debug] Optional libraries: Crypto, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [Picarto] Extracting URL: https://picarto.tv/DarkCookie/videos/616575
[Picarto] DarkCookie: Downloading JSON metadata
ERROR: [Picarto] DarkCookie: Stream is offline
  File "yt_dlp\extractor\common.py", line 616, in extract
  File "yt_dlp\extractor\picarto.py", line 50, in _real_extract
```



Discussion:
As a workaround, once u have the m3u8 url, you can just pass it to yt-dlp. No need to manually download and concat
```
❯ yt-dlp "https://recording-eu-1.picarto.tv/stream/hls/govod%2bgolive+DarkCookie_2022.01.13.20.05.50_nsfw.mkv/index.m3u8?video=maxbps"
[generic] index: Requesting header
[generic] index: Downloading m3u8 information
[info] index: Downloading 1 format(s): 2128
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3684
[download] Destination: index [index].mp4
[download]   0.1% of ~978.54MiB at 50.17KiB/s ETA 06:22:47 (frag 1/3684)
ERROR: Interrupted by user
```
> As a workaround, once u have the m3u8 url, you can just pass it to yt-dlp. No need to manually download and concat

Sometimes I like to see things through the hard way...just to make sure ;-)

Something to add: I've discovered that when a live stream is broken (stops due to streamer internet disconnection etc.), the video (recording) segments end prematurely, with the m3u8 handing out segments that don't exist on the server. If this gets added, it'll need to handle a consistently failed segment (usually in the 90th percentile completion) as the end of the video rather than an error.

Resolving patch:
diff --git a/yt_dlp/extractor/picarto.py b/yt_dlp/extractor/picarto.py
--- a/yt_dlp/extractor/picarto.py
+++ b/yt_dlp/extractor/picarto.py
@@ -1,7 +1,10 @@
+import urllib.parse
+
 from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
-    js_to_json,
+    str_or_none,
+    traverse_obj,
 )
 
 
@@ -84,7 +87,7 @@ def _real_extract(self, url):
 
 
 class PicartoVodIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www.)?picarto\.tv/videopopout/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?picarto\.tv/(?:videopopout|\w+/videos)/(?P<id>[^/?#&]+)'
     _TESTS = [{
         'url': 'https://picarto.tv/videopopout/ArtofZod_2017.12.12.00.13.23.flv',
         'md5': '3ab45ba4352c52ee841a28fb73f2d9ca',
@@ -94,6 +97,18 @@ class PicartoVodIE(InfoExtractor):
             'title': 'ArtofZod_2017.12.12.00.13.23.flv',
             'thumbnail': r're:^https?://.*\.jpg'
         },
+        'skip': 'The VOD does not exist',
+    }, {
+        'url': 'https://picarto.tv/ArtofZod/videos/772650',
+        'md5': '00067a0889f1f6869cc512e3e79c521b',
+        'info_dict': {
+            'id': '772650',
+            'ext': 'mp4',
+            'title': 'Art of Zod - Drawing and Painting',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'channel': 'ArtofZod',
+            'age_limit': 18,
+        }
     }, {
         'url': 'https://picarto.tv/videopopout/Plague',
         'only_matching': True,
@@ -102,21 +117,36 @@ class PicartoVodIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, video_id)
-
-        vod_info = self._parse_json(
-            self._search_regex(
-                r'(?s)#vod-player["\']\s*,\s*(\{.+?\})\s*\)', webpage,
-                'vod player'),
-            video_id, transform_source=js_to_json)
+        data = self._download_json(
+            'https://ptvintern.picarto.tv/ptvapi', video_id, query={
+                'query': f'''{{
+  video(id: "{video_id}") {{
+    id
+    title
+    adult
+    file_name
+    video_recording_image_url
+    channel {{
+      name
+    }}
+  }}
+}}'''
+            })['data']['video']
+
+        file_name = data['file_name']
+        netloc = urllib.parse.urlparse(data['video_recording_image_url']).netloc
 
         formats = self._extract_m3u8_formats(
-            vod_info['vod'], video_id, 'mp4', entry_protocol='m3u8_native',
-            m3u8_id='hls')
+            f'https://{netloc}/stream/hls/{file_name}/index.m3u8', video_id, 'mp4', m3u8_id='hls')
 
         return {
             'id': video_id,
-            'title': video_id,
-            'thumbnail': vod_info.get('vodThumb'),
+            **traverse_obj(data, {
+                'id': ('id', {str_or_none}),
+                'title': ('title', {str}),
+                'thumbnail': 'video_recording_image_url',
+                'channel': ('channel', 'name', {str}),
+                'age_limit': ('adult', {lambda x: 18 if x else 0}),
+            }),
             'formats': formats,
         }


Test patch: