Repository: yt-dlp/yt-dlp

Issue:
[mildom] Some mildom url formats are not supported
### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.01.21**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [x] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Any

### Description

While I was downloading videos from the Japanese streaming site mildom.com, I came across a few that refused to download with an extractor error.

Going through the whole playlist, it looks like some links have a video ID in the format:
https://www.mildom.com/playback/10882672/10882672-bu7um3rl5h2rpsddhmf0
some are in the format:
https://www.mildom.com/playback/10882672/10882672-1597662269
but a few (where I was getting the error) are formatted with three number blocks, like so:
https://www.mildom.com/playback/10882672/10882672-1597758589870-477
or
https://www.mildom.com/playback/10882672/10882672-1597845729830-1886

I noticed in the debug log that the ID was missing the third block of numbers. This hinted at the ID getting truncated somewhere in the extractor module, leading to a bad download URL and content the extractor could not parse.

So I tried fiddling a bit with the python code for the mildom extractor. I made a quick and dirty change to the URL validator in `class MildomVodIE(MildomBaseIE)` from
`_VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+)'`
to
`_VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9\-]+)'`
and video was able to download successfully.

Please, note that (dirty regex aside) this is not a complete fix. For instance, if I enable the download archive option, the video ID is still truncated before the third number group _somewhere else in the code_ when the download log is generated, and yt-dlp will happily keep trying to download the video instead of skipping it. So some additional changes might be required.

### Verbose log

```shell
[debug] Command-line config: ['-vU', 'https://www.mildom.com/playback/10882672/10882672-1597758589870-477']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, err UTF-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0]
[debug] Python version 3.6.9 (CPython 64bit) - Linux-4.4.0-43-Microsoft-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8, phantomjs .
[debug] Optional libraries: Cryptodome, secretstorage, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.01.21, Current version: 2022.01.21
yt-dlp is up to date (2022.01.21)
[debug] [mildom:vod] Extracting URL: https://www.mildom.com/playback/10882672/10882672-1597758589870-477
[mildom:vod] 10882672-1597758589870: Downloading webpage
[mildom:vod] initialization: Downloading dispatcher_config
[mildom:vod] initialization: Downloading guest token
[mildom:vod] 10882672-1597758589870: Downloading playback metadata
ERROR: 10882672-1597758589870: An extractor error has occurred. (caused by KeyError('body',)); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/mildom.py", line 173, in _real_extract
    'v_id': video_id,
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/mildom.py", line 27, in _call_api
    return self._download_json(url, video_id, note=note)['body']
KeyError: 'body'
Traceback (most recent call last):
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/mildom.py", line 173, in _real_extract
    'v_id': video_id,
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/mildom.py", line 27, in _call_api
    return self._download_json(url, video_id, note=note)['body']
KeyError: 'body'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/YoutubeDL.py", line 1367, in wrapper
    return func(self, *args, **kwargs)
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/YoutubeDL.py", line 1437, in __extract_info
    ie_result = ie.extract(url)
  File "/home/XXXXXXX/.local/lib/python3.6/site-packages/yt_dlp/extractor/common.py", line 642, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e, video_id=self.get_temp_id(url))
yt_dlp.utils.ExtractorError: 10882672-1597758589870: An extractor error has occurred. (caused by KeyError('body',)); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
```



Discussion:
Also, I noticed that the site support was pretty barebones, which is understandable since it is practically unknown out of Japan. So, since I was already doing some research on the site to debug the extractor, I set out to try improving it a little bit.

If you think it is better to split this off into a new site feature request, please feel free to do so (or to ask me to do it myself).

First of all, it looks like the cloudac.mildom.com addresses containing the channel and video metadata jsons are quite finicky with the referer, unless an extra `__platform=web` is added to the URL (I got the hint from https://github.com/NDSLib/mildom_dl).

   This link will show a "page not found" if you try to access it directly through the browser, wget or a normal python request:
   https://cloudac.mildom.com/nonolive/videocontent/playback/getPlaybackDetail?v_id=10882672-1597758589870-477
`{"code":1403,"message":"サインエラー"}`
   while this works from anywhere and gets you the whole json metadata:
   https://cloudac.mildom.com/nonolive/videocontent/playback/getPlaybackDetail?v_id=10882672-1597758589870-477&__platform=web
  And this is a link to a user's profile, also working normally with the extra parameter:
https://cloudac.mildom.com/nonolive/videocontent/profile/playbackList?user_id=10882672&page=1&limit=30&__platform=web

It might be interesting to add this parameter to the url generation throughout the extractor, just in case.

The second improvement I tried, once I found out how to access the json for the videos reliably, was to parse the rest of the metadata. The extractor initially parses id, title, description, uploader, uploader_id and formats, but it was pretty easy to add more. As a test (and because it was the minimum I needed for archiving videos) I added timestamp, duration and thumbnails to `class MildomVodIE(MildomBaseIE)`:

```
        timestamp = try_get(autoplay['publish_time'], compat_numeric_types)

        duration = try_get(autoplay['video_length'], compat_numeric_types)

        thumbnails=[]
        thumbnail_url=try_get(autoplay['upload_pic'], compat_str)
        thumbnails.append({
                'url': thumbnail_url,
            })

        return {
            'id': video_id,
            'title': title,
            'description': description,
            'timestamp': timestamp//1000,
            'duration': duration//1000,
            'thumbnails': thumbnails,
            'uploader': uploader,
            'uploader_id': user_id,
            'formats': formats,
        }
```
(I also needed to import compat_numeric_types to handle the timestamps and duration). The json does not have that much more information, but someone with more experience in how youtube-dl/yt-dlp handles metadata should be able to complete the extractor easily.

A further improvement I did not try (but which would be nice) is handling errors a bit more gracefully. On error the site returns a json that could be parsed, instead of raising an exception when the normal video metadata is not found. An incorrect video ID will return:
https://cloudac.mildom.com/nonolive/videocontent/playback/getPlaybackDetail?v_id=10882672-1597758589870&__platform=web
`{"code":35100,"message":"この動画は存在しません"}` (note that the message is "video does not exist", a bit different from the previous "page not found")
The same code appears when the video requires a paid subscription. I have not found a way to tell them apart, but I can leave a link to experiment:
https://www.mildom.com/playback/10882672/10882672-c4kfrmlaks9dkdqql4og

This is all the low-hanging fruit I have been able to find for now, I hope it is useful.

Cheers
Please make a PR if possible
If I find out what else is truncating the video IDs, I can give it a try.

Resolving patch:
diff --git a/yt_dlp/extractor/mildom.py b/yt_dlp/extractor/mildom.py
--- a/yt_dlp/extractor/mildom.py
+++ b/yt_dlp/extractor/mildom.py
@@ -12,6 +12,8 @@
     update_url_query,
     random_uuidv4,
     try_get,
+    float_or_none,
+    dict_get
 )
 from ..compat import (
     compat_str,
@@ -22,9 +24,18 @@ class MildomBaseIE(InfoExtractor):
     _GUEST_ID = None
     _DISPATCHER_CONFIG = None
 
-    def _call_api(self, url, video_id, query={}, note='Downloading JSON metadata', init=False):
+    def _call_api(self, url, video_id, query=None, note='Downloading JSON metadata', init=False):
+        query = query or {}
+        if query:
+            query['__platform'] = 'web'
         url = update_url_query(url, self._common_queries(query, init=init))
-        return self._download_json(url, video_id, note=note)['body']
+        content = self._download_json(url, video_id, note=note)
+        if content['code'] == 0:
+            return content['body']
+        else:
+            self.raise_no_formats(
+                f'Video not found or premium content. {content["code"]} - {content["message"]}',
+                expected=True)
 
     def _common_queries(self, query={}, init=False):
         dc = self._fetch_dispatcher_config()
@@ -148,6 +159,7 @@ def _real_extract(self, url):
             'id': result_video_id,
             'title': title,
             'description': description,
+            'timestamp': float_or_none(enterstudio.get('live_start_ms'), scale=1000),
             'uploader': uploader,
             'uploader_id': video_id,
             'formats': formats,
@@ -158,7 +170,50 @@ def _real_extract(self, url):
 class MildomVodIE(MildomBaseIE):
     IE_NAME = 'mildom:vod'
     IE_DESC = 'Download a VOD in Mildom'
-    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+)'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+-?[0-9]*)'
+    _TESTS = [{
+        'url': 'https://www.mildom.com/playback/10882672/10882672-1597662269',
+        'info_dict': {
+            'id': '10882672-1597662269',
+            'ext': 'mp4',
+            'title': '始めてのミルダム配信じゃぃ！',
+            'thumbnail': r're:^https?://.*\.(png|jpg)$',
+            'upload_date': '20200817',
+            'duration': 4138.37,
+            'description': 'ゲームをしたくて！',
+            'timestamp': 1597662269.0,
+            'uploader_id': '10882672',
+            'uploader': 'kson組長(けいそん)',
+        },
+    }, {
+        'url': 'https://www.mildom.com/playback/10882672/10882672-1597758589870-477',
+        'info_dict': {
+            'id': '10882672-1597758589870-477',
+            'ext': 'mp4',
+            'title': '【kson】感染メイズ！麻酔銃で無双する',
+            'thumbnail': r're:^https?://.*\.(png|jpg)$',
+            'timestamp': 1597759093.0,
+            'uploader': 'kson組長(けいそん)',
+            'duration': 4302.58,
+            'uploader_id': '10882672',
+            'description': 'このステージ絶対乗り越えたい',
+            'upload_date': '20200818',
+        },
+    }, {
+        'url': 'https://www.mildom.com/playback/10882672/10882672-buha9td2lrn97fk2jme0',
+        'info_dict': {
+            'id': '10882672-buha9td2lrn97fk2jme0',
+            'ext': 'mp4',
+            'title': '【kson組長】CART RACER!!!',
+            'thumbnail': r're:^https?://.*\.(png|jpg)$',
+            'uploader_id': '10882672',
+            'uploader': 'kson組長(けいそん)',
+            'upload_date': '20201104',
+            'timestamp': 1604494797.0,
+            'duration': 4657.25,
+            'description': 'WTF',
+        },
+    }]
 
     def _real_extract(self, url):
         m = self._match_valid_url(url)
@@ -213,6 +268,9 @@ def _real_extract(self, url):
             'id': video_id,
             'title': title,
             'description': description,
+            'timestamp': float_or_none(autoplay['publish_time'], scale=1000),
+            'duration': float_or_none(autoplay['video_length'], scale=1000),
+            'thumbnail': dict_get(autoplay, ('upload_pic', 'video_pic')),
             'uploader': uploader,
             'uploader_id': user_id,
             'formats': formats,
@@ -230,6 +288,13 @@ class MildomUserVodIE(MildomBaseIE):
             'title': 'Uploads from ねこばたけ',
         },
         'playlist_mincount': 351,
+    }, {
+        'url': 'https://www.mildom.com/profile/10882672',
+        'info_dict': {
+            'id': '10882672',
+            'title': 'Uploads from kson組長(けいそん)',
+        },
+        'playlist_mincount': 191,
     }]
 
     def _entries(self, user_id):


Test patch: