Repository: yt-dlp/yt-dlp

Issue:
NHK World HTTP Error 503: Service Unavailable
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that a **supported** site is broken
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

United States and most likely worldwide

### Provide a description that is worded well enough to be understood

Document 72 Hours is a weekly show. Two weeks ago (last week's show was a re-run) I successfully downloaded the then-latest show (https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026193/). Attempting to download the show today results in a 503 error. The same goes for the latest episode as well as a different show I tried for testing.

Episode that worked two weeks ago, April 17, 2023: https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026193/
Latest episode released May 2, 2023: https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026194/
Different show from the same site: https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2099001/

All attempted downloads result in 503 error.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026194/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a9421
yt-dlp is up to date (stable@2023.03.04)
[NhkVod] Extracting URL: https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026194/
[NhkVod] 4026-194: Downloading JSON metadata
[Piksel] Extracting URL: https://player.piksel.com/v/refid/nhkworld/prefid/nw_vod_v_en_4026_194_20230502113000_02_1683004957
[Piksel] nw_vod_v_en_4026_194_20230502113000_02_1683004957: Downloading webpage
ERROR: [Piksel] nw_vod_v_en_4026_194_20230502113000_02_1683004957: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: 'Service Unavailable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 694, in extract
  File "yt_dlp\extractor\piksel.py", line 79, in _real_extract
  File "yt_dlp\extractor\common.py", line 1097, in _download_webpage
  File "yt_dlp\extractor\common.py", line 1048, in download_content
  File "yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 839, in _request_webpage

  File "yt_dlp\extractor\common.py", line 821, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 3742, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 503: Service Unavailable
```



Discussion:
Looks like NHK uses their own API for video now instead of the Piksel API.

---
### Notes for devs:

Let's take this URL: https://www3.nhk.or.jp/nhkworld/en/ondemand/video/4026193/ 

The `NhkVod` extractor actually works up until it returns a url to the `Piksel` extractor. So we need to add code to handle the rest of the extraction within nhk.py

Instead of making calls to the Piksel API, the webpage first requests this javascript/html page:
https://movie-s.nhk.or.jp/v/refid/nhkworld/prefid/nw_vod_v_en_4026_193_20230418113000_01_1681787304?embed=js&targetId=videoplayer&de-responsive=true&de-callback-method=nwCustomCallback&de-appid=nw_vod_v_en_4026_193_20230418113000_01_1681787304&de-subtitle-on=false

In the extractor that could be done like this:
```py
        if is_video:
            vod_id = episode['vod_id']
            js_page = self._download_webpage(
                f'https://movie-s.nhk.or.jp/v/refid/nhkworld/prefid/{vod_id}', vod_id,
                'Downloading JS info', query={
                    'embed': 'js',
                    'targetId': 'videoplayer',
                    'de-responsive': True,
                    'de-callback-method': 'nwCustomCallback',
                    'de-appid': vod_id,
                    'de-subtitle-on': False,
                })
```

From that javascript, we need to extract the `data-de-api-key` and `data-de-program-uuid` values:
```js
newVideo.setAttribute('data-de-api-key','67f5b750-b419-11e9-8a16-0e45e8988f42');
newVideo.setAttribute('data-de-program-uuid','dr646i6d');
```
Then we can call this API endpoint, using the `data-de-api-key` value in the URL path and the `data-de-program-uuid` value in the query, which has all the metadata we need:
https://movie-s.nhk.or.jp/ws/ws_program/api/67f5b750-b419-11e9-8a16-0e45e8988f42/apiv/5/mode/json?v=dr646i6d


You're on the wrong track pal. They just switched to self-hosting.

https://github.com/ytdl-org/youtube-dl/pull/32213
@menschel thanks, pal
I just hacked the patch from @menschel into a clone of the yt-dlp repo, and I can confirm that this fixes the problem with downloading video from NHK. If you want, I can fork it into my own repo and submit a PR...
https://github.com/yt-dlp/yt-dlp/issues/4202#issuecomment-1167302440
> I just hacked the patch from @menschel into a clone of the yt-dlp repo, and I can confirm that this fixes the problem with downloading video from NHK. If you want, I can fork it into my own repo and submit a PR...

Comment referenced by pukkandan (yt-dlp member) seems to imply you should proceed with submitting a PR. Hope you will do so and we can see the fix merged into nightly soon. TIA
Some may not be aware but NHK World only leaves most videos on their site for usually up to a month before they are removed. Because of this unfortunate reality, there are many people desperately awaiting a fix so we can d/l videos which are important to us before they disappear and can no longer be streamed. Imagine you have an autistic child that becomes attached to a video and watches it every day... but, if that routine is disrupted because it's no longer available, they will become distraught. If someone could PLEASE move this forward it would be sincerely appreciated. Thank you!

@bashonly ping
@sjthespian ping
> Some may not be aware but NHK World only leaves most videos on their site for usually up to a month before they are removed. Because of this unfortunate reality, there are many people desperately awaiting a fix so we can d/l videos which are important to us before they disappear and can no longer be streamed. Imagine you have an autistic child that becomes attached to a video and watches it every day... but, if that routine is disrupted because it's no longer available, they will become distraught. If someone could PLEASE move this forward it would be sincerely appreciated.

use 
`pip install git+https://github.com/menschel/youtube-dl.git@dev_nhk`
and then download as usual with youtube-dl

I don't intend to push a PR to yt-dlp ever again.

Resolving patch:
diff --git a/yt_dlp/extractor/nhk.py b/yt_dlp/extractor/nhk.py
--- a/yt_dlp/extractor/nhk.py
+++ b/yt_dlp/extractor/nhk.py
@@ -67,7 +67,7 @@ def get_clean_field(key):
             info.update({
                 '_type': 'url_transparent',
                 'ie_key': 'Piksel',
-                'url': 'https://player.piksel.com/v/refid/nhkworld/prefid/' + vod_id,
+                'url': 'https://movie-s.nhk.or.jp/v/refid/nhkworld/prefid/' + vod_id,
                 'id': vod_id,
             })
         else:
@@ -94,6 +94,19 @@ class NhkVodIE(NhkBaseIE):
     # Content available only for a limited period of time. Visit
     # https://www3.nhk.or.jp/nhkworld/en/ondemand/ for working samples.
     _TESTS = [{
+        'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2061601/',
+        'info_dict': {
+            'id': 'yd8322ch',
+            'ext': 'mp4',
+            'description': 'md5:109c8b05d67a62d0592f2b445d2cd898',
+            'title': 'GRAND SUMO Highlights - [Recap] May Tournament Day 1 (Opening Day)',
+            'upload_date': '20230514',
+            'timestamp': 1684083791,
+            'series': 'GRAND SUMO Highlights',
+            'episode': '[Recap] May Tournament Day 1 (Opening Day)',
+            'thumbnail': 'https://mz-edge.stream.co.jp/thumbs/aid/t1684084443/4028649.jpg?w=1920&h=1080',
+        },
+    }, {
         # video clip
         'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/9999011/',
         'md5': '7a90abcfe610ec22a6bfe15bd46b30ca',
@@ -104,6 +117,9 @@ class NhkVodIE(NhkBaseIE):
             'description': 'md5:5aee4a9f9d81c26281862382103b0ea5',
             'timestamp': 1565965194,
             'upload_date': '20190816',
+            'thumbnail': 'https://mz-edge.stream.co.jp/thumbs/aid/t1567086278/3715195.jpg?w=1920&h=1080',
+            'series': 'Dining with the Chef',
+            'episode': 'Chef Saito\'s Family recipe: MENCHI-KATSU',
         },
     }, {
         # audio clip
@@ -114,10 +130,7 @@ class NhkVodIE(NhkBaseIE):
             'title': "Japan's Top Inventions - Miniature Video Cameras",
             'description': 'md5:07ea722bdbbb4936fdd360b6a480c25b',
         },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
+        'skip': '404 Not Found',
     }, {
         'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2015173/',
         'only_matching': True,
@@ -133,7 +146,6 @@ class NhkVodIE(NhkBaseIE):
     }, {
         # video, alphabetic character in ID #29670
         'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/9999a34/',
-        'only_matching': True,
         'info_dict': {
             'id': 'qfjay6cg',
             'ext': 'mp4',
@@ -142,7 +154,8 @@ class NhkVodIE(NhkBaseIE):
             'thumbnail': r're:^https?:/(/[a-z0-9.-]+)+\.jpg\?w=1920&h=1080$',
             'upload_date': '20210615',
             'timestamp': 1623722008,
-        }
+        },
+        'skip': '404 Not Found',
     }]
 
     def _real_extract(self, url):
@@ -153,12 +166,19 @@ class NhkVodProgramIE(NhkBaseIE):
     _VALID_URL = r'%s/program%s(?P<id>[0-9a-z]+)(?:.+?\btype=(?P<episode_type>clip|(?:radio|tv)Episode))?' % (NhkBaseIE._BASE_URL_REGEX, NhkBaseIE._TYPE_REGEX)
     _TESTS = [{
         # video program episodes
+        'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/program/video/sumo',
+        'info_dict': {
+            'id': 'sumo',
+            'title': 'GRAND SUMO Highlights',
+        },
+        'playlist_mincount': 12,
+    }, {
         'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/program/video/japanrailway',
         'info_dict': {
             'id': 'japanrailway',
             'title': 'Japan Railway Journal',
         },
-        'playlist_mincount': 1,
+        'playlist_mincount': 12,
     }, {
         # video program clips
         'url': 'https://www3.nhk.or.jp/nhkworld/en/ondemand/program/video/japanrailway/?type=clip',
diff --git a/yt_dlp/extractor/piksel.py b/yt_dlp/extractor/piksel.py
--- a/yt_dlp/extractor/piksel.py
+++ b/yt_dlp/extractor/piksel.py
@@ -7,8 +7,10 @@
     int_or_none,
     join_nonempty,
     parse_iso8601,
+    traverse_obj,
     try_get,
     unescapeHTML,
+    urljoin,
 )
 
 
@@ -63,11 +65,11 @@ class PikselIE(InfoExtractor):
         }
     ]
 
-    def _call_api(self, app_token, resource, display_id, query, fatal=True):
-        response = (self._download_json(
-            'http://player.piksel.com/ws/ws_%s/api/%s/mode/json/apiv/5' % (resource, app_token),
-            display_id, query=query, fatal=fatal) or {}).get('response')
-        failure = try_get(response, lambda x: x['failure']['reason'])
+    def _call_api(self, app_token, resource, display_id, query, host='https://player.piksel.com', fatal=True):
+        url = urljoin(host, f'/ws/ws_{resource}/api/{app_token}/mode/json/apiv/5')
+        response = traverse_obj(
+            self._download_json(url, display_id, query=query, fatal=fatal), ('response', {dict})) or {}
+        failure = traverse_obj(response, ('failure', 'reason')) if response else 'Empty response from API'
         if failure:
             if fatal:
                 raise ExtractorError(failure, expected=True)
@@ -83,7 +85,7 @@ def _real_extract(self, url):
         ], webpage, 'app token')
         query = {'refid': ref_id, 'prefid': display_id} if ref_id else {'v': display_id}
         program = self._call_api(
-            app_token, 'program', display_id, query)['WsProgramResponse']['program']
+            app_token, 'program', display_id, query, url)['WsProgramResponse']['program']
         video_id = program['uuid']
         video_data = program['asset']
         title = video_data['title']
@@ -129,7 +131,7 @@ def process_asset_files(asset_files):
                 process_asset_files(try_get(self._call_api(
                     app_token, 'asset_file', display_id, {
                         'assetid': asset_id,
-                    }, False), lambda x: x['WsAssetFileResponse']['AssetFiles']))
+                    }, url, False), lambda x: x['WsAssetFileResponse']['AssetFiles']))
 
         m3u8_url = dict_get(video_data, [
             'm3u8iPadURL',


Test patch: