Repository: yt-dlp/yt-dlp

Issue:
[Nebula] Extractor failed to obtain "id"
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that a **supported** site is broken
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

UK

### Provide a description that is worded well enough to be understood

I can still download older videos from Nebula, but recent videos from https://nebula.tv/videos/nothingbuttech-i-cloned-my-voice-to-expose-ai-drake on give an 'extractor failed to obtain "id"' error.

Please let me know if you require further information/login details.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '-f', 'bestvideo[height<=1440]+bestaudio/best[height<=1440]', '-o', 'H:/Videos/%(title)s-%(height)s.%(ext)s', '--embed-metadata', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.tv/videos/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back']
[debug] Encodings: locale cp932, fs utf-8, pref cp932, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (pip)
[debug] Python 3.10.0 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1l  24 Aug 2021)
[debug] exe versions: ffmpeg 5.1-essentials_build-www.gyan.dev (setts), ffprobe 5.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[Nebula] Logging in to Nebula with supplied credentials
[Nebula] Authorizing to Nebula
[Nebula] Extracting URL: https://nebula.tv/videos/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Fetching video meta data
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Fetching video stream info
[Nebula] rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back: Downloading m3u8 information
ERROR: [Nebula] Extractor failed to obtain "id"; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
Traceback (most recent call last):
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1518, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1615, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1674, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "C:\Users\mrhut\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2495, in process_video_result
    raise ExtractorError('Extractor failed to obtain "id"', ie=info_dict['extractor'])
yt_dlp.utils.ExtractorError: [Nebula] Extractor failed to obtain "id"; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```



Discussion:
If my understanding of nebula.py is correct (and it likely isn't), the issue is that that video's `zype_id` (fetched from https://content.watchnebula.com/video/rmtransit-how-we-used-to-build-subways-fast-and-why-we-should-go-back/) is `null`
~~cc @hheimbuerger~~

~~Or, if they are unavailable,~~ you can share account detail with me:
Contact: discord (pukkandan#4207) / reddit (/u/pukkandan) / email (pukkandan.ytdlp@gmail.com)
First problem is that there's a new API:

- current extractor uses https://content.watchnebula.com/video/...

- but the website started using https://content.api.nebula.app/video/...

And the old link still works, but actually returns 404 for videos posted in the last 24h.

The new API's reply is different, but the only important part is still there (manifest/m3u8).

Second problem: there's no zype_id for the main video. You can pull it from another API call, but I don't think it's necessary (it's not actually the real video id, it was used just to please yt-dlp).

If yt-dlp insists of having real ids, there actually are real video ids in Nebula (looks like "video_episode:8c2f6d8e-29ea-450f-bc16-51d896a2f59a" and is used in the API, like https://content.api.nebula.app/video_episodes/video_episode:8c2f6d8e-29ea-450f-bc16-51d896a2f59a/ - we might need to start using that API eventually). I'm removing the "video_episode:" part from the id in the patch, because it was getting mangled into "：" (also question marks in the titles are mangled into "？") when actually downloading.

Trivial patch working for now:
```diff
--- nebula.py@2023.03.04	2023-03-06 01:00:00.000000000 +0100
+++ nebula.py	2023-05-09 19:02:14.063160007 +0200
@@ -65,7 +65,7 @@
         return response['token']
 
     def _fetch_video_formats(self, slug):
-        stream_info = self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/stream/',
+        stream_info = self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/stream/',
                                             video_id=slug,
                                             auth_type='bearer',
                                             note='Fetching video stream info')
@@ -77,7 +77,7 @@
         channel_slug = episode['channel_slug']
         channel_title = episode['channel_title']
         return {
-            'id': episode['zype_id'],
+            'id': episode['id'].replace('video_episode:',''),
             'display_id': episode['slug'],
             'formats': fmts,
             'subtitles': subs,
@@ -186,7 +186,7 @@
     ]
 
     def _fetch_video_metadata(self, slug):
-        return self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/',
+        return self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/',
                                      video_id=slug,
                                      auth_type='bearer',
                                      note='Fetching video meta data')
```

> Second problem: there's no zype_id for the main video. You can pull it from another API call, but I don't think it's necessary (it's not actually the real video id, it was used just to please yt-dlp).

The original nebula extractor was using Zype and the id was kept for backward compatibility of archive. But if an additional request is needed just to fetch it, we can change it.

@Lamieur Will you be fixing tests and opening a PR?
I have no idea how to run the tests, `python test/test_download.py TestDownload.test_Nebula` doesn't read my .netrc and fails on login, never getting to pulling video info. Someone smarter needs to step in.

As usual, I won't be making a PR :)

I wonder though, is preserving the current id helpful to people? For example, if they're archiving videos automatically  I'm asking because it's still possible to grab that id with a different API call, which for my needs is a waste of time, so I just changed it to the Nebula UUID for myself (it's ugly, but already there in the json), but if preserving video ids is valuable in some use cases, perhaps my 3-line patch is not that good :)
Just dropping by to mention that I'm seeing the same issue here in the USA (unsurprisingly).
I'm not a Python dev, so forgive me if I'm missing something, please. @Lamieur, could you actually make that a patch?
> `python test/test_download.py TestDownload.test_Nebula` doesn't read my .netrc and fails on login, never getting to pulling video info.

https://github.com/yt-dlp/yt-dlp/blob/ddae33754ae1f32dd9c64cf895c47d20f6b5f336/CONTRIBUTING.md#L237




> I wonder though, is preserving the current id helpful to people?

In theory, yes. But idk if anyone's actually using the feature with nebula. If additional API call is needed, imo it's best to not implement it rn and wait for someone to request support.
@CarlFink https://github.com/yt-dlp/yt-dlp/issues/7017#issuecomment-1540085455
> @CarlFink [#7017 (comment)](https://github.com/yt-dlp/yt-dlp/issues/7017#issuecomment-1540085455)

Not sure of the significance. If you want my Nebula account detes, no problem.
I don't know how useful this might be to you, but I'm able to download the streams manually. 

First, locate two network streams named media.m3u8 in the browser developer console network tab. One represents the video and the other one, the audio. Then, download them both using VLC's Open Network Stream >> Convert action, and multiplex them using ffmpeg.

This is very annoying to do manually, but I'm pretty sure I can turn it into a script using VLC's command-line mode, or replacing VLC with ffmpeg again. Extracting the m3u8 URLs might be harder to automate. The thing I don't know how to do is extract the urls of the m3u8's from the page. Would I have to run an invisible browser instance to start the streams?

EDIT: 
ffmpeg -i "first.m3u8" -i "second.m3u8" -codec copy TitleOfVideo.mp4

Resolving patch:
diff --git a/yt_dlp/extractor/nebula.py b/yt_dlp/extractor/nebula.py
--- a/yt_dlp/extractor/nebula.py
+++ b/yt_dlp/extractor/nebula.py
@@ -3,7 +3,7 @@
 import urllib.error
 
 from .common import InfoExtractor
-from ..utils import ExtractorError, parse_iso8601
+from ..utils import ExtractorError, make_archive_id, parse_iso8601, remove_start
 
 _BASE_URL_RE = r'https?://(?:www\.|beta\.)?(?:watchnebula\.com|nebula\.app|nebula\.tv)'
 
@@ -65,19 +65,20 @@ def _fetch_nebula_bearer_token(self):
         return response['token']
 
     def _fetch_video_formats(self, slug):
-        stream_info = self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/stream/',
+        stream_info = self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/stream/',
                                             video_id=slug,
                                             auth_type='bearer',
                                             note='Fetching video stream info')
         manifest_url = stream_info['manifest']
-        return self._extract_m3u8_formats_and_subtitles(manifest_url, slug)
+        return self._extract_m3u8_formats_and_subtitles(manifest_url, slug, 'mp4')
 
     def _build_video_info(self, episode):
         fmts, subs = self._fetch_video_formats(episode['slug'])
         channel_slug = episode['channel_slug']
         channel_title = episode['channel_title']
+        zype_id = episode.get('zype_id')
         return {
-            'id': episode['zype_id'],
+            'id': remove_start(episode['id'], 'video_episode:'),
             'display_id': episode['slug'],
             'formats': fmts,
             'subtitles': subs,
@@ -99,6 +100,9 @@ def _build_video_info(self, episode):
             'uploader_url': f'https://nebula.tv/{channel_slug}',
             'series': channel_title,
             'creator': channel_title,
+            'extractor_key': NebulaIE.ie_key(),
+            'extractor': NebulaIE.IE_NAME,
+            '_old_archive_ids': [make_archive_id(NebulaIE, zype_id)] if zype_id else None,
         }
 
     def _perform_login(self, username=None, password=None):
@@ -113,7 +117,7 @@ class NebulaIE(NebulaBaseIE):
             'url': 'https://nebula.tv/videos/that-time-disney-remade-beauty-and-the-beast',
             'md5': '14944cfee8c7beeea106320c47560efc',
             'info_dict': {
-                'id': '5c271b40b13fd613090034fd',
+                'id': '84ed544d-4afd-4723-8cd5-2b95261f0abf',
                 'ext': 'mp4',
                 'title': 'That Time Disney Remade Beauty and the Beast',
                 'description': 'Note: this video was originally posted on YouTube with the sponsor read included. We weren’t able to remove it without reducing video quality, so it’s presented here in its original context.',
@@ -137,22 +141,22 @@ class NebulaIE(NebulaBaseIE):
             'url': 'https://nebula.tv/videos/the-logistics-of-d-day-landing-craft-how-the-allies-got-ashore',
             'md5': 'd05739cf6c38c09322422f696b569c23',
             'info_dict': {
-                'id': '5e7e78171aaf320001fbd6be',
+                'id': '7e623145-1b44-4ca3-aa0b-ed25a247ea34',
                 'ext': 'mp4',
                 'title': 'Landing Craft - How The Allies Got Ashore',
                 'description': r're:^In this episode we explore the unsung heroes of D-Day, the landing craft.',
                 'upload_date': '20200327',
                 'timestamp': 1585348140,
-                'channel': 'Real Engineering',
-                'channel_id': 'realengineering',
-                'uploader': 'Real Engineering',
-                'uploader_id': 'realengineering',
-                'series': 'Real Engineering',
+                'channel': 'Real Engineering — The Logistics of D-Day',
+                'channel_id': 'd-day',
+                'uploader': 'Real Engineering — The Logistics of D-Day',
+                'uploader_id': 'd-day',
+                'series': 'Real Engineering — The Logistics of D-Day',
                 'display_id': 'the-logistics-of-d-day-landing-craft-how-the-allies-got-ashore',
-                'creator': 'Real Engineering',
+                'creator': 'Real Engineering — The Logistics of D-Day',
                 'duration': 841,
-                'channel_url': 'https://nebula.tv/realengineering',
-                'uploader_url': 'https://nebula.tv/realengineering',
+                'channel_url': 'https://nebula.tv/d-day',
+                'uploader_url': 'https://nebula.tv/d-day',
                 'thumbnail': r're:https://\w+\.cloudfront\.net/[\w-]+\.jpeg?.*',
             },
         },
@@ -160,7 +164,7 @@ class NebulaIE(NebulaBaseIE):
             'url': 'https://nebula.tv/videos/money-episode-1-the-draw',
             'md5': 'ebe28a7ad822b9ee172387d860487868',
             'info_dict': {
-                'id': '5e779ebdd157bc0001d1c75a',
+                'id': 'b96c5714-9e2b-4ec3-b3f1-20f6e89cc553',
                 'ext': 'mp4',
                 'title': 'Episode 1: The Draw',
                 'description': r'contains:There’s free money on offer… if the players can all work together.',
@@ -190,7 +194,7 @@ class NebulaIE(NebulaBaseIE):
     ]
 
     def _fetch_video_metadata(self, slug):
-        return self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/',
+        return self._call_nebula_api(f'https://content.api.nebula.app/video/{slug}/',
                                      video_id=slug,
                                      auth_type='bearer',
                                      note='Fetching video meta data')


Test patch: