Repository: yt-dlp/yt-dlp

Issue:
Some Youtube m3u8 formats fail to download with a 404
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Netherlands

### Provide a description that is worded well enough to be understood

This video (https://www.youtube.com/watch?v=V5yN4xSKxRM) cannot be downloaded, exiting with the error 
`ERROR: unable to download video data: HTTP Error 404: Not Found`. 
I can play the video in a browser just fine, even without being logged in on Youtube. Other videos from the channel can be downloaded without issues. I remember encountering the same problem on another video some time ago (unfortunately I am not sure whether it was the same channel), but thought then it was just a temporary glitch.
 

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
~ yt-dlp -vU https://www.youtube.com/watch\?v\=V5yN4xSKxRM
[debug] Command-line config: ['--config-location', '/Users/alekoe/yt-dlp/config', '-vU', 'https://www.youtube.com/watch?v=V5yN4xSKxRM']
[debug] | Config "/Users/alekoe/yt-dlp/config": ['-o', '/Users/alekoe/Movies/Youtube/%(uploader)s/%(title)s.%(ext)s', '-f', 'bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b', '--download-archive', '/Users/alekoe/yt-dlp/archive.txt']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481] (zip)
[debug] Python 3.11.4 (CPython arm64 64bit) - macOS-13.4.1-arm64-arm-64bit (OpenSSL 3.1.1 30 May 2023)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Loading archive file '/Users/alekoe/yt-dlp/archive.txt'
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
Current Build Hash: cedf44a9cf3c241520a692c1006d276e6f434d4d945727c47cb1a70d32953f4a
yt-dlp is up to date (stable@2023.07.06)
[youtube] Extracting URL: https://www.youtube.com/watch?v=V5yN4xSKxRM
[youtube] V5yN4xSKxRM: Downloading webpage
[youtube] V5yN4xSKxRM: Downloading ios player API JSON
[youtube] V5yN4xSKxRM: Downloading android player API JSON
[debug] Loading youtube-nsig.0e6aaa83 from cache
[debug] [youtube] Decrypted nsig 3b4yG4cOBVYUqfLKTB => xREnxB6z0FDm4Q
[youtube] V5yN4xSKxRM: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[info] V5yN4xSKxRM: Downloading 1 format(s): 617+140
[debug] Invoking hlsnative downloader on "https://manifest.googlevideo.com/api/manifest/hls_playlist/expire/1690951820/ei/LIzJZOPlJMLQ1gLf5YHYDg/ip/2a02:a212:2241:3b80:8c17:9ec9:1d0:a29d/id/579c8de3148ac513/itag/617/source/youtube/requiressl/yes/ratebypass/yes/pfa/1/wft/1/sgovp/clen%3D441607860%3Bdur%3D1517.016%3Bgir%3Dyes%3Bitag%3D303%3Blmt%3D1690855624695842/hls_chunk_host/rr2---sn-5hneknee.googlevideo.com/mh/Un/mm/31,29/mn/sn-5hneknee,sn-5hne6nsy/ms/au,rdu/mv/m/mvi/2/pl/28/initcwndbps/2168750/bui/AYlvQAubp_5xZFxoJPwO7ClBPOCe_5Ur5ozlzjVc_eok-DdjFybKUS0ezqzuH9TPNWtm26AidxufaWGJioOKZMvZpvHXuF0Y/vprv/1/playlist_type/DVR/dover/13/txp/3309224/mt/1690929897/fvip/4/short_key/1/keepalive/yes/fexp/24007246,51000014/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,pfa,wft,sgovp,bui,vprv,playlist_type/sig/AOq0QJ8wRQIga-bVEuOKllOaG83nAEVb4_nto79dDK0EPCMhwGijWowCIQC4Vds1rER_wDnsZ3z2SIh9VTnoUM4CTUNJpDifNn44qg%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AG3C_xAwRQIgdZAKONrPWnHIAbaHkCP1PlquuGJIX9R2K4d-vbJ0zmwCIQDrH0ScbgVeyaxze8_DV5TQ7O7l65mAnQQtw1t0XjqEmA%3D%3D/playlist/index.m3u8"
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3345, in process_info
    partial_success, real_download = self.dl(fname, new_info)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3084, in dl
    return fd.download(name, new_info, subtitle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py", line 461, in download
    ret = self.real_download(filename, info_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/hls.py", line 77, in real_download
    urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3863, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
```



Discussion:
This is an issue on Youtube's end where newly uploaded videos have their iOS m3u8 formats listed as available before Youtube has finished processing those formats, resulting in the 404 error upon attempting to download them. After some hours have passed, format 6xx will have finished processing and be downloadable.

In the meantime, you can:
- use `--check-formats` to auto-select the next best format
- avoid the iOS formats altogether by passing `--extractor-args "youtube:player_client=android,web"`

NOTE: the nightly/master versions of yt-dlp now automatically check these formats before downloading. If you wish to disable the automatic check/reselection, then pass `--no-check-formats`
I just ran into this problem with a different video.
Is it possible to detect this case and give the user a better error message than a generic 404, or even fall back to another format automatically? (when enabled/not disabled by the user of course).

Debug log for reference:
```
[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch?v=Nj0M1E0mG6c']
[debug] User config "~/.config/yt-dlp/config": ['--prefer-free-formats', '--format', 'bestvideo+bestaudio/best', '--sub-langs', 'all,-live_chat', '--embed-subs', '--embed-thumbnail', '--embed-metadata', '--embed-chapters', '--embed-info-json', '--sponsorblock-mark', 'all', '--netrc-location=~/.config/yt-dlp/netrc', '--continue', '--netrc', '--cookies=~/.config/yt-dlp/cookie-jar']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481]
[debug] Python 3.11.3 (CPython x86_64 64bit) - Linux-6.1.39-1-lts-x86_64-with-glibc2.37 (OpenSSL 3.1.1 30 May 2023, glibc 2.37)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0
[debug] Optional libraries: certifi-2023.07.22, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
yt-dlp is up to date (stable@2023.07.06)
[youtube] Extracting URL: https://www.youtube.com/watch?v=Nj0M1E0mG6c
[youtube] Nj0M1E0mG6c: Downloading webpage
[youtube] Nj0M1E0mG6c: Downloading ios player API JSON
[youtube] Nj0M1E0mG6c: Downloading android player API JSON
[youtube] Nj0M1E0mG6c: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[SponsorBlock] Fetching SponsorBlock segments
[debug] SponsorBlock query: https://sponsor.ajay.app/api/skipSegments/a8d9?service=YouTube&categories=%5B%22selfpromo%22%2C+%22chapter%22%2C+%22intro%22%2C+%22sponsor%22%2C+%22preview%22%2C+%22outro%22%2C+%22filler%22%2C+%22interaction%22%2C+%22music_offtopic%22%2C+%22poi_highlight%22%5D&actionTypes=%5B%22skip%22%2C+%22poi%22%2C+%22chapter%22%5D
[SponsorBlock] Found 4 segments in the SponsorBlock database
[info] Nj0M1E0mG6c: Downloading 1 format(s): 616+251
[info] There are no subtitles for the requested languages
[info] Downloading video thumbnail 41 ...
[info] Writing video thumbnail 41 to: Wrong Call! The Incredible Story of Transair Flight 810 [Nj0M1E0mG6c].webp
WARNING: webm doesn't support embedding a thumbnail, mkv will be used
[debug] Invoking hlsnative downloader on "https://manifest.googlevideo.com/api/manifest/hls_playlist/expire/1691275147/ei/K3vOZIGmN7G7x_APwve22AM/ip/2a02:810c:4140:1053:417b:683b:3c5d:a615/id/363d0cd44d261ba7/itag/616/source/youtube/requiressl/yes/ratebypass/yes/pfa/1/wft/1/sgovp/clen%3D748529880%3Bdur%3D2135.267%3Bgir%3Dyes%3Bitag%3D356%3Blmt%3D1691242988963667/hls_chunk_host/rr1---sn-4g5e6nzl.googlevideo.com/mh/nt/mm/31,26/mn/sn-4g5e6nzl,sn-f5f7knee/ms/au,onr/mv/m/mvi/1/pl/40/initcwndbps/1892500/vprv/1/playlist_type/DVR/dover/13/txp/3308224/mt/1691253223/fvip/2/short_key/1/keepalive/yes/fexp/24007246,24363391/beids/24350017/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,pfa,wft,sgovp,vprv,playlist_type/sig/AOq0QJ8wRQIgUFOl3GzaGKaztF84op9lQcMLHNYBjUdIuYeYk8GUZ3oCIQDwlarzRISHLPNIdMgxH835R0Y4SZjIF5wjOy-O0tkF9g%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AG3C_xAwRQIhAJXwVPhsnbj5at9Ffqk7qTXBmo9nlnJHitTR7sdZsiafAiAT7IzLCZPECw5lI0wZVCSU9h16eJeomAwRfOSl86FVCA%3D%3D/playlist/index.m3u8"
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3345, in process_info
    partial_success, real_download = self.dl(fname, new_info)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3084, in dl
    return fd.download(name, new_info, subtitle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/downloader/common.py", line 461, in download
    ret = self.real_download(filename, info_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/downloader/hls.py", line 77, in real_download
    urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3863, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

```
>... possible to detect this case and ... even fall back to another format automatically

`--check-formats` with a format selection that doesn't specifically require the broken format will do that.
> > ... possible to detect this case and ... even fall back to another format automatically
> 
> `--check-formats` with a format selection that doesn't specifically require the broken format will do that.

That did not work for me... However `--extractor-args "youtube:player_client=android,web"` did work.
--check-formats is broken in last release
I would second the idea of having some way to automatically workaround this. I've encountered it a few times, and while it's good that the manual workaround exists (re. `--extractor-args "youtube:player_client=android,web"`), it's a bit annoying having to do it manually. I think a lot of people also just wouldn't know this workaround exists.
Still an issue on yt-dlp 2023.09.24

OPs video works for me, but https://www.youtube.com/watch?v=1YGD94lSor8 has the issue now. 
Seems it's back, this time for format 617:

`[debug] yt-dlp version master@2023.11.15.232812 from yt-dlp/yt-dlp-master-builds [15cb3528c] (zip)`

`[info] o4GZUCwVRLs: Downloading 1 format(s): 617+140`
`ERROR: unable to download video data: HTTP Error 404: Not Found`

Video: https://www.youtube.com/watch?v=o4GZUCwVRLs

Full debug output on request.
> [info] o4GZUCwVRLs: Downloading 1 format(s): 617+140
> ERROR: unable to download video data: HTTP Error 404: Not Found

can repro. 

```shell
$ yt-dlp -O upload_date o4GZUCwVRLs
20231115
```

it is outside the lookback window for automatic format testing, so I think we need to do this:
```diff
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
index f6caf0970..3328b3238 100644
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -4563,7 +4563,7 @@ def process_language(container, base_url, lang_code, sub_name, query):
         if upload_date and live_status not in ('is_live', 'post_live', 'is_upcoming'):
             # Newly uploaded videos' HLS formats are potentially problematic and need to be checked
             upload_datetime = datetime_from_str(upload_date).replace(tzinfo=datetime.timezone.utc)
-            if upload_datetime >= datetime_from_str('today-1day'):
+            if upload_datetime >= datetime_from_str('today-2days'):
                 for fmt in info['formats']:
                     if fmt.get('protocol') == 'm3u8_native':
                         fmt['__needs_testing'] = True
```

&nbsp;
as a workaround for now (and as mentioned earlier in the thread), pass `--check-formats`
Yes, this was uploaded on 2023-11-15 at 23:15 UTC or thereabouts (I recall posting a link to a friend, and it said "1 hour ago" at the time, which was 2023-11-16 at 01:14, GMT+01:00), so it's been up for 20 hours or so, making it likely that the date has changed -- so makes sense the detection code didn't catch it if the only thing you can get is the day of the upload, not the exact upload timestamp.

Is `--check-formats` something safe to use (and producing same effects otherwise) in day-to-day?
Forcing `--check-formats` is always "safe", it just may slow down extraction bc it will request each format as a sanity check
When it tests the formats, it downloads a (hopefully[^1]) small chunk of the target format to a tempfile in your temp dir and then immediately removes it. So if you are OK with the extra network overhead and disk writes, then it should be fine

[^1]: it tries for no larger than 10KB, but this may vary with fragmented formats: sometimes the first fragment can be several MB
Hm, I don't mind the disk and network use... but time I do. I tested it 10 minutes ago, and terminated the attempt after 2 minutes of it being stuck on `[info] Testing format 617`. Just tried it again, and it seems to work immediately now, so I don't know what happened there.

Thank you. I will add it to my invoke script.
> I tested it 10 minutes ago, and terminated the attempt after 2 minutes of it being stuck on `[info] Testing format 617`

This was likely caused by either https://github.com/yt-dlp/yt-dlp/issues/7894#issuecomment-1722904568 or #8579

Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -4563,7 +4563,7 @@ def process_language(container, base_url, lang_code, sub_name, query):
         if upload_date and live_status not in ('is_live', 'post_live', 'is_upcoming'):
             # Newly uploaded videos' HLS formats are potentially problematic and need to be checked
             upload_datetime = datetime_from_str(upload_date).replace(tzinfo=datetime.timezone.utc)
-            if upload_datetime >= datetime_from_str('today-1day'):
+            if upload_datetime >= datetime_from_str('today-2days'):
                 for fmt in info['formats']:
                     if fmt.get('protocol') == 'm3u8_native':
                         fmt['__needs_testing'] = True


Test patch: