Repository: yt-dlp/yt-dlp

Issue:
[Broken] Twitch livestream with mpv: [ytdl_hook] ERROR: Failed to download m3u8 playlist information: HTTP Error 403: Forbidden
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.03.01**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.03.01
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
PASTE VERBOSE LOG HERE

```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

- OS: Arch Linux

- `yt-dlp --version`:

```
2021.03.01
```

- `mpv --version`:

```
mpv 0.33.0-dirty Copyright © 2000-2020 mpv/MPlayer/mplayer2 projects
 built on UNKNOWN
FFmpeg library versions:
   libavutil       56.51.100
   libavcodec      58.91.100
   libavformat     58.45.100
   libswscale      5.7.100
   libavfilter     7.85.100
   libswresample   3.7.100
FFmpeg version: n4.3.1
```

- `mpv.conf`:

```
script-opts=ytdl_hook-ytdl_path="/usr/bin/yt-dlp"
```

Whenever I'm trying to play a Twitch livestream with mpv, I'm getting this:

- `mpv -v "https://www.twitch.tv/summit1g"`:

```sh
[cplayer] Command line options: '-v' 'https://www.twitch.tv/summit1g'
[cplayer] mpv 0.33.0-dirty Copyright © 2000-2020 mpv/MPlayer/mplayer2 projects
[cplayer]  built on UNKNOWN
[cplayer] FFmpeg library versions:
[cplayer]    libavutil       56.51.100
[cplayer]    libavcodec      58.91.100
[cplayer]    libavformat     58.45.100
[cplayer]    libswscale      5.7.100
[cplayer]    libavfilter     7.85.100
[cplayer]    libswresample   3.7.100
[cplayer] FFmpeg version: n4.3.1
[cplayer] 
[cplayer] Configuration: /usr/bin/waf configure --prefix=/usr --confdir=/etc/mpv --enable-cdda --enable-dvb --enable-dvdnav --enable-libarchive --enable-libmpv-shared --disable-build-date
[cplayer] List of enabled features: 52arch alsa asm caca cdda cplayer cplugins cuda-hwaccel cuda-interop debug-build drm dvbin dvdnav egl egl-drm egl-helpers egl-x11 ffmpeg ffnvcodec gbm gbm.h gl gl-wayland glibc-thread-name glob glob-posix gpl iconv jack javascript jpeg lcms2 libarchive libass libavdevice libbluray libdl libm libmpv-shared libplacebo librt linux-fstatfs linux-input-event-codes lua memfd_create optimize plain-gl posix posix-or-mingw pthreads pulse rubberband shaderc shaderc-shared stdatomic uchardet vaapi vaapi-drm vaapi-egl vaapi-vulkan vaapi-wayland vaapi-x-egl vaapi-x11 vdpau vt.h vulkan wayland wayland-protocols x11 xv zlib
[cplayer] Reading config file /etc/mpv/encoding-profiles.conf
[cplayer] Applying profile 'default'...
[cplayer] Reading config file /home/username/.config/mpv/mpv.conf
[cplayer] Applying profile 'default'...
[cplayer] Setting option 'script-opts' = 'ytdl_hook-ytdl_path="/usr/bin/yt-dlp"' (flags = 4)
[cplayer] Setting option 'v' = '' (flags = 8)
[ifo_dvdnav] Opening /home/username/.config/mpv/input.conf
[bdmv/bluray] Opening /home/username/.config/mpv/input.conf
[file] Opening /home/username/.config/mpv/input.conf
[input] Parsing input config file /home/username/.config/mpv/input.conf
[input] Input config file /home/username/.config/mpv/input.conf parsed: 23 binds
[cplayer] Waiting for scripts...
[osd/libass] Shaper: FriBidi 1.0.10 (SIMPLE) HarfBuzz-ng 2.7.4 (COMPLEX)
[osd/libass] Setting up fonts...
[progressbar] Opened config file script-opts/torque-progressbar/main.conf.
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Done loading scripts.
[cplayer] Running hook: ytdl_hook/on_load
[ytdl_hook] ytdl:// hook
[ytdl_hook] not a ytdl:// url
[ffmpeg] Opening https://www.twitch.tv/summit1g
[cplayer] Set property: shared-script-properties -> 1
[osd/libass] Shaper: FriBidi 1.0.10 (SIMPLE) HarfBuzz-ng 2.7.4 (COMPLEX)
[osd/libass] Setting up fonts...
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[cplayer] Set property: shared-script-properties -> 1
[ffmpeg] Mime-type: 'text/html'
[demux] Trying demuxers for level=normal.
[lavf] No format found, try lowering probescore or forcing the format.
[demux] Trying demuxers for level=unsafe.
[lavf] No format found, try lowering probescore or forcing the format.
[cplayer] Opening failed or was aborted: https://www.twitch.tv/summit1g
[cplayer] Running hook: ytdl_hook/on_load_fail
[ytdl_hook] full hook
[ytdl_hook] found youtube-dl at: /usr/bin/yt-dlp
[ytdl_hook] ERROR: Failed to download m3u8 playlist information: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[ytdl_hook] Traceback (most recent call last):
[ytdl_hook]   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
[ytdl_hook]     return _run_code(code, main_globals, None,
[ytdl_hook]   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
[ytdl_hook]     exec(code, run_globals)
[ytdl_hook]   File "/usr/bin/yt-dlp/__main__.py", line 19, in <module>
[ytdl_hook]   File "/usr/bin/yt-dlp/yt_dlp/__init__.py", line 606, in main
[ytdl_hook]   File "/usr/bin/yt-dlp/yt_dlp/__init__.py", line 596, in _real_main
[ytdl_hook]   File "/usr/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2503, in download
[ytdl_hook]   File "/usr/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2568, in post_extract
[ytdl_hook]   File "/usr/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2555, in actual_post_extract
[ytdl_hook] AttributeError: 'NoneType' object has no attribute 'get'
[ytdl_hook] youtube-dl failed: unexpected error occurred
[cplayer] finished playback, unrecognized file format (reason 4)
[cplayer] Failed to recognize file format.
[cplayer]
[cplayer] Exiting... (Errors when loading file)
[cplayer] Set property: shared-script-properties -> 1
```


It seems that `yt-dlp` cannot detect any format for Twitch livestream, unlike `youtube-dl`:

- `youtube-dl -F "https://www.twitch.tv/summit1g"`:

```
[twitch:stream] summit1g: Downloading stream GraphQL
[twitch:stream] summit1g: Downloading stream access token GraphQL
[twitch:stream] 41875779838: Downloading m3u8 information
[info] Available formats for 41875779838:
format code      extension  resolution note
audio_only       mp4        audio only  163k , mp4a.40.2
160p             mp4        284x160     230k , avc1.4D401F, 30.0fps, mp4a.40.2
360p             mp4        640x360     630k , avc1.4D401F, 30.0fps, mp4a.40.2
480p             mp4        852x480    1431k , avc1.4D401F, 30.0fps, mp4a.40.2
720p60           mp4        1280x720   3426k , avc1.4D401F, 60.0fps, mp4a.40.2
936p60__source_  mp4        1664x936   9037k , avc1.64002A, 60.0fps, mp4a.40.2 (best)
```

- `yt-dlp -F "https://www.twitch.tv/summit1g"`:

```
[twitch:stream] summit1g: Downloading stream GraphQL
[twitch:stream] summit1g: Downloading stream access token GraphQL
[twitch:stream] 41875779838: Downloading m3u8 information
ERROR: Failed to download m3u8 playlist information: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
```

It works fine when using `youtube-dl` instead of `yt-dlp`:


- `mpv "https://www.twitch.tv/summit1g"`:

```
(+) Video --vid=1 (h264 1920x1080)
 (+) Audio --aid=1 (aac 2ch 44100Hz)
[progressbar] Playing: "summit1g (live) 2021-03-01 11:03"
AO: [pulse] 44100Hz stereo 2ch float
VO: [gpu] 1920x1080 yuv420p
AV: 00:00:05 / 00:00:07 (64%) A-V:  0.000 Cache: 2.4s/1MB
```



Discussion:
Seems to be a bug introduced by https://github.com/yt-dlp/yt-dlp/commit/310c2ed2c648f4e1254e27d0e8901d629828b2ba

Untill we manage to fix it, you can use [2021.02.19](https://github.com/yt-dlp/yt-dlp/releases/tag/2021.02.19)
> Seems to be a bug introduced by [310c2ed](https://github.com/yt-dlp/yt-dlp/commit/310c2ed2c648f4e1254e27d0e8901d629828b2ba)
> 
> Untill we manage to fix it, you can use [2021.02.19](https://github.com/yt-dlp/yt-dlp/releases/tag/2021.02.19)

Can confirm it works fine with [2021.02.19](https://github.com/yt-dlp/yt-dlp/releases/tag/2021.02.19)

Cheers

Resolving patch:
diff --git a/yt_dlp/extractor/common.py b/yt_dlp/extractor/common.py
--- a/yt_dlp/extractor/common.py
+++ b/yt_dlp/extractor/common.py
@@ -1904,15 +1904,16 @@ def _parse_m3u8_formats(self, m3u8_doc, m3u8_url, ext=None,
         # media playlist and MUST NOT appear in master playlist thus we can
         # clearly detect media playlist with this criterion.
 
-        def _extract_m3u8_playlist_formats(format_url=None, m3u8_doc=None):
+        def _extract_m3u8_playlist_formats(format_url=None, m3u8_doc=None, video_id=None,
+                                           fatal=True, data=None, headers={}):
             if not m3u8_doc:
                 if not format_url:
                     return []
                 res = self._download_webpage_handle(
                     format_url, video_id,
                     note=False,
-                    errnote=errnote or 'Failed to download m3u8 playlist information',
-                    fatal=fatal, data=data, headers=headers, query=query)
+                    errnote='Failed to download m3u8 playlist information',
+                    fatal=fatal, data=data, headers=headers)
 
                 if res is False:
                     return []
@@ -1984,7 +1985,8 @@ def extract_media(x_media_line):
             if media_url:
                 manifest_url = format_url(media_url)
                 format_id = []
-                playlist_formats = _extract_m3u8_playlist_formats(manifest_url)
+                playlist_formats = _extract_m3u8_playlist_formats(manifest_url, video_id=video_id,
+                                                                  fatal=fatal, data=data, headers=headers)
 
                 for format in playlist_formats:
                     format_index = format.get('index')
@@ -2045,7 +2047,8 @@ def build_stream_name():
                     or last_stream_inf.get('BANDWIDTH'), scale=1000)
                 manifest_url = format_url(line.strip())
 
-                playlist_formats = _extract_m3u8_playlist_formats(manifest_url)
+                playlist_formats = _extract_m3u8_playlist_formats(manifest_url, video_id=video_id,
+                                                                  fatal=fatal, data=data, headers=headers)
 
                 for format in playlist_formats:
                     format_id = []


Test patch: