Repository: yt-dlp/yt-dlp

Issue:
Youtube :ytwatchlater [Broken]
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.06.09**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.06.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
[debug] Command-line config: ['--verbose', '--cookies', 'cookies.txt', '--skip-download', ':ytwatchlater']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.06.09
[debug] Python version 3.8.0 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-19-g8d172d9409, ffprobe n4.4-19-g8d172d9409
[debug] Proxy map: {}
[debug] [youtube:watchlater] Extracting URL: :ytwatchlater
[debug] [youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading webpage
[debug] [youtube:tab] Final URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading API JSON with unavailable videos
ERROR: YouTube said: The playlist does not exist.; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com
/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1133, in wrapper
    return func(self, *args, **kwargs)
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1154, in __extract_info
    ie_result = ie.extract(url)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\common.py", line 567, in extract
    ie_result = self._real_extract(url)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3642, in _real_extract
    info_dict = self.__real_extract(url, smuggled_data)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3742, in __real_extract
    data = self._reload_with_unavailable_videos(item_id, data, webpage) or data
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3556, in _reload_with_unavailable_videos
    return self._extract_response(
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3595, in _extract_response
    self._extract_and_report_alerts(response, expected=False)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3515, in _extract_and_report_alerts
    return self._report_alerts(self._extract_alerts(data), *args, **kwargs)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3512, in _report_alerts
    raise ExtractorError('YouTube said: %s' % errors[-1][1], expected=expected)
yt_dlp.utils.ExtractorError: YouTube said: The playlist does not exist.; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; s
ee  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

Trying to get a list of videos from my watchlater list, youtube-dl only listed 100 videos, googeling yt-dlp supposedly had this fixed but I'm getting this error instead.



Discussion:
Check your cookies? Seems like they don't contain your Youtube cookies. Also check the file path to the cookies.txt is correct.

----

 Works for me...

```
yt-dlp --cookies ~/Downloads/cookies.Youtube.txt --skip-download "https://www.youtube.com/playlist?list=WL" --verbose
[debug] Command-line config: ['--cookies', '/home/colethedj/Downloads/cookies.Youtube.txt', '--skip-download', 'https://www.youtube.com/playlist?list=WL', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] yt-dlp version 2021.06.09 
[debug] Python version 3.9.5 (CPython 64bit) - Linux-5.11.19-1-MANJARO-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[debug] [youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading webpage
[debug] [youtube:tab] Final URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading API JSON with unavailable videos
[download] Downloading playlist: Watch later
[youtube:tab] playlist Watch later: Downloading 33 videos
```
```
yt-dlp --cookies ~/Downloads/cookies.Youtube.txt --skip-download :ytwatchlater --verbose
[debug] Command-line config: ['--cookies', '/home/colethedj/Downloads/cookies.Youtube.txt', '--skip-download', ':ytwatchlater', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] yt-dlp version 2021.06.09 
[debug] Python version 3.9.5 (CPython 64bit) - Linux-5.11.19-1-MANJARO-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[debug] [youtube:watchlater] Extracting URL: :ytwatchlater
[debug] [youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading webpage
[debug] [youtube:tab] Final URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading API JSON with unavailable videos
[download] Downloading playlist: Watch later
[youtube:tab] playlist Watch later: Downloading 33 videos
[download] Downloading video 1 of 33
```
I'm running the same command but change to youtube-dl, same directory, same console and get this:
```
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'cookies.txt', '--skip-download', ':ytwatchlater', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg n4.4-19-g8d172d9409, ffprobe n4.4-19-g8d172d9409
[debug] Proxy map: {}
[youtube:tab] WL: Downloading webpage
[download] Downloading playlist: Watch later
[youtube:tab] Downloading page 1
[youtube:tab] playlist Watch later: Downloading 100 videos
[download] Downloading video 1 of 100
[youtube] fOnH8mbZO64: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[download] Downloading video 2 of 100
[youtube] 8o2FtM_oji8: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[download] Downloading video 3 of 100
[youtube] Bp_f_zhlwpM: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[download] Downloading video 4 of 100
[youtube] yTxkyoLxFmw: Downloading webpage
```
So the cookies file and path seem fine?
Hmm, yep seems fine. Can you try running with `--compat-options no-youtube-unavailable-videos`?

There's potentially a couple issues here. 

I notice it is working for the initial page but failing on the API call for the page with unavailable videos. Regardless of why, this call should not raise a fatal error. Seems to be triggered by the alert extraction.

Got this:
```
[debug] Command-line config: ['--compat-options', 'no-youtube-unavailable-videos', '--cookies', 'cookies.txt', '--skip-download', ':ytwatchlater', '--verbose']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.06.09
[debug] Compatibility options: no-youtube-unavailable-videos
[debug] Python version 3.8.0 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-19-g8d172d9409, ffprobe n4.4-19-g8d172d9409
[debug] Proxy map: {}
[debug] [youtube:watchlater] Extracting URL: :ytwatchlater
[debug] [youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=WL
[youtube:tab] WL: Downloading webpage
[debug] [youtube:tab] Final URL: https://www.youtube.com/playlist?list=WL
WARNING: [youtube:tab] YouTube said: INFO - Unavailable videos are hidden
[download] Downloading playlist: Watch later
[youtube:tab] WL page 1: Downloading API JSON
ERROR: YouTube said: The playlist does not exist.; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com
/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1133, in wrapper
    return func(self, *args, **kwargs)
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1165, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1270, in process_ie_result
    return self.__process_playlist(ie_result, download)
  File "d:\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1352, in __process_playlist
    entry = ie_entries[i - 1]
  File "d:\python\python38\lib\site-packages\yt_dlp\utils.py", line 3985, in __getitem__
    self.__cache.extend(itertools.islice(self.__iterable, n))
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3633, in _smuggle_data
    for entry in entries:
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3264, in _entries
    response = self._extract_response(
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3595, in _extract_response
    self._extract_and_report_alerts(response, expected=False)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3515, in _extract_and_report_alerts
    return self._report_alerts(self._extract_alerts(data), *args, **kwargs)
  File "d:\python\python38\lib\site-packages\yt_dlp\extractor\youtube.py", line 3512, in _report_alerts
    raise ExtractorError('YouTube said: %s' % errors[-1][1], expected=expected)
yt_dlp.utils.ExtractorError: YouTube said: The playlist does not exist.; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; s
ee  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
```
Ok, looks like my suspicion was correct. Seems like the API call isn't being authenticated correctly. :cry:

This might be a bit difficult to troubleshoot, since I haven't seen any issues like this since we implemented it. 

Trying to think of some cases we haven't tested:
- is the cookies for a third YouTube brand channel (3 channels under 1 account)?
- do you have any other accounts logged in maybe?
- does your cookie file contain a "SAPISID" entry? if not does it contain a "__Secure-3PAPISID" entry?

-----

So TL;DR we potentially have two issues:
- *use and prioritize "__Secure-3PAPISID" cookie over "SAPISID" cookie for API auth
- unavailable videos request should not raise an error on alert extraction


thanks for reporting this.


is the cookies for a third YouTube brand channel (3 channels under 1 account)?
Not sure, I seem to have 3 channels but know nothing about how that works.

do you have any other accounts logged in maybe?
I am logged in to Youtube on Kodi and on my Palemoon webbrowser.

does your cookie file contain a "SAPISID" entry? if not does it contain a "__Secure-3PAPISID" entry?
No "SAPISID" but there is a "__Secure-3PAPISID" cookie.


> No "SAPISID" but there is a "__Secure-3PAPISID" cookie.

Ah, that might be it. I removed the SAPISID cookie on my end and I get the same behavior. Not sure why yours doesn't have it, but yt-dlp only picks up the "SAPISID" cookie atm.

For now, can you try duplicating the entire line containing the "__Secure-3PAPISID" cookie and rename it to "SAPISID" in your cookie file and try again? 

From mine it looks like they are the same thing.
Thanks, Seems to work now :satisfied:
and I seem to get the full list that youtube-dl don't give me.
Awesome!  I'll get this fixed later. Thanks :smiley:

Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -301,12 +301,22 @@ def _real_initialize(self):
     _YT_INITIAL_BOUNDARY_RE = r'(?:var\s+meta|</script|\n)'
 
     def _generate_sapisidhash_header(self):
-        sapisid_cookie = self._get_cookies('https://www.youtube.com').get('SAPISID')
+        # Sometimes SAPISID cookie isn't present but __Secure-3PAPISID is.
+        # See: https://github.com/yt-dlp/yt-dlp/issues/393
+        yt_cookies = self._get_cookies('https://www.youtube.com')
+        sapisid_cookie = dict_get(
+            yt_cookies, ('__Secure-3PAPISID', 'SAPISID'))
         if sapisid_cookie is None:
             return
         time_now = round(time.time())
-        sapisidhash = hashlib.sha1((str(time_now) + " " + sapisid_cookie.value + " " + "https://www.youtube.com").encode("utf-8")).hexdigest()
-        return "SAPISIDHASH %s_%s" % (time_now, sapisidhash)
+        # SAPISID cookie is required if not already present
+        if not yt_cookies.get('SAPISID'):
+            self._set_cookie(
+                '.youtube.com', 'SAPISID', sapisid_cookie.value, secure=True, expire_time=time_now + 3600)
+        # SAPISIDHASH algorithm from https://stackoverflow.com/a/32065323
+        sapisidhash = hashlib.sha1(
+            f'{time_now} {sapisid_cookie.value} https://www.youtube.com'.encode('utf-8')).hexdigest()
+        return f'SAPISIDHASH {time_now}_{sapisidhash}'
 
     def _call_api(self, ep, query, video_id, fatal=True, headers=None,
                   note='Downloading API JSON', errnote='Unable to download API page',


Test patch: