Repository: yt-dlp/yt-dlp (Python)
Date: 2022-10-11 00:38:54+00:00
Task type: Bug Fix

Issue:
Downloads from Crunchyroll break if certain Optanon cookies are present
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.10.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

US

### Provide a description that is worded well enough to be understood

Apologies. Apparently I'm cursed to periodically find issues with Crunchyroll downloads.

Here's my (perhaps a little overzealously) redacted cookies.txt. You'll notice that the string yt-dlp breaks on is contained in some of those Optanon cookies. Deleting them from the file and retrying the download works, so I'm guessing it's *just* that. Still, a proper fix would be much appreciated.

```
# Netscape HTTP Cookie File
# This file is generated by yt-dlp.  Do not edit.

.beta.crunchyroll.com	TRUE	/	TRUE	1670409672	OptanonAlertBoxClosed	2021-12-07T10:41:12.992Z
.beta.crunchyroll.com	TRUE	/	TRUE	1670635235	OptanonConsent	isIABGlobal=false&datestamp=Fri+Dec+10+2021+02%3A20%3A35+GMT%2B0100+(Central+European+Standard+Time)&version=6.26.0&hosts=&consentId=-snip-&interactionCount=2&landingPath=NotLandingPage&groups=-snip-&AwaitingReconsent=false&geolocation=DE%3B
.beta.crunchyroll.com	TRUE	/	FALSE	1670409674	OptanonControl	ccc=DE&otvers=6.26.0&reg=gdpr&pctm=Tue Dec 07 2021 11:41:12 GMT+0100 (Central European Standard Time)&vers=3.0.3
.crunchyroll.com	TRUE	/	TRUE	1665269728	__cf_bm	-snip-
.crunchyroll.com	TRUE	/	TRUE	0	__cfruid	-snip-
.crunchyroll.com	TRUE	/	FALSE	1699827826	ab.storage.deviceId.-snip-	-snip-
.crunchyroll.com	TRUE	/	FALSE	1699827856	ab.storage.sessionId.-snip-	-snip-
.crunchyroll.com	TRUE	/	FALSE	1699827826	ab.storage.userId.-snip-	-snip-
.crunchyroll.com	TRUE	/	FALSE	1679595856	c_visitor	-snip-
.crunchyroll.com	TRUE	/	FALSE	1666477421	etp_rt	-snip-
.crunchyroll.com	TRUE	/	TRUE	0	session_id	-snip-
.www.crunchyroll.com	TRUE	/	TRUE	1670410351	OptanonConsent	isIABGlobal=false&datestamp=Tue+Dec+07+2021+11%3A52%3A31+GMT%2B0100+(Central+European+Standard+Time)&version=6.26.0&hosts=&consentId=-snip-&interactionCount=1&landingPath=https%3A%2F%2Fwww.crunchyroll.com%2Flogin&groups=-snip-
.www.crunchyroll.com	TRUE	/	FALSE	1670410341	OptanonControl	ccc=US&otvers=&reg=ccpa&pctm=0&vers=3.0.3
beta.crunchyroll.com	FALSE	/	FALSE	1695580687	_evidon_suppress_notification_cookie	{"date":"2022-09-24T18:38:07.351Z"}
www.crunchyroll.com	FALSE	/	FALSE	1670410345	crunchyroll_beta_hide_banner	1
www.crunchyroll.com	FALSE	/	FALSE	1695579859	initial_referrer	https%3A%2F%2Fwww.google.com%2F
www.crunchyroll.com	FALSE	/forumtopic-1068039	FALSE	1670410300	initial_referrer	https%3A%2F%2Fwww.google.com%2F
```

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '--cookies', '../crunchyroll.com_cookies.txt', '--write-subs', '--embed-subs', 'https://beta.crunchyroll.com/watch/G9DUEP2JW/pressure']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (pip) API
[debug] Python 3.10.7 (CPython 64bit) - macOS-12.6-arm64-arm-64bit
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.13.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.10.04, Current version: 2022.10.04
yt-dlp is up to date (2022.10.04)
[debug] [crunchyroll:beta] Extracting URL: https://beta.crunchyroll.com/watch/G9DUEP2JW/pressure
ERROR: Illegal key 'Time)&vers'
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1477, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1553, in __extract_info
    ie_result = ie.extract(url)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 672, in extract
    ie_result = self._real_extract(url)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/crunchyroll.py", line 814, in _real_extract
    api_domain, bucket, params = self._get_params(lang)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/crunchyroll.py", line 723, in _get_params
    if self._get_cookies(f'https://beta.crunchyroll.com/{lang}').get('etp_rt'):
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 3645, in _get_cookies
    return LenientSimpleCookie(self._downloader._calc_cookies(url))
  File "/opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/cookies.py", line 483, in __init__
    self.load(input)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/cookies.py", line 1087, in load
    morsel.set(key, real_value, coded_value)
  File "/opt/homebrew/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/cookies.py", line 353, in set
    raise CookieError('Illegal key %r' % (key,))
http.cookies.CookieError: Illegal key 'Time)&vers'
```



Issue discussion:
@Grub4K Isn't lenient cookies supposed to handle this?
I would call this a bug imported from the CPython code, since it clearly allows usage of `)` and `&` in its `_LEGAL_KEY_CHARS` which is used in the compiled regex but does NOT allow them while setting them in the morsel, since that uses `_LegalChars`.

As a workaround you can either remove all cookies that are not needed from the file (all except `etp_rt`) or try the following patch:
```diff
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -1084,8 +1084,9 @@ def load(self, data):
             elif value is not None:
                 morsel = self.get(key, http.cookies.Morsel())
                 real_value, coded_value = self.value_decode(value)
-                morsel.set(key, real_value, coded_value)
-                self[key] = morsel
+                with contextlib.suppress(http.cookies.CookieError):
+                    morsel.set(key, real_value, coded_value)
+                    self[key] = morsel
 
             else:
                 morsel = None
```
I will provide a proper implementation asap.

Solution patch:
diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -999,8 +999,9 @@ def _parse_browser_specification(browser_name, profile=None, keyring=None, conta
 class LenientSimpleCookie(http.cookies.SimpleCookie):
     """More lenient version of http.cookies.SimpleCookie"""
     # From https://github.com/python/cpython/blob/v3.10.7/Lib/http/cookies.py
-    _LEGAL_KEY_CHARS = r"\w\d!#%&'~_`><@,:/\$\*\+\-\.\^\|\)\(\?\}\{\="
-    _LEGAL_VALUE_CHARS = _LEGAL_KEY_CHARS + r"\[\]"
+    # We use Morsel's legal key chars to avoid errors on setting values
+    _LEGAL_KEY_CHARS = r'\w\d' + re.escape('!#$%&\'*+-.:^_`|~')
+    _LEGAL_VALUE_CHARS = _LEGAL_KEY_CHARS + re.escape('(),/<=>?@[]{}')
 
     _RESERVED = {
         "expires",
@@ -1046,25 +1047,17 @@ def load(self, data):
             return super().load(data)
 
         morsel = None
-        index = 0
-        length = len(data)
-
-        while 0 <= index < length:
-            match = self._COOKIE_PATTERN.search(data, index)
-            if not match:
-                break
-
-            index = match.end(0)
-            if match.group("bad"):
+        for match in self._COOKIE_PATTERN.finditer(data):
+            if match.group('bad'):
                 morsel = None
                 continue
 
-            key, value = match.group("key", "val")
+            key, value = match.group('key', 'val')
 
-            if key[0] == "$":
-                if morsel is not None:
-                    morsel[key[1:]] = True
-                continue
+            is_attribute = False
+            if key.startswith('$'):
+                key = key[1:]
+                is_attribute = True
 
             lower_key = key.lower()
             if lower_key in self._RESERVED:
@@ -1081,6 +1074,9 @@ def load(self, data):
 
                 morsel[key] = value
 
+            elif is_attribute:
+                morsel = None
+
             elif value is not None:
                 morsel = self.get(key, http.cookies.Morsel())
                 real_value, coded_value = self.value_decode(value)


Test patch:
diff --git a/test/test_cookies.py b/test/test_cookies.py
--- a/test/test_cookies.py
+++ b/test/test_cookies.py
@@ -277,9 +277,24 @@ def test_lenient_parsing(self):
                 "a=b; invalid; Version=1; c=d",
                 {"a": "b", "c": "d"},
             ),
+            (
+                "Reset morsel after invalid to not capture attributes",
+                "a=b; $invalid; $Version=1; c=d",
+                {"a": "b", "c": "d"},
+            ),
             (
                 "Continue after non-flag attribute without value",
                 "a=b; path; Version=1; c=d",
                 {"a": "b", "c": "d"},
             ),
+            (
+                "Allow cookie attributes with `$` prefix",
+                'Customer="WILE_E_COYOTE"; $Version=1; $Secure; $Path=/acme',
+                {"Customer": ("WILE_E_COYOTE", {"version": "1", "secure": True, "path": "/acme"})},
+            ),
+            (
+                "Invalid Morsel keys should not result in an error",
+                "Key=Value; [Invalid]=Value; Another=Value",
+                {"Key": "Value", "Another": "Value"},
+            ),
         )