Repository: yt-dlp/yt-dlp

Issue:
[Crunchyroll] Log In support
### Checklist

- [X] I'm reporting a site feature request
- [X] I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

_No response_

### Example URLs

https://beta.crunchyroll.com/it/watch/G8WUN2ND8/La-citt-delle-bestie-magiche-coperta-di-neve
https://beta.crunchyroll.com/watch/GJWU2J7Z8/Clash-Class-A-vs-Class-B

### Description

Like I said the in site support issue 
https://github.com/yt-dlp/yt-dlp/issues/915

It is possible to not only support the website but also the login for crunchyroll.
Access to beta crunchyroll should be default in US for free and paid account (so you should be able to test this with a free account).
Like I said in the support site issue we already got two example on how to implement the log in support
https://github.com/hyugogirubato/Kamyroll-Python
the old basic script (really easy to understand)
https://github.com/hyugogirubato/Kamyroll-Python/blob/ffba19ec6f8c81dae4e5f35e368748fe2fca6494/kamyroll.py
and my own C# implementation (based on that old kamyroll python version, easier to understand at least for me)
https://github.com/federicorosso1993/Kamyroll-CSharp

I think the old crunchyroll log in support was already half broken (it required a cookies.txt file and the user agent), my own script and kamyroll-python do not require anything like that (my own implementation and the old version of kamyroll used this kind of code

`var cookie = cookieContainer.GetCookies(uri).Cast().FirstOrDefault(x => x.Name == "etp_rt");` 
in C# and a code like this 
```
r = requests.get(endpoint)
etp_rt = r.cookies.get("etp_rt") 
```
in python

So it should be possible to avoid the request of a cookie file from the user. I know that the new kamyroll implemetation in python don't even need that since it use the mobile api to log in.
The old implementation use the desktop app api.
My own implementation got also a comment with the web API implementation to log in (I made it as a comment because the desktop app api it's already working as an alternative).
You can use this page
https://www.crunchyroll.com/welcome/login 
and this page
https://www.crunchyroll.com/originals/it/index.html

to find the basic auth code (in a get with this url https://api.crunchyroll.com/start_session.0.json?sess_id=1&device_type=com.crunchyroll.static&device_id=whatvalueshouldbeforweb&access_token=giKq5eY27ny3cqz )
It's always the same value and it will work to start the log in.

Not sure how to implement it in a large python app like yt-dlp (I'm not really skilled with python) but the code it's already done and it's working in another app (made just to fix youtube-dl and now yt-dlp).



Discussion:
Log in for old website is also broken so that'll need to be fixed too.
I don't want to go through the trouble of getting account, testing urls, etc etc so here's how a login would look like for dlp.
This is the code i'm modifying 
```py
def login(args_login, us_unblocker):
    try:
        email = args_login.split(':')[0].strip()
        password = args_login.split(':')[1].strip()
    except:
        print("Invalid login")
        sys.exit(0)

    endpoint = "https://api.crunchyroll.com/start_session.0.json?version=1.0&access_token={}&device_type={}&device_id={}".format(
        "LNDJgOit5yaRIWN", "com.crunchyroll.windows.desktop", "Az2srGnChW65fuxYz2Xxl1GcZQgtGgI")
    session = requests.session()

    if us_unblocker:
        session.proxies.update(proxies)

    r = session.get(endpoint)
    session_id = r.json().get("data").get("session_id")

    endpoint = "https://api.crunchyroll.com/login.0.json"
    data = {"session_id": session_id, "account": email, "password": password}
    r = requests.post(endpoint, data=data)

    start_session(session_id, us_unblocker)
```

Here's how it will broadly look for dlp
```py
def _login(self):
    username, password = self._get_login_info()
    if username is None:
        return
    session_id = self._download_json(f'https://api.crunchyroll.com/start_session.0.json?version=1.0&access_token=LNDJgOit5yaRIWN&device_type=com.crunchyroll.windows.desktop&device_id=Az2srGnChW65fuxYz2Xxl1GcZQgtGgI', None)['data']['session_id']
    # IDK why this request is being made if the session is not being used and the request is not being parsed
    r = self._download_json('https://api.crunchyroll.com/login.0.json', None, data=json.dump({"session_id": session_id, "account": username, "password": password}))
```
FYI, if you're ok with using cookies rather than a proper login, #2506 implements that (for US users. I know it already worked for non-US users). Any login system that gets you an `etp_rt` cookie should integrate with that without hassle as well. Other mechanisms would probably need to alter the way `auth_response` is generated.
Is including static 'access tokens' like kamyroll does a legal issue yt-dlp wants to avoid? If it isn't, I'd be happy to whip up a login mechanism for the beta extractor based on what the current version of kamyroll is doing. It's a pretty small addition to what I do in #2506. Basically just generate `auth_response` differently, supplying a hardcoded basic token copied from kamyroll in the `Authorization` header and putting `grant_type=password&username=<username>&password=<password>&scope=offline_access` in the post data. (kamyroll puts the scope key in, but in my experiments it doesn't seem to make any difference)
Is the "access_token" obtainable publically? ie, can it be deduced from the webpage or the requests that we get in the browser. If so, it is fine to hardcode it. If it is instead some sort of leaked key, that could have legal implications and is not something I want to deal with
> Is the "access_token" obtainable publically? ie, can it be deduced from the webpage or the requests that we get in the browser. If so, it is fine to hardcode it. If it is instead some sort of leaked key, that could have legal implications and is not something I want to deal with

Yes. You can get it from here
https://www.crunchyroll.com/welcome/login

The access token is always the same
https://api.crunchyroll.com/get_upsell_data.0.json?sess_id=1&device_id=whatvalueshouldbeforweb&device_type=com.crunchyroll.static&access_token=giKq5eY27ny3cqz&referer=https:%2F%2Fwww.crunchyroll.com%2Fit%2Fwelcome%2Flogin 
Then u can hardcode it. We do the same for other sites including youtube
The key used by the current kamyroll implementation (`aHJobzlxM2F3dnNrMjJ1LXRzNWE6cHROOURteXRBU2Z6QjZvbXVsSzh6cUxzYTczVE1TY1k=`) is datamined from the mobile app or its traffic, rather than the web site, afaik. The mobile app is freely available, but presumably has a EULA of some kind, unlike a web site. Still ok?
> Not sure how to implement it in a large python app like yt-dlp (I'm not really skilled with python) but the code it's already done and it's working in another app (made just to fix youtube-dl and now yt-dlp).

I quote myself
"Not sure how to implement it in a large python app like yt-dlp (I'm not really skilled with python) but the code it's already done and it's working in another app (made just to fix youtube-dl and now yt-dlp)."
My own implementation is in a C# app
https://github.com/federicorosso1993/Kamyroll-CSharp
Hmm, ok. After some inspection, and your comments, I'm starting to see how the web login works, so I think I could implement that directly, without any hardcoded tokens. I'll work on it soon.
I also find it amusing that `whatvalueshouldbeforweb` is literally the device id... somebody used a template and never changed it I guess :-P
I gave it a shot, but the problem I'm running into is that I get a 403 Forbidden response from cloudflare, that wants me to prove I'm human with a captcha, when I try to download the login page (https://www.crunchyroll.com/welcome/login). Afaict this happens even when I'm sending the exact same http request as the browser does (for example, using the 'copy as curl' option from my browser's inspector), so I'm a bit stumped as to how to get around it.

Resolving patch:
diff --git a/yt_dlp/extractor/crunchyroll.py b/yt_dlp/extractor/crunchyroll.py
--- a/yt_dlp/extractor/crunchyroll.py
+++ b/yt_dlp/extractor/crunchyroll.py
@@ -35,7 +35,6 @@
     sanitized_Request,
     traverse_obj,
     try_get,
-    urlencode_postdata,
     xpath_text,
 )
 from ..aes import (
@@ -44,8 +43,8 @@
 
 
 class CrunchyrollBaseIE(InfoExtractor):
-    _LOGIN_URL = 'https://www.crunchyroll.com/login'
-    _LOGIN_FORM = 'login_form'
+    _LOGIN_URL = 'https://www.crunchyroll.com/welcome/login'
+    _API_BASE = 'https://api.crunchyroll.com'
     _NETRC_MACHINE = 'crunchyroll'
 
     def _call_rpc_api(self, method, video_id, note=None, data=None):
@@ -62,50 +61,33 @@ def _login(self):
         username, password = self._get_login_info()
         if username is None:
             return
-
-        login_page = self._download_webpage(
-            self._LOGIN_URL, None, 'Downloading login page')
-
-        def is_logged(webpage):
-            return 'href="/logout"' in webpage
-
-        # Already logged in
-        if is_logged(login_page):
-            return
-
-        login_form_str = self._search_regex(
-            r'(?P<form><form[^>]+?id=(["\'])%s\2[^>]*>)' % self._LOGIN_FORM,
-            login_page, 'login form', group='form')
-
-        post_url = extract_attributes(login_form_str).get('action')
-        if not post_url:
-            post_url = self._LOGIN_URL
-        elif not post_url.startswith('http'):
-            post_url = compat_urlparse.urljoin(self._LOGIN_URL, post_url)
-
-        login_form = self._form_hidden_inputs(self._LOGIN_FORM, login_page)
-
-        login_form.update({
-            'login_form[name]': username,
-            'login_form[password]': password,
-        })
-
-        response = self._download_webpage(
-            post_url, None, 'Logging in', 'Wrong login info',
-            data=urlencode_postdata(login_form),
-            headers={'Content-Type': 'application/x-www-form-urlencoded'})
-
-        # Successful login
-        if is_logged(response):
+        if self._get_cookies(self._LOGIN_URL).get('etp_rt'):
             return
 
-        error = self._html_search_regex(
-            '(?s)<ul[^>]+class=["\']messages["\'][^>]*>(.+?)</ul>',
-            response, 'error message', default=None)
-        if error:
-            raise ExtractorError('Unable to login: %s' % error, expected=True)
-
-        raise ExtractorError('Unable to log in')
+        upsell_response = self._download_json(
+            f'{self._API_BASE}/get_upsell_data.0.json', None, 'Getting session id',
+            query={
+                'sess_id': 1,
+                'device_id': 'whatvalueshouldbeforweb',
+                'device_type': 'com.crunchyroll.static',
+                'access_token': 'giKq5eY27ny3cqz',
+                'referer': self._LOGIN_URL
+            })
+        if upsell_response['code'] != 'ok':
+            raise ExtractorError('Could not get session id')
+        session_id = upsell_response['data']['session_id']
+
+        login_response = self._download_json(
+            f'{self._API_BASE}/login.1.json', None, 'Logging in',
+            data=compat_urllib_parse_urlencode({
+                'account': username,
+                'password': password,
+                'session_id': session_id
+            }).encode('ascii'))
+        if login_response['code'] != 'ok':
+            raise ExtractorError('Login failed. Bad username or password?', expected=True)
+        if not self._get_cookies(self._LOGIN_URL).get('etp_rt'):
+            raise ExtractorError('Login succeeded but did not set etp_rt cookie')
 
     def _real_initialize(self):
         self._login()


Test patch: