Repository: streamlink/streamlink

Issue:
plugins.crunchyroll: Unauthenticated request error, login system may have changed ?
### Checklist

- [X] This is a plugin issue and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed plugin issues](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22plugin+issue%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest stable release

### Description

The Crunchyroll plugin worked well last Sunday. Now I get an Unauthenticated request message, even though my username/password haven't changed. I provide the log for an episode I know for a fact played well this weekend.

My best guess would be a change in the way Crunchyroll deals with logins, as I can now activate the CR app on my TV just by writing an activation code on my web browser session instead of writing the whole password, which was not an option weeks ago.

### Debug log

```text
C:\Users\Stella>streamlink --loglevel debug crunchyroll.com/fr/spy-x-family/episode-1-operation-strix-842454
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.9.10
[cli][debug] Streamlink: 3.2.0
[cli][debug] Requests(2.27.1), Socks(1.7.1), Websocket(1.2.3)
[cli][debug] Arguments:
[cli][debug]  url=crunchyroll.com/fr/spy-x-family/episode-1-operation-strix-842454
[cli][debug]  --loglevel=debug
[cli][debug]  --locale=en_US
[cli][debug]  --player="C:\Users\Stella\Documents\MPV\mpv.exe"
[cli][debug]  --player-args=--keep-open=yes --cache=yes
[cli][debug]  --player-no-close=True
[cli][debug]  --title={author} - {category} - {title}
[cli][debug]  --default-stream=['best']
[cli][debug]  --ringbuffer-size=134217728
[cli][debug]  --stream-segment-threads=3
[cli][debug]  --hls-segment-threads=3
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug]  --crunchyroll-username=********
[cli][debug]  --crunchyroll-password=********
[cli][info] Found matching plugin crunchyroll for URL crunchyroll.com/fr/spy-x-family/episode-1-operation-strix-842454
[utils.l10n][debug] Language code: en_US
[plugins.crunchyroll][debug] Creating session with locale: en_US
Traceback (most recent call last):
  File "runpy.py", line 197, in _run_module_as_main
  File "runpy.py", line 87, in _run_code
  File "C:\Program Files (x86)\Streamlink\bin\streamlink.exe\__main__.py", line 18, in <module>
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink_cli\main.py", line 1061, in main
    handle_url()
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink_cli\main.py", line 570, in handle_url
    streams = fetch_streams(plugin)
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink_cli\main.py", line 464, in fetch_streams
    return plugin.streams(stream_types=args.stream_types,
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\plugin\plugin.py", line 336, in streams
    ostreams = self._get_streams()
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\plugins\crunchyroll.py", line 307, in _get_streams
    api = self._create_api()
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\plugins\crunchyroll.py", line 372, in _create_api
    api.start_session()
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\plugins\crunchyroll.py", line 190, in start_session
    self.session_id = self._api_call("start_session", params, schema=_session_schema)
  File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\plugins\crunchyroll.py", line 166, in _api_call
    raise CrunchyrollAPIError(err_msg, err_code)
streamlink.plugins.crunchyroll.CrunchyrollAPIError: Unauthenticated request
```



Discussion:
Yes, same error here, it seems that Crunchyroll changed the access token again, tested on stable and nightly build
Same error, would be great to know how streamlink devs get that access token...
> would be great to know how streamlink devs get that access token

1. https://github.com/streamlink/streamlink/blame/e4490ad3519ff53ea53b3c8fc4356cf519dc5d1f/src/streamlink/plugins/crunchyroll.py#L102
2. https://github.com/streamlink/streamlink/commit/09f46a33b0b1918ce7e0ad0534788e0fd786caea
3. https://github.com/streamlink/streamlink/pull/2788
4. https://github.com/streamlink/streamlink/issues/2785#issuecomment-581081985
> > would be great to know how streamlink devs get that access token
> 
>     1. https://github.com/streamlink/streamlink/blame/e4490ad3519ff53ea53b3c8fc4356cf519dc5d1f/src/streamlink/plugins/crunchyroll.py#L102
> 
>     2. [09f46a3](https://github.com/streamlink/streamlink/commit/09f46a33b0b1918ce7e0ad0534788e0fd786caea)
> 
>     3. [Update Crunchyroll access token. Fixes streamlink/streamlink issue #2785. #2788](https://github.com/streamlink/streamlink/pull/2788)
> 
>     4. [Can't authenticate Crunchyroll, CrunchyrollAPIError: Unauthenticated request #2785 (comment)](https://github.com/streamlink/streamlink/issues/2785#issuecomment-581081985)

yeah, I've check all stuff related to crunchyroll plugin, but haven't found where can I get the new token when they change it yet.
Seems that in June this plugin will be gone (at least login logic), since crunchyroll is migrating to the new API. 
The access token comes from the Android app.  You'll need to either install the app on a phone and run it from there or perhaps it could be run via an Android emulator.  You'll need to configure a proxy and use something like mitmproxy to spy on the network requests.  Hopefully you will find a POST request with `access_token` in the payload data.

It's possible that things have changed now.  The plugin seems to be based on app version 2.1.0, but the current Crunchyroll app version appears to be 3.18.0 (I think).  It's possible they have not just changed the token value, but the whole API.  This failure could therefore be because they've turned the old API off.

> The access token comes from the Android app. You'll need to either install the app on a phone and run it from there or perhaps it could be run via an Android emulator. You'll need to configure a proxy and use something like mitmproxy to spy on the network requests. Hopefully you will find a POST request with `access_token` in the payload data.
> 
> It's possible that things have changed now. The plugin seems to be based on app version 2.1.0, but the current Crunchyroll app version appears to be 3.18.0 (I think). It's possible they have not just changed the token value, but the whole API. This failure could therefore be because they've turned the old API off.

Yeah, I've installed several versions on Crunchyroll app on android emulator, I run HttpToolkit to intercept that traffic, but the token seems encrypted `fgS4DSihTXaEj0HuWiaWlr:APA91bEjq73PQC327j0T1g5c41hzbo-bCW1Q3DCYzFepsczHDIBnmAMxNg1-eu56CLa5R6AINoL0TPavBtjB9ldm06uH_lI-m187ekOwehzMGxC6IvrHaaa5oVdwPtp1TF9ahWwRobYu`
So this method wouldn't be a workaround this time.
I installed the app on my phone and looked at the traffic.  I saw a similarly long token, but that appears to be a bearer token, which is another thing.  I don't have a Crunchyroll account anyway, so that doesn't help.

My phone is 32 bit ARMv7, with Android 7.1.  I tried using the last 2.x release I could find via APKPure (version 2.5.0), but it did not appear in the app menu, so I gave up with that one.  It seems to me, having looked at both APKPure and APKMirror that the 3.x releases target Android 6.0+ while 2.5.0 targets Android 5.0+.  If you can manage to get version 2.5.0 installed and running somewhere, it might yield the token you're looking for (if it works at all).  I did manage to get version 2.3.0 from APK mirror installed and it ran, but then produced an error message.  The network traffic did yield a token, but it was an old one, so no good.

So, I'm not sure about this, but it could be that with an Android 5.1 based device you might get a 2.x version, which might have the token we're all looking for.  Or it could be that the plugin needs a rewrite to do things the way they're done in the 3.x app releases.  I'm not sure why the plugin is based on the Android app, perhaps the website uses DRM or did other stuff in JS that would have made the plugin implementation a lot harder...

> I installed the app on my phone and looked at the traffic. I saw a similarly long token, but that appears to be a bearer token, which is another thing. I don't have a Crunchyroll account anyway, so that doesn't help.
> 
> My phone is 32 bit ARMv7, with Android 7.1. I tried using the last 2.x release I could find via APKPure (version 2.5.0), but it did not appear in the app menu, so I gave up with that one. It seems to me, having looked at both APKPure and APKMirror that the 3.x releases target Android 6.0+ while 2.5.0 targets Android 5.0+. If you can manage to get version 2.5.0 installed and running somewhere, it might yield the token you're looking for (if it works at all). I did manage to get version 2.3.0 from APK mirror installed and it ran, but then produced an error message. The network traffic did yield a token, but it was an old one, so no good.
> 
> So, I'm not sure about this, but it could be that with an Android 5.1 based device you might get a 2.x version, which might have the token we're all looking for. Or it could be that the plugin needs a rewrite to do things the way they're done in the 3.x app releases. I'm not sure why the plugin is based on the Android app, perhaps the website uses DRM or did other stuff in JS that would have made the plugin implementation a lot harder...

Didn't know about the android versions, I'll try to run a 5.1 based emulator an see what happens.
I had no luck , I tried v2.5.0 and v2.6.0 and both uses the old keys in the request, from v3.0.0 and newer releases points to the new API so can't get the token from this point. And the new login system seems to defer a lot. So a plugin rewrite is mandatory.
That's disappointing to hear, but thanks for trying it out.

There's one last chance to see if this works, seems that the 2.5.0 app version received the last update in 30th March, 2022. Just the day before this plugin stop working. I tried to download this version fom apkpure.com but this folks split the apk so I can't install it. If i could get the apk of this version and install it, maybe i'll be able to get the token. At least to make it work until June.
That is likely the same one I tried on my phone and could not get to work (the one I was talking about in my post, above).  Maybe you will have better luck with it.  As I guess you may have worked out, you'll need to install the APKPure app to install the split APKs from that site.

I'm confused about the dates shown - are they release dates or simply upload dates?  I was not able determine which.
One can use the token from Windows desktop client, instead of the android.

Changing these lines

https://github.com/streamlink/streamlink/blob/867b9b3b66aab57c0fcb3ab117a275f29a23b71a/src/streamlink/plugins/crunchyroll.py#L102-L103

to

``` python
_access_token = "LNDJgOit5yaRIWN"
_access_type = "com.crunchyroll.windows.desktop"
```

Seems to fix the issue. This token has not changed since at least 2020, so I think it's more stable.

### Debug log

```
[cli][debug] OS:         Windows 8.1
[cli][debug] Python:     3.10.3
[cli][debug] Streamlink: 4.0.0
[cli][debug] Requests(2.27.1), Socks(1.7.1), Websocket(1.3.2)
[cli][debug] Arguments:
[cli][debug]  url=https://www.crunchyroll.com/hunter-x-hunter/episode-148-past-x-and-x-future-654039
[cli][debug]  --loglevel=debug
[cli][debug]  --player=C:\mpv\mpv.com
[cli][debug]  --player-args=--pause
[cli][debug]  --verbose-player=True
[cli][debug]  --player-no-close=True
[cli][debug]  --title={title} -!- {author} -!- {category}
[cli][debug]  --stream-sorting-excludes=['None', '>=1440p60']
[cli][debug]  --ringbuffer-size=67108864
[cli][debug]  --stream-segment-threads=4
[cli][debug]  --mux-subtitles=True
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files\Streamlink\ffmpeg\ffmpeg.exe
[cli][info] Found matching plugin crunchyroll for URL https://www.crunchyroll.com/hunter-x-hunter/episode-148-past-x-and-x-future-654039
[utils.l10n][debug] Language code: pt_BR
[plugins.crunchyroll][debug] Creating session with locale: pt_BR
[plugins.crunchyroll][debug] Session created with ID: a552879f769b72538fdb60edf2945efd
[plugins.crunchyroll][warning] No authentication provided, you won't be able to access premium restricted content
[plugins.crunchyroll][debug] Loading streams from adaptive playlist
[utils.l10n][debug] Language code: pt_BR
[utils.l10n][debug] Language code: pt_BR
[utils.l10n][debug] Language code: pt_BR
Available streams: 240p_alt (worst), 240p, 360p_alt, 360p, 480p_alt, 480p, 720p_alt, 720p, 1080p_alt, 1080p (best)
```
> ```python
> LNDJgOit5yaRIWN
> ```

You have saved us we are eternally grateful. I didn't know that there was a Windows app. Thanks for sharing.
@kyldery, that's excellent, thanks very much.  I did not know there was a Windows app either!


Resolving patch:
diff --git a/src/streamlink/plugins/crunchyroll.py b/src/streamlink/plugins/crunchyroll.py
--- a/src/streamlink/plugins/crunchyroll.py
+++ b/src/streamlink/plugins/crunchyroll.py
@@ -96,16 +96,14 @@ def __init__(self, msg, code):
 class CrunchyrollAPI:
     _api_url = "https://api.crunchyroll.com/{0}.0.json"
     _default_locale = "en_US"
-    _user_agent = "Dalvik/1.6.0 (Linux; U; Android 4.4.2; Android SDK built for x86 Build/KK)"
-    _version_code = 444
-    _version_name = "2.1.10"
-    _access_token = "WveH9VkPLrXvuNm"
-    _access_type = "com.crunchyroll.crunchyroid"
+    _version_name = "1.3.1.0"
+    _access_token = "LNDJgOit5yaRIWN"
+    _access_type = "com.crunchyroll.windows.desktop"
 
     def __init__(self, cache, session, session_id=None, locale=_default_locale):
         """Abstract the API to access to Crunchyroll data.
 
-        Can take saved credentials to use on it's calls to the API.
+        Can take saved credentials to use on its calls to the API.
         """
         self.cache = cache
         self.session = session
@@ -116,17 +114,6 @@ def __init__(self, cache, session, session_id=None, locale=_default_locale):
             self.auth = cache.get("auth")
         self.device_id = cache.get("device_id") or self.generate_device_id()
         self.locale = locale
-        self.headers = {
-            "X-Android-Device-Is-GoogleTV": "0",
-            "X-Android-Device-Product": "google_sdk_x86",
-            "X-Android-Device-Model": "Android SDK built for x86",
-            "Using-Brightcove-Player": "1",
-            "X-Android-Release": "4.4.2",
-            "X-Android-SDK": "19",
-            "X-Android-Application-Version-Name": self._version_name,
-            "X-Android-Application-Version-Code": str(self._version_code),
-            'User-Agent': self._user_agent
-        }
 
     def _api_call(self, entrypoint, params=None, schema=None):
         """Makes a call against the api.
@@ -148,17 +135,17 @@ def _api_call(self, entrypoint, params=None, schema=None):
                 "device_id": self.device_id,
                 "device_type": self._access_type,
                 "access_token": self._access_token,
-                "version": self._version_code
             })
         params.update({
-            "locale": self.locale.replace('_', ''),
+            "locale": self.locale.replace("_", ""),
+            "version": self._version_name,
+            "connectivity_type": "ethernet",
         })
 
         if self.session_id:
             params["session_id"] = self.session_id
 
-        # The certificate used by Crunchyroll cannot be verified in some environments.
-        res = self.session.http.post(url, data=params, headers=self.headers, verify=False)
+        res = self.session.http.post(url, data=params)
         json_res = self.session.http.json(res, schema=_api_schema)
 
         if json_res["error"]:
@@ -175,7 +162,7 @@ def _api_call(self, entrypoint, params=None, schema=None):
     def generate_device_id(self):
         device_id = str(uuid4())
         # cache the device id
-        self.cache.set("device_id", 365 * 24 * 60 * 60)
+        self.cache.set("device_id", device_id, expires=365 * 24 * 60 * 60)
         log.debug("Device ID: {0}".format(device_id))
         return device_id
 
@@ -212,7 +199,7 @@ def authenticate(self):
             data = self._api_call("authenticate", {"auth": self.auth}, schema=_login_schema)
         except CrunchyrollAPIError:
             self.auth = None
-            self.cache.set("auth", None, expires_at=0)
+            self.cache.set("auth", None, expires=0)
             log.warning("Saved credentials have expired")
             return
 
@@ -248,10 +235,17 @@ def get_info(self, media_id, fields=None, schema=None):
     (?:
         /(en-gb|es|es-es|pt-pt|pt-br|fr|de|ar|it|ru)
     )?
-    (?:/[^/&?]+)?
-    /[^/&?]+-(?P<media_id>\d+)
+    (?:
+        (?:
+            (?:/[^/&?]+)?
+            /[^/&?]+-(?P<media_id>\d+)
+        )
+        |
+        /watch/(?P<beta_id>\w+)/[\w-]+
+    )
 """, re.VERBOSE))
 class Crunchyroll(Plugin):
+
     arguments = PluginArguments(
         PluginArgument(
             "username",
@@ -305,9 +299,34 @@ def stream_weight(cls, key):
         return Plugin.stream_weight(key)
 
     def _get_streams(self):
-        api = self._create_api()
-        media_id = int(self.match.group("media_id"))
+        beta_json_re = re.compile(r"window.__INITIAL_STATE__\s*=\s*({.*});")
+
+        beta_id = self.match.group("beta_id")
+        if beta_id:
+            json = self.session.http.get(self.url, schema=validate.Schema(
+                validate.parse_html(),
+                validate.xml_xpath_string(".//script[contains(text(), 'window.__INITIAL_STATE__')]/text()"),
+                validate.any(None, validate.all(
+                    validate.transform(beta_json_re.search),
+                    validate.any(None, validate.all(
+                        validate.get(1),
+                        validate.parse_json(),
+                        validate.any(None, validate.all(
+                            {"content": {"byId": {str: {"external_id": validate.all(
+                                validate.transform(lambda s: int(s.replace("EPI.", ""))),
+                            )}}}},
+                            validate.get(("content", "byId")),
+                        )),
+                    )),
+                )),
+            ))
+            if not json or beta_id not in json:
+                return
+            media_id = json[beta_id]["external_id"]
+        else:
+            media_id = int(self.match.group("media_id"))
 
+        api = self._create_api()
         try:
             # the media.stream_data field is required, no stream data is returned otherwise
             info = api.get_info(media_id, fields=["media.name", "media.series_name",
@@ -320,6 +339,7 @@ def _get_streams(self):
 
         streams = {}
 
+        self.id = media_id
         self.title = info.get("name")
         self.author = info.get("series_name")
         self.category = info.get("media_type")
@@ -352,14 +372,13 @@ def _get_streams(self):
         return streams
 
     def _create_api(self):
-        """Creates a new CrunchyrollAPI object, initiates it's session and
+        """Creates a new CrunchyrollAPI object, initiates its session and
         tries to authenticate it either by using saved credentials or the
         user's username and password.
         """
         if self.options.get("purge_credentials"):
-            self.cache.set("session_id", None, 0)
-            self.cache.set("auth", None, 0)
-            self.cache.set("session_id", None, 0)
+            self.cache.set("device_id", None, expires=0)
+            self.cache.set("auth", None, expires=0)
 
         # use the crunchyroll locale as an override, for backwards compatibility
         locale = self.get_option("locale") or self.session.localization.language_code


Test patch:
diff --git a/tests/plugins/test_crunchyroll.py b/tests/plugins/test_crunchyroll.py
--- a/tests/plugins/test_crunchyroll.py
+++ b/tests/plugins/test_crunchyroll.py
@@ -5,18 +5,25 @@
 class TestPluginCanHandleUrlCrunchyroll(PluginCanHandleUrl):
     __plugin__ = Crunchyroll
 
-    should_match = [
-        "http://www.crunchyroll.com/idol-incidents/episode-1-why-become-a-dietwoman-728233",
-        "http://www.crunchyroll.com/ru/idol-incidents/episode-1-why-become-a-dietwoman-728233",
-        "http://www.crunchyroll.com/idol-incidents/media-728233",
-        "http://www.crunchyroll.com/fr/idol-incidents/media-728233",
-        "http://www.crunchyroll.com/media-728233",
-        "http://www.crunchyroll.com/de/media-728233",
-        "http://www.crunchyroll.fr/media-728233",
-        "http://www.crunchyroll.fr/es/media-728233"
+    should_match_groups = [
+        ("http://www.crunchyroll.com/idol-incidents/episode-1-a-title-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.com/ru/idol-incidents/episode-1-a-title-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.com/idol-incidents/media-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.com/fr/idol-incidents/media-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.com/media-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.com/de/media-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.fr/media-728233", {"media_id": "728233"}),
+        ("http://www.crunchyroll.fr/es/media-728233", {"media_id": "728233"}),
+        ("https://beta.crunchyroll.com/watch/GRNQ5DDZR/Game-Over", {"beta_id": "GRNQ5DDZR"}),
+        ("https://beta.crunchyroll.com/watch/ValidID123/any/thing?x&y", {"beta_id": "ValidID123"}),
     ]
 
     should_not_match = [
         "http://www.crunchyroll.com/gintama",
         "http://www.crunchyroll.es/gintama",
+        "http://beta.crunchyroll.com/",
+        "http://beta.crunchyroll.com/something",
+        "http://beta.crunchyroll.com/watch/",
+        "http://beta.crunchyroll.com/watch/not-a-valid-id",
+        "http://beta.crunchyroll.com/watch/not-a-valid-id/a-title",
     ]