Repository: streamlink/streamlink

Issue:
No streams found for newest CR episodes that are only available to premium users
*Thanks for reporting an issue!*
*Please read the contribution guidelines first! (see the link above)*
*Also check the list of known issues before reporting an issue!*

*Feel free to use the following template. Be as detailed as possible.*
*Please see the text preview to avoid unnecessary formatting errors.*
*Don't forget to remove this text before submitting.*

----

### Checklist

- [x] This is a bug report.
- [ ] This is a feature request.
- [ ] This is a plugin (improvement) request.
- [ ] I have read the contribution guidelines.

### Description
I'm not sure if its fair to call this a bug as it might be an issue on Crunchyroll's end either purposeful or not. I'm not able to watch the latest episode in a series if it's only available to premium users using Streamlink, even logged in as a premium user as there are no streams available according to Streamlink.

error: No playable streams found on this URL: http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957

I can't find any recent issues that match this behaviour and I used to be able to play the latest episode with Streamlink though the last time I used Streamlink was about a month or so ago for new episodes I've mainly been rewatching older shows.

URL Tested with http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957
Tested watching other URLs which are not exclusive to premium users and they work fine.

...

### Expected / Actual behavior
A stream should be found and playable using Streamlink as there is no CR related issues open at present.
...

### Reproduction steps / Explicit stream URLs to test

1. Grab a URL (E.G http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957) for a new episode which is only available to premium users at present.
2. Use URL with Streamlink as usual.

### Logs

_Logs are always required for a bug report, use `-l debug` [(help)](https://streamlink.github.io/cli.html#cmdoption-l)
Make sure to **remove username and password**
You can upload your logs to https://gist.github.com/ or_

```
[console][debug] OS:         Windows 10
[console][debug] Python:     3.5.2
[console][debug] Streamlink: 0.13.0
[console][debug] Requests(2.18.4), Socks(1.6.7), Websocket(0.48.0)
[console][info] Found matching plugin crunchyroll for URL http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957
[console][debug] Plugin specific arguments:
[console][debug]  --crunchyroll-username=howdlej (username)
[console][debug]  --crunchyroll-password=******** (password)
[plugin.crunchyroll][debug] Creating session with locale: en_GB
[plugin.crunchyroll][debug] Session created
[plugin.crunchyroll][debug] Using saved credentials
C:\Program Files (x86)\Streamlink\pkgs\urllib3\connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
error: No playable streams found on this URL: http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957
```

### Comments, screenshots, etc.

If this is a known issue and something CR is doing specifically to counter Streamlink then please feel free to delete or mark as closed but I couldn't see anything about this specific issue or any other related CR issues.


[Love Streamlink? Please consider supporting our collective. Thanks!](https://opencollective.com/streamlink/donate)



Discussion:
I will take a look. Are you in the correct region? It’s not a geoblocked episode is it?
Nope, not geo-blocked, I'm able to watch it fine through the website. Thanks for the response.
We've had this happen in the past @beardypig: https://github.com/streamlink/streamlink/issues/1014. @Orsidus Can you try this: `streamlink --crunchyroll-purge-credentials --crunchyroll-username=USER --crunchyroll-password=PASSWORD http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957 best` but with your user and password.
Excluding the initial line for password reasons.

```
[console][info] Found matching plugin crunchyroll for URL http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957
C:\Program Files (x86)\Streamlink\pkgs\urllib3\connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
[plugin.crunchyroll][warning] Session ID does not match account, resetting session ID
[plugin.crunchyroll][info] Successfully logged in as 'howdlej'
C:\Program Files (x86)\Streamlink\pkgs\urllib3\connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
error: No playable streams found on this URL: http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957
```
@Orsidus Hmm, you know I just realized that you're using en_GB for your session. If you feel comfortable with it could you send me your current creds so I can try from a en_US locale? If you feel comfortable with it I'll be on gitter (https://gitter.im/streamlink/streamlink) for a bit and you can just log in there with your Github account and send me a private message or you can email me. I seem to remember this happening before and it was time sensitive (started working after a day or two for some weird reason).
@gravyboat Don't take this personally, but unfortunately I don't feel comfortable doing that. However, my suspicion of it being directly linked to the fact the episode is currently not available to free users yet and only to premium users seems to be close to the truth. The issue you referred to earlier, the person claimed it said it was working over 2 weeks later than when he made his post.

Episodes on CR switch from being premium to free after 1 week.  I've tried a number of currently premium-only episodes on CR and not a single one is playing. I will check Food Wars Episode 22 tomorrow after the new episode comes to see if it plays and that should confirm that its linked directly to that.

However this is something that I've only noticed happening recently (maybe it's been happening for no more than 2 months or so) but I did notice there was news on their website about changes to their backend recently so prehaps they've done something to counter however streamlink obtains the video streams for non-free episodes.

URLs Tested:
http://www.crunchyroll.com/darling-in-the-franxx/episode-21-for-you-my-love-769615
http://www.crunchyroll.com/boruto-naruto-next-generations/episode-62-the-otsutsuki-invasion-765641
http://www.crunchyroll.com/food-wars-shokugeki-no-soma/episode-22-to-the-final-battleground-769065 (This episode specifically is listed as becoming free tomorrow (Jun 17th))
http://www.crunchyroll.com/megalobox/episode-11-a-deadmarch-768819

@Orsidus No worries I totally understand. I'm pretty sure the last time this happened it was from outside the US as well which made me think something weird is going on with how their edge nodes work where the website falls back to other servers when it doesn't find the content, but something about how we make the request doesn't (sounds weird but honestly it's the only thing that I can think would cause this unless they have some weird mitigation in place as you noted). That food wars episode you linked is 6 days old at this point however so you would think it would have propagated.

Do you have the link to the blog post that talks about these recent changes? The last time I remember this happening was at least 4-6 months ago, if not more.
@gravyboat This is the blog post: https://blog.ellation.com/crunchyroll-maintenance-e11c4c065737 Though I had actually had this issue myself once last week before this rolled out so I don't think this has changed anything since once it happened again this week.

If there is any tests or anything you can think of that might give you some more information to work with feel free to let me know and I'll run what I can. I did notice that this new VRV thing they have going on is relatively new to me at least though I cannot access it from outside the U.S it might've caused them to do something to change how their premium streams work perhaps?
you could try to change your locale settings with `--locale en_US`
https://streamlink.github.io/cli.html#cmdoption-locale

but reset your session with `--crunchyroll-purge-credentials`

maybe it changes something.
```
streamlink --crunchyroll-purge-credentials --locale en_US --crunchyroll-username=howdlej --crunchyroll-password=<Ommited> http://www.crunchyroll.com/my-hero-academia/episode-49-one-for-all-768957 best
```

Same result. Unable to find any playable streams.
Seems like a similar issue to #1574. @Orsidus, do you have an Android smart phone? If so, could you test out the Android App and see if those premium streams play? 
@beardypig The only error I've ever gotten besides the https warning is "No playable streams found on this URL" I've never had the media look up error refeered to in that issue. To update, the link for Food Wars Episode 22 now works but the rest do not as predicted. I have an android phone and just tested that all episodes linked earlier work on the android app.
Thanks @Orsidus. It believe it is related to the session ID generation - which was the root cause of #1574 :)
@beardypig Fingers crossed! Thanks for the helped so far, let me know if there's anything else you need testing wise.

Resolving patch:
diff --git a/src/streamlink/cache.py b/src/streamlink/cache.py
--- a/src/streamlink/cache.py
+++ b/src/streamlink/cache.py
@@ -2,8 +2,8 @@
 import os
 import shutil
 import tempfile
+from time import time, mktime
 
-from time import time
 from .compat import is_win32
 
 if is_win32:
@@ -62,7 +62,7 @@ def _save(self):
         except (IOError, OSError):
             os.remove(tempname)
 
-    def set(self, key, value, expires=60 * 60 * 24 * 7):
+    def set(self, key, value, expires=60 * 60 * 24 * 7, expires_at=None):
         self._load()
         self._prune()
 
@@ -71,6 +71,9 @@ def set(self, key, value, expires=60 * 60 * 24 * 7):
 
         expires += time()
 
+        if expires_at:
+            expires = mktime(expires_at.timetuple())
+
         self._cache[key] = dict(value=value, expires=expires)
         self._save()
 
diff --git a/src/streamlink/plugins/crunchyroll.py b/src/streamlink/plugins/crunchyroll.py
--- a/src/streamlink/plugins/crunchyroll.py
+++ b/src/streamlink/plugins/crunchyroll.py
@@ -3,23 +3,16 @@
 import random
 import re
 import string
+import logging
+from uuid import uuid4
 
 from streamlink.plugin import Plugin, PluginError, PluginArguments, PluginArgument
-from streamlink.plugin.api import http, validate
+from streamlink.plugin.api import http, validate, useragents
 from streamlink.stream import HLSStream
 
-API_URL = "https://api.crunchyroll.com/{0}.0.json"
-API_DEFAULT_LOCALE = "en_US"
-API_USER_AGENT = "Mozilla/5.0 (iPhone; iPhone OS 8.3.0; {0})"
-API_HEADERS = {
-    "Host": "api.crunchyroll.com",
-    "Accept-Encoding": "gzip, deflate",
-    "Accept": "*/*",
-    "Content-Type": "application/x-www-form-urlencoded"
-}
-API_VERSION = "2313.8"
-API_ACCESS_TOKEN = "QWjz212GspMHH9h"
-API_DEVICE_TYPE = "com.crunchyroll.iphone"
+log = logging.getLogger(__name__)
+
+
 STREAM_WEIGHTS = {
     "low": 240,
     "mid": 420,
@@ -106,40 +99,70 @@ def __init__(self, msg, code):
 
 
 class CrunchyrollAPI(object):
-    def __init__(self, session_id=None, auth=None, locale=API_DEFAULT_LOCALE):
+    _api_url = "https://api.crunchyroll.com/{0}.0.json"
+    _default_locale = "en_US"
+    _user_agent = "Dalvik/1.6.0 (Linux; U; Android 4.4.2; Android SDK built for x86 Build/KK)"
+    _version_code = 444
+    _version_name = "2.1.10"
+    _access_token = "Scwg9PRRZ19iVwD"
+    _access_type = "com.crunchyroll.crunchyroid"
+
+    def __init__(self, cache, session_id=None, locale=_default_locale):
         """Abstract the API to access to Crunchyroll data.
 
         Can take saved credentials to use on it's calls to the API.
         """
+        self.cache = cache
         self.session_id = session_id
-        self.auth = auth
+        if self.session_id:  # if the session ID is setup don't use the cached auth token
+            self.auth = None
+        else:
+            self.auth = cache.get("auth")
+        self.device_id = cache.get("device_id") or self.generate_device_id()
         self.locale = locale
+        self.headers = {
+            "X-Android-Device-Is-GoogleTV": "0",
+            "X-Android-Device-Product": "google_sdk_x86",
+            "X-Android-Device-Model": "Android SDK built for x86",
+            "Using-Brightcove-Player": "1",
+            "X-Android-Release": "4.4.2",
+            "X-Android-SDK": "19",
+            "X-Android-Application-Version-Name": self._version_name,
+            "X-Android-Application-Version-Code": str(self._version_code),
+            'User-Agent': self._user_agent
+        }
 
-    def _api_call(self, entrypoint, params, schema=None):
+    def _api_call(self, entrypoint, params=None, schema=None):
         """Makes a call against the api.
 
         :param entrypoint: API method to call.
         :param params: parameters to include in the request data.
         :param schema: schema to use to validate the data
         """
-        url = API_URL.format(entrypoint)
+        url = self._api_url.format(entrypoint)
 
         # Default params
-        params = dict(params)
+        params = params or {}
+        if self.session_id:
+            params.update({
+                "session_id": self.session_id
+            })
+        else:
+            params.update({
+                "device_id": self.device_id,
+                "device_type": self._access_type,
+                "access_token": self._access_token,
+                "version": self._version_code
+            })
         params.update({
-            "version": API_VERSION,
             "locale": self.locale.replace('_', ''),
         })
 
         if self.session_id:
             params["session_id"] = self.session_id
 
-        # Headers
-        headers = dict(API_HEADERS)
-        headers['User-Agent'] = API_USER_AGENT.format(self.locale)
-
         # The certificate used by Crunchyroll cannot be verified in some environments.
-        res = http.get(url, params=params, headers=headers, verify=False)
+        res = http.post(url, data=params, headers=self.headers, verify=False)
         json_res = http.json(res, schema=_api_schema)
 
         if json_res["error"]:
@@ -153,47 +176,66 @@ def _api_call(self, entrypoint, params, schema=None):
 
         return data
 
-    def start_session(self, device_id, **kwargs):
-        """Starts a session against Crunchyroll's server.
+    def generate_device_id(self):
+        device_id = str(uuid4())
+        # cache the device id
+        self.cache.set("device_id", 365 * 24 * 60 * 60)
+        log.debug("Device ID: {0}".format(device_id))
+        return device_id
+
 
-        Is recommended that you call this method before making any other calls
-        to make sure you have a valid session against the server.
+    def start_session(self):
         """
-        if self.session_id:
-            http.cookies["sess_id"] = self.session_id
-        res = http.get("https://www.crunchyroll.com",
-                       headers={"User-Agent": API_USER_AGENT.format(self.locale)})
-        return res.cookies.get("sess_id", self.session_id)
-
-    def login(self, username, password, **kwargs):
-        """Authenticates the session to be able to access restricted data from
-        the server (e.g. premium restricted videos).
+            Starts a session against Crunchyroll's server.
+            Is recommended that you call this method before making any other calls
+            to make sure you have a valid session against the server.
+        """
+        params = {}
+        if self.auth:
+            params["auth"] = self.auth
+        self.session_id = self._api_call("start_session", params, schema=_session_schema)
+        log.debug("Session created with ID: {0}".format(self.session_id))
+        return self.session_id
+
+    def login(self, username, password):
+        """
+            Authenticates the session to be able to access restricted data from
+            the server (e.g. premium restricted videos).
         """
         params = {
             "account": username,
             "password": password
         }
 
-        return self._api_call("login", params, **kwargs)
+        login = self._api_call("login", params, schema=_login_schema)
+        self.auth = login["auth"]
+        self.cache.set("auth", login["auth"], expires_at=login["expires"])
+        return login
 
-    def get_info(self, media_id, fields=None, **kwargs):
-        """Returns the data for a certain media item.
+    def authenticate(self):
+        data = self._api_call("authenticate", {"auth": self.auth}, schema=_login_schema)
+        self.auth = data["auth"]
+        self.cache.set("auth", data["auth"], expires_at=data["expires"])
+        return data
 
-        :param media_id: id that identifies the media item to be accessed.
-        :param fields: list of the media"s field to be returned. By default the
-        API returns some fields, but others are not returned unless they are
-        explicity asked for. I have no real documentation on the fields, but
-        they all seem to start with the "media." prefix (e.g. media.name,
-        media.stream_data).
+    def get_info(self, media_id, fields=None, schema=None):
         """
-        params = {
-            "media_id": media_id
-        }
+            Returns the data for a certain media item.
+
+            :param media_id: id that identifies the media item to be accessed.
+            :param fields: list of the media"s field to be returned. By default the
+            API returns some fields, but others are not returned unless they are
+            explicity asked for. I have no real documentation on the fields, but
+            they all seem to start with the "media." prefix (e.g. media.name,
+            media.stream_data).
+            :param schema: validation schema to use
+        """
+        params = {"media_id": media_id}
 
         if fields:
             params["fields"] = ",".join(fields)
 
-        return self._api_call("info", params, **kwargs)
+        return self._api_call("info", params, schema=schema)
 
 
 class Crunchyroll(Plugin):
@@ -265,8 +307,8 @@ def _get_streams(self):
         media_id = int(match.group("media_id"))
 
         try:
-            info = api.get_info(media_id, fields=["media.stream_data"],
-                                schema=_media_schema)
+            # the media.stream_data field is required, no stream data is returned otherwise
+            info = api.get_info(media_id, fields=["media.stream_data"], schema=_media_schema)
         except CrunchyrollAPIError as err:
             raise PluginError(u"Media lookup error: {0}".format(err.msg))
 
@@ -300,18 +342,6 @@ def _get_streams(self):
 
         return streams
 
-    def _get_device_id(self):
-        """Returns the saved device id or creates a new one and saves it."""
-        device_id = self.cache.get("device_id")
-
-        if not device_id:
-            # Create a random device id and cache it for a year
-            char_set = string.ascii_letters + string.digits
-            device_id = "".join(random.sample(char_set, 32))
-            self.cache.set("device_id", device_id, 365 * 24 * 60 * 60)
-
-        return device_id
-
     def _create_api(self):
         """Creates a new CrunchyrollAPI object, initiates it's session and
         tries to authenticate it either by using saved credentials or the
@@ -322,50 +352,35 @@ def _create_api(self):
             self.cache.set("auth", None, 0)
             self.cache.set("session_id", None, 0)
 
-        current_time = datetime.datetime.utcnow()
-        device_id = self._get_device_id()
         # use the crunchyroll locale as an override, for backwards compatibility
         locale = self.get_option("locale") or self.session.localization.language_code
-        api = CrunchyrollAPI(
-            self.options.get("session_id") or self.cache.get("session_id"), self.cache.get("auth"), locale
-        )
-
-        self.logger.debug("Creating session with locale: {0}", locale)
-        api.session_id = api.start_session(device_id, schema=_session_schema)
-
-        # Save session and hope it lasts for a few hours
-        self.cache.set("session_id", api.session_id, 4 * 60 * 60)
-        self.logger.debug("Session created")
-
-        if api.auth:
-            self.logger.debug("Using saved credentials")
-        elif self.options.get("username"):
-            try:
-                self.logger.debug("Attempting to login using username and password")
-                login = api.login(
-                    self.options.get("username"),
-                    self.options.get("password"),
-                    schema=_login_schema
-                )
-                api.auth = login["auth"]
+        api = CrunchyrollAPI(self.cache, session_id=self.get_option("session_id"), locale=locale)
 
-                user_session_id = api.start_session(device_id, schema=_session_schema)
-                if user_session_id != api.session_id:
-                    self.logger.warning("Session ID does not match account, resetting session ID")
-                    api.session_id = user_session_id
+        if not self.get_option("session_id"):
+            self.logger.debug("Creating session with locale: {0}", locale)
+            api.start_session()
 
+            if api.auth:
+                self.logger.debug("Using saved credentials")
+                login = api.authenticate()
                 self.logger.info("Successfully logged in as '{0}'",
                                  login["user"]["username"] or login["user"]["email"])
-
-                expires = (login["expires"] - current_time).total_seconds()
-                self.cache.set("auth", login["auth"], expires)
-            except CrunchyrollAPIError as err:
-                raise PluginError(u"Authentication error: {0}".format(err.msg))
-        else:
-            self.logger.warning(
-                "No authentication provided, you won't be able to access "
-                "premium restricted content"
-            )
+            elif self.options.get("username"):
+                try:
+                    self.logger.debug("Attempting to login using username and password")
+                    api.login(self.options.get("username"),
+                              self.options.get("password"))
+                    login = api.authenticate()
+                    self.logger.info("Logged in as '{0}'",
+                                     login["user"]["username"] or login["user"]["email"])
+
+                except CrunchyrollAPIError as err:
+                    raise PluginError(u"Authentication error: {0}".format(err.msg))
+            else:
+                self.logger.warning(
+                    "No authentication provided, you won't be able to access "
+                    "premium restricted content"
+                )
 
         return api
 


Test patch:
diff --git a/tests/test_cache.py b/tests/test_cache.py
--- a/tests/test_cache.py
+++ b/tests/test_cache.py
@@ -3,8 +3,11 @@
 import tempfile
 import os.path
 
+import datetime
+
 import streamlink.cache
 from shutil import rmtree
+import time
 
 try:
     from unittest.mock import patch
@@ -52,9 +55,21 @@ def test_load_fail(self, exists_mock):
         self.assertEqual({}, self.cache._cache)
 
     def test_expired(self):
-        self.cache.set("value", 10, expires=-1)
+        self.cache.set("value", 10, expires=-20)
+        self.assertEqual(None, self.cache.get("value"))
+
+    def test_expired_at(self):
+        self.cache.set("value", 10, expires_at=datetime.datetime.now() - datetime.timedelta(seconds=20))
         self.assertEqual(None, self.cache.get("value"))
 
+    def test_not_expired(self):
+        self.cache.set("value", 10, expires=20)
+        self.assertEqual(10, self.cache.get("value"))
+
+    def test_expired_at(self):
+        self.cache.set("value", 10, expires_at=datetime.datetime.now() + datetime.timedelta(seconds=20))
+        self.assertEqual(10, self.cache.get("value"))
+
     def test_create_directory(self):
         try:
             streamlink.cache.cache_dir = os.path.join(tempfile.gettempdir(), "streamlink-test")