Repository: streamlink/streamlink
Pull request #484 (2017-01-25 13:25:48)

plugins.twitch: update public API calls to use v5 API
The main change from v3 was to look up the channel ID, the user endpoint is used to find the channel ID instead of the channel endpoint (thanks @bastimeyer). 

I grouped the functions in to private/public/unused so that it's easier to tell which ones need to work. I also added some warnings to the unused/unsupported methods in case anyone is importing them (unlikely, I know). 

As discussed in #377. 

codecov-io: ## [Current coverage](https://codecov.io/gh/streamlink/streamlink/pull/484?src=pr) is 47.22% (diff: 37.03%)
> Merging [#484](https://codecov.io/gh/streamlink/streamlink/pull/484?src=pr) into [master](https://codecov.io/gh/streamlink/streamlink/branch/master?src=pr) will decrease coverage by **0.05%**

```diff
@@             master       #484   diff @@
==========================================
  Files            59         59          
  Lines          4987       4999    +12   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           2358       2361     +3   
- Misses         2629       2638     +9   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [5b9f68a...f73aa4e](https://codecov.io/gh/streamlink/streamlink/compare/5b9f68ad45f6ad9f39993004648e486d86b77606...f73aa4ec5aa413ee16893503fa22235d6e123338?src=pr)
beardypig: Good point about `--twitch-disable-hosting`, that could definitely be optimised. 
Can do the other change, it doesn't really reduce the complexity, but probably one result is sufficient, it might be a slightly faster request though. 
bastimeyer: > My original thinking was the channel you wanted might not be first, I'm not sure exactly what it searches and in what order it sorts the results..

Hm, I think you're right. It doesn't always return the expected data. This is weird. See my examples:

Returns the correct channel data of twitch.tv/diablo (even though there are more popular channels)
```
curl -H "Accept: application/vnd.twitchtv.v5+json" -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" "https://api.twitch.tv/kraken/search/channels?query=diablo&offset=0&limit=1"
```

Doesn't return any channels, although there's twitch.tv/dota (weird)
```
curl -H "Accept: application/vnd.twitchtv.v5+json" -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" "https://api.twitch.tv/kraken/search/channels?query=dota&offset=0&limit=1"
```

Returns a different channel other than twitch.tv/arma (weird)
```
curl -H "Accept: application/vnd.twitchtv.v5+json" -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" "https://api.twitch.tv/kraken/search/channels?query=arma&offset=0&limit=1"
```
bastimeyer: Okay, had a look at the Twitch dev forums...
The correct way of getting a channel id in the v5 API is this:

```
curl -H "Accept: application/vnd.twitchtv.v5+json" -H "Client-ID: phiay4sq36lfv9zu7cbqwz2ndnesfd8" "https://api.twitch.tv/kraken/users?login=dota"
```

This returns a `user` record instead of a channel...
beardypig: Fair enough... I guess channel is equivalent to user, so that makes sense :) 
beardypig: Removing the hosted channel call when hosting is disabled changes the output slightly. 

If `--twitch-disable-hosting` is enabled; currently the channel is always tested to see if it is hosting (minimum 2 API calls), and if it's not hosting there is an HTTP call for the stream m3u8 (3 HTTP requests total). If the channel is hosting, then no streams are returned (2 HTTP requests).

The updated version doesn't make the hosting check, but always makes the m3u8 request, even if the channel is hosting (1 HTTP request, always). As the general case is that the channel is not hosting, it is a reduction of 2 HTTP calls. 

Additionally the stream token/signature request has been moved to after resolution of the hosted channels has been done, this reduced the number of requests if the channel is hosting. 
beardypig: I've made some changes based on what has been discussed. I also fixed the new VOD URLs as reported in #496, but we can skip that commit and create new PR if that is preferred :) 
gravyboat: I'm fine with this one, opinions @bastimeyer and @thinkpad4?
thinkpad4: Unless the channel is hosting someone it shouldn't be displayed, example: 
[cli][info] Found matching plugin twitch for URL twitch.tv/crream
[plugin.twitch][info] Attempting to authenticate using OAuth token
[plugin.twitch][info] Successfully logged in as Thinkpad4
[plugin.twitch][info] hosting was disabled by command line option
error: No streams found on this URL: twitch.tv/crream

See, they aren't hosting someone so informing me that I disabled hosting in the command line is not needed.  Only when I load a channel that IS hosting someone should the [plugin.twitch][info] hosting was disabled by command line option be displayed informing me that I won't see the hosted channel.

Patch:
diff --git a/docs/plugin_matrix.rst b/docs/plugin_matrix.rst
index 51af2a96..c5c08654 100644
--- a/docs/plugin_matrix.rst
+++ b/docs/plugin_matrix.rst
@@ -63,6 +63,7 @@ dogus               - startv.com.tr      Yes   No
                     - ntvspor.net
                     - kralmuzik.com.tr
                     - ntv.com.tr
+                    - eurostartv.com.tr
 dommune             dommune.com          Yes   --
 douyutv             douyutv.com          Yes   --
 dplay               - dplay.se           --    Yes   Streams may be geo-restricted.
diff --git a/src/streamlink/plugins/dogus.py b/src/streamlink/plugins/dogus.py
index 9eabfd24..4d58d33b 100644
--- a/src/streamlink/plugins/dogus.py
+++ b/src/streamlink/plugins/dogus.py
@@ -7,6 +7,7 @@ from streamlink.plugin.api import http
 from streamlink.plugin.api import validate
 from streamlink.stream import HDSStream
 from streamlink.stream import HLSStream
+from streamlink.utils import update_scheme
 
 
 class Dogus(Plugin):
@@ -20,13 +21,14 @@ class Dogus(Plugin):
             ntv.com.tr/canli-yayin/ntv|
             ntvspor.net/canli-yayin|
             kralmuzik.com.tr/tv/kral-tv|
-            kralmuzik.com.tr/tv/kral-pop-tv
+            kralmuzik.com.tr/tv/kral-pop-tv|
+            eurostartv.com.tr/canli-izle
         )/?""", re.VERBOSE)
-    mobile_url_re = re.compile(r"""(?P<q>[\"'])(?P<url>https?://[^'"]*?/live/hls/[^'"]*?\?token=)
+    mobile_url_re = re.compile(r"""(?P<q>["'])(?P<url>(https?:)?//[^'"]*?/live/hls/[^'"]*?\?token=)
                                    (?P<token>[^'"]*?)(?P=q)""", re.VERBOSE)
-    desktop_url_re = re.compile(r"""(?P<q>[\"'])(?P<url>https?://[^'"]*?/live/hds/[^'"]*?\?token=)
+    desktop_url_re = re.compile(r"""(?P<q>["'])(?P<url>(https?:)?//[^'"]*?/live/hds/[^'"]*?\?token=)
                                     (?P<token>[^'"]*?)(?P=q)""", re.VERBOSE)
-    token_re = re.compile(r"""token=(?P<q>[\"'])(?P<token>[^'"]*?)(?P=q)""")
+    token_re = re.compile(r"""token=(?P<q>["'])(?P<token>[^'"]*?)(?P=q)""")
 
     hds_schema = validate.Schema(validate.all(
         {
@@ -69,8 +71,8 @@ class Dogus(Plugin):
         mobile_url_m = self.mobile_url_re.search(res.text)
         desktop_url_m = self.desktop_url_re.search(res.text)
 
-        desktop_url = desktop_url_m and desktop_url_m.group("url")
-        mobile_url = mobile_url_m and mobile_url_m.group("url")
+        desktop_url = desktop_url_m and  update_scheme(self.url, desktop_url_m.group("url"))
+        mobile_url = mobile_url_m and  update_scheme(self.url, mobile_url_m.group("url"))
 
         token = (desktop_url_m and desktop_url_m.group("token")) or (mobile_url_m and mobile_url_m.group("token"))
         if not token:
diff --git a/src/streamlink/plugins/twitch.py b/src/streamlink/plugins/twitch.py
index 8f916bca..e8d21f6d 100644
--- a/src/streamlink/plugins/twitch.py
+++ b/src/streamlink/plugins/twitch.py
@@ -1,10 +1,9 @@
 # coding=utf-8
 import re
-
-import requests
-
+import warnings
 from random import random
 
+import requests
 from streamlink.compat import urlparse
 from streamlink.exceptions import NoStreamsError, PluginError, StreamError
 from streamlink.plugin import Plugin, PluginOptions
@@ -40,9 +39,11 @@ _url_re = re.compile(r"""
         (?P<subdomain>[\w\-]+)
         \.
     )?
-    twitch.tv
-    /
-    (?P<channel>[^/]+)
+    twitch.tv/
+    (?:
+        videos/(?P<videos_id>\d+)|
+        (?P<channel>[^/]+)
+    )
     (?:
         /
         (?P<video_type>[bcv])
@@ -180,9 +181,10 @@ class UsherService(object):
 
 
 class TwitchAPI(object):
-    def __init__(self, beta=False):
+    def __init__(self, beta=False, version=3):
         self.oauth_token = None
         self.subdomain = beta and "betaapi" or "api"
+        self.version = version
 
     def add_cookies(self, cookies):
         http.parse_cookies(cookies, domain="twitch.tv")
@@ -198,7 +200,7 @@ class TwitchAPI(object):
         else:
             url = "https://{0}.twitch.tv{1}".format(self.subdomain, path)
 
-        headers = {'Accept': 'application/vnd.twitchtv.v3+json',
+        headers = {'Accept': 'application/vnd.twitchtv.v{0}+json'.format(self.version),
                    'Client-ID': TWITCH_CLIENT_ID}
 
         # The certificate used by Twitch cannot be verified on some OpenSSL versions.
@@ -216,27 +218,28 @@ class TwitchAPI(object):
         self.subdomain = subdomain_buffer
         return response
 
-    def access_token(self, endpoint, asset, **params):
-        return self.call("/api/{0}/{1}/access_token".format(endpoint, asset), **params)
+    # Public API calls
+
+    def user(self, **params):
+        return self.call("/kraken/user", **params)
+
+    def users(self, **params):
+        return self.call("/kraken/users", **params)
+
+    def videos(self, video_id, **params):
+        return self.call("/kraken/videos/{0}".format(video_id), **params)
 
     def channel_info(self, channel, **params):
         return self.call("/kraken/channels/{0}".format(channel), **params)
 
-    def channel_subscription(self, channel, **params):
-        return self.call("/api/channels/{0}/subscription".format(channel), **params)
+    # Private API calls
 
-    def channel_viewer_info(self, channel, **params):
-        return self.call("/api/channels/{0}/viewer".format(channel), **params)
+    def access_token(self, endpoint, asset, **params):
+        return self.call("/api/{0}/{1}/access_token".format(endpoint, asset), **params)
 
     def token(self, **params):
         return self.call("/api/viewer/token", **params)
 
-    def user(self, **params):
-        return self.call("/kraken/user", **params)
-
-    def videos(self, video_id, **params):
-        return self.call("/kraken/videos/{0}".format(video_id), **params)
-
     def viewer_info(self, **params):
         return self.call("/api/viewer/info", **params)
 
@@ -246,6 +249,17 @@ class TwitchAPI(object):
     def clip_status(self, channel, clip_name, schema):
         return http.json(self.call_subdomain("clips", "/api/v1/clips/" + channel + "/" + clip_name + "/status", format=""), schema=schema)
 
+    # Unsupported/Removed private API calls
+
+    def channel_viewer_info(self, channel, **params):
+        warnings.warn("The channel_viewer_info API call is unsupported and may stop working at any time")
+        return self.call("/api/channels/{0}/viewer".format(channel), **params)
+
+    def channel_subscription(self, channel, **params):
+        warnings.warn("The channel_subscription API call has been removed and no longer works",
+                      category=DeprecationWarning)
+        return self.call("/api/channels/{0}/subscription".format(channel), **params)
+
 
 class Twitch(Plugin):
     options = PluginOptions({
@@ -269,19 +283,66 @@ class Twitch(Plugin):
     def __init__(self, url):
         Plugin.__init__(self, url)
         match = _url_re.match(url).groupdict()
-        self.channel = match.get("channel").lower()
+        self._channel = match.get("channel") and match.get("channel").lower()
+        self._channel_id = None
         self.subdomain = match.get("subdomain")
         self.video_type = match.get("video_type")
-        self.video_id = match.get("video_id")
+        if match.get("videos_id"):
+            self.video_type = "v"
+        self.video_id = match.get("video_id") or match.get("videos_id")
         self.clip_name = match.get("clip_name")
         self._hosted_chain = []
 
         parsed = urlparse(url)
         self.params = parse_query(parsed.query)
 
-        self.api = TwitchAPI(beta=self.subdomain == "beta")
+        self.api = TwitchAPI(beta=self.subdomain == "beta", version=5)
         self.usher = UsherService()
 
+    @property
+    def channel(self):
+        if not self._channel:
+            if self.video_id:
+                cdata = self._channel_from_video_id(self.video_id)
+                self._channel = cdata["name"].lower()
+                self._channel_id = cdata["_id"]
+        return self._channel
+
+    @channel.setter
+    def channel(self, channel):
+        self._channel = channel
+        # channel id becomes unknown
+        self._channel_id = None
+
+    @property
+    def channel_id(self):
+        if not self._channel_id:
+            # If the channel name is set, use that to look up the ID
+            if self._channel:
+                cdata = self._channel_from_login(self._channel)
+                self._channel_id = cdata["_id"]
+
+            # If the channel name is not set but the video ID is,
+            # use that to look up both ID and name
+            elif self.video_id:
+                cdata = self._channel_from_video_id(self.video_id)
+                self._channel = cdata["name"].lower()
+                self._channel_id = cdata["_id"]
+        return self._channel_id
+
+    def _channel_from_video_id(self, video_id):
+        vdata = self.api.videos(video_id)
+        if "channel" not in vdata:
+            raise PluginError("Unable to find video: {0}".format(video_id))
+        return vdata["channel"]
+
+    def _channel_from_login(self, channel):
+        cdata = self.api.users(login=channel)
+        if len(cdata["users"]):
+            return cdata["users"][0]
+        else:
+            raise PluginError("Unable to find channel: {0}".format(channel))
+
     def _authenticate(self):
         if self.api.oauth_token:
             return
@@ -374,7 +435,7 @@ class Twitch(Plugin):
             chunk_stop = chunk_start + chunk_length
             chunk_stream = HTTPStream(self.session, chunk_url)
 
-            if start_offset >= chunk_start and start_offset <= chunk_stop:
+            if chunk_start <= start_offset <= chunk_stop:
                 try:
                     headers = extract_flv_header_tags(chunk_stream)
                 except IOError as err:
@@ -411,7 +472,7 @@ class Twitch(Plugin):
                 playlist_streams.append(chunk_stream)
                 for tag in headers:
                     playlist_tags.append(tag)
-            elif chunk_start >= start_offset and chunk_start < stop_offset:
+            elif start_offset <= chunk_start < stop_offset:
                 playlist_streams.append(chunk_stream)
 
             playlist_offset += chunk_length
@@ -463,33 +524,34 @@ class Twitch(Plugin):
         return sig, token
 
     def _check_for_host(self):
-        channel_id = self.api.channel_info(self.channel)["_id"]
-        host_info = self.api.hosted_channel(include_logins=1, host=channel_id).json()["hosts"][0]
-        if "target_login" in host_info:
+        host_info = self.api.hosted_channel(include_logins=1, host=self.channel_id).json()["hosts"][0]
+        if "target_login" in host_info and host_info["target_login"].lower() != self.channel.lower():
             self.logger.info("{0} is hosting {1}".format(self.channel, host_info["target_login"]))
             return host_info["target_login"]
 
     def _get_hls_streams(self, stream_type="live"):
         self.logger.debug("Getting {} HLS streams for {}".format(stream_type, self.channel))
         self._authenticate()
-        sig, token = self._access_token(stream_type)
+        self._hosted_chain.append(self.channel)
+
         if stream_type == "live":
             hosted_channel = self._check_for_host()
             if hosted_channel and self.options.get("disable_hosting"):
                 self.logger.info("hosting was disabled by command line option")
-                return {}
             elif hosted_channel:
                 self.logger.info("switching to {}", hosted_channel)
                 if hosted_channel in self._hosted_chain:
-                    self._hosted_chain.append(hosted_channel)
                     self.logger.error(u"A loop of hosted channels has been detected, "
-                                      "cannot find a playable stream. ({})".format(u" -> ".join(self._hosted_chain)))
+                                      "cannot find a playable stream. ({})".format(u" -> ".join(self._hosted_chain + [hosted_channel])))
                     return {}
-                self._hosted_chain.append(hosted_channel)
                 self.channel = hosted_channel
                 return self._get_hls_streams(stream_type)
+
+            # only get the token once the channel has been resolved
+            sig, token = self._access_token(stream_type)
             url = self.usher.channel(self.channel, sig=sig, token=token)
         elif stream_type == "video":
+            sig, token = self._access_token(stream_type)
             url = self.usher.video(self.video_id, nauthsig=sig, nauth=token)
         else:
             self.logger.debug("Unknown HLS stream type: {}".format(stream_type))
diff --git a/src/streamlink/utils/__init__.py b/src/streamlink/utils/__init__.py
index ff6e63af..774bad79 100644
--- a/src/streamlink/utils/__init__.py
+++ b/src/streamlink/utils/__init__.py
@@ -134,6 +134,21 @@ def rtmpparse(url):
     return (tcurl, playpath)
 
 
+def update_scheme(current, target):
+    """
+    Take the scheme from the current URL and applies it to the
+    target URL if the target URL startswith //
+    :param current: current URL
+    :param target: target URL
+    :return: target URL with the current URLs schema
+    """
+    scheme = urlparse(current).scheme
+    if target.startswith("//"):
+        return "{0}:{1}".format(scheme, target)
+    else:
+        return target
+
+
 #####################################
 # Deprecated functions, do not use. #
 #####################################


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 4495b0c2..c8cd19dc 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,6 +1,7 @@
 import sys
 
 from streamlink.plugin.api.validate import xml_element, text
+from streamlink.utils import update_scheme
 
 try:
     import xml.etree.cElementTree as ET
@@ -79,3 +80,17 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(
             {"test": "1", "foo": "bar"},
             parse_qsd("test=1&foo=bar", schema=validate.Schema({"test": validate.text, "foo": "bar"})))
+
+    def test_update_scheme(self):
+        self.assertEqual(
+            "https://example.com/foo",
+            update_scheme("https://other.com/bar", "//example.com/foo")
+        )
+        self.assertEqual(
+            "http://example.com/foo",
+            update_scheme("http://other.com/bar", "//example.com/foo")
+        )
+        self.assertEqual(
+            "http://example.com/foo",
+            update_scheme("https://other.com/bar", "http://example.com/foo")
+        )