Repository: streamlink/streamlink (python)

Issue:
Non unicode escaped API breaks the plugin
I have a problem on Streamlink 0.0.2 with Twitch plugin. Luckily I figured out what's going on and fixed my own copy of code. So I would like to share my fix though I am not sure with my decision.

My question is:
> Is it safe to assume that JSON APIs other than Twitch, use particular text encoding?


Here is some error log.
```
> streamlink.exe --player <some my player> --twitch-oauth-token <redacted> https://twitch.tv/<readacted> source
[cli][info] Found matching plugin twitch for URL https://twitch.tv/<readacted>
[plugin.twitch][info] Attempting to authenticate using OAuth token
error: 'cp949' codec can't encode character '\u011b' in position 48: illegal multibyte sequence
```

Little backgrounds.

1. Twitch API does not escape 'non-ascii' characters.
2. Twitch API does not state `charset` in the `Content-type` header. Only states MIME type `application/json`.
3. I am running Korean version of Windows 10, and the plugin tries to decode response body with OS default encoding(presumably) which is CP949 in my case.

Let me follow the code flow.

1. Twitch plugin tries to authenticate with my OAuth token, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L264
2. ...which is `user` method in helper class, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L208
3. ...which requests API with GET method, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L167
4. I know Twitch API returns in JSON format, so going https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L182
5. ...which tries to parse response body text in JSON, implicitly tries to decode response body from bytes to str, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugin/api/http_session.py#L70

I could simply edit `http_session.py` and fix it by stating 'known' charset before accessing `text` property, something like:
```
+        if res.encoding is None:
+            res.encoding = 'utf8'
         return parse_json(res.text, *args, **kwargs)
```


Pull request:
Automatically determine the encoding when parsing JSON with HTTPSession
This should fix #120 and if I understand #19 it should fix that too. 

According to [RFC4627: The application/json Media Type for JavaScript Object Notation (JSON)
](http://www.ietf.org/rfc/rfc4627.txt) JSON is always encoded as Unicode (see section 3.  Encoding) and the specific Unicode encoding can be determined by looking at the pattern of NULL bytes in the first 4 bytes. 

I have added a method that implements this pattern matching and added a couple of tests for it, the tests use `unittest.mock` which is not available in Python 2 - I added `mock` to the `before_install` command for travis. 

Props to @cyluss 👍 

Patch:
diff --git a/.travis.yml b/.travis.yml
index 57c30c83..c678e2da 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,7 +11,7 @@ matrix:
     env: BUILD_DOCS=yes
 
 before_install:
-  - pip install pytest pytest-cov codecov coverage
+  - pip install pytest pytest-cov codecov coverage mock
 
 install:
   - python setup.py install
diff --git a/src/streamlink/plugin/api/http_session.py b/src/streamlink/plugin/api/http_session.py
index f5b6879c..0b1caa7a 100644
--- a/src/streamlink/plugin/api/http_session.py
+++ b/src/streamlink/plugin/api/http_session.py
@@ -66,9 +66,34 @@ class HTTPSession(Session):
             self.mount("http://", HTTPAdapterWithReadTimeout())
             self.mount("https://", HTTPAdapterWithReadTimeout())
 
+    @classmethod
+    def determine_json_encoding(cls, sample):
+        """
+        Determine which Unicode encoding the JSON text sample is encoded with
+
+        RFC4627 (http://www.ietf.org/rfc/rfc4627.txt) suggests that the encoding of JSON text can be determined
+        by checking the pattern of NULL bytes in first 4 octets of the text.
+        :param sample: a sample of at least 4 bytes of the JSON text
+        :return: the most likely encoding of the JSON text
+        """
+        nulls_at = [i for i, j in enumerate(bytearray(sample[:4])) if j == 0]
+        if nulls_at == [0, 1, 2]:
+            return "UTF-32BE"
+        elif nulls_at == [0, 2]:
+            return "UTF-16BE"
+        elif nulls_at == [1, 2, 3]:
+            return "UTF-32LE"
+        elif nulls_at == [1, 3]:
+            return "UTF-16LE"
+        else:
+            return "UTF-8"
+
     @classmethod
     def json(cls, res, *args, **kwargs):
         """Parses JSON from a response."""
+        # if an encoding is already set then use the provided encoding
+        if res.encoding is None:
+            res.encoding = cls.determine_json_encoding(res.content[:4])
         return parse_json(res.text, *args, **kwargs)
 
     @classmethod
diff --git a/src/streamlink/plugins/livecodingtv.py b/src/streamlink/plugins/livecodingtv.py
index b5ad3a08..74522af7 100644
--- a/src/streamlink/plugins/livecodingtv.py
+++ b/src/streamlink/plugins/livecodingtv.py
@@ -1,12 +1,20 @@
 import re
 from streamlink.plugin import Plugin
+from streamlink.stream import HLSStream
 from streamlink.stream import RTMPStream, HTTPStream
 from streamlink.plugin.api import http
 
 
-_vod_re = re.compile('\"(http(s)?://.*\.mp4\?t=.*)\"')
-_rtmp_re = re.compile('rtmp://[^"]+/(?P<channel>\w+)+[^/"]+')
-_url_re = re.compile('http(s)?://(?:\w+.)?\livecoding\.tv')
+_streams_re = re.compile(r"""
+    src:\s+"(
+        rtmp://.*?\?t=.*?|                      # RTMP stream
+        https?://.*?playlist.m3u8.*?\?t=.*?|    # HLS stream
+        https?://.*?manifest.mpd.*?\?t=.*?|     # DASH stream
+        https?://.*?.mp4\?t=.*?                 # HTTP stream
+        )".*?
+     type:\s+"(.*?)"                            # which stream type it is
+     """, re.M | re.DOTALL | re.VERBOSE)
+_url_re = re.compile(r"http(s)?://(?:\w+\.)?livecoding\.tv")
 
 
 class LivecodingTV(Plugin):
@@ -16,18 +24,19 @@ class LivecodingTV(Plugin):
 
     def _get_streams(self):
         res = http.get(self.url)
-        match = _rtmp_re.search(res.content.decode('utf-8'))
-        if match:
-            params = {
-                "rtmp": match.group(0),
-                "pageUrl": self.url,
-                "live": True,
-            }
-            yield 'live', RTMPStream(self.session, params)
-            return
-
-        match = _vod_re.search(res.content.decode('utf-8'))
-        if match:
-            yield 'vod', HTTPStream(self.session, match.group(1))
+        match = _streams_re.findall(res.content.decode('utf-8'))
+        for url, stream_type in match:
+            if stream_type == "rtmp/mp4" and RTMPStream.is_usable(self.session):
+                params = {
+                    "rtmp": url,
+                    "pageUrl": self.url,
+                    "live": True,
+                }
+                yield 'live', RTMPStream(self.session, params)
+            elif stream_type == "application/x-mpegURL":
+                for s in HLSStream.parse_variant_playlist(self.session, url).items():
+                    yield s
+            elif stream_type == "video/mp4":
+                yield 'vod', HTTPStream(self.session, url)
 
 __plugin__ = LivecodingTV
diff --git a/src/streamlink/plugins/tvplayer.py b/src/streamlink/plugins/tvplayer.py
index ffe8f191..a16d6d34 100644
--- a/src/streamlink/plugins/tvplayer.py
+++ b/src/streamlink/plugins/tvplayer.py
@@ -11,7 +11,7 @@ class TVPlayer(Plugin):
         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) "
         "Chrome/43.0.2357.65 Safari/537.36")
     API_URL = "http://api.tvplayer.com/api/v2/stream/live"
-    _url_re = re.compile(r"http://(?:www.)?tvplayer.com/watch/(.+)")
+    _url_re = re.compile(r"https?://(?:www.)?tvplayer.com/(:?watch/?|watch/(.+)?)")
     _stream_attrs_ = re.compile(r'var\s+(validate|platform|resourceId)\s+=\s*(.*?);', re.S)
     _stream_schema = validate.Schema({
         "tvplayer": validate.Schema({
@@ -25,11 +25,12 @@ class TVPlayer(Plugin):
     @classmethod
     def can_handle_url(cls, url):
         match = TVPlayer._url_re.match(url)
-        return match
+        return match is not None
 
     def _get_streams(self):
         # find the list of channels from the html in the page
-        res = http.get(self.url)
+        self.url = self.url.replace("https", "http")  # https redirects to http
+        res = http.get(self.url, headers={"User-Agent": TVPlayer._user_agent})
         stream_attrs = dict((k, v.strip('"')) for k, v in TVPlayer._stream_attrs_.findall(res.text))
 
         # get the stream urls


Test patch:
diff --git a/tests/test_plugin_api_http_session.py b/tests/test_plugin_api_http_session.py
index 163a8ee8..bf3296aa 100644
--- a/tests/test_plugin_api_http_session.py
+++ b/tests/test_plugin_api_http_session.py
@@ -1,5 +1,13 @@
+# coding=utf-8
 import unittest
 
+import requests
+
+try:
+    from unittest.mock import patch, PropertyMock
+except ImportError:
+    from mock import patch, PropertyMock
+
 from streamlink.exceptions import PluginError
 from streamlink.plugin.api.http_session import HTTPSession
 
@@ -16,5 +24,26 @@ class TestPluginAPIHTTPSession(unittest.TestCase):
 
         self.assertRaises(PluginError, stream_data)
 
+    def test_json_encoding(self):
+        json_str = u"{\"test\": \"Α and Ω\"}"
+
+        # encode the json string with each encoding and assert that the correct one is detected
+        for encoding in ["UTF-32BE", "UTF-32LE", "UTF-16BE", "UTF-16LE", "UTF-8"]:
+            with patch('requests.Response.content', new_callable=PropertyMock) as mock_content:
+                mock_content.return_value = json_str.encode(encoding)
+                res = requests.Response()
+
+                self.assertEqual(HTTPSession.json(res), {u"test": u"\u0391 and \u03a9"})
+
+    def test_json_encoding_override(self):
+        json_text = u"{\"test\": \"Α and Ω\"}".encode("cp949")
+
+        with patch('requests.Response.content', new_callable=PropertyMock) as mock_content:
+            mock_content.return_value = json_text
+            res = requests.Response()
+            res.encoding = "cp949"
+
+            self.assertEqual(HTTPSession.json(res), {u"test": u"\u0391 and \u03a9"})
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/tests/test_plugin_tvplayer.py b/tests/test_plugin_tvplayer.py
new file mode 100644
index 00000000..b3f1a6a3
--- /dev/null
+++ b/tests/test_plugin_tvplayer.py
@@ -0,0 +1,21 @@
+import unittest
+
+from streamlink.plugins.tvplayer import TVPlayer
+
+
+class TestPluginTVPlayer(unittest.TestCase):
+    def test_can_handle_url(self):
+        # should match
+        self.assertTrue(TVPlayer.can_handle_url("http://tvplayer.com/watch/"))
+        self.assertTrue(TVPlayer.can_handle_url("http://www.tvplayer.com/watch/"))
+        self.assertTrue(TVPlayer.can_handle_url("http://tvplayer.com/watch"))
+        self.assertTrue(TVPlayer.can_handle_url("http://www.tvplayer.com/watch"))
+        self.assertTrue(TVPlayer.can_handle_url("http://tvplayer.com/watch/dave"))
+        self.assertTrue(TVPlayer.can_handle_url("http://www.tvplayer.com/watch/itv"))
+        self.assertTrue(TVPlayer.can_handle_url("https://www.tvplayer.com/watch/itv"))
+        self.assertTrue(TVPlayer.can_handle_url("https://tvplayer.com/watch/itv"))
+
+        # shouldn't match
+        self.assertFalse(TVPlayer.can_handle_url("http://www.tvplayer.com/"))
+        self.assertFalse(TVPlayer.can_handle_url("http://www.tvcatchup.com/"))
+        self.assertFalse(TVPlayer.can_handle_url("http://www.youtube.com/"))