Repository: streamlink/streamlink

Issue:
Non unicode escaped API breaks the plugin
I have a problem on Streamlink 0.0.2 with Twitch plugin. Luckily I figured out what's going on and fixed my own copy of code. So I would like to share my fix though I am not sure with my decision.

My question is:
> Is it safe to assume that JSON APIs other than Twitch, use particular text encoding?


Here is some error log.
```
> streamlink.exe --player <some my player> --twitch-oauth-token <redacted> https://twitch.tv/<readacted> source
[cli][info] Found matching plugin twitch for URL https://twitch.tv/<readacted>
[plugin.twitch][info] Attempting to authenticate using OAuth token
error: 'cp949' codec can't encode character '\u011b' in position 48: illegal multibyte sequence
```

Little backgrounds.

1. Twitch API does not escape 'non-ascii' characters.
2. Twitch API does not state `charset` in the `Content-type` header. Only states MIME type `application/json`.
3. I am running Korean version of Windows 10, and the plugin tries to decode response body with OS default encoding(presumably) which is CP949 in my case.

Let me follow the code flow.

1. Twitch plugin tries to authenticate with my OAuth token, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L264
2. ...which is `user` method in helper class, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L208
3. ...which requests API with GET method, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L167
4. I know Twitch API returns in JSON format, so going https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugins/twitch.py#L182
5. ...which tries to parse response body text in JSON, implicitly tries to decode response body from bytes to str, in https://github.com/streamlink/streamlink/blob/0.0.2/src/streamlink/plugin/api/http_session.py#L70

I could simply edit `http_session.py` and fix it by stating 'known' charset before accessing `text` property, something like:
```
+        if res.encoding is None:
+            res.encoding = 'utf8'
         return parse_json(res.text, *args, **kwargs)
```



Discussion:
I guess #19 is related. Because it also happened to me with Livestreamer.


Resolving patch:
diff --git a/src/streamlink/plugin/api/http_session.py b/src/streamlink/plugin/api/http_session.py
--- a/src/streamlink/plugin/api/http_session.py
+++ b/src/streamlink/plugin/api/http_session.py
@@ -66,9 +66,34 @@ def __init__(self, *args, **kwargs):
             self.mount("http://", HTTPAdapterWithReadTimeout())
             self.mount("https://", HTTPAdapterWithReadTimeout())
 
+    @classmethod
+    def determine_json_encoding(cls, sample):
+        """
+        Determine which Unicode encoding the JSON text sample is encoded with
+
+        RFC4627 (http://www.ietf.org/rfc/rfc4627.txt) suggests that the encoding of JSON text can be determined
+        by checking the pattern of NULL bytes in first 4 octets of the text.
+        :param sample: a sample of at least 4 bytes of the JSON text
+        :return: the most likely encoding of the JSON text
+        """
+        nulls_at = [i for i, j in enumerate(bytearray(sample[:4])) if j == 0]
+        if nulls_at == [0, 1, 2]:
+            return "UTF-32BE"
+        elif nulls_at == [0, 2]:
+            return "UTF-16BE"
+        elif nulls_at == [1, 2, 3]:
+            return "UTF-32LE"
+        elif nulls_at == [1, 3]:
+            return "UTF-16LE"
+        else:
+            return "UTF-8"
+
     @classmethod
     def json(cls, res, *args, **kwargs):
         """Parses JSON from a response."""
+        # if an encoding is already set then use the provided encoding
+        if res.encoding is None:
+            res.encoding = cls.determine_json_encoding(res.content[:4])
         return parse_json(res.text, *args, **kwargs)
 
     @classmethod


Test patch:
diff --git a/tests/test_plugin_api_http_session.py b/tests/test_plugin_api_http_session.py
--- a/tests/test_plugin_api_http_session.py
+++ b/tests/test_plugin_api_http_session.py
@@ -1,5 +1,13 @@
+# coding=utf-8
 import unittest
 
+import requests
+
+try:
+    from unittest.mock import patch, PropertyMock
+except ImportError:
+    from mock import patch, PropertyMock
+
 from streamlink.exceptions import PluginError
 from streamlink.plugin.api.http_session import HTTPSession
 
@@ -16,5 +24,26 @@ def stream_data():
 
         self.assertRaises(PluginError, stream_data)
 
+    def test_json_encoding(self):
+        json_str = u"{\"test\": \"Α and Ω\"}"
+
+        # encode the json string with each encoding and assert that the correct one is detected
+        for encoding in ["UTF-32BE", "UTF-32LE", "UTF-16BE", "UTF-16LE", "UTF-8"]:
+            with patch('requests.Response.content', new_callable=PropertyMock) as mock_content:
+                mock_content.return_value = json_str.encode(encoding)
+                res = requests.Response()
+
+                self.assertEqual(HTTPSession.json(res), {u"test": u"\u0391 and \u03a9"})
+
+    def test_json_encoding_override(self):
+        json_text = u"{\"test\": \"Α and Ω\"}".encode("cp949")
+
+        with patch('requests.Response.content', new_callable=PropertyMock) as mock_content:
+            mock_content.return_value = json_text
+            res = requests.Response()
+            res.encoding = "cp949"
+
+            self.assertEqual(HTTPSession.json(res), {u"test": u"\u0391 and \u03a9"})
+
 if __name__ == "__main__":
     unittest.main()