Repository: streamlink/streamlink

Issue:
Schoolism Plugin Error with newer streams
<!--
Thanks for reporting a plugin issue!
USE THE TEMPLATE. Otherwise your plugin issue may be rejected.

First, see the contribution guidelines:
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink

Also check the list of open and closed plugin issues:
https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22plugin+issue%22

Please see the text preview to avoid unnecessary formatting errors.
-->


## Plugin Issue

<!-- Replace [ ] with [x] in order to check the box -->
- [ X] This is a plugin issue and I have read the contribution guidelines.


### Description
The plugin is working with older streams but not working with newer streams like "Deconstructed: Drawing People"

Description

Schoolism plugin doesnt work with the following

Deconstructed: Drawing People

I'm using streamlink 1.4.1 latest installation


<!-- Explain the plugin issue as thoroughly as you can. -->


### Reproduction steps / Explicit stream URLs to test

<!-- How can we reproduce this? Please note the exact steps below using the list format supplied. If you need more steps please add them. -->

1. ... Using Username and password the plugin logs in correctly
2. ... Accessing the path of the course either to stream or to output doesn't work https://www.schoolism.com/watchLesson.php?type=st&id=596
3. ... It throws the following error "Unable to parse JSON" (see log)


### Log output

<!--
TEXT LOG OUTPUT IS REQUIRED for a plugin issue!
Use the `--loglevel debug` parameter and avoid using parameters which suppress log output.
https://streamlink.github.io/cli.html#cmdoption-l

Make sure to **remove usernames and passwords**
You can copy the output to https://gist.github.com/ or paste it below.
-->

```
Using the log i get the following:

C:\Users\PC>streamlink -l debug "https://www.schoolism.com/watchLesson.php?type=
st&id=596" --schoolism-email --schoolism-password --schoolism-part 1
[cli][debug] OS: Windows 7
[cli][debug] Python: 3.6.6
[cli][debug] Streamlink: 1.4.1
[cli][debug] Requests(2.23.0), Socks(1.7.1), Websocket(0.57.0)
[cli][info] Found matching plugin schoolism for URL https://www.schoolism.com/wa
tchLesson.php?type=st&id=596
[cli][debug] Plugin specific arguments:
[cli][debug] --schoolism-email= (email)
[cli][debug] --schoolism-password= (password)
[cli][debug] --schoolism-part=1 (part)
[plugin.schoolism][debug] Logged in to Schoolism as
error: Unable to parse JSON: Expecting ',' delimiter: line 1 column 1053 (char 1
052) ('[{"sources":[{"type":"application/ ...)
```


### Additional comments, screenshots, etc.



[Love Streamlink? Please consider supporting our collective. Thanks!](https://opencollective.com/streamlink/donate)



Discussion:
I can confirm that this is happening on my end as well. I've tested old and new courses URLs and it only happens to this course: "Deconstructed: Drawing People with Viktor Kalvachev"

### Log output
```
> C:\Users\Test\Lesson 1 - Crosshatching\Lesson>streamlink --schoolism-email email --schoolism-password password --schoolism-part 1 -o "1.mp4" "https://www.schoolism.com/watchLesson.php?type=st&id=593" best
> [cli][info] Found matching plugin schoolism for URL https://www.schoolism.com/watchLesson.php?type=st&id=593
> error: Unable to parse JSON: Expecting ',' delimiter: line 1 column 1053 (char 1052) ('[{"sources":[{"type":"application/ ...)
```

Do let me know if I'm not supposed to post here, just wanted to highlight the issue.
I'd like to try and fix this, but I don't have an account.  I presume it would also be necessary to have a subscription to see this content.  It's possible it could be fixed with help from one or both of you though.

Initially, if you could visit a working video page, right click, select view source, then copy and paste the HTML here (formatted nicely with the GH markup, please), then repeat that on the non-working URL you have identified, then I can perhaps use that to make a fix.  Another option, that may or may not be better would be to post the HTML output on a pastebin site like https://pastebin.com/ and just post the links in the issue instead.

I note that the plugin uses a User-Agent string for Safari 8, so it's hard to know if the output will differ enough in other browsers that it matters, but hopefully not.  If you're willing to do so, you could install a browser extension that allows you to easily change the user agent string and set it up to use:
```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12
```
which is what the plugin sends.  Cheers.

basically https://github.com/streamlink/streamlink/issues/1683#issuecomment-397264747
I had the same problem so I looked into this issue and found the bug on line 18 in the [schoolism plugin](https://github.com/streamlink/streamlink/blob/master/src/streamlink/plugins/schoolism.py)

Line 18:
```js_to_json = partial(re.compile(r'(?!<")(\w+):(?!/)').sub, r'"\1":')```

Basically a regex sub is used to transform a raw javascript object into python parsable JSON. This breaks for this video because the javascript object in question looks like
```{title: "Deconstructed: Drawing People"}```
which then gets transformed into
```{"title": ""Deconstructed": Drawing People"}```
since the colon inside the string is tripping up the pattern.

Regex patterns can't be created to fix this since the underlying data is nesting. So you need a slightly beefier implementation like [hjson](https://github.com/hjson/hjson) to do this correctly.

Personally I don't know much about the streamlink codebase so maybe this is an issue that's been fixed before in a different plugin?

Resolving patch:
diff --git a/src/streamlink/plugins/schoolism.py b/src/streamlink/plugins/schoolism.py
--- a/src/streamlink/plugins/schoolism.py
+++ b/src/streamlink/plugins/schoolism.py
@@ -17,6 +17,7 @@ class Schoolism(Plugin):
     playlist_re = re.compile(r"var allVideos\s*=\s*(\[.*\]);", re.DOTALL)
     js_to_json = partial(re.compile(r'(?!<")(\w+):(?!/)').sub, r'"\1":')
     fix_brackets = partial(re.compile(r',\s*\}').sub, r'}')
+    fix_colon_in_title = partial(re.compile(r'"title":""(.*?)":(.*?)"').sub, r'"title":"\1:\2"')
     playlist_schema = validate.Schema(
         validate.transform(playlist_re.search),
         validate.any(
@@ -25,6 +26,7 @@ class Schoolism(Plugin):
                 validate.get(1),
                 validate.transform(js_to_json),
                 validate.transform(fix_brackets),  # remove invalid ,
+                validate.transform(fix_colon_in_title),
                 validate.transform(parse_json),
                 [{
                     "sources": validate.all([{


Test patch:
diff --git a/tests/plugins/test_schoolism.py b/tests/plugins/test_schoolism.py
--- a/tests/plugins/test_schoolism.py
+++ b/tests/plugins/test_schoolism.py
@@ -50,3 +50,20 @@ def test_playlist_parse(self):
 
         self.assertIsNotNone(data)
         self.assertEqual(2, len(data))
+
+    def test_playlist_parse_colon_in_title(self):
+        colon_in_title = """var allVideos=[
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part1.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 1",playlistTitle:"Part 1",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part2.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 2",playlistTitle:"Part 2",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part3.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 3",playlistTitle:"Part 3",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part4.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 4",playlistTitle:"Part 4",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part5.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 5",playlistTitle:"Part 5",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part6.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 6",playlistTitle:"Part 6",}],},
+            {sources:[{type:"application/x-mpegurl",src:"https://d8u31iyce9xic.cloudfront.net/52/1/part7.m3u8?Policy=TOKEN&Signature=TOKEN&Key-Pair-Id=TOKEN",title:"Deconstructed: Drawing People - Lesson 1 - Part 7",playlistTitle:"Part 7",}]}
+            ];
+        """  # noqa: E501
+
+        data = Schoolism.playlist_schema.validate(colon_in_title)
+
+        self.assertIsNotNone(data)
+        self.assertEqual(7, len(data))