Repository: learningequality/kolibri

Issue:
'Server 500' response when trying to view any of the H5P content
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->

Server 500 response within the content viewport when trying to view any H5P content imported from *H5P Samples* channel.

![image](https://user-images.githubusercontent.com/2577060/80483443-b8080880-8973-11ea-8bc1-d1b82ed8fb44.png)


### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

H5P interactive content should have loaded correctly.

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

Can't use the content.

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->

Logs herein -
[kolibri.txt](https://github.com/learningequality/kolibri/files/4545476/kolibri.txt)


### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

1. Enable h5p_viewer plugin
2. Restart Kolibri
3. Import content from H5P Samples channel 
4. Navigate in Learn to view content

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

Firefox on Windows 10.

Some information from Device - Info:
```Version:           0.13.2
OS:                Windows-8-6.2.9200 
Python:            3.4.3
Installer:         Windows
Server:            Kolibri internal server
```


Discussion:
This works fine in the following environment -
```
Version:           0.13.2
OS:                Linux-5.3.7-050307-generic-x86_64-with-Ubuntu-18.04-bionic
Python:            3.6.9
Installer:         kolibri(apt) with kolibri-server Version: 0.3.8~beta2-0ubuntu1
Server:            nginx/1.14.0 (Ubuntu)
```

Thanks for the feedback @intelliant01  - sounds like there is an issue with this on Windows, very helpful to know.
@mrpau-dev would you mind investigating?
FYI, the command for Step 1 is 

```
kolibri plugin enable kolibri.plugins.h5p_viewer
```

and Step 2 restart is needed for the plugin-initialization-code to run.

And the channel with H5P samples has the token `piman-fapum`

I reproduced this in Windows 7, Upgrading the Python version from 3.4 to 3.6 will fix the issue on Windows. 
Do you have a fuller traceback @mrpau-richard - presumably we could fix the underlying code as well?
For reference, here's the last few lines of the `kolibri.txt` log file attached by @intelliant01 above:

```
INFO 2020-04-28 08:06:44,128 cherrypy.access.115203152 127.0.0.1 - - "GET /static/kolibri.plugins.h5p_viewer.main/kolibri.plugins.h5p_viewer.main-0.13.2.js HTTP/1.1" 200 5730 "http://127.0.0.1:8080/en/learn/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0"
ERROR 2020-04-28 08:06:44,171 exception Internal Server Error: /zipcontent/dda40b9ee291ef85d0911ccba551aba4.h5p/
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\kolibri\dist\django\core\handlers\exception.py", line 41, in inner
    response = get_response(request)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\core\handlers\base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\core\handlers\base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\views\generic\base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\views\generic\base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\utils\decorators.py", line 67, in _wrapper
    return bound_func(*args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\views\decorators\http.py", line 99, in inner
    response = func(request, *args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\utils\decorators.py", line 63, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "C:\Python34\lib\site-packages\kolibri\core\content\api.py", line 75, in wrapper_func
    response = some_func(*args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\dist\django\views\decorators\clickjacking.py", line 58, in wrapped_view
    resp = view_func(*args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\core\content\decorators.py", line 14, in wrapper_func
    response = some_func(request, *args, **kwargs)
  File "C:\Python34\lib\site-packages\kolibri\core\content\views.py", line 269, in get
    response = get_h5p(zf, embedded_filepath)
  File "C:\Python34\lib\site-packages\kolibri\core\content\views.py", line 151, in get_h5p
    h5pdata = json.loads(zf.open(zf.getinfo("h5p.json")).read())
  File "C:\Python34\lib\json\__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
```

This has the same exception to this issue: https://github.com/learningequality/kolibri/issues/6615 - I'm not sure if they are related or the same @indirectlylit.

@mrpau-richard You may be right that upgrading to Python 3.6 fixes the issue.  The [Python 3.6 release notes mentions `json` improvements](https://docs.python.org/3/whatsnew/3.6.html#json) very similar to this issue.

> I reproduced this in Windows 7, Upgrading the Python version from 3.4 to 3.6 will fix the issue on Windows.

Please note however that we had an issue using Kolibri on Windows 10 with Python 3.6 last year: https://github.com/learningequality/kolibri/issues/5733 

I see two approaches to this:

1. Test if upgrading to Python 3.6 indeed fixes this issue (and the other one too) and make a PR for such.  We hope to enter beta testing soon so we can thoroughly test this "upgrade fix" too.
2. Fix the underlying code as @rtibbles suggested.

Or we can do both 😺
I think both sounds good to me - but mostly because 3.4 has reached EOL, so we should upgrade the windows installer anyway!
I can fix 2, if you guys can tackle 1.
Hrm, investigating further, I'm not sure why upgrading to Python 3.6 fixes this - as in both 3.4 and 3.6, the read returns a byte object, rather than a string, so I think this is more of a Py2/3 compatibility issue than anything else.

Resolving patch:
diff --git a/kolibri/core/content/views.py b/kolibri/core/content/views.py
--- a/kolibri/core/content/views.py
+++ b/kolibri/core/content/views.py
@@ -73,6 +73,11 @@ def get_path_or_404(zipped_filename):
         )
 
 
+def load_json_from_zipfile(zf, filepath):
+    with zf.open(filepath, "r") as f:
+        return json.load(f)
+
+
 def recursive_h5p_dependencies(zf, data, prefix=""):
 
     jsfiles = OrderedDict()
@@ -82,8 +87,7 @@ def recursive_h5p_dependencies(zf, data, prefix=""):
     for dep in data.get("preloadedDependencies", []):
         packagepath = "{machineName}-{majorVersion}.{minorVersion}/".format(**dep)
         librarypath = packagepath + "library.json"
-        info = zf.getinfo(librarypath)
-        content = json.loads(zf.open(info).read())
+        content = load_json_from_zipfile(zf, librarypath)
         newjs, newcss = recursive_h5p_dependencies(zf, content, packagepath)
         cssfiles.update(newcss)
         jsfiles.update(newjs)
@@ -148,11 +152,14 @@ def get_h5p(zf, embedded_filepath):
     if not embedded_filepath:
         # Get the h5p bootloader, and then run it through our hashi templating code.
         # return the H5P bootloader code
-        h5pdata = json.loads(zf.open(zf.getinfo("h5p.json")).read())
+        try:
+            h5pdata = load_json_from_zipfile(zf, "h5p.json")
+            contentdata = load_json_from_zipfile(zf, "content/content.json")
+        except KeyError:
+            raise Http404("No valid h5p file was found at this location")
         jsfiles, cssfiles = recursive_h5p_dependencies(zf, h5pdata)
         jsfiles = jsfiles.keys()
         cssfiles = cssfiles.keys()
-        contentdata = zf.open(zf.getinfo("content/content.json")).read()
         path_includes_version = (
             "true"
             if "-" in [name for name in zf.namelist() if "/" in name][0]
@@ -168,7 +175,9 @@ def get_h5p(zf, embedded_filepath):
             {
                 "jsfiles": jsfiles,
                 "cssfiles": cssfiles,
-                "content": contentdata,
+                "content": json.dumps(
+                    json.dumps(contentdata, separators=(",", ":"), ensure_ascii=False)
+                ),
                 "library": "{machineName} {majorVersion}.{minorVersion}".format(
                     **main_library_data
                 ),
@@ -183,6 +192,8 @@ def get_h5p(zf, embedded_filepath):
     elif embedded_filepath.startswith("dist/"):
         # return static H5P dist resources
         path = finders.find("assets/h5p-standalone-" + embedded_filepath)
+        if path is None:
+            raise Http404("{} not found".format(embedded_filepath))
         # try to guess the MIME type of the embedded file being referenced
         content_type = (
             mimetypes.guess_type(embedded_filepath)[0] or "application/octet-stream"


Test patch: