Repository: learningequality/kolibri

Issue:
When upgrading from 0.4 to 0.5, get ImportError
## Summary

Upgrading from 0.4 to 0.5 produces an import error due to the disappearance of two plugins.

## System information

 - Version: v0.5.0-beta4
 - Operating system: Windows 10
 - Browser: N/A

## Traceback or relevant snippet from server.log or browser console

```
C:\Python27\Scripts>kolibri start
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\kolibri.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\kolibri\utils\cli.py", line 532, in main
    initialize(debug=debug)
  File "c:\python27\lib\site-packages\kolibri\utils\cli.py", line 134, in initialize
    django.setup()
  File "c:\python27\lib\site-packages\kolibri\dist\django\__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "c:\python27\lib\site-packages\kolibri\dist\django\apps\registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "c:\python27\lib\site-packages\kolibri\dist\django\apps\config.py", line 123, in create
    import_module(entry)
  File "c:\python27\lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named video_mp4_render
```

## How to reproduce

1. Install v0.4.x
2. Upgrade to v0.5.0-beta4
3. Try to run the server

## Real-life consequences (anything community should be aware of, for instance how it affects your deployment)

Upgrading becomes significantly more painful. Should rather show a warning about non-existent plugins and remove them from the configuration.

Originally reported by @laurenlichtman 


Discussion:
Yeah, that seems like the most expected behaviour, but it would be nice if we could also get the warnings to display in the browser.. do we use `django.contrib.messages` by any chance?
I don't think we are currently doing so. Not sure how useful this warning is to the end user, as opposed to the person upgrading, who presumably would see the terminal or log? 
> opposed to the person upgrading, who presumably would see the terminal or log

Nah, if the install view .deb or .exe and have a script that just starts KA Lite on boot, they won't see these messages.
Currently, `conf` sets `config['INSTALLED_APPS']` - and this is appended in `settings.INSTALLED_APPS`.

This is a bit sub-optimal, because while we might wanna switch off plugins from our core that have been removed, we should also fail when other critical aspects are unavailable.

I'll just add a comment in the `conf` module about this for now, since we are going to do a real overhaul of the way that settings work anyways.

Resolving patch:
diff --git a/kolibri/utils/compat.py b/kolibri/utils/compat.py
new file mode 100644
--- /dev/null
+++ b/kolibri/utils/compat.py
@@ -0,0 +1,35 @@
+"""
+Compatibility layer for Python 2+3
+"""
+from __future__ import absolute_import, print_function, unicode_literals
+
+import sys
+
+
+def module_exists(module_path):
+    """
+    Determines if a module exists without loading it (Python 3)
+    In Python 2, the module will be loaded
+    """
+    if sys.version_info >= (3, 4):
+        from importlib.util import find_spec
+        try:
+            if "." in module_path:
+                find_spec(module_path)
+                return True
+            else:
+                return find_spec(module_path) is not None
+        except ImportError:
+            return False
+    elif sys.version_info < (3,):
+        from imp import find_module
+        try:
+            if "." in module_path:
+                __import__(module_path)
+            else:
+                find_module(module_path)
+            return True
+        except ImportError:
+            return False
+    else:
+        raise NotImplementedError("No compatibility with Python 3.0 and 3.2")
diff --git a/kolibri/utils/conf.py b/kolibri/utils/conf.py
--- a/kolibri/utils/conf.py
+++ b/kolibri/utils/conf.py
@@ -20,6 +20,7 @@
 import json
 import logging
 import os
+from kolibri.utils.compat import module_exists
 
 logger = logging.getLogger(__name__)
 
@@ -84,3 +85,30 @@ def save(first_run=False):
     # Open up the config file and overwrite defaults
     with open(conf_file, 'r') as kolibri_conf_file:
         config.update(json.load(kolibri_conf_file))
+
+
+def autoremove_unavailable_plugins():
+    """
+    Sanitize INSTALLED_APPS - something that should be done separately for all
+    build in plugins, but we should not auto-remove plugins that are actually
+    configured by the user or some other kind of hard dependency that should
+    make execution stop if not loadable.
+    """
+    global config
+    changed = False
+    for module_path in config['INSTALLED_APPS']:
+        if not module_exists(module_path):
+            config['INSTALLED_APPS'].remove(module_path)
+            logger.error(
+                (
+                    "Plugin {mod} not found and disabled. To re-enable it, run:\n"
+                    "   $ kolibri plugin {mod} enable"
+                ).format(mod=module_path)
+            )
+            changed = True
+
+    if changed:
+        save()
+
+
+autoremove_unavailable_plugins()


Test patch:
diff --git a/kolibri/utils/tests/test_cli.py b/kolibri/utils/tests/test_cli.py
--- a/kolibri/utils/tests/test_cli.py
+++ b/kolibri/utils/tests/test_cli.py
@@ -23,6 +23,28 @@ def conf():
     conf.save()
 
 
+def test_bogus_plugin_autoremove(conf):
+    """
+    Checks that a plugin is auto-removed when it cannot be imported
+    """
+    plugin_name = "giraffe.horse"
+    conf.config["INSTALLED_APPS"].append(plugin_name)
+    conf.save()
+    conf.autoremove_unavailable_plugins()
+    assert plugin_name not in conf.config["INSTALLED_APPS"]
+
+
+def test_bogus_plugin_autoremove_no_path(conf):
+    """
+    Checks that a plugin without a dotted path is also auto-removed
+    """
+    plugin_name = "giraffehorse"
+    conf.config["INSTALLED_APPS"].append(plugin_name)
+    conf.save()
+    conf.autoremove_unavailable_plugins()
+    assert plugin_name not in conf.config["INSTALLED_APPS"]
+
+
 def test_bogus_plugin_disable(conf):
     installed_apps_before = conf.config["INSTALLED_APPS"][:]
     cli.plugin("i_do_not_exist", disable=True)