Repository: learningequality/kolibri

Issue:
installers don't work on Python 3
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->

When using pex/whl files from the latest commits of develop branch, I'm not able to run it successfully with Python 3. 
This issue happened before: https://github.com/learningequality/kolibri/issues/1797

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

Installers should work with python 3 as well

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

If users only have Python 3 installed in their machines, they are not able to run Kolibri

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/kolibri.log
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->

```
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/kolibri/__main__.py", line 7, in <module>
    from kolibri.utils.cli import main
  File "/usr/local/lib/python3.6/site-packages/kolibri/utils/cli.py", line 17, in <module>
    from django.core.management import call_command  # noqa
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/core/management/__init__.py", line 13, in <module>
    from django.core.management.base import (
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/core/management/base.py", line 17, in <module>
    from django.db.migrations.exceptions import MigrationSchemaMissing
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/migrations/__init__.py", line 2, in <module>
    from .operations import *  # NOQA
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/migrations/operations/__init__.py", line 1, in <module>
    from .fields import AddField, AlterField, RemoveField, RenameField
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/migrations/operations/fields.py", line 4, in <module>
    from django.db.models.fields import NOT_PROVIDED
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/models/__init__.py", line 3, in <module>
    from django.db.models.aggregates import *  # NOQA
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/models/aggregates.py", line 5, in <module>
    from django.db.models.expressions import Func, Star
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/models/expressions.py", line 6, in <module>
    from django.db.models import fields
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/db/models/fields/__init__.py", line 14, in <module>
    from django import forms
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/forms/__init__.py", line 6, in <module>
    from django.forms.boundfield import *  # NOQA
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/forms/boundfield.py", line 6, in <module>
    from django.forms.utils import flatatt, pretty_name
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/forms/utils.py", line 10, in <module>
    from django.utils.html import escape, format_html, format_html_join, html_safe
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/utils/html.py", line 15, in <module>
    from django.utils.text import normalize_newlines
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/utils/text.py", line 14, in <module>
    from django.utils.six.moves import html_entities
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/utils/six.py", line 92, in __get__
    result = self._resolve()
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/utils/six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/django/utils/six.py", line 82, in _import_module
    __import__(name)
  File "/usr/local/lib/python3.6/site-packages/kolibri/dist/html/__init__.py", line 7, in <module>
    raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
```

### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

1. Get the pex/whl from develop branch
2. Run kolibri

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

Kolibri version: 0.8.0
Operating system: Mac OS X




Discussion:
It seems that this issue is introduced by https://github.com/learningequality/kolibri/pull/2808, which was created to fix the openssl issue in Ubuntu. I will look into this more
The issue is introduced when we moved `kolibri/dist` to the front of PYTHONPATH. In this case, Kolibri doesn't work with Python 3 on OS X nor Ubuntu 14.04, showing the error described above (haven't tried other versions of Ubuntu yet, but I assume they would have the similar result). This error is because the [`future` package](https://github.com/PythonCharmers/python-future/blob/v0.16.0/setup.py#L63) (from `morango`) depends on the `html` module for python 2, which installs `html` in `kolibri/dist`. However, this `html` module shouldn't be used for Python 3, as Python 3 is supposed to use its default `html` module. Having `kolibri/dist` in the front of PYTHONPATH makes kolibri import html from `kolibri/dist` instead of Python 3's own packages, which caused the error.

I tried to move the `kolibri/dist` to the end of PYTHONPATH, as we did before. In this case, it works on OS X but not on Ubuntu 14.04. This is because Ubuntu 14.04 has its system library `requests` with version 2.2.1, which is older than the one we have in `kolibri/dist`. This system `requests` module doesn't have `packages.py`, which is used in `utils/Transfer.py` to download channels. Therefore, when we move the `kolibri/dist` to the end of PYTHONPATH, it will cause an error on Ubuntu when we trying to import channels from Studio.
  
This should be opened as a Morango issue possibly? Maybe we should create a test case that reproduces it in the Travis build of Morango?

Also similar (but not the same):

https://github.com/learningequality/iceqube/issues/31
Hi @benjaoming @ralphiee22 I wrote some [code](https://github.com/learningequality/kolibri/compare/release-v0.7.x...lyw07:move_future_modules) to fix this issue and https://github.com/learningequality/iceqube/issues/31. It moves the directory of those python 2 modules, for example `html` from `future` and `concurrent` from `futures` to a subdirectory called `py2only` inside `kolibri/dist`. It also adds the directory of `py2only` to the PYTHONPATH only when running in Python 2. 

So when it's in Python 2, the PYTHONPATH will be something like
`["kolibri/dist", <other paths>, "kolibri/dist/py2only"]`
to make those python 2 modules available.

When it's in Python 3, the PYTHONPATH will be
`["kolibri/dist", <other paths>]`
to make those python 2 modules unavailable so that kolibri will use the Python 3 built-in modules.

I tested on OS X with the pex file and on Ubuntu 14.04/16.04/17.04/17.10 with the whl file under both Python 2 and Python 3. They all worked without both issues we have for `future` and `futures`. 

However, I'm not sure if this is a good approach and it's worth it to create a PR for this, since I feel like this code has some weird design and seems a bit hard-coded. But this is the way I can think of to fix both issues currently. When you have time, could you please take a look at the code and tell me about your thoughts? If you think there are other ways to fix those issues, please let me know :) Thank you!

>This should be opened as a Morango issue possibly?

Hmm, doesn't sound like it to me. The only thing Morango is doing is requiring the `future` module, which isn't problematic in itself (and will work fine for normal non-bundled distributions). It's the path modifications in Kolibri that are causing the broader issue, since we have a single `dist` folder that is created by installing modules using Python 2, which is then also used when running with Python 3. It's the same issue we had back in the day with `cherrypy` and other libs installing themselves differently depending on the Python version, which is why we were pointing our requirements to the git version rather than pypi for a while (until they had fixed this on their side).
@lyw07 Your approach looks reasonable, but as you say, it's a bit hard-coded and specific to `futures`, when there might also be other packages with this type of issue. Would it be possible to generalize this a bit by leveraging some of the same systems you did for `cext` to have Python-version-specific dist folders? It's a similar type of problem (some libraries or library versions that should only be added to the path for a particular architecture/Python version/OS). Not sure how easy that would be, but might be a bit cleaner, until we're able to take a step back and reassess our overall approach to static bundling.
@jamalex Thank you for your suggestions! If I understand correctly, since `futures` is a dependency from `iceqube` and thus is installed along with `iceqube`, so I need to have this script that changes the `futures` directory after we install it inside `kolibri/dist` from "kolibri/dist/futures" to `kolibri/dist/futures/python2/futures` (I assume `kolibri/dist/futures/python3/futures` shouldn't be created)? And this applies to `future` from `morango` as well.
I feel that this approach is still specific to `future` and `futures`. Do you think it is generalized enough? Thank you so much!
@lyw07 I think your approach sounds reasonable. We only have 2 packages `future` and `futures` that are annoying enough to raise these issues.

Another important part is to ensure that these failures show up in tests, so we can maintain this "dynamic" path in case other libraries suffer the same shortcoming. Our tests have to run the built environment, not the dev environment... so simulating installing the Kolibri source dist, and then running tests against this. I think adding `make staticdeps` to populate `kolibri/dist` before running tox tests (tox.ini has `usedevelop=true`) would suffice for now. But we should figure out how to make tox run the real `setup.py`.. which might be as simple as specifying `usedevelop=false` and running `make assets` before invoking tox.. but I haven't tried.

Btw notice that `future` places all of these packages in the root namespace: https://github.com/PythonCharmers/python-future/tree/master/src
Oh and one more thing! The reason why we have Python 2 dependencies is because we invoke `make dist` with Python 2 -- we should actually assert that it is Python 3, so we never by mistake start creating distributions with Python 3, because that would leave out the Python 2-specific reqs!
@benjaoming sorry do you mean that putting the command `make staticdeps` in the `before_script` section inside `.travis.yml` file? If we populate `kolibri/dist` before running tests, will these failures still show in the travisCI? Sorry I'm not familiar with tests setup, so I'm confused about this part... If you have time, could you please explain a bit? Thank you!

I also have [an updated version](https://github.com/learningequality/kolibri/compare/release-v0.7.x...lyw07:move_future_modules) that is a more generalized process of finding python 2 modules and moving them to `py2only` directory. In this code, I'm finding python 2 modules by:
1. Going inside `metadata.json` of each `dist-info` directory, and find if they have defined `Programming Language :: Python :: 2 :: Only` as one of the classifiers. (in our case, `futures` has defined it as one of its classifiers)
2. Try to import all the modules in the `kolibri/dist` directory under Python 3. For each module that can't be imported successfully and cause an ImportError, I move them to `py2only` directory. (in our case, `future` has defined its classifiers as both Python 2 and Python 3, so this is the only way I can think of to find the Python 2 packages inside `future`..)

The problem with this way of implementation is that the file `py2only.py` needs to be run with Python 3 to find those modules that can't be imported (the second point I mentioned above), so I put the command `python3 py2only.py` in `Makefile`. However, we always use python 2 to run our builds. Having Python 3 to run a file seems to be very inconsistent. 

Maybe @jamalex when you have some time, could you please take a look at this updated version as well? Thank you!
Hi @lyw07 I just saw you had made a branch with the `py2only.py` approach, I have ~2~ 3 comments, but otherwise I think it's the right solution

1. Instead of the very advanced identification of Python 2 classifiers in setup.py, I think you should hard-code the names of the packages to move. It will be way easier to configure and understand.
   
   This approach will eliminate the string code + the need to try-fail importing things in Python 3. These to assertions are no guarantees that a package is Python 2-only because 1) package maintainers are often sloppy with the classifiers, and 2) a package might import file and fail later at runtime.
   
   I know that the list of packages in `future` is long, but it should be manageable :)
   
1. Because the root-level of our repository is becoming extremely cluttered, I think you should consider renaming both install_cext.py and py2only.py to something else, possibly moving them to `build_tools/` folder.
   
1. I think we need the tests, as well. So we can fail+fix in a single PR with 2 commits. For proof of concept.
@benjaoming thank you for the comments! They are really helpful. I think I will then stick to the [first approach](https://github.com/learningequality/kolibri/compare/release-v0.7.x...lyw07:move_future_modules). 

And sorry I'm still confused about the third point where you said "fail+fix" on the tests. Could you please explain this more? Thank you!
The fail+fix PR looks like this... it's test-driven btw :)

1. You create a commit with the tests that are breaking, proving the issue ✘
1. Then add the fix, making it evident that the tests work ✓

To create the failure, I would suggest the approach of removing `usedevelop` from `tox.ini` + all the requirements, and then only install the built Kolibri in the tox env. If you can't get it to work quickly, then forget what I said :)
thank you! I will give it a try
I think @benjaoming is much more qualified than I am to advise on this, but @aronasorman's take once he's back next week would be great too.
@jamalex That sounds good to me! I will write the tests and create a PR first, and then wait for Aron's thoughts.

Resolving patch:
diff --git a/install_cexts.py b/build_tools/install_cexts.py
similarity index 100%
rename from install_cexts.py
rename to build_tools/install_cexts.py
diff --git a/build_tools/py2only.py b/build_tools/py2only.py
new file mode 100644
--- /dev/null
+++ b/build_tools/py2only.py
@@ -0,0 +1,49 @@
+import os
+import shutil
+import sys
+
+dest = 'py2only'
+futures_dirname = 'concurrent'
+DIST_DIR = os.path.realpath('kolibri/dist')
+
+
+def hide_py2_modules():
+    """
+    Move the directory of 'futures' and python2-only modules of 'future'
+    inside the directory 'py2only'
+    """
+
+    # Move the directory of 'futures' inside the directory 'py2only'
+    _move_modules_to_py2only(futures_dirname)
+
+    # Future's submodules are not downloaded in Python 3 but only in Python 2
+    if sys.version_info[0] == 2:
+        from future.standard_library import TOP_LEVEL_MODULES
+
+        for module in TOP_LEVEL_MODULES:
+            if module == 'test':
+                continue
+
+            # Move the directory of submodules of 'future' inside 'py2only'
+            _move_modules_to_py2only(module)
+
+
+def _move_modules_to_py2only(module_name):
+    module_src_path = os.path.join(DIST_DIR, module_name)
+    module_dst_path = os.path.join(DIST_DIR, dest, module_name)
+    shutil.move(module_src_path, module_dst_path)
+
+
+if __name__ == '__main__':
+    # Temporarily add `kolibri/dist` to PYTHONPATH to import future
+    sys.path = sys.path + [os.path.realpath(os.path.join(DIST_DIR))]
+
+    try:
+        os.makedirs(os.path.join(DIST_DIR, dest))
+    except OSError:
+        raise
+
+    hide_py2_modules()
+
+    # Remove `kolibri/dist` from PYTHONPATH
+    sys.path = sys.path[:-1]
diff --git a/kolibri/utils/env.py b/kolibri/utils/env.py
--- a/kolibri/utils/env.py
+++ b/kolibri/utils/env.py
@@ -60,6 +60,12 @@ def set_env():
         reload(sys)  # noqa
         sys.setdefaultencoding('utf8')
 
+        # Dynamically add the path of `py2only` to PYTHONPATH in Python 2 so that
+        # we only import the `future` and `futures` packages from system packages when
+        # running with Python 3. Please see `build_tools/py2only.py` for details.
+        sys.path = sys.path + [os.path.join(
+            os.path.realpath(os.path.dirname(kolibri_dist.__file__)), 'py2only')]
+
     try:
         from .build_config.default_settings import settings_path
     except ImportError:


Test patch:
diff --git a/test/test_future_and_futures.py b/test/test_future_and_futures.py
new file mode 100644
--- /dev/null
+++ b/test/test_future_and_futures.py
@@ -0,0 +1,35 @@
+import os
+import sys
+import imp
+from django.test import TestCase
+from kolibri import dist as kolibri_dist
+
+dist_dir = os.path.realpath(os.path.dirname(kolibri_dist.__file__))
+
+class FutureAndFuturesTestCase(TestCase):
+
+    def test_import_concurrent_py3(self):
+        import concurrent
+        if sys.version_info[0] == 3:
+            # Python 3 is supposed to import its builtin package `concurrent`
+            # instead of being inside kolibri/dist/py2only or kolibri/dist
+            concurrent_parent_path = os.path.realpath(
+                os.path.dirname(os.path.dirname(concurrent.__file__)))
+
+            self.assertNotEqual(dist_dir, concurrent_parent_path)
+            self.assertNotEqual(os.path.join(dist_dir, 'py2only'), concurrent_parent_path)
+
+    def test_import_future_py2(self):
+        from future.standard_library import TOP_LEVEL_MODULES
+
+        if sys.version_info[0] == 2:
+            for module_name in TOP_LEVEL_MODULES:
+                if 'test' in module_name:
+                    continue
+
+                module_parent_path = os.path.realpath(
+                    os.path.dirname(imp.find_module(module_name)[1]))
+                # future's standard libraries such as `html` should not be found
+                # at the same level as kolibri/dist; otherwise, python3 will try to
+                # import them from kolibri/dist instead of its builtin packages
+                self.assertNotEqual(dist_dir, module_parent_path)