Repository: learningequality/kolibri

Issue:
Kolibri 0.15.2 Python 3.10 error on Ubuntu 22.04 "ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)"
This promising work from earlier in April seemed tailored to getting Kolibri 0.15.2 working with Python 3.10?  _[CLARIF: No, Kolibri seems to be deferring support for Python 3.10 until a later date]_

- #9295 
- PR #9296

Somehow however the just-released Kolibri 0.15.2 (https://learningequality.org/r/kolibri-deb-latest) appears to install on Ubuntu Desktop 22.04 on x86_64 (as part of [Internet-in-a-Box](http://github.com/iiab/iiab)) &mdash; but then serious Python 3.10 problems appear:

```
TASK [kolibri : Provision Kolibri, while setting: facility name, admin acnt / password, preset type, and language] *****************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "export KOLIBRI_HOME=\"/library/kolibri\" && \"/usr/bin/kolibri\" manage provisiondevice --facility \"Kolibri-in-a-Box\" --superusername \"Admin\" --superuserpassword \"changeme\" --preset \"formal\" --language_id \"en\" #--preset \"formal\" --language_id \"en\" --verbosity 0 --noinput\n", "delta": "0:00:01.342052", "end": "2022-04-29 21:47:01.194557", "msg": "non-zero return code", "rc": 1, "start": "2022-04-29 21:46:59.852505", "stderr": "INFO: No C extensions are available for this platform\nTraceback (most recent call last):\n  File \"/usr/bin/kolibri\", line 11, in <module>\n    load_entry_point('kolibri==0.15.2', 'console_scripts', 'kolibri')()\n  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 479, in load_entry_point\n    return get_distribution(dist).load_entry_point(group, name)\n  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2861, in load_entry_point\n    return ep.load()\n  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2465, in load\n    return self.resolve()\n  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2471, in resolve\n    module = __import__(self.module_name, fromlist=['__name__'], level=0)\n  File \"/usr/lib/python3/dist-packages/kolibri/utils/cli.py\", line 12, in <module>\n    from django.core.management import execute_from_command_line\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py\", line 13, in <module>\n    from django.core.management.base import (\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py\", line 17, in <module>\n    from django.db.migrations.exceptions import MigrationSchemaMissing\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/__init__.py\", line 2, in <module>\n    from .operations import *  # NOQA\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/__init__.py\", line 1, in <module>\n    from .fields import AddField, AlterField, RemoveField, RenameField\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/fields.py\", line 4, in <module>\n    from django.db.models.fields import NOT_PROVIDED\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/__init__.py\", line 5, in <module>\n    from django.db.models.deletion import (\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/deletion.py\", line 5, in <module>\n    from django.db.models import signals, sql\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/__init__.py\", line 2, in <module>\n    from django.db.models.sql.query import *  # NOQA\n  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/query.py\", line 11, in <module>\n    from collections import Counter, Iterator, Mapping, OrderedDict\nImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)", "stderr_lines": ["INFO: No C extensions are available for this platform", "Traceback (most recent call last):", "  File \"/usr/bin/kolibri\", line 11, in <module>", "    load_entry_point('kolibri==0.15.2', 'console_scripts', 'kolibri')()", "  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 479, in load_entry_point", "    return get_distribution(dist).load_entry_point(group, name)", "  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2861, in load_entry_point", "    return ep.load()", "  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2465, in load", "    return self.resolve()", "  File \"/usr/lib/python3/dist-packages/pkg_resources/__init__.py\", line 2471, in resolve", "    module = __import__(self.module_name, fromlist=['__name__'], level=0)", "  File \"/usr/lib/python3/dist-packages/kolibri/utils/cli.py\", line 12, in <module>", "    from django.core.management import execute_from_command_line", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py\", line 13, in <module>", "    from django.core.management.base import (", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py\", line 17, in <module>", "    from django.db.migrations.exceptions import MigrationSchemaMissing", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/__init__.py\", line 2, in <module>", "    from .operations import *  # NOQA", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/__init__.py\", line 1, in <module>", "    from .fields import AddField, AlterField, RemoveField, RenameField", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/fields.py\", line 4, in <module>", "    from django.db.models.fields import NOT_PROVIDED", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/__init__.py\", line 5, in <module>", "    from django.db.models.deletion import (", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/deletion.py\", line 5, in <module>", "    from django.db.models import signals, sql", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/__init__.py\", line 2, in <module>", "    from django.db.models.sql.query import *  # NOQA", "  File \"/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/query.py\", line 11, in <module>", "    from collections import Counter, Iterator, Mapping, OrderedDict", "ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)"], "stdout": "", "stdout_lines": []}
```

Likewise `systemctl start kolibri` fails, leading to this `systemctl status kolibri` output/trace:

```
Apr 29 21:55:00 box kolibri[3985]: INFO: No C extensions are available for this platform
Apr 29 21:55:01 box kolibri[3985]: Traceback (most recent call last):
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/bin/kolibri", line 11, in <module>
Apr 29 21:55:01 box kolibri[3985]:     load_entry_point('kolibri==0.15.2', 'console_scripts', 'kolibri')()
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 479, in load_entry_point
Apr 29 21:55:01 box kolibri[3985]:     return get_distribution(dist).load_entry_point(group, name)
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2861, in load_entry_point
Apr 29 21:55:01 box kolibri[3985]:     return ep.load()
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2465, in load
Apr 29 21:55:01 box kolibri[3985]:     return self.resolve()
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2471, in resolve
Apr 29 21:55:01 box kolibri[3985]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 12, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.core.management import execute_from_command_line
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 13, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.core.management.base import (
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 17, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.db.migrations.exceptions import MigrationSchemaMissing
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/__init__.py", line 2, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from .operations import *  # NOQA
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/__init__.py", line 1, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from .fields import AddField, AlterField, RemoveField, RenameField
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/fields.py", line 4, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.db.models.fields import NOT_PROVIDED
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/__init__.py", line 5, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.db.models.deletion import (
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/deletion.py", line 5, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.db.models import signals, sql
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/__init__.py", line 2, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from django.db.models.sql.query import *  # NOQA
Apr 29 21:55:01 box kolibri[3985]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/query.py", line 11, in <module>
Apr 29 21:55:01 box kolibri[3985]:     from collections import Counter, Iterator, Mapping, OrderedDict
Apr 29 21:55:01 box kolibri[3985]: ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
Apr 29 21:55:01 box systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Apr 29 21:55:01 box systemd[1]: kolibri.service: Failed with result 'exit-code'.
Apr 29 21:55:01 box systemd[1]: Failed to start Kolibri.
```

Any suggestions?

Thanks if possible!

Earlier/Stale references to this same topic:

- "Kolibri doesn’t currently support Python 3.10.0 or higher"
  https://kolibri-dev.readthedocs.io/en/develop/getting_started.html#python-and-pip
- "ImportError: cannot import name ‘Iterator’ from ‘collections’ (Py 3.10)
  https://community.learningequality.org/t/importerror-cannot-import-name-iterator-from-collections-py-3-10/2266
- PR iiab/iiab#3189


Discussion:
Or is it the case that Ubuntu 22.04 LTS (Python 3.10) users need to wait for Kolibri 0.16.0 &mdash; around the end of 2022 possibly?

Resolving patch:
diff --git a/kolibri/utils/compat.py b/kolibri/utils/compat.py
--- a/kolibri/utils/compat.py
+++ b/kolibri/utils/compat.py
@@ -82,3 +82,22 @@ def parse_version(v):
     parsed = _parse_version(v)
 
     return VersionCompat(parsed)
+
+
+def monkey_patch_collections():
+    """
+    Monkey-patching for the collections module is required for Python 3.10
+    and above.
+    Prior to 3.10, the collections module still contained all the entities defined in
+    collections.abc from Python 3.3 onwards. Here we patch those back into main
+    collections module.
+    This can be removed when we upgrade to a version of Django that is Python 3.10 compatible.
+    """
+    if sys.version_info < (3, 10):
+        return
+    import collections
+    from collections import abc
+
+    for name in dir(abc):
+        if not hasattr(collections, name):
+            setattr(collections, name, getattr(abc, name))
diff --git a/kolibri/utils/env.py b/kolibri/utils/env.py
--- a/kolibri/utils/env.py
+++ b/kolibri/utils/env.py
@@ -18,6 +18,7 @@
     ColoredFormatter = None
 
 from .logger import LOG_COLORS
+from kolibri.utils.compat import monkey_patch_collections
 
 
 logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.INFO)
@@ -124,6 +125,8 @@ def set_env():
 
     check_python_versions()
 
+    monkey_patch_collections()
+
     sys.path = [os.path.realpath(os.path.dirname(kolibri_dist.__file__))] + sys.path
 
     # Add path for c extensions to sys.path
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -107,5 +107,5 @@ def run(self):
         "Programming Language :: Python :: Implementation :: PyPy",
     ],
     cmdclass={"install_scripts": gen_windows_batch_files},
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.10",
+    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <3.11",
 )


Test patch:
diff --git a/test/patch_pytest.py b/test/patch_pytest.py
new file mode 100644
--- /dev/null
+++ b/test/patch_pytest.py
@@ -0,0 +1,38 @@
+"""
+This script backports this Python 3.10 compatibility fix https://github.com/pytest-dev/pytest/pull/8540
+in order to allow pytest to run in Python 3.10 without upgrading to version 6.2.5 which does not support 2.7
+"""
+import os
+import subprocess
+
+import pytest
+
+site_packages_dir = os.path.dirname(pytest.__file__)
+
+patch_file = os.path.join(os.path.dirname(__file__), "pytest_3.10.patch")
+
+print("Applying patch: " + str(patch_file))
+
+# -N: insist this is FORWARD patch, don't reverse apply
+# -p1: strip first path component
+# -t: batch mode, don't ask questions
+patch_command = ["patch", "-N", "-p1", "-d", site_packages_dir, "-t", "-i", patch_file]
+print(" ".join(patch_command))
+try:
+    # Use a dry run to establish whether the patch is already applied.
+    # If we don't check this, the patch may be partially applied (which is bad!)
+    subprocess.check_output(patch_command + ["--dry-run"])
+except subprocess.CalledProcessError as e:
+    if e.returncode == 1:
+        # Return code 1 means not all hunks could be applied, this usually
+        # means the patch is already applied.
+        print(
+            "Warning: failed to apply patch (exit code 1), "
+            "assuming it is already applied: ",
+            str(patch_file),
+        )
+    else:
+        raise e
+else:
+    # The dry run worked, so do the real thing
+    subprocess.check_output(patch_command)
diff --git a/test/pytest_3.10.patch b/test/pytest_3.10.patch
new file mode 100644
--- /dev/null
+++ b/test/pytest_3.10.patch
@@ -0,0 +1,38 @@
+diff --git a/_pytest/assertion/rewrite.py b/_pytest/assertion/rewrite.py
+index 4f96b9e8c..1aa5b12de 100644
+--- a/_pytest/assertion/rewrite.py
++++ b/_pytest/assertion/rewrite.py
+@@ -587,10 +587,6 @@ class AssertionRewriter(ast.NodeVisitor):
+             return
+         # Insert some special imports at the top of the module but after any
+         # docstrings and __future__ imports.
+-        aliases = [
+-            ast.alias(py.builtin.builtins.__name__, "@py_builtins"),
+-            ast.alias("_pytest.assertion.rewrite", "@pytest_ar"),
+-        ]
+         doc = getattr(mod, "docstring", None)
+         expect_docstring = doc is None
+         if doc is not None and self.is_rewrite_disabled(doc):
+@@ -617,6 +613,22 @@ class AssertionRewriter(ast.NodeVisitor):
+             pos += 1
+         else:
+             lineno = item.lineno
++        # Now actually insert the special imports.
++        if sys.version_info >= (3, 10):
++            aliases = [
++                ast.alias("builtins", "@py_builtins", lineno=lineno, col_offset=0),
++                ast.alias(
++                    "_pytest.assertion.rewrite",
++                    "@pytest_ar",
++                    lineno=lineno,
++                    col_offset=0,
++                ),
++            ]
++        else:
++            aliases = [
++                ast.alias("builtins", "@py_builtins"),
++                ast.alias("_pytest.assertion.rewrite", "@pytest_ar"),
++            ]
+         imports = [
+             ast.Import([alias], lineno=lineno, col_offset=0) for alias in aliases
+         ]
diff --git a/test/test_future_and_futures.py b/test/test_future_and_futures.py
--- a/test/test_future_and_futures.py
+++ b/test/test_future_and_futures.py
@@ -2,9 +2,10 @@
 import os
 import sys
 
-from django.test import TestCase
+# Import from kolibri first to ensure Kolibri's monkey patches are applied.
+from kolibri import dist as kolibri_dist  # noreorder
 
-from kolibri import dist as kolibri_dist
+from django.test import TestCase  # noreorder
 
 dist_dir = os.path.realpath(os.path.dirname(kolibri_dist.__file__))