Repository: learningequality/kolibri

Issue:
"sqlite3.OperationalError: too many SQL variables" when trying to import large channels
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->

When trying to import significantly large amounts of contents have experienced this error -

```
ERROR 2020-08-17 09:20:24,790 kolibri.core.tasks.worker Job 737e2e6f419f498181e9c9cdb629102c raised an exception: Traceback (most recent call last):
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1283, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/default.py", line 590, in do_execute
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kivy/build/android/kolibri/core/tasks/worker.py", line 72, in handle_finished_future
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/_base.py", line 432, in result
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/_base.py", line 388, in __get_result
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/thread.py", line 57, in run
  File "/home/kivy/build/android/kolibri/core/tasks/worker.py", line 217, in wrap
  File "/home/kivy/build/android/kolibri/core/tasks/job.py", line 194, in y
  File "/home/kivy/build/android/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
  File "/home/kivy/build/android/kolibri/dist/django/core/management/base.py", line 330, in execute
  File "/home/kivy/build/android/kolibri/core/tasks/management/commands/base.py", line 113, in handle
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 451, in handle_async
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 180, in copy_content
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 366, in _transfer
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 659, in set_content_visibility
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 389, in mark_local_files_as_available
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 409, in mark_local_files_availability
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1020, in execute
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1323, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1517, in _handle_dbapi_exception
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/util/compat.py", line 178, in raise_
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1283, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/default.py", line 590, in do_execute
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables
[SQL: UPDATE content_localfile SET available=? WHERE content_localfile.id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 
```

"Large amount" could be resources totalling to around 100 MB+ from channels such as "Khan Academy" or "CK-12". It is mostly the number of resources that seem to matter than the size.

The imports are reported as failed in the UI.
If upon subsequent attempts we import smaller chunks (<100 MB) of resources, the imports progress much quickly and are reported as successful in the UI as well. This also indicates that the content transfer is happening properly and it is only the post processing part that seems to fail.

Have experienced this in imports irrespective of source and destination platforms. Either could be Windows, Linux or Android. 
 
### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

Channel resource imports should not fail.

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

Very inconvenient to get started when large amounts of resources can't be imported at once.

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->

Attached log contains multiple instances of `sqlite3.OperationalError: too many SQL variable` errors.

[kolibri.txt](https://github.com/learningequality/kolibri/files/5085999/kolibri.txt)


### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

Initiate an import of large number of resources i.e. at least size worth of 100 MB from channels such as KA or CK-12. For example, all Math Grade 1 content from KA.

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

Kolibri 0.14.1
Irrespective of environment.



Discussion:

Resolving patch:
diff --git a/kolibri/core/content/utils/annotation.py b/kolibri/core/content/utils/annotation.py
--- a/kolibri/core/content/utils/annotation.py
+++ b/kolibri/core/content/utils/annotation.py
@@ -25,6 +25,7 @@
 from kolibri.core.content.models import ContentNode
 from kolibri.core.content.models import File
 from kolibri.core.content.models import LocalFile
+from kolibri.core.content.utils.sqlalchemybridge import filter_by_checksums
 from kolibri.core.device.models import ContentCacheKey
 
 logger = logging.getLogger(__name__)
@@ -408,7 +409,11 @@ def mark_local_files_availability(checksums, availability, destination=None):
         for i in range(0, len(checksums), CHUNKSIZE):
             connection.execute(
                 LocalFileTable.update()
-                .where(LocalFileTable.c.id.in_(checksums[i : i + CHUNKSIZE]))
+                .where(
+                    filter_by_checksums(
+                        LocalFileTable.c.id, checksums[i : i + CHUNKSIZE]
+                    )
+                )
                 .values(available=availability)
             )
 
@@ -417,55 +422,71 @@ def mark_local_files_availability(checksums, availability, destination=None):
         bridge.end()
 
 
+def _check_file_availability(files):
+    checksums_to_set_available = []
+    checksums_to_set_unavailable = []
+    for file in files:
+        try:
+            # Update if the file exists, *and* the localfile is set as unavailable.
+            if os.path.exists(
+                get_content_storage_file_path(
+                    get_content_file_name({"id": file[0], "extension": file[2]})
+                )
+            ):
+                if not file[1]:
+                    checksums_to_set_available.append(file[0])
+            # Update if the file does not exist, *and* the localfile is set as available.
+            else:
+                if file[1]:
+                    checksums_to_set_unavailable.append(file[0])
+        except InvalidStorageFilenameError:
+            continue
+
+    return checksums_to_set_available, checksums_to_set_unavailable
+
+
 def set_local_file_availability_from_disk(checksums=None, destination=None):
+    if type(checksums) == list and len(checksums) > CHUNKSIZE:
+        for i in range(0, len(checksums), CHUNKSIZE):
+            set_local_file_availability_from_disk(
+                checksums=checksums[i : i + CHUNKSIZE], destination=destination
+            )
+        return
+
     bridge = Bridge(app_name=CONTENT_APP_NAME, sqlite_file_path=destination)
 
-    LocalFileClass = bridge.get_class(LocalFile)
+    LocalFileTable = bridge.get_table(LocalFile)
+
+    query = select(
+        [LocalFileTable.c.id, LocalFileTable.c.available, LocalFileTable.c.extension]
+    )
 
     if checksums is None:
         logger.info(
             "Setting availability of LocalFile objects based on disk availability"
         )
-        files = bridge.session.query(
-            LocalFileClass.id, LocalFileClass.available, LocalFileClass.extension
-        ).all()
     elif type(checksums) == list:
         logger.info(
             "Setting availability of {number} LocalFile objects based on disk availability".format(
                 number=len(checksums)
             )
         )
-        files = (
-            bridge.session.query(
-                LocalFileClass.id, LocalFileClass.available, LocalFileClass.extension
-            )
-            .filter(LocalFileClass.id.in_(checksums))
-            .all()
-        )
+        query = query.where(filter_by_checksums(LocalFileTable.c.id, checksums))
     else:
         logger.info(
             "Setting availability of LocalFile object with checksum {checksum} based on disk availability".format(
                 checksum=checksums
             )
         )
-        files = [bridge.session.query(LocalFileClass).get(checksums)]
+        query = query.where(LocalFileTable.c.id == checksums)
 
-    checksums_to_set_available = []
-    checksums_to_set_unavailable = []
-    for file in files:
-        try:
-            # Update if the file exists, *and* the localfile is set as unavailable.
-            if os.path.exists(
-                get_content_storage_file_path(get_content_file_name(file))
-            ):
-                if not file.available:
-                    checksums_to_set_available.append(file.id)
-            # Update if the file does not exist, *and* the localfile is set as available.
-            else:
-                if file.available:
-                    checksums_to_set_unavailable.append(file.id)
-        except InvalidStorageFilenameError:
-            continue
+    connection = bridge.get_connection()
+
+    files = connection.execute(query).fetchall()
+
+    checksums_to_set_available, checksums_to_set_unavailable = _check_file_availability(
+        files
+    )
 
     bridge.end()
 
diff --git a/kolibri/core/content/utils/sqlalchemybridge.py b/kolibri/core/content/utils/sqlalchemybridge.py
--- a/kolibri/core/content/utils/sqlalchemybridge.py
+++ b/kolibri/core/content/utils/sqlalchemybridge.py
@@ -1,6 +1,7 @@
 import importlib
 import logging
 import os
+import re
 from uuid import UUID
 
 from django.apps import apps
@@ -413,6 +414,15 @@ def _by_uuids(field, ids, validate, include, vendor=None):
     return UnaryExpression(field, modifier=operators.custom_op(empty_query))
 
 
+checksum_re = re.compile("^[0-9a-f]{32}$")
+
+
+def _validate_checksums(checksums):
+    for checksum in checksums:
+        if not checksum_re.match(checksum):
+            raise ValueError("Invalid checksum: {}".format(checksum))
+
+
 def filter_by_checksums(field, checksums):
     query = "IN ("
     # trick to workaround postgresql, it does not allow returning ():
@@ -425,10 +435,17 @@ def filter_by_checksums(field, checksums):
                 these should be batched into separate querysets to avoid SQL Query too large errors in SQLite
             """
             )
-        checksums_list = ["'{}'".format(identifier) for identifier in checksums]
-        return UnaryExpression(
-            field, modifier=operators.custom_op(query + ",".join(checksums_list) + ")")
-        )
+        try:
+            _validate_checksums(checksums)
+            checksums_list = ["'{}'".format(identifier) for identifier in checksums]
+            return UnaryExpression(
+                field,
+                modifier=operators.custom_op(query + ",".join(checksums_list) + ")"),
+            )
+        except ValueError:
+            # the value is not a valid hex code for a checksum, so fall through to the
+            # empty case and don't return any results
+            pass
     return UnaryExpression(field, modifier=operators.custom_op(empty_query))
 
 


Test patch: