Repository: learningequality/kolibri

Issue:
On Windows, importing or exporting a large number of items (~1000) can fail
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
When trying to download content i get a transfer failed error.  First channel downloaded in bulk ok but cannot download further channelsin bulk. If i go in and download sub categories i can eventually pull down the whole channel

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

channel downloads as a whole not 1 by 1 resource sub sets


### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/kolibri.log
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->

```
[33mWARNING  Job d416c174faeb4648b22bd1420a75f2a0 raised an exception: Traceback (most recent call last):
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\_base.py", line 455, in result
    return self.__get_result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 149, in wrap
    raise e
OperationalError: too many SQL variables
[0m
WARNING:iceqube.worker.backends.base:Job d416c174faeb4648b22bd1420a75f2a0 raised an exception: Traceback (most recent call last):
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\_base.py", line 455, in result
    return self.__get_result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 149, in wrap
    raise e
OperationalError: too many SQL variables
```


### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

win 10 running kolibri 0.7.0.  all browsers.




Discussion:
@edunetck Would you mind sharing details of which channels and sub-categories you attempted to import? Which worked, and which did not work?
Hi
Importing from Koilbri Online.  Touchable Earth (337 resources) and Be Strong (65 resources) imoprted ok.  The other channels wont bulk import - im guessing it may have something to do with the number of resources in each channel.  For example i have nearly finished importing EngageNY but i have had  to do it a couple of chapters at a time - each chapter having X number of resources.  If i select too many chapters it will fail so have to reduce to 1,2 or 3 chapters at a time and the import works. If i select say 5 chapters and it fails i can go back and import each chapter individually andf it runs ok......  

@edunetck Thanks for the info. We will try to reproduce it on our side. I'll check the Engage NY channel to see what 1-5 chapters translates to in terms of resources.

To clarify, the largest number of chapters on Engage NY has been 3? But attempting 5 chapters fails?
it seems to fail if the total number of resources, irrespective of the number of chapters gets close to 1000 in one download. EngageNY has 8678 resources, each chapter is between 250 and 450 resources.  sometimes i can get 3, sometimes 2 chapters. ANd yes - have not been able to get more than 3 chapters but ihavent speciffically gone and looked for all the 200 resource chapters just to try that scenario.
Grade 5 mathematics in EnganeNY (EN) wont import at all, get same failed error.  All other chapters imported ok.

And now when Exporting facing the same issue.
@edunetck I just tried an import of about 1200 resources from EngageNY and it was successful (on a MacOS computer). We will try to reproduce this on a Windows machine.
@edunetck @radinamatic Was able to reproduce the same exact error on a Windows 10 machine. In her case, the threshold was slightly under 1000 resources from EngageNY.

I will rename this issue to make it easier for others with the same problem to find it and we'll keep you posted on when it's resolved.
@jonboiser for reference, this was raised by a couple people on the forum as well: Threads are here: https://community.learningequality.org/t/ck-12-and-ka-lite-channels-failing-to-download-0-7-0/467 and https://community.learningequality.org/t/is-kolibri-central-server-down/482/3
Looks like it's due to the implementation of `_get_leaves_ids` in https://github.com/learningequality/kolibri/pull/2703/files (CC @lyw07 @aronasorman). We shouldn't be creating a list of leaf IDs, we should leverage django-mptt's hierarchical queries. I'll give it a shot; do we have any test coverage on this?

Resolving patch:
diff --git a/kolibri/content/utils/import_export_content.py b/kolibri/content/utils/import_export_content.py
--- a/kolibri/content/utils/import_export_content.py
+++ b/kolibri/content/utils/import_export_content.py
@@ -1,16 +1,16 @@
 from django.db.models import Sum
-from kolibri.content.models import LocalFile, ContentNode
+from kolibri.content.models import ContentNode, LocalFile
 
 
 def get_files_to_transfer(channel_id, node_ids, exclude_node_ids, available):
     files_to_transfer = LocalFile.objects.filter(files__contentnode__channel_id=channel_id, available=available)
 
     if node_ids:
-        leaf_node_ids = _get_leaves_ids(node_ids)
+        leaf_node_ids = _get_leaf_node_ids(node_ids)
         files_to_transfer = files_to_transfer.filter(files__contentnode__in=leaf_node_ids)
 
     if exclude_node_ids:
-        exclude_leaf_node_ids = _get_leaves_ids(exclude_node_ids)
+        exclude_leaf_node_ids = _get_leaf_node_ids(exclude_node_ids)
         files_to_transfer = files_to_transfer.exclude(files__contentnode__in=exclude_leaf_node_ids)
 
     # Make sure the files are unique, to avoid duplicating downloads
@@ -21,12 +21,10 @@ def get_files_to_transfer(channel_id, node_ids, exclude_node_ids, available):
     return files_to_transfer, total_bytes_to_transfer
 
 
-def _get_leaves_ids(node_ids):
-    leaf_node_ids = []
-    for node_id in node_ids:
-        node_leaves = ContentNode.objects.get(pk=node_id) \
-            .get_descendants(include_self=True) \
-            .filter(children__isnull=True) \
-            .values_list('id', flat=True)
-        leaf_node_ids += node_leaves
-    return leaf_node_ids
+def _get_leaf_node_ids(node_ids):
+
+    return ContentNode.objects \
+        .filter(pk__in=node_ids) \
+        .get_descendants(include_self=True) \
+        .filter(children__isnull=True) \
+        .values_list('id', flat=True)


Test patch: