Repository: learningequality/kolibri

Issue:
Very long, CPU-intensive process after asking to import selected Khan Academy resources
### Observed behavior

Need more info on this, but this is the best I can produce right now.. observe the 566m of CPU usage:

```
top - 01:39:58 up 2 days, 11:30,  1 user,  load average: 2.10, 2.17, 2.17
Tasks: 287 total,   1 running, 286 sleeping,   0 stopped,   0 zombie
%Cpu(s): 45.6 us, 12.9 sy,  0.0 ni, 41.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8054696 total,   904492 free,  5504828 used,  1645376 buff/cache
KiB Swap: 16381948 total, 14482384 free,  1899564 used.  1499508 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                            
15123 kolibri   20   0 3474360 1.112g  10440 S 201.0 14.5 566:35.82 kolibri                         
```

### Steps to reproduce

Not sure, but I have asked to download a select sub-part of the KA resources, and my laptop is stuck for hours.


### Context

Kolibri 0.9.0, debian installation source, Ubuntu 16.04



Discussion:
Some more debugging: After monitoring bandwidth usage for >10m, the process still hasn't requested anything online. So I would assume that there is some kind of endless loop going on here.

Log doesn't say anything:

```
INFO 2018-04-07 18:41:17,811 cli Going to daemon mode, logging to /var/kolibri/.kolibri/server.log
INFO 2018-04-07 18:41:17,823 ping Attempting a ping.
INFO 2018-04-07 18:41:17,843 ping data: {"language": "en", "sysversion": "3.5.2 (default, Nov 23 2017, 16:37:01) \n[GCC 5.4.0 20160609]", "node_id": "0906dca0fbae53e9abe7", "database_id": "db4903d6f2b84001aaf0ee01ecf45b00", "uptime": 0, "mode": "", "version": "0.9.0", "instance_id": "600bc8a0a69b1dae1d24cf0afa33d34d", "platform": "Linux-4.4.0-119-generic-x86_64-with-Ubuntu-16.04-xenial", "system_id": ""}
```

```
kolibri@localhost:~$ kolibri start --foreground
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Running update routines for new version...

0 static files copied to '/var/kolibri/.kolibri/static', 588 unmodified.
Operations to perform:
  Apply all migrations: admin, auth, content, contenttypes, device, exams, kolibriauth, lessons, logger, morango, sessions
Running migrations:
  No migrations to apply.
INFO     Running 'kolibri start' in foreground...
INFO     Kolibri running on:

	http://192.168.1.100:8080/
	http://192.168.1.102:8080/
	http://127.0.0.1:8080/

INFO     Attempting a ping.
INFO     data: {"database_id": "db4903d6f2b84001aaf0ee01ecf45b00", "instance_id": "600bc8a0a69b1dae1d24cf0afa33d34d", "sysversion": "3.5.2 (default, Nov 23 2017, 16:37:01) \n[GCC 5.4.0 20160609]", "platform": "Linux-4.4.0-119-generic-x86_64-with-Ubuntu-16.04-xenial", "mode": "", "system_id": "", "language": "en", "uptime": 0, "version": "0.9.0", "node_id": "0906dca0fbae53e9abe7"}
INFO     Started new MESSAGEPROCESSOR thread ID#thirteen-pluto-don-neptune
INFO     Started new SCHEDULER thread ID#juliet-georgia-eight-social
INFO     Started new WORKERMESSAGEHANDLER thread ID#uranus-zebra-magazine-bulldog
INFO:cherrypy.error:[08/Apr/2018:02:16:40] ENGINE Serving on http://0.0.0.0:8080
INFO:cherrypy.error:[08/Apr/2018:02:16:40] ENGINE Bus STARTED
INFO     Ping succeeded! (response: {}) Sleeping for 1440.0 minutes.
INFO:kolibri.core.analytics.management.commands.ping:Ping succeeded! (response: {}) Sleeping for 1440.0 minutes.
WARNING  Did not find language code None in our le_utils.constants!
WARNING:kolibri.content.api:Did not find language code None in our le_utils.constants!
INFO     Downloading data for channel id 1ceff53605e55bef987d88e0908658c5
INFO:kolibri.content.management.commands.importchannel:Downloading data for channel id 1ceff53605e55bef987d88e0908658c5
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 105951232/105951232 [00:29<00:00, 3535773.07it/s]INFO     Channel 1ceff53605e55bef987d88e0908658c5 already exists in database, cleaning up ContentNodes
INFO:kolibri.content.utils.channel_import:Channel 1ceff53605e55bef987d88e0908658c5 already exists in database, cleaning up ContentNodes
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 105951232/105951232 [00:40<00:00, 3535773.07it/s]INFO     Importing ContentTag data
INFO:kolibri.content.utils.channel_import:Importing ContentTag data
INFO     Importing ContentNode_has_prerequisite data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_has_prerequisite data
INFO     Importing ContentNode_related data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_related data
INFO     Importing ContentNode_tags data
INFO:kolibri.content.utils.channel_import:Importing ContentNode_tags data
INFO     Importing ContentNode data
INFO:kolibri.content.utils.channel_import:Importing ContentNode data
INFO     Importing Language data
INFO:kolibri.content.utils.channel_import:Importing Language data
INFO     Importing File data
INFO:kolibri.content.utils.channel_import:Importing File data
INFO     Importing LocalFile data
INFO:kolibri.content.utils.channel_import:Importing LocalFile data
INFO     Importing AssessmentMetaData data
INFO:kolibri.content.utils.channel_import:Importing AssessmentMetaData data
INFO     Importing ChannelMetadata data
INFO:kolibri.content.utils.channel_import:Importing ChannelMetadata data
INFO     Setting availability of File objects based on LocalFile availability
INFO:kolibri.content.utils.annotation:Setting availability of File objects based on LocalFile availability
INFO     Setting availability of non-topic ContentNode objects based on File availability
INFO:kolibri.content.utils.annotation:Setting availability of non-topic ContentNode objects based on File availability
INFO     Setting availability of ContentNode objects with children for 5 levels
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for 5 levels
INFO     Setting availability of ContentNode objects with children for level 5
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 5
INFO     Setting availability of ContentNode objects with children for level 4
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 4
INFO     Setting availability of ContentNode objects with children for level 3
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 3
INFO     Setting availability of ContentNode objects with children for level 2
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 2
INFO     Setting availability of ContentNode objects with children for level 1
INFO:kolibri.content.utils.annotation:Setting availability of ContentNode objects with children for level 1

```

![image](https://user-images.githubusercontent.com/374612/38461593-d1b41e80-3ad4-11e8-9585-6c06434f01a0.png)

Can you reproduce this? I sense it happens every time I just select the first 4 top-level items of Khan Academy.. math, science etc.. nothing happens after. In the log, nothing is shown, in the UI, it shows "preparing".
Yeah, I've had this. I think I reported it somewhere too. It's not
*literally* forever for me, but I think it took like an hour or something
once too.

It happens for huge content DBs, with lots of content nodes and/or file
nodes. It's either the tree importation or the availability annotation.
Doesn't always happen. And sometimes it's slow via the web ui and fast via
the command-line import commands. Other times it's also slow via command
line, especially when annotating Level 4 or 5.

tl;dr: more investigation needed -- perhaps the perf team can take a peek
too.

On Sat, Apr 7, 2018, 2:32 PM Benjamin Bach <notifications@github.com> wrote:

> Can you reproduce this? I sense it happens every time I just select the
> first 4 top-level items of Khan Academy.. math, science etc.. nothing
> happens after. In the log, nothing is shown, in the UI, it shows
> "preparing".
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/learningequality/kolibri/issues/3497#issuecomment-379509247>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAlvsP3ViOF3X9bXIDDXymICMfygQYNPks5tmVqfgaJpZM4TLTXm>
> .
>

> tl;dr: more investigation needed -- perhaps the perf team can take a peek
> too.

ACK, it goes on the list.
I believe this has been very slow for large channels ever since we introduced availability annotation
> It's either the tree importation or the availability annotation. Doesn't always happen.

IIRC I had most of Sikana already installed when importing KA. After canceling the import, nothing had happened, the database was still intact, so that's a bit of good news :)

As you can see, I had spent almost 10h of CPU time, hence "endless" :) I was starting to sense that it wouldn't complete.
I have the same issue on ubuntu 16.04, UI is stuck on 'prepaing' when trying to import from kolibri studio just the Maths package. Let me know when resolved please.

Resolving patch:
diff --git a/kolibri/content/management/commands/importcontent.py b/kolibri/content/management/commands/importcontent.py
--- a/kolibri/content/management/commands/importcontent.py
+++ b/kolibri/content/management/commands/importcontent.py
@@ -67,7 +67,7 @@ def add_arguments(self, parser):
         parser.add_argument(
             "--include-unrenderable-content",
             action='store_false',
-            default=True,
+            default=False,
             dest="renderable_only",
             help="Import all content, not just that which this Kolibri instance can render"
         )


Test patch: