Repository: learningequality/kolibri

Issue:
sqlite3-wal got too big file size.
### Observed behavior
After I import the Khan Academy(English) channel and browse its downloadable contents the `sqlite3-wal` changed the file size to almost 1gb. This happened without downloading any contents from Khan Academy(English).

### Errors and logs
![screen shot 2018-07-10 at 2 24 37 pm](https://user-images.githubusercontent.com/8663934/42493017-cd59d98e-844d-11e8-9430-df5a847ce10f.png)

### Context

 * Kolibri version: 0.11-0a4
 * Operating system: Windows 10
 * Browser: firefox 61.0



Discussion:
This is quite easy to reproduce, using a clean database (time on the left)
(**Note**: done using Kolibri from develop, in a powerful computer and a nice Internet fiber connection)
19:21 wal file size= 3 Kb
19:21 Manage Content - Import - Khan Akademy
19:23 choose Math
wal file size = 1.3 Gb
Select Early Math - Counting
Select Numbers 0 to 120 => 6 resources (4MB)
19:25 Import 
Finished in less than 10 seconds
wal file size = 1.3 Gb

Then,
19:25 KHan Academy -> Delete the content imported previously
Preparing .....
19:37.....
19:38 Removing channel 0.10%
wal file size =  4.1 Gb
19:40 100% Finished
wal file size = 4.1 Gb


Analyzing the problem, it's clearly a problem in the wal file getting very fragmented. Doing a VACUUM instruction makes wal pass from 4.1 Gb to 0 bytes.

We have two options:
a. Set a pragma with AUTO_VACUUM=Full;
b. Do a DB vacuum when Kolibri starts and after an operation of importing/deleting content from a channel, as this is the operation that produces more fragmentation

Option 'a' has two important problems:
1. It might delay inserts as vacuum needs to move all the data to a temp wal file to unframent.
2. It does not work if the tables are already created nor any connection is done to the db. It will require a special migration for existing databases.

For this reason I am working in option b to solve the problem, unless a better idea is suggested (@rtibbles , @bonidjukic , @jamalex )

Resolving patch:
diff --git a/kolibri/core/deviceadmin/management/commands/vacuumsqlite.py b/kolibri/core/deviceadmin/management/commands/vacuumsqlite.py
new file mode 100644
--- /dev/null
+++ b/kolibri/core/deviceadmin/management/commands/vacuumsqlite.py
@@ -0,0 +1,49 @@
+import logging
+import time
+
+from django.core.management.base import BaseCommand
+from django.db import connections
+from django.db import DEFAULT_DB_ALIAS
+
+logger = logging.getLogger(__name__)
+
+
+class Command(BaseCommand):
+    help = "Vacuum Kolibri's SQLite database to optimize it and reduce the .wal file size"
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--database', action='store', dest='database', default=DEFAULT_DB_ALIAS,
+            help='Specifies the database to vacuum. Defaults to the "default" database.',
+        )
+        parser.add_argument(
+            '--interval', action='store', dest='interval', default=0, type=int,
+            help='Specifies the interval (in minutes) to run the process continuosly. If 0, no repetition will happen',
+        )
+
+    def handle(self, *args, **options):
+        database = options['database']
+        connection = connections[database]
+        interval = options['interval']
+        if connection.vendor == "sqlite":
+            while True:
+                self.perform_vacuum(connection)
+                if not interval:
+                    break
+                logger.info("Next Vacuum in {interval} minutes.".format(interval=interval))
+                time.sleep(interval * 60)
+
+    def perform_vacuum(self, connection):
+        try:
+            cursor = connection.cursor()
+            cursor.execute('vacuum;')
+            connection.close()
+        except Exception as e:
+            new_msg = (
+                "Vacuum of database {db_name} couldn't be executed. Possible reasons:\n"
+                "  * There is an open transaction in the db.\n"
+                "  * There are one or more active SQL statements.\n"
+                "The full error: {error_msg}").format(db_name=connection.settings_dict['NAME'], error_msg=e)
+            logger.error(new_msg)
+        else:
+            logger.info("Sqlite database Vacuum finished.")
diff --git a/kolibri/utils/cli.py b/kolibri/utils/cli.py
--- a/kolibri/utils/cli.py
+++ b/kolibri/utils/cli.py
@@ -646,6 +646,9 @@ def main(args=None):  # noqa: max-complexity=13
         # Check if the content directory exists when Kolibri runs after the first time.
         check_content_directory_exists_and_writable()
 
+        # Defragment the db
+        call_command("vacuumsqlite")
+
         # Clear old sessions up
         call_command("clearsessions")
 
diff --git a/kolibri/utils/server.py b/kolibri/utils/server.py
--- a/kolibri/utils/server.py
+++ b/kolibri/utils/server.py
@@ -65,6 +65,9 @@ def start(port=8080):
     # start the pingback thread
     PingbackThread.start_command()
 
+    # Do a db vacuum periodically
+    VacuumThread.start_command()
+
     # Write the new PID
     with open(PID_FILE, 'w') as f:
         f.write("%d\n%d" % (os.getpid(), port))
@@ -100,6 +103,19 @@ def run(self):
         call_command("ping")
 
 
+class VacuumThread(threading.Thread):
+
+    @classmethod
+    def start_command(cls):
+        thread = cls()
+        thread.daemon = True
+        thread.start()
+
+    def run(self):
+        # Try to do the vacuum every 3 hours
+        call_command("vacuumsqlite", interval=10)
+
+
 def stop(pid=None, force=False):
     """
     Stops the kalite server, either from PID or through a management command


Test patch: