Repository: learningequality/kolibri

Issue:
job_storage DB upgrading unnecessarily - fails to start devserver
## Observed behavior

This has happened to me several times in the last week where I stop my Kolibri dev server with CTRL+C and/or `killall -9 kolibri` and go to restart it and my job_storage DB is borked. When I run `rm $KOLIBRI_HOME/job_storage.sqlite3` and start it up again, it works just fine. 

## Errors and logs

```
Error: Traceback (most recent call last):
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: index queue__scheduled_time already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/utils/cli.py", line 193, in invoke
    initialize(**get_initialize_params())
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/utils/main.py", line 291, in initialize
    update(version, kolibri.__version__)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/utils/main.py", line 349, in update
    run_upgrades(old_version, new_version)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/core/upgrade.py", line 156, in run_upgrades
    version_upgrade()
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/core/upgrade.py", line 51, in __call__
    return self.upgrade()
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/core/tasks/upgrade.py", line 24, in drop_old_iceqube_tables
    job_storage.recreate_tables()
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/kolibri/core/tasks/storage.py", line 113, in recreate_tables
    self.Base.metadata.create_all(self.engine)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4465, in create_all
    ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2104, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1663, in _run_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 144, in traverse_single
    return meth(obj, **kw)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 787, in visit_metadata
    _is_metadata_operation=True,
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 144, in traverse_single
    return meth(obj, **kw)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 839, in visit_table
    self.traverse_single(index)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 144, in traverse_single
    return meth(obj, **kw)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 868, in visit_index
    self.connection.execute(CreateIndex(index))
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1082, in _execute_ddl
    compiled,
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
    cursor, statement, parameters, context
  File "/home/jacob/Code/LE/kolibri/refactor-library-page/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) index queue__scheduled_time already exists
[SQL: CREATE INDEX queue__scheduled_time ON jobs (queue, scheduled_time)]
(Background on this error at: http://sqlalche.me/e/e3q8)
```

## Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

I shouldn't have to delete a DB to start/restart the dev server (this often anyway).

## User-facing consequences

Difficulties / unexpected errors for devs.

## Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

Kill your devserver and start it up again? 

## Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

`develop` branch
Fedora


Discussion:
Interesting - it looks like the upgrade operation that I thought shouldn't affect devs is being run too aggressively. The job storage database is actually not broken, it's just that the upgrade is trying to run without regard to whether it needs to.

Resolving patch:
diff --git a/kolibri/core/tasks/upgrade.py b/kolibri/core/tasks/upgrade.py
--- a/kolibri/core/tasks/upgrade.py
+++ b/kolibri/core/tasks/upgrade.py
@@ -3,6 +3,8 @@
 """
 import logging
 
+from sqlalchemy.exc import OperationalError
+
 from kolibri.core.tasks.main import job_storage
 from kolibri.core.upgrade import version_upgrade
 
@@ -21,4 +23,9 @@ def drop_old_iceqube_tables():
     Rather than write a migration for the iceqube database, it is simpler to just drop the tables
     and let iceqube reinitialize the tables from scratch.
     """
-    job_storage.recreate_tables()
+    try:
+        job_storage.recreate_tables()
+    except OperationalError:
+        logger.warning(
+            "Could not recreate job storage table. This is probably because the database already exists and did not need to be recreated."
+        )


Test patch: