Repository: learningequality/kolibri

Issue:
Upgrading a channel using 'importchannel' command fails with sqlalchemy errors
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior

I get errors trying to update a channel metadata database using the `importchannel` command, regardless of whether this is via the command line or through the REST API.

Traceback comes from SQLalchemy module:

```
packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite
```

@rtibbles You hypothesized that this might be due to incomplete cleanup on the original transfer of the metadata DB.


### Expected behavior

No errors when running the command.

### User-facing consequences

The channel upgrade workflow does not work (UI still WIP). The Task will start, but fail immediately and return a 500 error on the client.

### Errors and logs

Main error after upgrade attempt:

```
INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importchannel network 8169e866a541498e88e8eb724c878c01
INFO     Downloading data for channel id 8169e866a541498e88e8eb724c878c01
INFO     Older version 3 of channel 8169e866a541498e88e8eb724c878c01 already exists in database; removing old entries so we can upgrade to version 4
ERROR    An error occurred during channel import: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite
```

Full terminal output in the 'details'

<details>
INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importchannel network 8169e866a541498e88e8eb724c878c01
INFO     Downloading data for channel id 8169e866a541498e88e8eb724c878c01
INFO     Older version 3 of channel 8169e866a541498e88e8eb724c878c01 already exists in database; removing old entries so we can upgrade to version 4
ERROR    An error occurred during channel import: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri', 'console_scripts', 'kolibri')()
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jon/Github/kolibri/kolibri/utils/cli.py", line 237, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jon/Github/kolibri/kolibri/utils/cli.py", line 642, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/jon/Github/kolibri/kolibri/core/tasks/management/commands/base.py", line 96, in handle
    return self.handle_async(*args, **options)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 185, in handle_async
    options["channel_id"], options["baseurl"], options["upgrade"]
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 92, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl, upgrade=upgrade)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 126, in _transfer
    filetransfer, channel_id, dest, upgrade=upgrade
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 171, in _start_file_transfer
    retry_import(e, skip_404=False)
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/import_export_content.py", line 159, in retry_import
    raise e
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 163, in _start_file_transfer
    import_channel_by_id(channel_id, self.is_cancelled)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 25, in import_channel_by_id
    channel_id, cancel_check=cancel_check
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 799, in import_channel_from_local_db
    import_manager.import_channel_data()
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 622, in import_channel_data
    raise e
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 603, in import_channel_data
    if self.check_and_delete_existing_channel():
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 483, in check_and_delete_existing_channel
    self.delete_old_channel_data(root_node.tree_id)
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 561, in delete_old_channel_data
    self.destination.get_connection().execute(text(query))
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
</details>


### Steps to reproduce

1. Import a channel using `kolibri manage importchannel network <channel token>`
1. Upgrade and publish the channel on Kolibri Studio
1. Try to import the channel again -> get error.

To try this out with a UI, you can clone my fork here

https://github.com/jonboiser/kolibri/tree/diff-stats-ui

1. Go to device > channels
1. Click options dropdown > manage
1. If a new version is on studio, you should see a "new version banner", follow the link.
1. Click "update channel" and confirm modal on the new channel version changelog page.

### Context

Kolibri 0.13 beta 3

Upgrading a channel using 'importchannel' command fails with sqlalchemy errors
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior

I get errors trying to update a channel metadata database using the `importchannel` command, regardless of whether this is via the command line or through the REST API.

Traceback comes from SQLalchemy module:

```
packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite
```

@rtibbles You hypothesized that this might be due to incomplete cleanup on the original transfer of the metadata DB.


### Expected behavior

No errors when running the command.

### User-facing consequences

The channel upgrade workflow does not work (UI still WIP). The Task will start, but fail immediately and return a 500 error on the client.

### Errors and logs

Main error after upgrade attempt:

```
INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importchannel network 8169e866a541498e88e8eb724c878c01
INFO     Downloading data for channel id 8169e866a541498e88e8eb724c878c01
INFO     Older version 3 of channel 8169e866a541498e88e8eb724c878c01 already exists in database; removing old entries so we can upgrade to version 4
ERROR    An error occurred during channel import: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite
```

Full terminal output in the 'details'

<details>
INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importchannel network 8169e866a541498e88e8eb724c878c01
INFO     Downloading data for channel id 8169e866a541498e88e8eb724c878c01
INFO     Older version 3 of channel 8169e866a541498e88e8eb724c878c01 already exists in database; removing old entries so we can upgrade to version 4
ERROR    An error occurred during channel import: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: sourcedb.content_contentnode_has_prerequisite

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri', 'console_scripts', 'kolibri')()
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jon/Github/kolibri/kolibri/utils/cli.py", line 237, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jon/Github/kolibri/kolibri/utils/cli.py", line 642, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/jon/Github/kolibri/kolibri/core/tasks/management/commands/base.py", line 96, in handle
    return self.handle_async(*args, **options)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 185, in handle_async
    options["channel_id"], options["baseurl"], options["upgrade"]
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 92, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl, upgrade=upgrade)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 126, in _transfer
    filetransfer, channel_id, dest, upgrade=upgrade
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 171, in _start_file_transfer
    retry_import(e, skip_404=False)
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/import_export_content.py", line 159, in retry_import
    raise e
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 163, in _start_file_transfer
    import_channel_by_id(channel_id, self.is_cancelled)
  File "/Users/jon/Github/kolibri/kolibri/core/content/management/commands/importchannel.py", line 25, in import_channel_by_id
    channel_id, cancel_check=cancel_check
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 799, in import_channel_from_local_db
    import_manager.import_channel_data()
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 622, in import_channel_data
    raise e
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 603, in import_channel_data
    if self.check_and_delete_existing_channel():
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 483, in check_and_delete_existing_channel
    self.delete_old_channel_data(root_node.tree_id)
  File "/Users/jon/Github/kolibri/kolibri/core/content/utils/channel_import.py", line 561, in delete_old_channel_data
    self.destination.get_connection().execute(text(query))
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/jon/.local/share/virtualenvs/kolibri-UH8N_Uf0/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: sourcedb.content_contentnode_has_prerequisite [SQL: "\n            DELETE FROM content_contentnode_has_prerequisite\n                WHERE from_contentnode_id IN (\n                    SELECT id FROM content_contentnode WHERE tree_id = '1'\n                )\n         AND NOT id IN (SELECT id FROM sourcedb.content_contentnode_has_prerequisite)"] (Background on this error at: http://sqlalche.me/e/e3q8)
</details>


### Steps to reproduce

1. Import a channel using `kolibri manage importchannel network <channel token>`
1. Upgrade and publish the channel on Kolibri Studio
1. Try to import the channel again -> get error.

To try this out with a UI, you can clone my fork here

https://github.com/jonboiser/kolibri/tree/diff-stats-ui

1. Go to device > channels
1. Click options dropdown > manage
1. If a new version is on studio, you should see a "new version banner", follow the link.
1. Click "update channel" and confirm modal on the new channel version changelog page.

### Context

Kolibri 0.13 beta 3



Discussion:

Resolving patch:
diff --git a/kolibri/core/content/utils/channel_import.py b/kolibri/core/content/utils/channel_import.py
--- a/kolibri/core/content/utils/channel_import.py
+++ b/kolibri/core/content/utils/channel_import.py
@@ -558,7 +558,7 @@ def delete_old_channel_data(self, old_tree_id):
                 self.check_cancelled()
 
                 # execute the actual query
-                self.destination.get_connection().execute(text(query))
+                self.destination.session.execute(text(query))
 
     def check_cancelled(self):
         if callable(self.cancel_check):


Test patch: