Repository: learningequality/kolibri

Issue:
Kolibri shutdown blocked by running cancellable task
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
Kolibri wouldn't shutdown while a content import task was running. Observe the timestamps around `^C` in the logs below-- above it was also already during shutdown.

## Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->
```
INFO     2023-03-14 13:22:59,426 Keyboard interrupt caught. Exiting.
INFO     2023-03-14 13:22:59,426 Bus state: STOP
INFO     2023-03-14 13:22:59,588 HTTP Server kolibri.utils.server.Server(('0.0.0.0', 8080)) shut down
INFO     2023-03-14 13:23:00,034 HTTP Server kolibri.utils.server.Server(('0.0.0.0', 34365)) shut down
INFO     2023-03-14 13:23:00,034 Joining 'ProcessControlPlugin'
INFO     2023-03-14 13:23:00,404 Stopped thread 'ProcessControlPlugin'.
INFO     2023-03-14 13:23:00,404 Joining 'ZeroConfPlugin'
INFO     2023-03-14 13:23:01,399 Stopped thread 'ZeroConfPlugin'.
INFO     2023-03-14 13:23:01,654 Asking job schedulers to shut down.
INFO     2023-03-14 13:23:01,654 Canceling job id 6ead5d86df944986848a66e1585c6a48.
^C
^[[1;5F
INFO     2023-03-14 13:27:29,520 Setting availability to True of 1474 LocalFile objects based on passed in checksums
INFO     2023-03-14 13:27:29,527 Setting availability of non-topic ContentNode objects based on LocalFile availability in 1 batches of 10000
INFO     2023-03-14 13:27:29,537 Annotating ContentNode objects with children for 3 levels
INFO     2023-03-14 13:27:29,540 Annotating ContentNode objects with children for level 3
INFO     2023-03-14 13:27:29,542 Annotating ContentNode objects with children for level 2
INFO     2023-03-14 13:27:29,542 Annotating ContentNode objects with children for level 1
INFO     2023-03-14 13:27:29,569 Bus state: IDLE
INFO     2023-03-14 13:27:29,569 Bus state: EXIT
INFO     2023-03-14 13:27:29,569 Waiting for child threads to terminate...
INFO     2023-03-14 13:27:29,569 Bus state: EXITED
```

## Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
It seems that this should not have blocked Kolibri shutdown, since the task is cancellable. So for some reason the cancellation of the job, as captured by the logs, didn't actually stop the content import task.

## User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->
In ability to shutdown Kolibri until tasks are complete

## Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->
1) Start a content import that should take a while
2) Attempt to shutdown Kolibri

## Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Ubuntu 22.04
Kolibri develop pex https://github.com/learningequality/kolibri/suites/11465479013/artifacts/591721049
Chrome




Discussion:
@bjester I would like to work on this issue
Hi @photon0205, I have assigned you. First step would be to replicate this error (it looks like @bjester was importing resources for a channel).

The next step would be to look at the worker implementation and see how it handles shutdown to see why his might be happening. 
We run into this all the time if there's a worker downloading content from a task. In the log it reports that the job is being cancelled, but nothing actually happens.

First, the task is received and it starts downloading the items:

```
DEBUG 2023-08-17 07:12:10,950 cherrypy.access 127.0.0.1 - - "POST /api/tasks/tasks/" 200 0 "" "curl/7.74.0"
DEBUG 2023-08-17 07:12:10,972 urllib3.connectionpool Starting new HTTPS connection (1): studio.learningequality.org:443
DEBUG 2023-08-17 07:12:11,225 urllib3.connectionpool https://studio.learningequality.org:443 "GET /api/public/v1/channels/lookup/2f95235c3709511fa12d007f31ed6
a7b HTTP/1.1" 200 None
DEBUG 2023-08-17 07:12:11,307 urllib3.connectionpool Starting new HTTPS connection (1): studio.learningequality.org:443
DEBUG 2023-08-17 07:12:11,465 urllib3.connectionpool https://studio.learningequality.org:443 "HEAD /content/storage/e/6/e610bd5a54c93a5550a77a8828a915d5.png H
TTP/1.1" 200 0
DEBUG 2023-08-17 07:12:11,556 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/e/6/e610bd5a54c93a5550a77a8828a915d5.png HT
TP/1.1" 206 133718
```

Then I interrupt the server to stop it and it says it's canceling the task:

```
INFO 2023-08-17 07:12:14,033 kolibri.utils.server Caught signal SIGINT.
DEBUG 2023-08-17 07:12:14,034 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/1/1/11acc0389000ac01255a56088a9bf73e.png HTTP/1.1" 206 128702
INFO 2023-08-17 07:12:14,035 kolibri.utils.server Keyboard interrupt caught. Exiting.
INFO 2023-08-17 07:12:14,037 kolibri.utils.server Bus state: STOP
INFO 2023-08-17 07:12:14,116 kolibri.utils.server HTTP Server kolibri.utils.server.Server(('0.0.0.0', 39877)) shut down
DEBUG 2023-08-17 07:12:14,364 urllib3.connectionpool https://studio.learningequality.org:443 "HEAD /content/storage/2/4/2460a226770606206260106e22e743ba.png HTTP/1.1" 200 0
DEBUG 2023-08-17 07:12:14,443 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/2/4/2460a226770606206260106e22e743ba.png HTTP/1.1" 206 138061
INFO 2023-08-17 07:12:14,479 kolibri.utils.server HTTP Server kolibri.utils.server.Server(('0.0.0.0', 8080)) shut down
INFO 2023-08-17 07:12:14,480 kolibri.utils.server Joining 'ZeroConfPlugin'
DEBUG 2023-08-17 07:12:14,758 urllib3.connectionpool https://studio.learningequality.org:443 "HEAD /content/storage/f/1/f12a24c01ad4feebc55f1c915a2cdc6f.png HTTP/1.1" 200 0
DEBUG 2023-08-17 07:12:14,844 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/f/1/f12a24c01ad4feebc55f1c915a2cdc6f.png HTTP/1.1" 206 66215
INFO 2023-08-17 07:12:14,998 kolibri.utils.server Stopped thread 'ZeroConfPlugin'.
DEBUG 2023-08-17 07:12:15,017 urllib3.connectionpool https://studio.learningequality.org:443 "HEAD /content/storage/7/2/7291ae066b1a2086cfd3528cfe94c51c.png HTTP/1.1" 200 0
DEBUG 2023-08-17 07:12:15,228 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/7/2/7291ae066b1a2086cfd3528cfe94c51c.png HTTP/1.1" 206 35127
INFO 2023-08-17 07:12:15,265 kolibri.core.tasks.worker Asking job schedulers to shut down.
INFO 2023-08-17 07:12:15,266 kolibri.core.tasks.worker Canceling job id a8531ca3de91493e84b4cc221f537790.
DEBUG 2023-08-17 07:12:15,373 kolibri JOBCHECKER shut down event received; closing.
```

But the downloads keep running. Finally when they're done the server shuts down:

```
DEBUG 2023-08-17 07:15:37,625 urllib3.connectionpool https://studio.learningequality.org:443 "GET /content/storage/d/d/dde21e80075dce78d0e4028519c197f6.mp4 HTTP/1.1" 206 50140116
INFO 2023-08-17 07:15:41,723 kolibri.core.content.utils.annotation Setting availability to True of 202 LocalFile objects based on passed in checksums
INFO 2023-08-17 07:15:41,731 kolibri.core.content.utils.annotation Setting availability of non-topic ContentNode objects based on LocalFile availability in 1 batches of 10000
INFO 2023-08-17 07:15:41,738 kolibri.core.content.utils.annotation Annotating ContentNode objects with children for 2 levels
INFO 2023-08-17 07:15:41,741 kolibri.core.content.utils.annotation Annotating ContentNode objects with children for level 2
INFO 2023-08-17 07:15:41,743 kolibri.core.content.utils.annotation Annotating ContentNode objects with children for level 1
DEBUG 2023-08-17 07:15:41,746 kolibri.core.content.utils.annotation Recursive topic tree annotation took 0 seconds
INFO 2023-08-17 07:15:41,789 kolibri.utils.server Joining 'ProcessControlPlugin'
INFO 2023-08-17 07:15:42,254 kolibri.utils.server Stopped thread 'ProcessControlPlugin'.
INFO 2023-08-17 07:15:42,254 kolibri.utils.server Bus state: IDLE
INFO 2023-08-17 07:15:42,254 kolibri.utils.server Bus state: EXIT
INFO 2023-08-17 07:15:42,264 kolibri.utils.server Waiting for child threads to terminate...
INFO 2023-08-17 07:15:42,265 kolibri.utils.server Bus state: EXITED
```
@rtibbles what do you think is going on here? Does the `Future` used for content imports not support cancelling?
Oh, `Future`s just can't be cancelled once they're running - https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.cancel. I confirmed with the following patch:

```
diff --git a/kolibri/core/tasks/worker.py b/kolibri/core/tasks/worker.py
index 9861a441c4..a0f1164a09 100644
--- a/kolibri/core/tasks/worker.py
+++ b/kolibri/core/tasks/worker.py
@@ -72,7 +72,8 @@ class Worker(object):
         job_ids = list(self.future_job_mapping.keys())
         for job_id in job_ids:
             logger.info("Canceling job id {}.".format(job_id))
-            self.cancel(job_id)
+            if not self.cancel(job_id):
+                logger.warning("Job id {} could not be cancelled".format(job_id))
         # Now shutdown the workers
         self.workers.shutdown(wait=wait)
 
```

Which resulted in:

```
WARNING  2023-08-17 07:27:39,566 Job id 100b46ee3a0845ef99ff9d1102269dd9 could not be cancelled
```

This [stack overflow post](https://stackoverflow.com/questions/42782953/python-concurrent-futures-how-to-make-it-cancelable) has some more details about it. Presumably this is because you can't cancel a thread. You could workaround it by using a process pool, but you'd have to have a custom executor to kill the running processes.

I guess to do this generically in Kolibri there would have to be a separate mechanism to signal a running task that it needs to stop. Possibly how you'd do it is that you pass an [`Event`](https://docs.python.org/3/library/threading.html#event-objects) into the task function which it could monitor. When cancelling tasks, you'd signal the event if the task was running and block until the future completed.

Resolving patch:
diff --git a/kolibri/core/discovery/utils/network/broadcast.py b/kolibri/core/discovery/utils/network/broadcast.py
--- a/kolibri/core/discovery/utils/network/broadcast.py
+++ b/kolibri/core/discovery/utils/network/broadcast.py
@@ -370,9 +370,13 @@ def start_broadcast(self):
         self.register()
 
         # manually add our service browser to Zeroconf so it's automatically cleaned up on close
-        self.zeroconf.browsers["bus"] = ServiceBrowser(
-            self.zeroconf, SERVICE_TYPE, handlers=[self.events.publish_zeroconf_change]
-        )
+        # Check that the zeroconf attribute has not been set to None by a previous call to stop_broadcast
+        if self.zeroconf is not None:
+            self.zeroconf.browsers["bus"] = ServiceBrowser(
+                self.zeroconf,
+                SERVICE_TYPE,
+                handlers=[self.events.publish_zeroconf_change],
+            )
 
     def update_broadcast(self, instance=None, interfaces=None):
         """
@@ -504,7 +508,8 @@ def unregister(self):
 
         # very important to publish the event first, to avoid race conditions
         self.events.publish(EVENT_UNREGISTER_INSTANCE, self.instance)
-        self.zeroconf.unregister_service(self.instance.service_info)
+        if self.instance.service_info is not None:
+            self.zeroconf.unregister_service(self.instance.service_info)
         self.instance.reset_broadcasting()
 
     def add_listener(self, listener_cls):
diff --git a/kolibri/core/tasks/storage.py b/kolibri/core/tasks/storage.py
--- a/kolibri/core/tasks/storage.py
+++ b/kolibri/core/tasks/storage.py
@@ -23,6 +23,7 @@
 from kolibri.core.tasks.exceptions import JobNotRestartable
 from kolibri.core.tasks.exceptions import JobNotRunning
 from kolibri.core.tasks.exceptions import JobRunning
+from kolibri.core.tasks.exceptions import UserCancelledError
 from kolibri.core.tasks.hooks import StorageHook
 from kolibri.core.tasks.job import Job
 from kolibri.core.tasks.job import Priority
@@ -493,6 +494,8 @@ def retry_job_in(self, job_id, delta_t, **kwargs):
         if not isinstance(delta_t, timedelta):
             raise TypeError("Time argument must be a timedelta object.")
         orm_job = self.get_orm_job(job_id)
+        if orm_job.state == State.CANCELING:
+            raise UserCancelledError("Job has been marked for cancellation.")
         if orm_job.state != State.RUNNING:
             raise JobNotRunning("Job is not running, cannot retry.")
         if orm_job.retry_interval is not None or orm_job.repeat != 0:
diff --git a/kolibri/core/tasks/worker.py b/kolibri/core/tasks/worker.py
--- a/kolibri/core/tasks/worker.py
+++ b/kolibri/core/tasks/worker.py
@@ -69,10 +69,11 @@ def shutdown_workers(self, wait=True):
         # First cancel all running jobs
         # Coerce to a list, as otherwise the iterable can change size
         # during iteration, as jobs are cancelled and removed from the mapping
-        job_ids = list(self.future_job_mapping.keys())
-        for job_id in job_ids:
-            logger.info("Canceling job id {}.".format(job_id))
-            self.cancel(job_id)
+        for job in self.storage.get_running_jobs():
+            logger.info("Canceling job id {}.".format(job.job_id))
+            self.storage.mark_job_as_canceling(job.job_id)
+            if self.cancel(job.job_id):
+                self.storage.mark_job_as_canceled(job.job_id)
         # Now shutdown the workers
         self.workers.shutdown(wait=wait)
 
@@ -96,9 +97,11 @@ def handle_finished_future(self, future):
     def shutdown(self, wait=True):
         logger.info("Asking job schedulers to shut down.")
         self.job_checker.stop()
-        self.shutdown_workers(wait=wait)
+        # Wait for the job checker to finish
+        # before attempting to pause any running jobs
         if wait:
             self.job_checker.join()
+        self.shutdown_workers(wait=wait)
 
     def start_job_checker(self):
         """


Test patch: