Repository: learningequality/kolibri

Issue:
"Restarting" a Task normally means deleting it and triggering a new one with same parameters -> UI consequences
### Observed behavior

I don't believe there's a way to simply restart a Task once it's in the CANCELED/FAILED status. So to implement the "retry" feature for failed facility syncs/imports, we actually delete the original one and start a new one with data that is tacked on to the task metadata.

Having to do things this way has a couple of consequences for the UI code and actual UX:

1. "Restarting" a task with our standard "poll for tasks" framework means 1) delete it, 2) see it disappear, 3) start a new one, 4) see an identical-looking task reappear
1. The API calls to restart a task require the same full payload, as opposed to a more minimal one with just the TaskID. This is why we do not allow restarting Facility Import tasks, because this would require us tacking on plaintext credentials onto metadata to use the same approach

### Expected behavior

If it doesn't already exist, it would be nice to have a simpler interface to restart a Task and reduce the process from 2 steps (delete + create) to just 1 (restart).

And concretely, this would mean that the Facility Import/Sync Task Manager UIs in the Facility Page and Setup Wizard wouldn't have the disappear-then-reappear behavior it does now.

### User-facing consequences

### Errors and logs

### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

…

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

…



Discussion:
Hi!
I was looking around to understand the TaskAPI. From what I understand the TaskAPI has a [BaseViewSet](https://github.com/learningequality/kolibri/blob/release-v0.14.x/kolibri/core/tasks/api.py#L180) that handles the creation/deletion of tasks.

A way to implement the RestartTask functionality from the backend would be to add an endpoint to clear the CANCELED/FAILED status marked task and then enqueuing it again.

Would like your inputs and pick this up!
> A way to implement the RestartTask functionality from the backend would be to add an endpoint to clear the CANCELED/FAILED status marked task and then enqueuing it again.

I think that's basically the interface I would like. Where you can POST to something like

```
POST /api/task/restarttask/<task_id>
```

And the original job with be enqueued with the same exact parameters, _but the parameters do not need to be sent as part of the POST request_. That is, all the client needs to know to restart a task/job is the task ID.

I'm not sure if this is possible, by simply modifying the API layer. You might need to see if there is a way to restart jobs within the queue library we're using: "iceqube".

https://github.com/learningequality/iceqube 

Here is the interface for queues

https://github.com/learningequality/iceqube/blob/master/src/iceqube/queue.py


A few questions. I cannot find exactly how [Iceqube](https://github.com/learningequality/iceqube) is being used in Kolibri?
Is it just a framework whose code is copied into the [Task Monolithic](https://github.com/learningequality/kolibri/tree/release-v0.14.x/kolibri/core/tasks)?

Cause I don't see it being added as pip requirement.

I'll have a look and understand the system better. I have created a simple explanatory design document for the TaskAPI (iceqube) which can be found here - [Dropbox Paper](https://paper.dropbox.com/doc/Kolibri-TaskAPI-Docs-WIP--BHO5slA8cTYMvnmnSD6mneGiAQ-y4TJykzXIwrUhwWo4rQSP)
From what I have understood, [Queue](https://github.com/learningequality/kolibri/blob/release-v0.14.x/kolibri/core/tasks/queue.py) is the interface for the [TaskAPI](https://github.com/learningequality/kolibri/blob/release-v0.14.x/kolibri/core/tasks/api.py) to enqueue/schedule tasks.

To implement restart, a possible solution is defining a function in `Queue`as `restart_job(self, job_id)`

`restart_job(self, job_id)` would use `Storage.get_job(job_id)` to get the Job object that was saved by `Storage` after pickling. 

Then the `restart` function can delete the existing record using `Storage.clear(job_id)` if `Job.State in [State.FAILED, State.CANCELED]`

Then re queue the Job using `enqueue(func, *args, **kwargs):` Since we will be submitting the same Job Object that was used last time it should contain the same parameters. 
@ashmeet13 - yes, that could definitely work, we could also just take the existing job object, clear any exceptions or results from it and then resave it.

However, I think your solution is cleaner, and means that we don't have to keep a running list of all things we need to clear, we just need to recreate the desired `enqueue` parameters. The one other thing that we should do is make a copy of the Job by passing the existing Job to the Job constructor: `new_job = Job(old_job)`

This should clear any extraneous information (see here: https://github.com/learningequality/kolibri/blob/release-v0.14.x/kolibri/core/tasks/job.py#L74), like any previous raised exceptions.

Please go ahead with your solution, with the additional proviso that I mention above ^
Got it. Thank you for the help!

Resolving patch:
diff --git a/kolibri/core/tasks/api.py b/kolibri/core/tasks/api.py
--- a/kolibri/core/tasks/api.py
+++ b/kolibri/core/tasks/api.py
@@ -54,6 +54,7 @@
 from kolibri.core.discovery.utils.network.errors import URLParseError
 from kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES
 from kolibri.core.tasks.exceptions import JobNotFound
+from kolibri.core.tasks.exceptions import JobNotRestartable
 from kolibri.core.tasks.exceptions import UserCancelledError
 from kolibri.core.tasks.job import State
 from kolibri.core.tasks.main import facility_queue
@@ -221,6 +222,30 @@ def retrieve(self, request, pk=None):
 
         return Response(task)
 
+    @decorators.action(methods=["post"], detail=False)
+    def restarttask(self, request):
+        """
+        Restart a task with its task id given in the task_id parameter.
+        """
+
+        if "task_id" not in request.data:
+            raise serializers.ValidationError("The 'task_id' field is required.")
+        if not isinstance(request.data["task_id"], string_types):
+            raise serializers.ValidationError("The 'task_id' should be a string.")
+
+        resp = {}
+        for _queue in self.queues:
+            try:
+                task_id = _queue.restart_job(request.data["task_id"])
+                resp = _job_to_response(_queue.fetch_job(task_id))
+                break
+            except JobNotFound:
+                continue
+            except JobNotRestartable as e:
+                raise serializers.ValidationError(str(e))
+
+        return Response(resp)
+
     def destroy(self, request, pk=None):
         # unimplemented for now.
         pass
@@ -284,7 +309,9 @@ def deletefinishedtasks(self, request):
 
 
 class TasksViewSet(BaseViewSet):
-    queues = [queue, priority_queue]
+    @property
+    def queues(self):
+        return [queue, priority_queue]
 
     def default_permission_classes(self):
         # exclusive permission for facility management
diff --git a/kolibri/core/tasks/exceptions.py b/kolibri/core/tasks/exceptions.py
--- a/kolibri/core/tasks/exceptions.py
+++ b/kolibri/core/tasks/exceptions.py
@@ -11,3 +11,7 @@ class UserCancelledError(CancelledError):
 
 class JobNotFound(Exception):
     pass
+
+
+class JobNotRestartable(Exception):
+    pass
diff --git a/kolibri/core/tasks/queue.py b/kolibri/core/tasks/queue.py
--- a/kolibri/core/tasks/queue.py
+++ b/kolibri/core/tasks/queue.py
@@ -1,3 +1,4 @@
+from kolibri.core.tasks.exceptions import JobNotRestartable
 from kolibri.core.tasks.job import Job
 from kolibri.core.tasks.job import State
 from kolibri.core.tasks.storage import Storage
@@ -66,6 +67,23 @@ def update_progress(progress, total_progress, stage=""):
         job_id = self.storage.enqueue_job(job, self.name)
         return job_id
 
+    def restart_job(self, job_id):
+        """
+        Given a job_id, restart the job for that id. A job will only be restarted if
+        in CANCELED or FAILED state.
+
+        This will create a new job with a new job_id.
+        """
+        old_job = self.fetch_job(job_id)
+        if old_job.state in [State.CANCELED, State.FAILED]:
+            self.clear_job(job_id)
+            job = Job(old_job)
+            return self.enqueue(job)
+        else:
+            raise JobNotRestartable(
+                "Cannot restart job with state={}".format(old_job.state)
+            )
+
     def cancel(self, job_id):
         """
         Mark a job as canceling, and let the worker pick this up to initiate


Test patch:
diff --git a/kolibri/core/tasks/test/test_api.py b/kolibri/core/tasks/test/test_api.py
--- a/kolibri/core/tasks/test/test_api.py
+++ b/kolibri/core/tasks/test/test_api.py
@@ -118,6 +118,26 @@ def assert_clearable(index, expected):
             for i in [4, 5, 6]:
                 assert_clearable(i, True)
 
+    def test_restart_task(self, queue_mock):
+        queue_mock.restart_job.return_value = 2
+        queue_mock.fetch_job.return_value = fake_job(state=State.QUEUED, job_id=2)
+
+        response = self.client.post(
+            reverse("kolibri:core:task-restarttask"), {"task_id": "1"}, format="json"
+        )
+
+        expected_response = {
+            "status": "QUEUED",
+            "exception": "",
+            "traceback": "",
+            "percentage": 0,
+            "id": 2,
+            "cancellable": False,
+            "clearable": False,
+        }
+
+        self.assertDictEqual(response.data, expected_response)
+
 
 class TaskAPIPermissionsTestCase(APITestCase):
     def setUp(self):
diff --git a/kolibri/core/tasks/test/test_job_running.py b/kolibri/core/tasks/test/test_job_running.py
--- a/kolibri/core/tasks/test/test_job_running.py
+++ b/kolibri/core/tasks/test/test_job_running.py
@@ -7,6 +7,7 @@
 from sqlalchemy.pool import NullPool
 
 from kolibri.core.tasks.compat import Event
+from kolibri.core.tasks.exceptions import JobNotRestartable
 from kolibri.core.tasks.job import Job
 from kolibri.core.tasks.job import State
 from kolibri.core.tasks.queue import Queue
@@ -179,6 +180,19 @@ def update_progress_cancelable_job():
             return
 
 
+def wait_for_state_change(inmem_queue, job_id, state):
+    interval = 0.1
+    time_spent = 0
+
+    job = inmem_queue.fetch_job(job_id)
+
+    while job.state != state:
+        time.sleep(interval)
+        time_spent += interval
+        job = inmem_queue.fetch_job(job_id)
+        assert time_spent < 5
+
+
 class TestQueue(object):
     def test_enqueues_a_function(self, inmem_queue):
         job_id = inmem_queue.enqueue(id, 1)
@@ -301,3 +315,92 @@ def test_can_cancel_a_job_that_updates_progress(self, inmem_queue):
             assert time_spent < 5
         # and hopefully it's canceled by this point
         assert job.state == State.CANCELED
+
+    def test_failed_job_can_restart(self, inmem_queue):
+        # Start a failng function and check for failure and
+        # the case of it being present in jobs as a failed state job
+        old_job_id = inmem_queue.enqueue(failing_func)
+
+        wait_for_state_change(inmem_queue, old_job_id, State.FAILED)
+
+        job = inmem_queue.fetch_job(old_job_id)
+        assert len(inmem_queue.jobs) == 1
+        assert job.state == State.FAILED
+
+        # Restart the function and assert the same case as above along with
+        # the new created job should have an another unique id.
+        new_job_id = inmem_queue.restart_job(old_job_id)
+
+        assert new_job_id != old_job_id
+
+        wait_for_state_change(inmem_queue, new_job_id, State.FAILED)
+
+        job = inmem_queue.fetch_job(new_job_id)
+
+        assert len(inmem_queue.jobs) == 1
+        assert job.state == State.FAILED
+
+    def test_cancelled_job_can_restart(self, inmem_queue):
+        # Start a function waiting to be cancelled. Once cancelled check
+        # the case of it being present in jobs as a cancelled state job
+        old_job_id = inmem_queue.enqueue(cancelable_job, cancellable=True)
+
+        # The job should go from Queued to Running. At that point we mark it for
+        # cancellation. Then the state should go from Cancelling to Cancelled.
+        wait_for_state_change(inmem_queue, old_job_id, State.RUNNING)
+
+        inmem_queue.cancel(old_job_id)
+        job = inmem_queue.fetch_job(old_job_id)
+        assert job.state == State.CANCELING
+
+        wait_for_state_change(inmem_queue, old_job_id, State.CANCELED)
+        job = inmem_queue.fetch_job(old_job_id)
+        assert job.state == State.CANCELED
+
+        # Restart the job, check that the new created job should have a
+        # unique id.
+        new_job_id = inmem_queue.restart_job(old_job_id)
+        assert new_job_id != old_job_id
+
+        # Test the remaing cases same as above.
+        wait_for_state_change(inmem_queue, new_job_id, State.RUNNING)
+
+        inmem_queue.cancel(new_job_id)
+        job = inmem_queue.fetch_job(new_job_id)
+        assert job.state == State.CANCELING
+
+        wait_for_state_change(inmem_queue, new_job_id, State.CANCELED)
+        job = inmem_queue.fetch_job(new_job_id)
+        assert job.state == State.CANCELED
+
+    def test_queued_job_cannot_restart(self, inmem_queue):
+        # Start a failing function task, ut do not wait for it to change it's
+        # state from QUEUED. Initating an immediate restart raises the
+        # JobNotRestartable Exception
+        old_job_id = inmem_queue.enqueue(failing_func)
+
+        job = inmem_queue.fetch_job(old_job_id)
+        assert job.state == State.QUEUED
+
+        with pytest.raises(JobNotRestartable) as excinfo:
+            inmem_queue.restart_job(old_job_id)
+
+        assert str(excinfo.value) == "Cannot restart job with state=QUEUED"
+
+    def test_successful_job_cannot_restart(self, inmem_queue, flag):
+        # Start a function and wait for it to be completed. Initiate a restart
+        # request for the task which should raise a JobNotRestartable Exception
+        old_job_id = inmem_queue.enqueue(set_flag, flag)
+
+        flag.wait(timeout=5)
+        assert flag.is_set()
+
+        time.sleep(0.5)
+
+        job = inmem_queue.fetch_job(old_job_id)
+        assert job.state == State.COMPLETED
+
+        with pytest.raises(JobNotRestartable) as excinfo:
+            inmem_queue.restart_job(old_job_id)
+
+        assert str(excinfo.value) == "Cannot restart job with state=COMPLETED"