Repository: pulp/pulpcore

Issue:
Tasks API should report a `finished_at` date/time for cancelled tasks
Cancelled tasks should have `finished_at` set at cancel time.
Ideally any attempt to set `finished_at` from something different than `None` in the database should be reported as a `RuntimeError`. (audit of all `Task.save()` and `Task.objects.update()` calls)

Original wording: Tasks API should return a `cancelled_at` date/time for tasks that were cancelled. The `finished_at` should be set to `null` when `cancelled_at` is not `null`.


Discussion:
We have tasks in multiple final states - completed, failed, cancelled, skipped. Regardless on what state it is getting in the `finished_at` gets populated.
While your request makes sense, to make everything consistent we'd need to introduce besides `cancelled_at` also `failed_at` and `skipped_at`.

My preference would be keep things as they are  so we don't need to manage fields in addition. The combination of `state` and `finished_at` fields makes it pretty self explanatory of what happened to the task.
So i read that canceling would set the `finished_at` field as well.
I agree with @ipanova, I'm not sure I've seen justification for adding a new field just yet.  Is there a gap with `state: canceled, finished_at: $date` doesn't cover?
The thing is that we seem to have canceled tasks with `finished_at: null` and `finished_at: <date>`. That should probably not be the case. `started_at` should then tell us whether the task was aborted or canceled before being started.
> The thing is that we seem to have canceled tasks with `finished_at: null` and `finished_at: <date>`. That should probably not be the case. `started_at` should then tell us whether the task was aborted or canceled before being started.

I was not clear on this issue from the original report. It seems like we always set `finished_at` for canceled tasks https://github.com/pulp/pulpcore/blob/main/pulpcore/tasking/pulpcore_worker.py#L206 am i looking into wrong place?

@mdellweg For the sake of readability and preserving history - can you avoid editing original messages and rather add a comment?
>The thing is that we seem to have canceled tasks with finished_at: null and finished_at: <date>. That should probably not be the case. started_at should then tell us whether the task was aborted or canceled before being started.

Definitely agree that should not be the case, but that's a bug that can be fixed, I'm not sure I understand why a new field is required here.
> > The thing is that we seem to have canceled tasks with `finished_at: null` and `finished_at: <date>`. That should probably not be the case. `started_at` should then tell us whether the task was aborted or canceled before being started.
> 
> I was not clear on this issue from the original report. It seems like we always set `finished_at` for canceled tasks https://github.com/pulp/pulpcore/blob/main/pulpcore/tasking/pulpcore_worker.py#L206 am i looking into wrong place?
> 
> @mdellweg For the sake of readability and preserving history - can you avoid editing original messages and rather add a comment?

I recall that we edited the plan of an issue after the discussion took it into a different direction in the first message back on plan.io. And it made sense to me. So you didn't get confused reading the original request and then need to extract what actually to do from a lengthy discussion. For histroy purpose i kept the original text. Sorry if i confused you.

Resolving patch:
diff --git a/pulpcore/app/models/task.py b/pulpcore/app/models/task.py
--- a/pulpcore/app/models/task.py
+++ b/pulpcore/app/models/task.py
@@ -4,6 +4,7 @@
 import logging
 import traceback
 import os
+from contextlib import suppress
 from datetime import timedelta
 from gettext import gettext as _
 
@@ -18,7 +19,7 @@
     BaseModel,
     GenericRelationModel,
 )
-from pulpcore.constants import TASK_CHOICES, TASK_FINAL_STATES, TASK_STATES
+from pulpcore.constants import TASK_CHOICES, TASK_INCOMPLETE_STATES, TASK_STATES
 from pulpcore.exceptions import AdvisoryLockError, exception_to_dict
 from pulpcore.tasking.constants import TASKING_CONSTANTS
 
@@ -239,8 +240,13 @@ def set_running(self):
             state=TASK_STATES.RUNNING, started_at=timezone.now()
         )
         if rows != 1:
-            _logger.warning(_("Task __call__() occurred but Task %s is not at WAITING") % self.pk)
-        self.refresh_from_db()
+            raise RuntimeError(
+                _("Task set_running() occurred but Task {} is not WAITING").format(self.pk)
+            )
+        with suppress(AttributeError):
+            del self.state
+        with suppress(AttributeError):
+            del self.started_at
 
     def set_completed(self):
         """
@@ -248,18 +254,19 @@ def set_completed(self):
 
         This updates the :attr:`finished_at` and sets the :attr:`state` to :attr:`COMPLETED`.
         """
-        # Only set the state to finished if it's not already in a complete state. This is
-        # important for when the task has been canceled, so we don't move the task from canceled
-        # to finished.
-        rows = (
-            Task.objects.filter(pk=self.pk)
-            .exclude(state__in=TASK_FINAL_STATES)
-            .update(state=TASK_STATES.COMPLETED, finished_at=timezone.now())
+        # Only set the state to finished if it's running. This is important for when the task has
+        # been canceled, so we don't move the task from canceled to finished.
+        rows = Task.objects.filter(pk=self.pk, state=TASK_STATES.RUNNING).update(
+            state=TASK_STATES.COMPLETED, finished_at=timezone.now()
         )
         if rows != 1:
-            msg = "Task set_completed() occurred but Task %s is already in final state"
-            _logger.warning(msg % self.pk)
-        self.refresh_from_db()
+            raise RuntimeError(
+                _("Task set_completed() occurred but Task {} is not RUNNING.").format(self.pk)
+            )
+        with suppress(AttributeError):
+            del self.state
+        with suppress(AttributeError):
+            del self.finished_at
 
     def set_failed(self, exc, tb):
         """
@@ -273,18 +280,70 @@ def set_failed(self, exc, tb):
             tb (traceback): Traceback instance for the current exception.
         """
         tb_str = "".join(traceback.format_tb(tb))
-        rows = (
-            Task.objects.filter(pk=self.pk)
-            .exclude(state__in=TASK_FINAL_STATES)
-            .update(
-                state=TASK_STATES.FAILED,
-                finished_at=timezone.now(),
-                error=exception_to_dict(exc, tb_str),
-            )
+        rows = Task.objects.filter(pk=self.pk, state=TASK_STATES.RUNNING).update(
+            state=TASK_STATES.FAILED,
+            finished_at=timezone.now(),
+            error=exception_to_dict(exc, tb_str),
+        )
+        if rows != 1:
+            raise RuntimeError(_("Attempt to set a not running task to failed."))
+        with suppress(AttributeError):
+            del self.state
+        with suppress(AttributeError):
+            del self.finished_at
+        with suppress(AttributeError):
+            del self.error
+
+    def set_canceling(self):
+        """
+        Set this task to canceling from either waiting, running or canceling.
+
+        This is the only valid transition without holding the task lock.
+        """
+        rows = Task.objects.filter(pk=self.pk, state__in=TASK_INCOMPLETE_STATES).update(
+            state=TASK_STATES.CANCELING,
+        )
+        if rows != 1:
+            raise RuntimeError(_("Attempt to cancel a finished task."))
+        with suppress(AttributeError):
+            del self.state
+
+    def set_canceled(self, final_state=TASK_STATES.CANCELED, reason=None):
+        """
+        Set this task to canceled or failed from canceling.
+        """
+        # Make sure this function was called with a proper final state
+        assert final_state in [TASK_STATES.CANCELED, TASK_STATES.FAILED]
+        task_data = {}
+        if reason:
+            task_data["error"] = {"reason": reason}
+        rows = Task.objects.filter(pk=self.pk, state=TASK_STATES.CANCELING).update(
+            state=final_state,
+            finished_at=timezone.now(),
+            **task_data,
         )
         if rows != 1:
-            raise RuntimeError("Attempt to set a finished task to failed.")
-        self.refresh_from_db()
+            raise RuntimeError(_("Attempt to mark a task canceled that is not in canceling state."))
+        with suppress(AttributeError):
+            del self.state
+        with suppress(AttributeError):
+            del self.finished_at
+        with suppress(AttributeError):
+            del self.error
+
+    # Example taken from here:
+    # https://docs.djangoproject.com/en/3.2/ref/models/instances/#refreshing-objects-from-database
+    def refresh_from_db(self, using=None, fields=None, **kwargs):
+        # fields contains the name of the deferred field to be
+        # loaded.
+        if fields is not None:
+            fields = set(fields)
+            deferred_fields = self.get_deferred_fields()
+            # If any deferred field is going to be loaded
+            if fields.intersection(deferred_fields):
+                # then load all of them
+                fields = fields.union(deferred_fields)
+        super().refresh_from_db(using, fields, **kwargs)
 
     class Meta:
         indexes = [models.Index(fields=["pulp_created"])]
diff --git a/pulpcore/tasking/pulpcore_worker.py b/pulpcore/tasking/pulpcore_worker.py
--- a/pulpcore/tasking/pulpcore_worker.py
+++ b/pulpcore/tasking/pulpcore_worker.py
@@ -184,32 +184,24 @@ def cancel_abandoned_task(self, task, final_state, reason=None):
         Return ``True`` if the task was actually canceled, ``False`` otherwise.
         """
         # A task is considered abandoned when in running state, but no worker holds its lock
-        Task.objects.filter(pk=task.pk, state=TASK_STATES.RUNNING).update(
-            state=TASK_STATES.CANCELING
-        )
-        task.refresh_from_db()
-        if task.state == TASK_STATES.CANCELING:
-            if reason:
-                _logger.info(
-                    "Cleaning up task %s and marking as %s. Reason: %s",
-                    task.pk,
-                    final_state,
-                    reason,
-                )
-            else:
-                _logger.info(_("Cleaning up task %s and marking as %s."), task.pk, final_state)
-            _delete_incomplete_resources(task)
-            if task.reserved_resources_record:
-                self.notify_workers()
-            task_data = {
-                "state": final_state,
-                "finished_at": timezone.now(),
-            }
-            if reason:
-                task_data["error"] = {"reason": reason}
-            Task.objects.filter(pk=task.pk, state=TASK_STATES.CANCELING).update(**task_data)
-            return True
-        return False
+        try:
+            task.set_canceling()
+        except RuntimeError:
+            return False
+        if reason:
+            _logger.info(
+                "Cleaning up task %s and marking as %s. Reason: %s",
+                task.pk,
+                final_state,
+                reason,
+            )
+        else:
+            _logger.info(_("Cleaning up task %s and marking as %s."), task.pk, final_state)
+        _delete_incomplete_resources(task)
+        task.set_canceled(final_state=final_state, reason=reason)
+        if task.reserved_resources_record:
+            self.notify_workers()
+        return True
 
     def iter_tasks(self):
         """Iterate over ready tasks and yield each task while holding the lock."""
diff --git a/pulpcore/tasking/util.py b/pulpcore/tasking/util.py
--- a/pulpcore/tasking/util.py
+++ b/pulpcore/tasking/util.py
@@ -5,7 +5,7 @@
 from django.db import connection
 
 from pulpcore.app.models import Task
-from pulpcore.constants import TASK_FINAL_STATES, TASK_INCOMPLETE_STATES, TASK_STATES
+from pulpcore.constants import TASK_FINAL_STATES, TASK_STATES
 
 _logger = logging.getLogger(__name__)
 
@@ -15,7 +15,7 @@ def cancel(task_id):
     Cancel the task that is represented by the given task_id.
 
     This method cancels only the task with given task_id, not the spawned tasks. This also updates
-    task's state to either 'canceled' or 'canceling'.
+    task's state to 'canceling'.
 
     Args:
         task_id (str): The ID of the task you wish to cancel
@@ -23,30 +23,25 @@ def cancel(task_id):
     Raises:
         rest_framework.exceptions.NotFound: If a task with given task_id does not exist
     """
-    task_status = Task.objects.get(pk=task_id)
+    task = Task.objects.get(pk=task_id)
 
-    if task_status.state in TASK_FINAL_STATES:
+    if task.state in TASK_FINAL_STATES:
         # If the task is already done, just stop
         _logger.debug(
             "Task [{task_id}] already in a final state: {state}".format(
-                task_id=task_id, state=task_status.state
+                task_id=task_id, state=task.state
             )
         )
-        return task_status
+        return task
 
     _logger.info(_("Canceling task: {id}").format(id=task_id))
 
-    task = task_status
     # This is the only valid transition without holding the task lock
-    rows = Task.objects.filter(pk=task.pk, state__in=TASK_INCOMPLETE_STATES).update(
-        state=TASK_STATES.CANCELING
-    )
+    task.set_canceling()
     # Notify the worker that might be running that task and other workers to clean up
     with connection.cursor() as cursor:
         cursor.execute("SELECT pg_notify('pulp_worker_cancel', %s)", (str(task.pk),))
         cursor.execute("NOTIFY pulp_worker_wakeup")
-    if rows == 1:
-        task.refresh_from_db()
     return task
 
 
@@ -57,8 +52,8 @@ def _delete_incomplete_resources(task):
     Args:
         task (Task): A task.
     """
-    if task.state not in [TASK_STATES.CANCELED, TASK_STATES.CANCELING]:
-        raise RuntimeError(_("Task must be canceled."))
+    if task.state != TASK_STATES.CANCELING:
+        raise RuntimeError(_("Task must be canceling."))
     for model in (r.content_object for r in task.created_resources.all()):
         try:
             if model.complete:


Test patch:
diff --git a/pulpcore/tests/functional/api/test_tasking.py b/pulpcore/tests/functional/api/test_tasking.py
--- a/pulpcore/tests/functional/api/test_tasking.py
+++ b/pulpcore/tests/functional/api/test_tasking.py
@@ -91,28 +91,27 @@ def test_delete_cancel_waiting_task(dispatch_task, tasks_api_client):
 
     # Now cancel the task
     task = tasks_api_client.tasks_cancel(task_href, {"state": "canceled"})
-
-    assert task.started_at is None
-    assert task.finished_at is None
-    assert task.state == "canceling"
-
     # cancel the blocking task
-    task = tasks_api_client.tasks_cancel(blocking_task_href, {"state": "canceled"})
+    tasks_api_client.tasks_cancel(blocking_task_href, {"state": "canceled"})
 
-    for i in range(10):
-        task = tasks_api_client.read(task_href)
-        if task.state != "canceling":
-            break
-        time.sleep(1)
+    if task.state == "canceling":
+        assert task.started_at is None
+        assert task.finished_at is None
+
+        for i in range(10):
+            if task.state != "canceling":
+                break
+            time.sleep(1)
+            task = tasks_api_client.read(task_href)
 
     assert task.state == "canceled"
+    assert task.started_at is None
+    assert task.finished_at is not None
 
 
-@pytest.mark.skip(
-    reason="This is a flaky test that fails from time to time. Probably gonna be fixed by 3319"
-)
+@pytest.mark.parallel
 def test_delete_cancel_running_task(dispatch_task, tasks_api_client):
-    task_href = dispatch_task("time.sleep", args=(6000,))
+    task_href = dispatch_task("time.sleep", args=(600,))
 
     for i in range(10):
         task = tasks_api_client.read(task_href)
@@ -130,17 +129,19 @@ def test_delete_cancel_running_task(dispatch_task, tasks_api_client):
     # Now cancel the task
     task = tasks_api_client.tasks_cancel(task_href, {"state": "canceled"})
 
-    assert task.started_at is not None
-    assert task.finished_at is None
-    assert task.state == "canceling"
+    if task.state == "canceling":
+        assert task.started_at is not None
+        assert task.finished_at is None
 
-    for i in range(10):
-        task = tasks_api_client.read(task_href)
-        if task.state != "canceling":
-            break
-        time.sleep(1)
+        for i in range(10):
+            if task.state != "canceling":
+                break
+            time.sleep(1)
+            task = tasks_api_client.read(task_href)
 
     assert task.state == "canceled"
+    assert task.started_at is not None
+    assert task.finished_at is not None
 
 
 @pytest.mark.parallel