Repository: pulp/pulpcore

Issue:
Possible race on task dispatch
We identified that there may be a possible race when dispatching new tasks.
With two processes adding a new task they become visible to the workers after finishing their transaction, but the created_at fields may actually report a different order. This can lead to  two workers picking up conflicting tasks concurrently.
This may be even worse on distributed systems where you can have a clockscrew on top.

We believe the window to already be very small, so any attempt to fix is should be like a vacuum seal.

Current working idea: Wrap adding a task to the table in a transaction-based-exclusive-wait-for advisory-lock. Check that the current latest task is actually from the past and add the new task afterwards.


Discussion:
Wild suggestion here, perhaps not even something we can do currently:

UUIDv7 embeds a timestamp and is designed to have a monotonically increasing unique value.   Theoretically we could work with the PK instead of raw timestamps (so long as no ancient tasks utilizing UUIDv4 PKs remain) to obtain a complete table ordering.  Postgresql has functions for generating UUIDs (but not UUIDv7 [currently](https://commitfest.postgresql.org/47/4388/), you need an extension for it), so if [0] we were able to utilize those for sourcing the PK then we could mostly [1] eliminate clock skew as an issue, as only the database timestamp would matter

[0] we currently "expect" the PK to have been generated in Python prior to any attempts to save IIRC
[1] assuming the database isn't itself split across a blue/green deployment, and/or nobody is messing with the clocks on those systems.  I'm not sure how time would be handled in that scenario.
> Wild suggestion here, perhaps not even something we can do currently:
> 
> UUIDv7 embeds a timestamp and is designed to have a monotonically increasing unique value. Theoretically we could work with the PK instead of raw timestamps (so long as no ancient tasks utilizing UUIDv4 PKs remain) to obtain a complete table ordering. Postgresql has functions for generating UUIDs (but not UUIDv7 [currently](https://commitfest.postgresql.org/47/4388/), you need an extension for it), so if [0] we were able to utilize those for sourcing the PK then we could mostly [1] eliminate clock skew as an issue, as only the database timestamp would matter
> 
> [0] we currently "expect" the PK to have been generated in Python prior to any attempts to save IIRC [1] assuming the database isn't itself split across a blue/green deployment, and/or nobody is messing with the clocks on those systems. I'm not sure how time would be handled in that scenario.

I'm not entirely convinced this would solve the issue, which is basically that there is a time gap between creating the task and committing the transaction it is in. And while once all tasks are committed (after say 5 s) we do have a definite order, by that time we'd hope some worker would have started on it already.

Resolving patch:
diff --git a/pulpcore/constants.py b/pulpcore/constants.py
--- a/pulpcore/constants.py
+++ b/pulpcore/constants.py
@@ -5,6 +5,13 @@
 
 VAR_TMP_PULP = Path("/var/tmp/pulp")
 
+# Special purpose advisory locks for use with the two number variant.
+# The group will be 0.
+# The numbers are randomly chosen.
+# !!! Never change these values !!!
+TASK_DISPATCH_LOCK = 21
+TASK_SCHEDULING_LOCK = 42
+
 
 #: All valid task states.
 TASK_STATES = SimpleNamespace(
diff --git a/pulpcore/tasking/tasks.py b/pulpcore/tasking/tasks.py
--- a/pulpcore/tasking/tasks.py
+++ b/pulpcore/tasking/tasks.py
@@ -5,15 +5,21 @@
 import logging
 import sys
 import traceback
+from datetime import timedelta
 from gettext import gettext as _
 
 from django.db import connection, transaction
-from django.db.models import Model
+from django.db.models import Model, Max
 from django_guid import get_guid
 from pulpcore.app.apps import MODULE_PLUGIN_VERSIONS
 from pulpcore.app.models import Task
 from pulpcore.app.util import current_task, get_domain, get_url
-from pulpcore.constants import TASK_FINAL_STATES, TASK_INCOMPLETE_STATES, TASK_STATES
+from pulpcore.constants import (
+    TASK_FINAL_STATES,
+    TASK_INCOMPLETE_STATES,
+    TASK_STATES,
+    TASK_DISPATCH_LOCK,
+)
 
 _logger = logging.getLogger(__name__)
 
@@ -148,6 +154,13 @@ def dispatch(
     notify_workers = False
     with contextlib.ExitStack() as stack:
         with transaction.atomic():
+            # Task creation need to be serialized so that pulp_created will provide a stable order
+            # at every time. We specifically need to ensure that each task, when commited to the
+            # task table will be the newest with respect to `pulp_created`.
+            with connection.cursor() as cursor:
+                # Wait for exclusive access and release automatically after transaction.
+                cursor.execute("SELECT pg_advisory_xact_lock(%s, %s)", [0, TASK_DISPATCH_LOCK])
+            newest_created = Task.objects.aggregate(Max("pulp_created"))["pulp_created__max"]
             task = Task.objects.create(
                 state=TASK_STATES.WAITING,
                 logging_cid=(get_guid()),
@@ -159,6 +172,15 @@ def dispatch(
                 reserved_resources_record=resources,
                 versions=versions,
             )
+            if newest_created and task.pulp_created <= newest_created:
+                # Let this workaround not row forever into the future.
+                if newest_created - task.pulp_created > timedelta(seconds=1):
+                    # Do not commit the transaction if this condition is not met.
+                    # If we ever hit this, think about delegating the timestamping to PostgresQL.
+                    raise RuntimeError("Clockscrew detected. Task dispatching would be dangerous.")
+                # Try to work around the smaller glitch
+                task.pulp_created = newest_created + timedelta(milliseconds=1)
+                task.save()
             if immediate:
                 # Grab the advisory lock before the task hits the db.
                 stack.enter_context(task)
diff --git a/pulpcore/tasking/worker.py b/pulpcore/tasking/worker.py
--- a/pulpcore/tasking/worker.py
+++ b/pulpcore/tasking/worker.py
@@ -16,7 +16,7 @@
 from django.db import connection
 from django.utils import timezone
 
-from pulpcore.constants import TASK_STATES, TASK_INCOMPLETE_STATES
+from pulpcore.constants import TASK_STATES, TASK_INCOMPLETE_STATES, TASK_SCHEDULING_LOCK
 from pulpcore.exceptions import AdvisoryLockError
 from pulpcore.app.apps import pulp_plugin_configs
 from pulpcore.app.models import Worker, Task, ApiAppStatus, ContentAppStatus
@@ -40,8 +40,6 @@
 TASK_KILL_INTERVAL = 1
 # Number of heartbeats between cleaning up worker processes (approx)
 WORKER_CLEANUP_INTERVAL = 100
-# Randomly chosen
-TASK_SCHEDULING_LOCK = 42
 
 
 class PulpcoreWorker:


Test patch: