Repository: pulp/pulpcore

Issue:
bulk_update() in content-stages can cause (very rare) deadlock
**Version**
3.14

**Describe the bug**
In high-concurrency environments, with overlapping content, calling bulk_update() can cause a deadlock. Specifically, this call:

https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/stages/content_stages.py#L158-L164

Ordering the list-to-be-updated does not, alas, protect us - because Postgres doesn't guarantee order when doing an update like this.

**To Reproduce**
We have only seen this "in the wild" once, syncing 8-10 repos with similar content at the same time with 10 workers available.

**Expected behavior**
Don't deadlock.

**Additional context**
This is the traceback from the initial description for

https://bugzilla.redhat.com/show_bug.cgi?id=2062526

We fixed the deadlock noted in https://bugzilla.redhat.com/show_bug.cgi?id=2062526#c2 under #2420 




Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/stages/content_stages.py b/pulpcore/plugin/stages/content_stages.py
--- a/pulpcore/plugin/stages/content_stages.py
+++ b/pulpcore/plugin/stages/content_stages.py
@@ -155,13 +155,39 @@ def process_batch():
                         content_artifact.artifact = to_update_ca_artifact[key]
                         to_update_ca_bulk.append(content_artifact)
 
-                    # Sort the lists we're about to do bulk updates/creates on.
-                    # We know to_update_ca_bulk entries already are in the DB, so we can enforce
-                    # order just using pulp_id.
-                    to_update_ca_bulk.sort(key=lambda x: x.pulp_id)
-                    content_artifact_bulk.sort(key=lambda x: ContentArtifact.sort_key(x))
+                    # to_update_ca_bulk are the CAs that we know are already persisted.
+                    # We need to update their artifact_ids, and wish to do it in bulk to
+                    # avoid hundreds of round-trips to the database.
+                    #
+                    # To avoid deadlocks in high-concurrency environments with overlapping
+                    # content, we need to update the rows in some defined order. Unfortunately,
+                    # postgres doesn't support order-on-update - but it *does* support ordering
+                    # on select-for-update. So, we select-for-update, in pulp_id order, the
+                    # rows we're about to update as one db-call, and then do the update in a
+                    # second.
+                    ids = [k.pulp_id for k in to_update_ca_bulk]
+                    with transaction.atomic():
+                        # "len()" forces the QA to be evaluated. Using exist() or count() won't
+                        # work for us - Django is smart enough to either not-order, or even
+                        # not-emit, a select-for-update in these cases.
+                        #
+                        # To maximize performance, we make sure to only ask for pulp_ids, and
+                        # avoid instantiating a python-object for the affected CAs by using
+                        # values_list()
+                        len(
+                            ContentArtifact.objects.filter(pulp_id__in=ids)
+                            .only("pulp_id")
+                            .order_by("pulp_id")
+                            .select_for_update()
+                            .values_list()
+                        )
+                        ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
 
-                    ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
+                    # To avoid a similar deadlock issue when calling get_or_create, we sort the
+                    # "new" CAs to make sure inserts happen in a defined order. Since we can't
+                    # trust the pulp_id (by the time we go to create a CA, it may already exist,
+                    # and be replaced by the 'real' one), we sort by their "natural key".
+                    content_artifact_bulk.sort(key=lambda x: ContentArtifact.sort_key(x))
                     ContentArtifact.objects.bulk_get_or_create(content_artifact_bulk)
                     self._post_save(batch)
 


Test patch: