Repository: pulp/pulpcore

Issue:
bulk_update() can still deadlock in content_stages
**Version**
pulpcore/3.14.16 (but problem exists in main)

**Describe the bug**
Syncing multiple repositories with high content-overlaps under high concurrency, continues to hit occasional deadlock.

Traceback and postgres log for the particular failure:
```
2022-08-10 15:51:29 EDT ERROR:  deadlock detected
2022-08-10 15:51:29 EDT DETAIL:  Process 55740 waits for ShareLock on transaction 61803; blocked by process 55746.
        Process 55746 waits for ShareLock on transaction 61805; blocked by process 55740.
        Process 55740: SELECT ....
        Process 55746: COMMIT
2022-08-10 15:51:29 EDT HINT:  See server log for query details.
2022-08-10 15:51:29 EDT CONTEXT:  while locking tuple (209,51) in relation "core_contentartifact"
```

```
pulpcore-worker-5[54158]: pulp [88649f1c-3393-4693-aab7-d73b62eeda62]: pulpcore.tasking.pulpcore_worker:INFO: Task 407bb67b-65d0-4d65-b9c8-b1aa1f2c87fd failed (deadlock detected
pulpcore-worker-5[54158]: DETAIL:  Process 55740 waits for ShareLock on transaction 61803; blocked by process 55746.
pulpcore-worker-5[54158]: Process 55746 waits for ShareLock on transaction 61805; blocked by process 55740.
pulpcore-worker-5[54158]: HINT:  See server log for query details.
pulpcore-worker-5[54158]: CONTEXT:  while locking tuple (209,51) in relation "core_contentartifact"
pulpcore-worker-5[54158]: )
pulpcore-worker-5[54158]: pulp [88649f1c-3393-4693-aab7-d73b62eeda62]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 342, in _perform_task
pulpcore-worker-5[54158]: result = func(*args, **kwargs)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 494, in synchronize
pulpcore-worker-5[54158]: version = dv.create()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
pulpcore-worker-5[54158]: loop.run_until_complete(pipeline)
pulpcore-worker-5[54158]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulpcore-worker-5[54158]: return future.result()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
pulpcore-worker-5[54158]: await asyncio.gather(*futures)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulpcore-worker-5[54158]: await self.run()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 178, in run
pulpcore-worker-5[54158]: .order_by("pulp_id")
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-worker-5[54158]: self._fetch_all()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-worker-5[54158]: self._result_cache = list(self._iterable_class(self))
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 144, in __iter__
pulpcore-worker-5[54158]: return compiler.results_iter(tuple_expected=True, chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1094, in results_iter
pulpcore-worker-5[54158]: results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-worker-5[54158]: cursor.execute(sql, params)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-worker-5[54158]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-worker-5[54158]: return executor(sql, params, many, context)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-5[54158]: return self.cursor.execute(sql, params)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-worker-5[54158]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-5[54158]: return self.cursor.execute(sql, params)
```
**To Reproduce**
See the QE test-setup from https://bugzilla.redhat.com/show_bug.cgi?id=2062526. We have not been able to force the problem with a synthetic test case.

**Expected behavior**
All the repositories should sync all their content without any of the sync-processes being failed due to detected deadlocks.


Discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=2082209

Resolving patch:
diff --git a/pulpcore/plugin/stages/content_stages.py b/pulpcore/plugin/stages/content_stages.py
--- a/pulpcore/plugin/stages/content_stages.py
+++ b/pulpcore/plugin/stages/content_stages.py
@@ -165,23 +165,26 @@ def process_batch():
                     # on select-for-update. So, we select-for-update, in pulp_id order, the
                     # rows we're about to update as one db-call, and then do the update in a
                     # second.
+                    #
+                    # NOTE: select-for-update requires being in an atomic-transaction. We are
+                    # **already in an atomic transaction** at this point as a result of the
+                    # "with transaction.atomic():", above.
                     ids = [k.pulp_id for k in to_update_ca_bulk]
-                    with transaction.atomic():
-                        # "len()" forces the QA to be evaluated. Using exist() or count() won't
-                        # work for us - Django is smart enough to either not-order, or even
-                        # not-emit, a select-for-update in these cases.
-                        #
-                        # To maximize performance, we make sure to only ask for pulp_ids, and
-                        # avoid instantiating a python-object for the affected CAs by using
-                        # values_list()
-                        len(
-                            ContentArtifact.objects.filter(pulp_id__in=ids)
-                            .only("pulp_id")
-                            .order_by("pulp_id")
-                            .select_for_update()
-                            .values_list()
-                        )
-                        ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
+                    # "len()" forces the QuerySet to be evaluated. Using exist() or count() won't
+                    # work for us - Django is smart enough to either not-order, or even
+                    # not-emit, a select-for-update in these cases.
+                    #
+                    # To maximize performance, we make sure to only ask for pulp_ids, and
+                    # avoid instantiating a python-object for the affected CAs by using
+                    # values_list()
+                    subq = (
+                        ContentArtifact.objects.filter(pulp_id__in=ids)
+                        .only("pulp_id")
+                        .order_by("pulp_id")
+                        .select_for_update()
+                    )
+                    len(subq.values_list())
+                    ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
 
                     # To avoid a similar deadlock issue when calling get_or_create, we sort the
                     # "new" CAs to make sure inserts happen in a defined order. Since we can't


Test patch: