Repository: pulp/pulpcore

Issue:
bulk_update() can still deadlock in content_stages
**Version**
pulpcore/3.14.16 (but problem exists in main)

**Describe the bug**
Syncing multiple repositories with high content-overlaps under high concurrency, continues to hit occasional deadlock.

Traceback and postgres log for the particular failure:
```
2022-08-10 15:51:29 EDT ERROR:  deadlock detected
2022-08-10 15:51:29 EDT DETAIL:  Process 55740 waits for ShareLock on transaction 61803; blocked by process 55746.
        Process 55746 waits for ShareLock on transaction 61805; blocked by process 55740.
        Process 55740: SELECT ....
        Process 55746: COMMIT
2022-08-10 15:51:29 EDT HINT:  See server log for query details.
2022-08-10 15:51:29 EDT CONTEXT:  while locking tuple (209,51) in relation "core_contentartifact"
```

```
pulpcore-worker-5[54158]: pulp [88649f1c-3393-4693-aab7-d73b62eeda62]: pulpcore.tasking.pulpcore_worker:INFO: Task 407bb67b-65d0-4d65-b9c8-b1aa1f2c87fd failed (deadlock detected
pulpcore-worker-5[54158]: DETAIL:  Process 55740 waits for ShareLock on transaction 61803; blocked by process 55746.
pulpcore-worker-5[54158]: Process 55746 waits for ShareLock on transaction 61805; blocked by process 55740.
pulpcore-worker-5[54158]: HINT:  See server log for query details.
pulpcore-worker-5[54158]: CONTEXT:  while locking tuple (209,51) in relation "core_contentartifact"
pulpcore-worker-5[54158]: )
pulpcore-worker-5[54158]: pulp [88649f1c-3393-4693-aab7-d73b62eeda62]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 342, in _perform_task
pulpcore-worker-5[54158]: result = func(*args, **kwargs)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 494, in synchronize
pulpcore-worker-5[54158]: version = dv.create()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
pulpcore-worker-5[54158]: loop.run_until_complete(pipeline)
pulpcore-worker-5[54158]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulpcore-worker-5[54158]: return future.result()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
pulpcore-worker-5[54158]: await asyncio.gather(*futures)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulpcore-worker-5[54158]: await self.run()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 178, in run
pulpcore-worker-5[54158]: .order_by("pulp_id")
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
pulpcore-worker-5[54158]: self._fetch_all()
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
pulpcore-worker-5[54158]: self._result_cache = list(self._iterable_class(self))
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 144, in __iter__
pulpcore-worker-5[54158]: return compiler.results_iter(tuple_expected=True, chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1094, in results_iter
pulpcore-worker-5[54158]: results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
pulpcore-worker-5[54158]: cursor.execute(sql, params)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-worker-5[54158]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-worker-5[54158]: return executor(sql, params, many, context)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-5[54158]: return self.cursor.execute(sql, params)
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-worker-5[54158]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-worker-5[54158]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-5[54158]: return self.cursor.execute(sql, params)
```
**To Reproduce**
See the QE test-setup from https://bugzilla.redhat.com/show_bug.cgi?id=2062526. We have not been able to force the problem with a synthetic test case.

**Expected behavior**
All the repositories should sync all their content without any of the sync-processes being failed due to detected deadlocks.


Discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=2082209

Resolving patch:
diff --git a/pulpcore/plugin/stages/content_stages.py b/pulpcore/plugin/stages/content_stages.py
--- a/pulpcore/plugin/stages/content_stages.py
+++ b/pulpcore/plugin/stages/content_stages.py
@@ -163,23 +163,26 @@ async def run(self):
                 # on select-for-update. So, we select-for-update, in pulp_id order, the
                 # rows we're about to update as one db-call, and then do the update in a
                 # second.
+                #
+                # NOTE: select-for-update requires being in an atomic-transaction. We are
+                # **already in an atomic transaction** at this point as a result of the
+                # "with transaction.atomic():", above.
                 ids = [k.pulp_id for k in to_update_ca_bulk]
-                with transaction.atomic():
-                    # "len()" forces the QA to be evaluated. Using exist() or count() won't
-                    # work for us - Django is smart enough to either not-order, or even
-                    # not-emit, a select-for-update in these cases.
-                    #
-                    # To maximize performance, we make sure to only ask for pulp_ids, and
-                    # avoid instantiating a python-object for the affected CAs by using
-                    # values_list()
-                    len(
-                        ContentArtifact.objects.filter(pulp_id__in=ids)
-                        .only("pulp_id")
-                        .order_by("pulp_id")
-                        .select_for_update()
-                        .values_list()
-                    )
-                    ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
+                # "len()" forces the QuerySet to be evaluated. Using exist() or count() won't
+                # work for us - Django is smart enough to either not-order, or even
+                # not-emit, a select-for-update in these cases.
+                #
+                # To maximize performance, we make sure to only ask for pulp_ids, and
+                # avoid instantiating a python-object for the affected CAs by using
+                # values_list()
+                subq = (
+                    ContentArtifact.objects.filter(pulp_id__in=ids)
+                    .only("pulp_id")
+                    .order_by("pulp_id")
+                    .select_for_update()
+                )
+                len(subq.values_list())
+                ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
 
                 # To avoid a similar deadlock issue when calling get_or_create, we sort the
                 # "new" CAs to make sure inserts happen in a defined order. Since we can't


Test patch: