Repository: pulp/pulpcore

Issue:
bulk_create() deadlock
**Version**
3.14.12  (reproduceable in main)

**Describe the bug**
ArtifactSaver can deadlock on bulk_get_or_create() call in the presence of high concurrency and overlapping content.

Deadlock will be some variant of the following:
```
Exception ignored in thread started by: <function bulk_doit at 0x7f878f388dc0>
Traceback (most recent call last):
  File "<ipython-input-1-297483773c63>", line 29, in bulk_doit
  File "/home/vagrant/devel/pulpcore/pulpcore/app/models/content.py", line 87, in bulk_get_or_create
    return super().bulk_create(objs, batch_size=batch_size)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/query.py", line 502, in bulk_create
    returned_columns = self._batched_insert(
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/query.py", line 1287, in _batched_insert
    inserted_rows.extend(self._insert(
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/pulp/lib64/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: deadlock detected
DETAIL:  Process 20326 waits for ShareLock on transaction 4661315; blocked by process 20320.
Process 20320 waits for ShareLock on transaction 4661333; blocked by process 20326.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1221,10) in relation "core_artifact_pkey"
```
See https://bugzilla.redhat.com/show_bug.cgi?id=2062526 for more details.

**To Reproduce**
Steps to reproduce the behavior:

Reproducing "in the wild" is difficuilt. The follwoing steps will reliably hit the deadlock:
1) Start with a clean pulp database (at a minimum, none of the Artifacts created at Step 5, below, should exist in the db)
2) make sure you have at least 10 pulpcore-workers up. This command will do the trick:
```
sudo systemctl start pulpcore-worker@1 pulpcore-worker@2 \
  pulpcore-worker@3 pulpcore-worker@4 pulpcore-worker@5 pulpcore-worker@6 \
  pulpcore-worker@7 pulpcore-worker@8 pulpcore-worker@9 pulpcore-worker@10
```
3) Create "a lot" of Artifacts - 10K is a good choice.
4) Arrange to sort those artifacts by sha256 (the collision point) ascending, descending, and mixed.
5) Arrange to bulk_get_or_create() those three sets, in independent threads, at least 4-5 threads per sort.

You can accomplish this with the following script in pulpcore-manager shell:
```
from pulpcore.app.models.content import Artifact
from hashlib import sha256
import _thread
artifacts = []
print(">>>BUILDING ARTIFACTS...")
for i in range(9999):    
  filename = f'/tmp/{i:06d}.txt'
  with open(filename, "w") as f:
    f.write(filename)
  with open(filename, "rb") as f:
    sum256 = sha256(f.read()).hexdigest()         
  attrs = {"file": filename, "sha256": sum256, "size": i}
  artifacts.append(Artifact (**attrs))

print(">>> SORT ASC...")
artifacts_asc = sorted(artifacts, key=lambda a: a.sha256)
print(">>> SORT DSC...")
artifacts_dsc = sorted(artifacts, key=lambda a: a.sha256, reverse=True)
print(">>> SHUFFLE...")
artifacts_mix = []
for i in range(9999):    
  artifacts_mix.append(artifacts_asc[i])
  artifacts_mix.append(artifacts_dsc[i])

def bulk_doit(art_list):
  print(">>> ENTER...")
  Artifact.objects.bulk_get_or_create(art_list)
  print(">>> EXIT...")

print(">>> STARTING LOOP...")
for i in range(5):
  _thread.start_new_thread(bulk_doit, (artifacts_asc,))
  _thread.start_new_thread(bulk_doit, (artifacts_dsc,))
  _thread.start_new_thread(bulk_doit, (artifacts_mix,))
```

**Expected behavior**
All the threads complete without error.

**Additional context**
https://bugzilla.redhat.com/show_bug.cgi?id=2062526



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/stages/artifact_stages.py b/pulpcore/plugin/stages/artifact_stages.py
--- a/pulpcore/plugin/stages/artifact_stages.py
+++ b/pulpcore/plugin/stages/artifact_stages.py
@@ -236,12 +236,13 @@ async def run(self):
                     if d_artifact.artifact._state.adding and not d_artifact.deferred_download:
                         d_artifact.artifact.file = str(d_artifact.artifact.file)
                         da_to_save.append(d_artifact)
+            da_to_save_ordered = sorted(da_to_save, key=lambda x: x.artifact.sha256)
 
             if da_to_save:
                 for d_artifact, artifact in zip(
-                    da_to_save,
+                    da_to_save_ordered,
                     await sync_to_async(Artifact.objects.bulk_get_or_create)(
-                        d_artifact.artifact for d_artifact in da_to_save
+                        d_artifact.artifact for d_artifact in da_to_save_ordered
                     ),
                 ):
                     d_artifact.artifact = artifact


Test patch: