Repository: pulp/pulpcore

Issue:
Duplicate Key Exception Raised during Sync related to duplicate RemoteArtifact objects
**Version**
pulpcore 3.18 at 3e5b68377088616c9ec55884f614b00c4a5dfc4e and pulp_container at d801f93ee4f50b08bd9e8c699be668e2ed5732c1

**Describe the bug**
Sync with pulp_container until you receive the duplicate RemoteArtifact exception:

```
pulp [bf9ddb0e6acf4ade98840d4890e01b6e]: pulpcore.tasking.pulpcore_worker:INFO: Task 24e40894-bd11-446e-bccb-08de4bbfdce4 failed (duplicate key value violates unique constraint "core_remoteartifact_content_artifact_id_remote_id_be64c19e_uniq"
DETAIL:  Key (content_artifact_id, remote_id)=(6225848a-55ca-4635-862c-e77db4bc98b7, c60e2039-a95e-4480-bbf7-329158ae4b17) already exists.
)
pulp [bf9ddb0e6acf4ade98840d4890e01b6e]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 442, in _perform_task
    result = func(*args, **kwargs)
  File "/home/vagrant/devel/pulp_container/pulp_container/app/tasks/synchronize.py", line 44, in synchronize
    return dv.create()
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.10/asyncio/base_events.py", line 641, in run_until_complete 
    return future.result()
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 279, in run
    await self._handle_remote_artifacts(batch)
  File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 399, in _handle_remote_artifacts
    await sync_to_async(RemoteArtifact.objects.bulk_create)(ras_to_create_ordered)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/asgiref/sync.py", line 414, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib64/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/asgiref/sync.py", line 455, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/query.py", line 502, in bulk_create
    returned_columns = self._batched_insert(
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/query.py", line 1287, in _batched_insert
    inserted_rows.extend(self._insert(
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)  
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
```

**To Reproduce**
Run the pulp_container sync `pytest -v pulp_container/tests/functional/ -k test_sync` over and over until you receive an exception and the sync fails.

**Expected behavior**
For the sync to not fail on a duplicate key error

**Additional context**
It's easier to reproduce with the refactored pulp_container sync pipeline contained in d801f93ee4f50b08bd9e8c699be668e2ed5732c1



Discussion:
The reason is the prefetcher provides caching, see this bug https://code.djangoproject.com/ticket/33596 I learned from @dkliban on matrix that the last time we had this problem we used a solution [like this](https://github.com/pulp/pulpcore/commit/8dcdcd2dfb6d1f217e9725db9bbead1eef86c079). I'm going to try to put that into a PR now.

Resolving patch:
diff --git a/pulpcore/plugin/stages/artifact_stages.py b/pulpcore/plugin/stages/artifact_stages.py
--- a/pulpcore/plugin/stages/artifact_stages.py
+++ b/pulpcore/plugin/stages/artifact_stages.py
@@ -284,12 +284,17 @@ def _handle_remote_artifacts(self, batch):
         """
         remotes_present = set()
         for d_content in batch:
+            # The fix described here solves two sepearate bugs:
+            #
             # If the attribute is set in a previous batch on the very first item in this batch, the
             # rest of the items in this batch will not get the attribute set during prefetch.
             # https://code.djangoproject.com/ticket/32089
+
+            # https://code.djangoproject.com/ticket/33596
+            # If the content was pre-fetched previously, remove that cached data, which could be out
+            # of date.
             if hasattr(d_content.content, "_remote_artifact_saver_cas"):
                 delattr(d_content.content, "_remote_artifact_saver_cas")
-
             for d_artifact in d_content.d_artifacts:
                 if d_artifact.remote:
                     remotes_present.add(d_artifact.remote)


Test patch: