Repository: pulp/pulpcore

Issue:
Concurrent PulpImports with iverlapping  content can still fail.
**Version**
core/3.14

**Describe the bug**
PulpImport imports repositories in parallel. If the repos in a single import-file have overlapping content, two repo-import-threads can attempt to create the same object at the same time, and collide. This is handled by retry-logic, that notices the collision and retries in the failing task.

HOWEVER - if "enough" repositories, with "enough" overlapping content, are imported in a system with "enough" available pulpcore-workers, then **more than two** tasks can collide. The first one "wins", the second fails, retries, and "wins", the third fails *twice* and aborts.  We need to let PulpImport retry more than once.

Note that there needs to be a finite number of retries, because it's better to eventually give up, than to fall into an infinite retry-hole due to some equally-rare edge case.

**To Reproduce**
Steps to reproduce the behavior:

See discussion in https://bugzilla.redhat.com/show_bug.cgi?id=2056702 . We have seen this happen to exactly one user to date. We have not been able to reproduce it "in captivity" even once (but the hole clearly exists).

**Expected behavior**
PulpImport should retry an import more than once, in the event of an object-creation collision.

**Additional context**
Original collision-problem-report : https://pulp.plan.io/issues/8633
Retry fix : https://github.com/pulp/pulpcore/pull/1278



Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -44,6 +44,10 @@
 VERSIONS_FILE = "versions.json"
 CONTENT_MAPPING_FILE = "content_mapping.json"
 
+# Concurrent imports w/ overlapping content can collide - how many attempts are we willing to
+# make before we decide this is a fatal error?
+MAX_ATTEMPTS = 3
+
 
 def _destination_repo(importer, source_repo_name):
     """Find the destination repository based on source repo's name."""
@@ -62,31 +66,34 @@ def _import_file(fpath, resource_class, retry=False):
             resource = resource_class()
             log.info(_("...Importing resource {}.").format(resource.__class__.__name__))
             if retry:
-                # django import-export can have a problem with concurrent-imports that are
-                # importing the same 'thing' (e.g., a Package that exists in two different
-                # repo-versions that are being imported at the same time). If we're asked to
-                # retry, we will try an import that will simply record errors as they happen
-                # (rather than failing with an exception) first. If errors happen, we'll do one
-                # retry before we give up on this repo-version's import.
-                a_result = resource.import_data(data, raise_errors=False)
-                if a_result.has_errors():
-                    log.info(
-                        "...{} import-errors encountered importing {}, retrying".format(
-                            a_result.totals["error"], fpath
+                curr_attempt = 1
+                while curr_attempt < MAX_ATTEMPTS:
+                    curr_attempt += 1
+                    # django import-export can have a problem with concurrent-imports that are
+                    # importing the same 'thing' (e.g., a Package that exists in two different
+                    # repo-versions that are being imported at the same time). If we're asked to
+                    # retry, we will try an import that will simply record errors as they happen
+                    # (rather than failing with an exception) first. If errors happen, we'll
+                    # retry before we give up on this repo-version's import.
+                    a_result = resource.import_data(data, raise_errors=False)
+                    if a_result.has_errors():
+                        total_errors = a_result.totals["error"]
+                        log.info(
+                            f"...{total_errors} import-errors encountered importing "
+                            "{fpath}, attempt {curr_attempt}, retrying"
                         )
-                    )
-                    # Second attempt, we raise an exception on any problem.
-                    # This will either succeed, or log a fatal error and fail.
-                    try:
-                        a_result = resource.import_data(data, raise_errors=True)
-                    except Exception as e:  # noqa log on ANY exception and then re-raise
-                        log.error(_("FATAL import-failure importing {}").format(fpath))
-                        raise
+                # Last attempt, we raise an exception on any problem.
+                # This will either succeed, or log a fatal error and fail.
+                try:
+                    a_result = resource.import_data(data, raise_errors=True)
+                except Exception as e:  # noqa log on ANY exception and then re-raise
+                    log.error(f"FATAL import-failure importing {fpath}")
+                    raise
             else:
                 a_result = resource.import_data(data, raise_errors=True)
             return a_result
     except AttributeError:
-        log.error(_("FAILURE importing file {}!").format(fpath))
+        log.error(f"FAILURE loading import-file {fpath}!")
         raise
 
 


Test patch: