Repository: pulp/pulpcore

Issue:
Occasional UQ error core_repositoryversion_repository_id_number_3c54ce50_uniq on DistTree PulpImport
Rare failure, in this occasion invoked during a pulp-import testcase:

`pulp_rpm/tests/functional/api/test_pulpimport.py::DistributionTreePulpImportTestCase::test_import FAILED [ 81%]`

```
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.tasking.pulpcore_worker:INFO: Starting task c455492e-bc13-4517-b86a-6069bca53ed7
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 023517b6-401d-4f7e-be44-1d5814d0fe0c
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.PackageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.PackageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.ModulemdResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ModulemdResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.ModulemdDefaultsResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ModulemdDefaultsResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.PackageGroupResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageGroupResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.PackageCategoryResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageCategoryResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.PackageEnvironmentResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageEnvironmentResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.PackageLangpacksResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageLangpacksResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.UpdateRecordResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateRecordResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.RepoMetadataFileResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource RepoMetadataFileResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.DistributionTreeResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource DistributionTreeResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.DistributionTreeRepositoryResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource DistributionTreeRepositoryResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.ChecksumResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ChecksumResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.ImageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ImageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.AddonResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource AddonResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.ModulemdResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ModulemdResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.ModulemdDefaultsResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ModulemdDefaultsResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.PackageGroupResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageGroupResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.VariantResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource VariantResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.PackageCategoryResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageCategoryResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.UpdateReferenceResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateReferenceResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.UpdateCollectionResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateCollectionResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulp_rpm.app.modelresource.UpdateCollectionPackageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateCollectionPackageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpopm4e5hz/repository-7206edf0-b89d-4da5-93d0-dcae838399c5_1/pulpcore.app.modelresource.ContentArtifactResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ContentArtifactResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.PackageEnvironmentResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageEnvironmentResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.PackageLangpacksResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource PackageLangpacksResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.UpdateRecordResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateRecordResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.RepoMetadataFileResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource RepoMetadataFileResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.DistributionTreeResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource DistributionTreeResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.DistributionTreeRepositoryResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource DistributionTreeRepositoryResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.ChecksumResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ChecksumResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.ImageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ImageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.AddonResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource AddonResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.VariantResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource VariantResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.UpdateReferenceResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateReferenceResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.UpdateCollectionResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateCollectionResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulp_rpm.app.modelresource.UpdateCollectionPackageResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource UpdateCollectionPackageResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: Importing file /tmp/tmpg9flo4a_/repository-868191e5-b1bb-49f7-86b8-efe70ecb0329_1/pulpcore.app.modelresource.ContentArtifactResource.json.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.app.tasks.importer:INFO: ...Importing resource ContentArtifactResource.
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.tasking.pulpcore_worker:INFO: Task c455492e-bc13-4517-b86a-6069bca53ed7 failed (duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_3c54ce50_uniq"
DETAIL:  Key (repository_id, number)=(e84ff294-5d1b-4bfe-b5fb-e98d0c328537, 2) already exists.
)
pulp [2edbce945b1e462193b75d622f48aa44]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 362, in _perform_task
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/tasks/importer.py", line 206, in import_repository_version
    with repo.new_version() as new_version:
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 126, in new_version
    version.save()
  File "/usr/lib64/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.8/site-packages/django_lifecycle/mixins.py", line 134, in save
    save(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
```


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -54,14 +54,37 @@ def _destination_repo(importer, source_repo_name):
     return Repository.objects.get(name=dest_repo_name)
 
 
-def _import_file(fpath, resource_class, do_raise=True):
+def _import_file(fpath, resource_class, retry=False):
     try:
         log.info(_("Importing file {}.").format(fpath))
         with open(fpath, "r") as json_file:
             data = Dataset().load(json_file, format="json")
             resource = resource_class()
             log.info(_("...Importing resource {}.").format(resource.__class__.__name__))
-            return resource.import_data(data, raise_errors=do_raise)
+            if retry:
+                # django import-export can have a problem with concurrent-imports that are
+                # importing the same 'thing' (e.g., a Package that exists in two different
+                # repo-versions that are being imported at the same time). If we're asked to
+                # retry, we will try an import that will simply record errors as they happen
+                # (rather than failing with an exception) first. If errors happen, we'll do one
+                # retry before we give up on this repo-version's import.
+                a_result = resource.import_data(data, raise_errors=False)
+                if a_result.has_errors():
+                    log.info(
+                        _("...{} import-errors encountered importing {}, retrying").format(
+                            a_result.totals["error"], fpath
+                        )
+                    )
+                    # Second attempt, we raise an exception on any problem.
+                    # This will either succeed, or log a fatal error and fail.
+                    try:
+                        a_result = resource.import_data(data, raise_errors=True)
+                    except Exception as e:  # noqa log on ANY exception and then re-raise
+                        log.error(_("FATAL import-failure importing {}").format(fpath))
+                        raise
+            else:
+                a_result = resource.import_data(data, raise_errors=True)
+            return a_result
     except AttributeError:
         log.error(_("FAILURE importing file {}!").format(fpath))
         raise
@@ -157,36 +180,14 @@ def import_repository_version(importer_pk, destination_repo_pk, source_repo_name
         resulting_content_ids = []
         for res_class in cfg.exportable_classes:
             filename = f"{res_class.__module__}.{res_class.__name__}.json"
-            a_result = _import_file(os.path.join(rv_path, filename), res_class, do_raise=False)
-            # django import-export can have a problem with concurrent-imports that are
-            # importing the same 'thing' (e.g., a Package that exists in two different
-            # repo-versions that are being imported at the same time). We will try an import
-            # that will simply record errors as they happen (rather than failing with an exception)
-            # first. If errors happen, we'll do one retry before we give up on this repo-version's
-            # import.
-            if a_result.has_errors():
-                log.info(
-                    _("...{} import-errors encountered importing {} from {}, retrying").format(
-                        a_result.totals["error"], filename, rv_name
-                    )
-                )
-                # Second attempt, we allow to raise an exception on any problem.
-                # This will either succeed, or log a fatal error and fail.
-                try:
-                    a_result = _import_file(os.path.join(rv_path, filename), res_class)
-                except Exception as e:  # noqa log on ANY exception and then re-raise
-                    log.error(
-                        _("FATAL import-failure importing {} from {}").format(filename, rv_name)
-                    )
-                    raise
-
+            a_result = _import_file(os.path.join(rv_path, filename), res_class, retry=True)
             resulting_content_ids.extend(
                 row.object_id for row in a_result.rows if row.import_type in ("new", "update")
             )
 
         # Once all content exists, create the ContentArtifact links
         ca_path = os.path.join(rv_path, CA_FILE)
-        _import_file(ca_path, ContentArtifactResource)
+        _import_file(ca_path, ContentArtifactResource, retry=True)
 
         # see if we have a content mapping
         mapping_path = f"{rv_name}/{CONTENT_MAPPING_FILE}"


Test patch: