Repository: pulp/pulpcore

Issue:
PulpExport ArtifactResource.json has duplicate entries for overlapping content.
**Version**
main

**Describe the bug**
PulpExport is typically used to export the content of many repositories. It gathers up the unique Artifacts themselves into a single directory. However, the ArtifactResource.json file which is used to import the datasbe metadata is just a **list** of artifacts found in all the repositories.

In the presence of exported repositories with duplicate content (e.g., Satellite CVs with many of the same base repositories in them), the resulting Artifact.json file can get very large. The deduplication is handled correctly, but it takes up more (sometimes A LOT more) space and takes a lot more time to import as the import-export engine figures out whether a row is a duplicate or not.

**To Reproduce**
```
pulp file remote create --name many --policy immediate --url https://fixtures.pulpproject.org/file-many/PULP_MANIFEST
for r in {0..100}; do   \
  pulp file repository create --name "repo-$r" --remote many  \
  pulp file repository sync --name "repo-$r"  \
done
pulp exporter pulp create --name many --path /tmp/exports/ --repository (href-of-one-repo) --repository...
pulp export pulp run --exporter many
# extract the Artifact.json from the resulting tar.gz
tar xvf  export-01899388-9d86-75d1-8da0-8c2ebf466d87-20230726_1849.tar.gz  pulpcore.app.modelresource.ArtifactResource.json
cat pulpcore.app.modelresource.ArtifactResource.json | jq '.[] | .sha256' | sort | uniq -c
```
**Expected behavior**
Artifact,json should show 1 entry per sha256.

**Actual behavior**
Artifact.json shows **number-of-repositories** copies of a given sha256.

**Additional context**
To create the exporter with all the repositories, I did something like the following:
```
# get all the repo-hrefs into a file, "hrefs"
pulp file repository list --limit 200 --field pulp_href | jq -r '.[] | .pulp_href' > hrefs
# Edit the file to start each line with ` --repository ` and end it with `\$`
# Add this as the first line
pulp exporter pulp create --name many --path /tmp/exports/ \                          
# Now run as a single command
source ./hrefs
```

There are several end-users being affected by this, even though they may not know it yet. The larger the installation, the more painful this bug becomes.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -94,14 +94,14 @@ def export_artifacts(export, artifacts):
 
     Args:
         export (django.db.models.PulpExport): export instance that's doing the export
-        artifacts (django.db.models.Artifacts): list of artifacts in all repos being exported
+        artifacts (django.db.models.Artifacts): QuerySet of artifacts in all repos being exported
 
     Raises:
         ValidationError: When path is not in the ALLOWED_EXPORT_PATHS setting
     """
     data = dict(message="Exporting Artifacts", code="export.artifacts", total=len(artifacts))
     with ProgressReport(**data) as pb:
-        for artifact in pb.iter(artifacts):
+        for artifact in artifacts.iterator():  # chunk_size= defaults to 2000 at a fetch
             dest = artifact.file.name
             if settings.DEFAULT_FILE_STORAGE != "pulpcore.app.models.storage.FileSystem":
                 with tempfile.TemporaryDirectory(dir=".") as temp_dir:
@@ -112,6 +112,7 @@ def export_artifacts(export, artifacts):
                         export.tarfile.add(temp_file.name, dest)
             else:
                 export.tarfile.add(artifact.file.path, dest)
+            pb.increment()
 
     resource = ArtifactResource()
     resource.queryset = artifacts
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -495,7 +495,7 @@ def _do_export(pulp_exporter, tar, the_export):
     starting_versions = _get_starting_versions(do_incremental, pulp_exporter, the_export)
     vers_match = _version_match(ending_versions, starting_versions)
     # Gather up versions and artifacts
-    artifacts = []
+    artifacts = None  # Will be a QuerySet selecting the Artifacts that need to be exported
     for version in ending_versions:
         # Check version-content to make sure we're not being asked to export
         # an on_demand repo
@@ -504,10 +504,15 @@ def _do_export(pulp_exporter, tar, the_export):
             raise RuntimeError(_("Remote artifacts cannot be exported."))
 
         if do_incremental:
-            vers_artifacts = version.artifacts.difference(vers_match[version].artifacts).all()
+            vers_artifacts = version.artifacts.difference(vers_match[version].artifacts)
         else:
-            vers_artifacts = version.artifacts.all()
-        artifacts.extend(vers_artifacts)
+            vers_artifacts = version.artifacts
+
+        if artifacts:
+            artifacts.union(vers_artifacts)
+        else:
+            artifacts = vers_artifacts
+
     # export plugin-version-info
     export_versions(the_export, plugin_version_info)
     # Export the top-level entities (artifacts and repositories)


Test patch: