Repository: pulp/pulpcore

Issue:
Can't rerun a failed content-import task if it was exported using chunks
**Version**

All?

**Describe the bug**


When importing content that was exported in chunks, the importer process concatenate the chunks into a single file in order to import.

If that import task fails for some reason, after the chunks were already combined into a single file, the user can't simply re-run the same command to retry the import. Satellite will complain that the chunks are missing.

Checking the directory, we can see that the chunks are gone and only the master file is present (together with metadata.json and TOC file).


**To Reproduce**


Steps to Reproduce:
1. Export something using chunks
2. Start the import on another satellite. Monitor the data directory until all the chunk files are gone and only the main file is present. At this moment, kill the pulpcore-worker which is processing the import. Wait until the task returns with error.
3. Repeat the same import command. Pulp will error out with an error like this:



~~~
Feb 27 16:38:56 reproducer-import pulpcore-worker-3[156191]: pulp [6c91f855-9959-43b1-864f-925393ae025a]: pulpcore.tasking.pulpcore_worker:INFO: Task ddf45eb1-adea-480e-9114-748baa7bd7bf failed ([ErrorDetail(string="Missing import-chunks named in table-of-contents: ['export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0000', 'export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0001', 'export-fc4273a4-6320-4fd5-98c9-6adfe9461781-20230227_2132.tar.gz.0002'].", code='invalid')])
~~~

Actual results:
Re-running same import fails, complaining about chunks missing. User needs to modify the TOC file manually OR split the master file in chunks again OR copy the files again in order to run the import.

**Expected behavior**

Pulp could be smart enough to identify either all the chunks are present OR the global file file. If the checksum matches, move forward with the import.


**Additional context**
https://bugzilla.redhat.com/show_bug.cgi?id=2173757



Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -337,6 +337,18 @@ def validate_toc(toc_filename):
                 raise ValidationError(_("Missing 'files' or 'meta' keys in table-of-contents!"))
 
             base_dir = os.path.dirname(toc_filename)
+
+            # Regardless of what the TOC says, it's possible for a previous import to have
+            # failed after successfully creating the combined file. If the TOC specifies multiple
+            # chunks, but the "expected result" exists, ignore the chunk-list and process as if
+            # it's all there ever was.
+            top_level_file = os.path.join(base_dir, the_toc["meta"]["file"])
+            if len(the_toc["files"]) > 1 and os.path.isfile(top_level_file):
+                the_toc["files"] = {the_toc["meta"]["file"]: the_toc["meta"]["global_hash"]}
+
+            # At this point, we either have the original chunks, or we're validating the
+            # full-file as a single chunk. Validate the hash(es).
+
             # Points at chunks that exist?
             missing_files = []
             for f in sorted(the_toc["files"].keys()):
@@ -371,17 +383,7 @@ def validate_toc(toc_filename):
 
         return the_toc
 
-    def validate_and_assemble(toc_filename):
-        """Validate checksums of, and reassemble, chunks in table-of-contents file."""
-        the_toc = validate_toc(toc_filename)
-        toc_dir = os.path.dirname(toc_filename)
-        result_file = os.path.join(toc_dir, the_toc["meta"]["file"])
-
-        # if we have only one entry in "files", it must be the full .tar.gz - return it
-        if len(the_toc["files"]) == 1:
-            return os.path.join(toc_dir, list(the_toc["files"].keys())[0])
-
-        # We have multiple chunks.
+    def reassemble(the_toc, toc_dir, result_file):
         # reassemble into one file 'next to' the toc and return the resulting full-path
         chunk_size = int(the_toc["meta"]["chunk_size"])
         offset = 0
@@ -434,6 +436,20 @@ def validate_and_assemble(toc_filename):
         # Let the rest of the import process do its thing on the new combined-file.
         return result_file
 
+    def validate_and_assemble(toc_filename):
+        """Validate checksums of, and reassemble, chunks in table-of-contents file."""
+        the_toc = validate_toc(toc_filename)
+        toc_dir = os.path.dirname(toc_filename)
+        result_file = os.path.join(toc_dir, the_toc["meta"]["file"])
+
+        # if we have only one entry in "files", it must be the full .tar.gz.
+        # Return the filename from the meta-section.
+        if len(the_toc["files"]) == 1:
+            return result_file
+
+        # We have multiple chunks. Reassemble them and return the result.
+        return reassemble(the_toc, toc_dir, result_file)
+
     if toc:
         log.info(_("Validating TOC {}.").format(toc))
         path = validate_and_assemble(toc)


Test patch: