Repository: pulp/pulpcore

Issue:
Memory error when importing large repository
**Description of problem:**
Getting MemoryError when importing large repository, such as rhel-7-server-rpms repo.

The PackageResource.json file of the rhel-7-server-rpms repo is about 5.5GB. Pulp uses "json.load" method to decode the json file. When decoding the json, it will use up to 18GB+ memory and returns a 5.5GB python dictionary. If the system doesn't have enough memory, the json decode will fail with MemoryError.


hammer content-import version --path /var/lib/pulp/imports/Default_Organization/rhel-7-imported/1.0/2022-03-04T17-17-09-11-00/ --organization-id 1
[................................................                                                                                                                                                           ] [24%]
Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/ebc7514a-f606-4965-a599-eab74101e9b0/.


**/var/log/messages**
pulpcore-worker-2: pulp [6eda5a34-529f-42b9-909b-fb526c9f35e0]: pulpcore.tasking.pulpcore_worker:INFO: Task d5f26164-087a-4acb-aca1-7aed30851040 failed ()
pulpcore-worker-2: pulp [6eda5a34-529f-42b9-909b-fb526c9f35e0]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-2: result = func(*args, **kwargs)
pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 161, in import_repository_version
pulpcore-worker-2: a_result = _import_file(os.path.join(rv_path, filename), res_class, do_raise=False)
pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/importer.py", line 62, in _import_file
pulpcore-worker-2: data = Dataset().load(json_file.read(), format="json")
pulpcore-worker-2: File "/usr/lib64/python3.6/codecs.py", line 321, in decode
pulpcore-worker-2: (result, consumed) = self._buffer_decode(data, self.errors, final)


**Output of top command**
16011 pulp      20   0 5924132   2.5g   2496 D  25.0 16.7  21:43.78 pulpcore-worker                                                                                                                                                                                                                                                              
16011 pulp      20   0 5924132   2.8g   2496 D   6.2 18.7  21:44.76 pulpcore-worker                                                                                                                                                                                                                                                             
16011 pulp      20   0 5924132   3.1g   2496 D  12.5 20.7  21:45.55 pulpcore-worker                                                                                                                                                                                                                                                              
16011 pulp      20   0 5924132   3.5g   2496 D  12.5 23.2  21:46.38 pulpcore-worker                                                                                                                                                                                                                                                               
16011 pulp      20   0 5924132   3.9g   2496 R  25.0 25.7  21:47.39 pulpcore-worker                                                                                                                                                                                                                                                               
16011 pulp      20   0 5924132   4.2g   2496 D  12.5 27.7  21:48.38 pulpcore-worker                                                                                                                                                                                                                                                            
16011 pulp      20   0 5924132   4.5g   2496 R  66.7 29.5  21:49.35 pulpcore-worker                                                                                                                                                                                                                                                               
16011 pulp      20   0 5924132   4.9g   2496 D  18.8 32.2  21:50.58 pulpcore-worker                                                                                                                                                                                                                                                               
16011 pulp      20   0 5924132   5.3g   2496 R  26.7 34.7  21:51.53 pulpcore-worker


**Output of free command**
              total        used        free      shared  buff/cache   available
Mem:       15879256    10213472      167648      193168     5498136     5135336
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    10516964      162816      193168     5199476     4832256
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    10774956      179184      193168     4925116     4573800
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    11009476      168300      193168     4701480     4339684
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    11354424      179644      193168     4345188     3994460
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    11653416      154880      193168     4070960     3695840
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    11960864      150900      193168     3767492     3388684
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    12341972      150652      193168     3386632     3007040
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    12741088      157716      193168     2980452     2608400
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    13012212      159016      193168     2708028     2337108
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    13309296      171804      193168     2398156     2039872
Swap:       8060924         352     8060572
              total        used        free      shared  buff/cache   available
Mem:       15879256    13726688      180136      193168     1972432     1622400
              total        used        free      shared  buff/cache   available
Mem:       15879256    14151480      169480      193168     1558296     1197956
Swap:       8060924         352     8060572
 

**Test to decode the PackageResource.json file in the python console.**
```
python3
>>> fh = open("/var/lib/pulp/imports/Default_Organization/rhel-7-imported/1.0/2022-03-04T17-17-09-11-00/repository-Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server_13/pulp_rpm.app.modelresource.PackageResource.json")
>>> import json
>>> json.load(fh)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/json/__init__.py", line 296, in load
    return loads(fp.read(),
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError  <===========
```

```
top  -o %MEM
top - 11:50:42 up 1 day, 14:42,  3 users,  load average: 0.49, 0.15, 0.09
Tasks: 134 total,   3 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  3.3 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  3.7 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  7.0 sy,  0.0 ni, 93.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 31.8 us, 68.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 99.2/19906644 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
KiB Swap:  1.0/8060924  [|                                                                                                   ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                        
10034 root      20   0   20.8g  18.1g   3268 R 100.0 95.4   1:23.00 python3  <==========
```

**Steps to Reproduce:**
1. Prepare a disconnected Satellite with only 15GB RAM to easily reproduce the issue
2. Import a content view with only the rhel-7-server-rpms repo to the disconnected Satellite


Actual results:
Failed with memory error

Expected results:
No error and should consume only reasonable amount of memory.

**Research and Proposing potential solutions:**
1) Split the resource.json file into a number of chunk files when exporting the contents. 
2) Or still have one large resource.json (non-standard json file). Each chunk is split into newline so that import can read them line by line.
3) Or use a non-standard json library, such as ijson to read the json file in chunk to save memory.


https://bugzilla.redhat.com/show_bug.cgi?id=2061224


Discussion:
Is this issue distinct from https://github.com/pulp/pulpcore/issues/2072?

What version are you seeing this on?  Is there a BZ that can be attached?
@dralley: Yes. This is different issue with #2072.  #2072 fixed the export part only but unfortunately not the import part.  The import still having memory issue after fixing the below line.

https://github.com/pulp/pulpcore/pull/1782/files#diff-06badc442eb525dc6f2de7f4e19a7762ce20a972b79b6a90cf1ee80211053ab5L62-R62


I had created a bugzilla for it:

https://bugzilla.redhat.com/show_bug.cgi?id=2061224

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -8,10 +8,10 @@
 from gettext import gettext as _
 from logging import getLogger
 
-from django.conf import settings
 from django.core.files.storage import default_storage
 from django.db.models import F
-
+from naya.json import stream_array, tokenize
+from io import StringIO
 from pkg_resources import DistributionNotFound, get_distribution
 from rest_framework.serializers import ValidationError
 from tablib import Dataset
@@ -44,6 +44,8 @@
 CA_FILE = "pulpcore.app.modelresource.ContentArtifactResource.json"
 VERSIONS_FILE = "versions.json"
 CONTENT_MAPPING_FILE = "content_mapping.json"
+# How many entities from an import-file should be processed at one time
+IMPORT_BATCH_SIZE = 100
 
 # Concurrent imports w/ overlapping content can collide - how many attempts are we willing to
 # make before we decide this is a fatal error?
@@ -59,22 +61,52 @@ def _destination_repo(importer, source_repo_name):
     return Repository.objects.get(name=dest_repo_name)
 
 
+def _impfile_iterator(fd):
+    """
+    Iterate over an import-file returning batches of rows as a json-array-string.
+
+    We use naya.json.stream_array() to get individual rows; once a batch is gathered,
+    we yield the result of json.dumps() for that batch. Repeat until all rows have been
+    called for.
+    """
+    eof = False
+    batch = []
+    rows = stream_array(tokenize(fd))
+    while not eof:
+        try:
+            while len(batch) < IMPORT_BATCH_SIZE:
+                batch.append(next(rows))
+        except StopIteration:
+            eof = True
+        yield json.dumps(batch)
+        batch.clear()
+
+
 def _import_file(fpath, resource_class, retry=False):
+    """
+    Import the specified resource-file in batches to limit memory-use.
+
+    We process resource-files one "batch" at a time. Because of the way django-import's
+    internals work, we have to feed it batches as StringIO-streams of json-formatted strings.
+    The file-to-json-to-string-to-import is ovehead, but it lets us put an upper bound on the
+    number of entities in memory at any one time at import-time.
+    """
     try:
-        log.info(_("Importing file {}.").format(fpath))
+        log.info(f"Importing file {fpath}.")
         with open(fpath, "r") as json_file:
-            data = Dataset().load(json_file.read(), format="json")
             resource = resource_class()
-            log.info(_("...Importing resource {}.").format(resource.__class__.__name__))
-            if retry:
-                curr_attempt = 1
-                while curr_attempt < MAX_ATTEMPTS:
-                    curr_attempt += 1
+            log.info("...Importing resource {resource.__class__.__name__}.")
+            # Load one batch-sized chunk of the specified import-file at a time. If requested,
+            # retry a batch if it looks like we collided with some other repo being imported with
+            # overlapping content.
+            for batch_str in _impfile_iterator(json_file):
+                data = Dataset().load(StringIO(batch_str))
+                if retry:
                     # django import-export can have a problem with concurrent-imports that are
                     # importing the same 'thing' (e.g., a Package that exists in two different
                     # repo-versions that are being imported at the same time). If we're asked to
                     # retry, we will try an import that will simply record errors as they happen
-                    # (rather than failing with an exception) first. If errors happen, we'll
+                    # (rather than failing with an exception) first. If errors happen, we'll do one
                     # retry before we give up on this repo-version's import.
                     a_result = resource.import_data(data, raise_errors=False)
                     if a_result.has_errors():
@@ -83,16 +115,16 @@ def _import_file(fpath, resource_class, retry=False):
                             f"...{total_errors} import-errors encountered importing "
                             "{fpath}, attempt {curr_attempt}, retrying"
                         )
-                # Last attempt, we raise an exception on any problem.
-                # This will either succeed, or log a fatal error and fail.
-                try:
+                        # Second attempt, we raise an exception on any problem.
+                        # This will either succeed, or log a fatal error and fail.
+                        try:
+                            a_result = resource.import_data(data, raise_errors=True)
+                        except Exception as e:  # noqa log on ANY exception and then re-raise
+                            log.error(f"FATAL import-failure importing {fpath}")
+                            raise
+                else:
                     a_result = resource.import_data(data, raise_errors=True)
-                except Exception as e:  # noqa log on ANY exception and then re-raise
-                    log.error(f"FATAL import-failure importing {fpath}")
-                    raise
-            else:
-                a_result = resource.import_data(data, raise_errors=True)
-            return a_result
+                yield a_result
     except AttributeError:
         log.error(f"FAILURE loading import-file {fpath}!")
         raise
@@ -195,15 +227,19 @@ def import_repository_version(importer_pk, destination_repo_pk, source_repo_name
 
         resulting_content_ids = []
         for res_class in cfg.exportable_classes:
+            content_count = 0
             filename = f"{res_class.__module__}.{res_class.__name__}.json"
-            a_result = _import_file(os.path.join(rv_path, filename), res_class, retry=True)
-            resulting_content_ids.extend(
-                row.object_id for row in a_result.rows if row.import_type in ("new", "update")
-            )
+            for a_result in _import_file(os.path.join(rv_path, filename), res_class, retry=True):
+                content_count += len(a_result.rows)
+                resulting_content_ids.extend(
+                    row.object_id for row in a_result.rows if row.import_type in ("new", "update")
+                )
 
         # Once all content exists, create the ContentArtifact links
         ca_path = os.path.join(rv_path, CA_FILE)
-        _import_file(ca_path, ContentArtifactResource, retry=True)
+        # We don't do anything with the imported batches, we just need to get them imported
+        for a_batch in _import_file(ca_path, ContentArtifactResource, retry=True):
+            pass
 
         # see if we have a content mapping
         mapping_path = f"{rv_name}/{CONTENT_MAPPING_FILE}"
@@ -404,21 +440,24 @@ def validate_and_assemble(toc_filename):
             _check_versions(version_json)
 
         # Artifacts
-        ar_result = _import_file(os.path.join(temp_dir, ARTIFACT_FILE), ArtifactResource)
         data = dict(
-            message="Importing Artifacts", code="import.artifacts", total=len(ar_result.rows)
+            message="Importing Artifacts",
+            code="import.artifacts",
         )
         with ProgressReport(**data) as pb:
-            for row in pb.iter(ar_result.rows):
-                artifact = Artifact.objects.get(pk=row.object_id)
-                base_path = os.path.join("artifact", artifact.sha256[0:2], artifact.sha256[2:])
-                src = os.path.join(temp_dir, base_path)
-                dest = os.path.join(settings.MEDIA_ROOT, base_path)
-
-                if not default_storage.exists(dest):
-                    with open(src, "rb") as f:
-                        default_storage.save(dest, f)
-
+            # Import artifacts, and place their binary blobs, one batch at a time.
+            # Skip artifacts that already exist in storage.
+            for ar_result in _import_file(os.path.join(temp_dir, ARTIFACT_FILE), ArtifactResource):
+                for row in pb.iter(ar_result.rows):
+                    artifact = Artifact.objects.get(pk=row.object_id)
+                    base_path = os.path.join("artifact", artifact.sha256[0:2], artifact.sha256[2:])
+                    src = os.path.join(temp_dir, base_path)
+
+                    if not default_storage.exists(base_path):
+                        with open(src, "rb") as f:
+                            default_storage.save(base_path, f)
+
+        # Now import repositories, in parallel.
         with open(os.path.join(temp_dir, REPO_FILE), "r") as repo_data_file:
             data = json.load(repo_data_file)
             gpr = GroupProgressReport(
@@ -444,7 +483,7 @@ def validate_and_assemble(toc_filename):
                 dispatch(
                     import_repository_version,
                     exclusive_resources=[dest_repo],
-                    args=[importer.pk, dest_repo.pk, src_repo["name"], path],
+                    args=(importer.pk, dest_repo.pk, src_repo["name"], path),
                     task_group=task_group,
                 )
 


Test patch: