Repository: pulp/pulpcore

Issue:
System runs out of memory when exporting repository with many packages, such as rhel 7 server rpms repo
Author: @hao-yu (hyu)


Redmine Issue: 9645, https://pulp.plan.io/issues/9645

---

Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2033847

Description of problem:

Pulp raises memory error when exporting repositories with large contents.

hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2
[....................................................................................................................................................................] [100%]
Error: Pulp task error
undefined method `first' for nil:NilClas

~~~ python
1. Traceback in /var/log/messages
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed ()  <========= Raised error without message which is normally MemoryError
...
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-7: result = func(*args, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-7: export_content(the_export, version)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-7: data = dataset.json.encode("utf8")
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__
pulpcore-worker-7: return self._format.export_set(obj, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set
pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False
pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
pulpcore-worker-7: **kw).encode(obj)
pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode
pulpcore-worker-7: return ''.join(chunks)
~~~



~~~ python
1. Add a logging to Pulp so that it will print the Exception type;
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed ()
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-6: result = func(*args, **kwargs)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-6: export_content(the_export, version)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-6: data = dataset.json.encode("utf8")
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'>  <=======
~~~


~~~ text
1. While the task is running, we can see that the memory consumption is very high

              total        used        free      shared  buff/cache   available
Mem:       33010716    19796992      221744      579536    12991980    12230452
Swap:      12058620     3917188     8141432

              total        used        free      shared  buff/cache   available
Mem:       33010716    20831688      331752      579536    11847276    11196116
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    21926716      224460      579540    10859540    10100732
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    23173152      219596      579540     9617968     8854304
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24317448      238176      579660     8455092     7709900
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    25737804      230592      579612     7042320     6289292
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    27344320      226584      579612     5439812     4683004
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    30193200      234504      579612     2583012     1834164    <=============== Observed free memory dropped to 1.8GB from 20GB
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24422544     7876732      334776      711440     7886420
Swap:      12058620     8729032     3329588             <===================== Free swap dropped 3G

              total        used        free      shared  buff/cache   available
Mem:       33010716    23213200     9030436      336976      767080     9067412
Swap:      12058620     8674504     3384116

              total        used        free      shared  buff/cache   available
Mem:       33010716    22871236     9357400      339484      782080     9400532
Swap:      12058620     8637896     3420724

              total        used        free      shared  buff/cache   available
Mem:       33010716     7216620    25034040      339872      760056    25053004   <======== Memory freed
Swap:      12058620     8431560     3627060

              total        used        free      shared  buff/cache   available
Mem:       33010716     7234448    25014092      339928      762176    25033868
Swap:      12058620     8414408     3644212
~~~



~~~ text
1. As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file

top - 00:31:24 up 1 day, 12:52,  6 users,  load average: 60.50, 26.92, 13.02
Tasks: 373 total,   1 running, 372 sleeping,   0 stopped,   0 zombie
KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
KiB Swap: 52.1/12058620 [||||||||||||||||||||||||||||||||||||||||||||||||||||                                                ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
 8442 pulp      20   0   23.9g  22.3g    636 D  17.2 70.7  13:38.63 pulpcore-worker  <=============


And it consumes 10G of RAM when writing a smaller file (I think)
KiB Mem : 56.3/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                            ]
KiB Swap: 34.2/12058620 [||||||||||||||||||||||||||||||||||                                                                  ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 8442 pulp      20   0   10.4g  10.0g    856 R  99.7 31.7  17:25.61 pulpcore-worker  <=============
~~~



~~~ text
Steps to Reproduce:
1. Enable and sync the following 2 repos

Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

2. Create a content view and add the above repos
3. Publish the content view
4. Export the content view

hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2


Actual results:
Error: Pulp task error
undefined method `first' for nil:NilClass

Expected results:
Success
~~~





Discussion:
From: pulpbot (pulpbot)
Date: 2021-12-19T12:07:07Z

---

PR: https://github.com/pulp/pulpcore/pull/1782
From: Johndavidson9 (Johndavidson9)
Date: 2022-01-10T12:30:27Z

---

I have the same issue. Any tips for solution?

https://get-9apps.com
https://cartoonhd.onl/index/

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -3,8 +3,10 @@
 import json
 import tarfile
 import tempfile
+import logging
 
 from django.conf import settings
+from django.db.models.query import QuerySet
 
 from pulpcore.app.apps import get_plugin_config
 from pulpcore.app.models.progress import ProgressReport
@@ -14,7 +16,9 @@
     ContentArtifactResource,
     RepositoryResource,
 )
-from pulpcore.constants import TASK_STATES
+from pulpcore.constants import TASK_STATES, EXPORT_BATCH_SIZE
+
+log = logging.getLogger(__name__)
 
 
 def _write_export(the_tarfile, resource, dest_dir=None):
@@ -32,16 +36,37 @@ def _write_export(the_tarfile, resource, dest_dir=None):
         dest_dir str(directory-path): directory 'inside' the tarfile to write to
     """
     filename = "{}.{}.json".format(resource.__module__, type(resource).__name__)
-    dataset = resource.export(resource.queryset)
     if dest_dir:
         dest_filename = os.path.join(dest_dir, filename)
     else:
         dest_filename = filename
 
-    data = dataset.json.encode("utf8")
-    info = tarfile.TarInfo(name=dest_filename)
-    info.size = len(data)
-    the_tarfile.addfile(info, io.BytesIO(data))
+    # If the resource is the type of QuerySet, then export the data in batch to save memory.
+    # Otherwise, export all data in oneshot. This is because the underlying libraries
+    # (json; django-import-export) do not support to stream the output to file, we export
+    # the data in batches to memory and concatenate the json lists via string manipulation.
+    with tempfile.NamedTemporaryFile(dir=os.getcwd(), mode="w", encoding="utf8") as temp_file:
+        if isinstance(resource.queryset, QuerySet):
+            temp_file.write("[")
+            total = resource.queryset.count()
+            for i in range(0, total, EXPORT_BATCH_SIZE):
+                current_batch = i + EXPORT_BATCH_SIZE
+                dataset = resource.export(resource.queryset[i:current_batch])
+                # Strip "[" and "]" as we are writing the dataset in batch
+                temp_file.write(dataset.json.lstrip("[").rstrip("]"))
+                if current_batch < total:
+                    # Write "," if not last loop
+                    temp_file.write(", ")
+            temp_file.write("]")
+        else:
+            dataset = resource.export(resource.queryset)
+            temp_file.write(dataset.json)
+
+        temp_file.flush()
+        info = tarfile.TarInfo(name=dest_filename)
+        info.size = os.path.getsize(temp_file.name)
+        with open(temp_file.name, "rb") as fd:
+            the_tarfile.addfile(info, fd)
 
 
 def export_versions(export, version_info):
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -58,7 +58,7 @@ def _import_file(fpath, resource_class, do_raise=True):
     try:
         log.info(_("Importing file {}.").format(fpath))
         with open(fpath, "r") as json_file:
-            data = Dataset().load(json_file.read(), format="json")
+            data = Dataset().load(json_file, format="json")
             resource = resource_class()
             log.info(_("...Importing resource {}.").format(resource.__class__.__name__))
             return resource.import_data(data, raise_errors=do_raise)
diff --git a/pulpcore/constants.py b/pulpcore/constants.py
--- a/pulpcore/constants.py
+++ b/pulpcore/constants.py
@@ -63,3 +63,5 @@
     (FS_EXPORT_METHODS.HARDLINK, "Export by hardlinking"),
     (FS_EXPORT_METHODS.SYMLINK, "Export by symlinking"),
 )
+
+EXPORT_BATCH_SIZE = 2000


Test patch: