Repository: pulp/pulpcore

Issue:
Uploading large files fails on the "Too many open files" error
Author: @lubosmj (lmjachky)


Redmine Issue: 9634, https://pulp.plan.io/issues/9634

---

Steps to reproduce the behaviour:
```
(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$  truncate -s 3G gentoo_root.img

(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$ pulp artifact upload --file gentoo_root.img
Uploading file gentoo_root.img
................[truncated the number of dots]...................Upload complete. Creating artifact.
Started background task /pulp/api/v3/tasks/2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30/
........Error: Task /pulp/api/v3/tasks/2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30/ failed: '[Errno 24] Too many open files: '/var/lib/pulp/media/upload/ec2d5c92-57e9-4191-ae95-ff4b9ec94353''

(pulp) [vagrant@pulp3-source-fedora34 pulp_ostree]$ ls -la gentoo_root.img
-rw-r--r--. 1 vagrant vagrant 3221225472 Dec 13 11:32 gentoo_root.img
```

Traceback:
```
Task 2d3cf569-2d5c-449d-9ac6-7a53ab5f3a30 failed ([Errno 24] Too many open files: '/var/lib/pulp/media/upload/ec2d5c92-57e9-4191-ae95-ff4b9ec94353')
pulp [3a3a9ea662994f609eea7d43ac8f30aa]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 362, in _perform_task
    result = func(*args, **kwargs)
  File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/upload.py", line 31, in commit
    temp_file.write(chunk.file.read())
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/utils.py", line 42, in <lambda>
    read = property(lambda self: self.file.read)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/fields/files.py", line 45, in _get_file
    self._file = self.storage.open(self.name, 'rb')
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/files/storage.py", line 238, in _open
    return File(open(self.path(name), mode))
```





Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -83,6 +83,7 @@ def export_artifacts(export, artifacts):
                     with tempfile.NamedTemporaryFile(dir=temp_dir) as temp_file:
                         temp_file.write(artifact.file.read())
                         temp_file.flush()
+                        artifact.file.close()
                         export.tarfile.add(temp_file.name, dest)
             else:
                 export.tarfile.add(artifact.file.path, dest)
diff --git a/pulpcore/app/tasks/upload.py b/pulpcore/app/tasks/upload.py
--- a/pulpcore/app/tasks/upload.py
+++ b/pulpcore/app/tasks/upload.py
@@ -29,6 +29,7 @@ def commit(upload_id, sha256):
     with NamedTemporaryFile("ab") as temp_file:
         for chunk in chunks:
             temp_file.write(chunk.file.read())
+            chunk.file.close()
         temp_file.flush()
 
         file = files.PulpTemporaryUploadedFile.from_file(File(open(temp_file.name, "rb")))


Test patch: