internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
Export artifact-dedup fix from #4210 incomplete
Version
main, 18, 21, 22, 23, 28, 29

Describe the bug
The change for https://github.com/pulp/pulpcore/issues/4210 failed to completely fix the problem uncovered in pulp_rpm.

`django-importexport.export()` apparently **really** wants either a QuerySet or a **list** (specifically).

Fix and backport to the above versions.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -94,7 +94,7 @@ def export_artifacts(export, artifacts):
 
     Args:
         export (django.db.models.PulpExport): export instance that's doing the export
-        artifacts (django.db.models.Artifacts): Set of artifacts in all repos being exported
+        artifacts (django.db.models.Artifacts): List of artifacts in all repos being exported
 
     Raises:
         ValidationError: When path is not in the ALLOWED_EXPORT_PATHS setting
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -512,14 +512,13 @@ def _do_export(pulp_exporter, tar, the_export):
             vers_artifacts = version.artifacts.difference(vers_match[version].artifacts).all()
         else:
             vers_artifacts = version.artifacts.all()
-
         artifacts.update(vers_artifacts)
 
     # export plugin-version-info
     export_versions(the_export, plugin_version_info)
     # Export the top-level entities (artifacts and repositories)
     # Note: we've already handled "what about incrementals" when building the 'artifacts' list
-    export_artifacts(the_export, artifacts)
+    export_artifacts(the_export, list(artifacts))
     # Export the repository-version data, per-version
     for version in ending_versions:
         export_content(the_export, version)


Test patch:
Repository: pulp/pulpcore

Issue:
Export artifact-dedup fix from #4210 incomplete
Version
main, 18, 21, 22, 23, 28, 29

Describe the bug
The change for https://github.com/pulp/pulpcore/issues/4210 failed to completely fix the problem uncovered in pulp_rpm.

`django-importexport.export()` apparently **really** wants either a QuerySet or a **list** (specifically).

Fix and backport to the above versions.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -94,7 +94,7 @@ def export_artifacts(export, artifacts):
 
     Args:
         export (django.db.models.PulpExport): export instance that's doing the export
-        artifacts (django.db.models.Artifacts): Set of artifacts in all repos being exported
+        artifacts (django.db.models.Artifacts): List of artifacts in all repos being exported
 
     Raises:
         ValidationError: When path is not in the ALLOWED_EXPORT_PATHS setting
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -507,14 +507,13 @@ def _do_export(pulp_exporter, tar, the_export):
             vers_artifacts = version.artifacts.difference(vers_match[version].artifacts).all()
         else:
             vers_artifacts = version.artifacts.all()
-
         artifacts.update(vers_artifacts)
 
     # export plugin-version-info
     export_versions(the_export, plugin_version_info)
     # Export the top-level entities (artifacts and repositories)
     # Note: we've already handled "what about incrementals" when building the 'artifacts' list
-    export_artifacts(the_export, artifacts)
+    export_artifacts(the_export, list(artifacts))
     # Export the repository-version data, per-version
     for version in ending_versions:
         export_content(the_export, version)


Test patch:
Repository: pulp/pulpcore

Issue:
Import-check may return error saying "director does not exist" when it actually just lack permissions
Endpoint /pulp/api/v3/importers/core/pulp/import-check/ returns error saying "Directory does not exist" when pulp user lack permissions to read said directory.

**To Reproduce**

Try importing content from a directory where pulp user doesn't have read access.

**Expected behavior**
Error returned should indicate the permission error.

**Additional context**

Pulp is using os.path.exists() method to verify if the directory exists: https://github.com/pulp/pulpcore/blob/main/pulpcore/app/views/importer.py#L44-L45

However, the method can return false if permission is not granted to access the directory even if the directory exists

~~~
os.path.exists(path)
Return True if path refers to an existing path or an open file descriptor. Returns False for broken symbolic links. On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
~~~

os.path method documentation -> https://docs.python.org/3/library/os.path.html



Discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=2232625

Resolving patch:
diff --git a/pulpcore/app/views/importer.py b/pulpcore/app/views/importer.py
--- a/pulpcore/app/views/importer.py
+++ b/pulpcore/app/views/importer.py
@@ -40,11 +40,14 @@ def _validate_file(in_param, data):
         return rc, [msg]
 
     # check directory-sanity, leave if failed
+    # use os.stat to ensure directory exists and pulp has read-access
+    # return any errors received from os.stat to the user
+
     owning_dir = os.path.dirname(real_file)
-    if not os.path.exists(owning_dir):
-        return False, [_("directory {} does not exist").format(owning_dir)]
-    if not os.access(owning_dir, os.R_OK):
-        return False, [_("directory {} does not allow read-access").format(owning_dir)]
+    try:
+        os.stat(owning_dir)
+    except OSError as e:
+        return False, [_("{}").format(e)]
 
     # check file-exists, leave if failed
     if not os.path.exists(real_file):


Test patch:
Repository: pulp/pulpcore

Issue:
Distribution Serializer consults the instance when validating even when not partial
This, for example prevents you to change a distribution from using a repository to using a publication in a single PUT call.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/publication.py b/pulpcore/app/serializers/publication.py
--- a/pulpcore/app/serializers/publication.py
+++ b/pulpcore/app/serializers/publication.py
@@ -240,13 +240,13 @@ def validate(self, data):
         super().validate(data)
 
         repository_provided = data.get(
-            "repository", (self.instance and self.instance.repository) or None
+            "repository", (self.partial and self.instance.repository) or None
         )
         repository_version_provided = data.get(
-            "repository_version", (self.instance and self.instance.repository_version) or None
+            "repository_version", (self.partial and self.instance.repository_version) or None
         )
         publication_provided = data.get(
-            "publication", (self.instance and self.instance.publication) or None
+            "publication", (self.partial and self.instance.publication) or None
         )
 
         if publication_provided and repository_version_provided:


Test patch:
Repository: pulp/pulpcore

Issue:
Burst mode for pulpcore-worker
When upgrading, it is sometimes not safe to bring "old" task unfinished to the new version. Though we have a backwards compatibility claim, some admins may decide they want to drain the task queue anyway.
Providing a "--burst" option to the pulpcore worker that will tell it to exit instead of going to sleep will help there.
Also this may be useful in case you have a system stuck on a bunch of sync/import/export tasks to add a few workers that will disconnect themselves once the congestion is over.


Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/entrypoint.py b/pulpcore/tasking/entrypoint.py
--- a/pulpcore/tasking/entrypoint.py
+++ b/pulpcore/tasking/entrypoint.py
@@ -14,9 +14,12 @@
 _logger = logging.getLogger(__name__)
 
 
-@click.option("--pid", help="Write the process ID number to a file at the specified path")
+@click.option("--pid", help="Write the process ID number to a file at the specified path.")
+@click.option(
+    "--burst/--no-burst", help="Run in burst mode; terminate when no more tasks are available."
+)
 @click.command()
-def worker(pid):
+def worker(pid, burst):
     """A Pulp worker."""
 
     if pid:
@@ -25,4 +28,4 @@ def worker(pid):
 
     _logger.info("Starting distributed type worker")
 
-    NewPulpWorker().run_forever()
+    NewPulpWorker().run(burst=burst)
diff --git a/pulpcore/tasking/pulpcore_worker.py b/pulpcore/tasking/pulpcore_worker.py
--- a/pulpcore/tasking/pulpcore_worker.py
+++ b/pulpcore/tasking/pulpcore_worker.py
@@ -393,22 +393,26 @@ def supervise_task(self, task):
             self.notify_workers()
         self.task = None
 
-    def run_forever(self):
+    def run(self, burst=False):
         with WorkerDirectory(self.name):
             signal.signal(signal.SIGINT, self._signal_handler)
             signal.signal(signal.SIGTERM, self._signal_handler)
             signal.signal(signal.SIGHUP, self._signal_handler)
             # Subscribe to pgsql channels
             connection.connection.add_notify_handler(self._pg_notify_handler)
-            self.cursor.execute("LISTEN pulp_worker_wakeup")
             self.cursor.execute("LISTEN pulp_worker_cancel")
-            while not self.shutdown_requested:
+            if burst:
                 for task in self.iter_tasks():
                     self.supervise_task(task)
-                if not self.shutdown_requested:
-                    self.sleep()
+            else:
+                self.cursor.execute("LISTEN pulp_worker_wakeup")
+                while not self.shutdown_requested:
+                    for task in self.iter_tasks():
+                        self.supervise_task(task)
+                    if not self.shutdown_requested:
+                        self.sleep()
+                self.cursor.execute("UNLISTEN pulp_worker_wakeup")
             self.cursor.execute("UNLISTEN pulp_worker_cancel")
-            self.cursor.execute("UNLISTEN pulp_worker_wakeup")
             self.shutdown()
 
 


Test patch:
Repository: pulp/pulpcore

Issue:
rotate-db-key corrupts data
There are two issues:
1. Our encrypted JSON/text fields do not work with bulk_update.
2. EncryptedJSONField uses repr and eval for transformations which is not the exact inverse.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/fields.py b/pulpcore/app/models/fields.py
--- a/pulpcore/app/models/fields.py
+++ b/pulpcore/app/models/fields.py
@@ -1,3 +1,4 @@
+import json
 import logging
 import os
 from gettext import gettext as _
@@ -94,14 +95,16 @@ def _fernet(self):
         with open(settings.DB_ENCRYPTION_KEY, "rb") as key_file:
             return Fernet(key_file.read())
 
-    def get_db_prep_save(self, value, connection):
-        value = super().get_db_prep_save(value, connection)
+    def get_prep_value(self, value):
         if value is not None:
-            return force_str(self._fernet.encrypt(force_bytes(value)))
+            assert isinstance(value, str)
+            value = force_str(self._fernet.encrypt(force_bytes(value)))
+        return super().get_prep_value(value)
 
     def from_db_value(self, value, expression, connection):
         if value is not None:
-            return force_str(self._fernet.decrypt(force_bytes(value)))
+            value = force_str(self._fernet.decrypt(force_bytes(value)))
+        return value
 
 
 class EncryptedJSONField(JSONField):
@@ -136,16 +139,23 @@ def decrypt(self, value):
         elif isinstance(value, (list, tuple, set)):
             return [self.decrypt(v) for v in value]
 
-        return eval(force_str(self._fernet.decrypt(force_bytes(value))))
+        dec_value = force_str(self._fernet.decrypt(force_bytes(value)))
+        try:
+            return json.loads(dec_value, cls=self.decoder)
+        except json.JSONDecodeError:
+            return eval(dec_value)
 
-    def get_db_prep_save(self, value, connection):
-        value = self.encrypt(value)
-        return super().get_db_prep_save(value, connection)
+    def get_prep_value(self, value):
+        if value is not None:
+            if hasattr(value, "as_sql"):
+                return value
+            value = self.encrypt(value)
+        return super().get_prep_value(value)
 
     def from_db_value(self, value, expression, connection):
         if value is not None:
-            value = super().from_db_value(value, expression, connection)
-            return self.decrypt(value)
+            value = self.decrypt(super().from_db_value(value, expression, connection))
+        return value
 
 
 @Field.register_lookup


Test patch:
Repository: pulp/pulpcore

Issue:
missing command line options in new app entrypoint
**Version**
3.33.0

**Describe the bug**
in 4def68e1f1bd4ffcae5352ee9d6ff7404759a19e a new app entrypoint was added to be used instead of calling gunicorn directly, however it is missing (at least) the `--preload` switch that gunicorn supports and Katello uses

https://github.com/benoitc/gunicorn/blob/master/gunicorn/config.py#L992

https://github.com/theforeman/puppet-pulpcore/commit/b8189466629b05df91fe90fbb7f741e2e9c27786

**To Reproduce**
try to call `pulpcore-content` with `--preload`

**Expected behavior**
app boots

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/entrypoint.py b/pulpcore/app/entrypoint.py
--- a/pulpcore/app/entrypoint.py
+++ b/pulpcore/app/entrypoint.py
@@ -69,7 +69,6 @@ def init_process(self):
             "Api App '{name}' failed to write a heartbeat to the database, sleeping for "
             "'{interarrival}' seconds."
         ).format(name=self.name, interarrival=self.timeout)
-        using_pulp_api_worker.set(True)
         super().init_process()
 
     def run(self):
@@ -96,8 +95,11 @@ def load_config(self):
         self.cfg.set("worker_class", PulpApiWorker.__module__ + "." + PulpApiWorker.__qualname__)
 
     def load(self):
+        using_pulp_api_worker.set(True)
+
         import pulpcore.app.wsgi
 
+        using_pulp_api_worker.set(False)
         return pulpcore.app.wsgi.application
 
 
@@ -132,6 +134,7 @@ def load(self):
 @click.option("--reload/--no-reload")
 @click.option("--reload-engine", type=click.Choice(["auto", "poll", "inotify"]))
 @click.option("--reload-extra-file", "reload_extra_files", multiple=True)
+@click.option("--preload/--no-preload", "preload_app")
 @click.option("--reuse-port/--no-reuse-port")
 @click.option("--chdir")
 @click.option("--user", "-u")
diff --git a/pulpcore/content/entrypoint.py b/pulpcore/content/entrypoint.py
--- a/pulpcore/content/entrypoint.py
+++ b/pulpcore/content/entrypoint.py
@@ -42,6 +42,7 @@ def load(self):
 @click.option("--reload/--no-reload")
 @click.option("--reload-engine", type=click.Choice(["auto", "poll", "inotify"]))
 @click.option("--reload-extra-file", "reload_extra_files", multiple=True)
+@click.option("--preload/--no-preload", "preload_app")
 @click.option("--reuse-port/--no-reuse-port")
 @click.option("--chdir")
 @click.option("--user", "-u")


Test patch:
Repository: pulp/pulpcore

Issue:
Export artifact-dedup optimizations break pulp_rpm export.
**Version**
main, 18, 21, 22, 23, 28, 29

**Describe the bug**
After the changes for #4159 were released, pulp_rpm nightly tests started failing. This turns out to be the result of a subtle bug in the optimizations introduced for that fix.

This has to be fixed and released ASAP to all versions #4159 was releasxed into.

**To Reproduce**
* Run https://github.com/pulp/pulp_rpm/blob/main/pulp_rpm/tests/functional/api/test_pulpimport.py#L287





Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -94,7 +94,7 @@ def export_artifacts(export, artifacts):
 
     Args:
         export (django.db.models.PulpExport): export instance that's doing the export
-        artifacts (django.db.models.Artifacts): list of artifacts in all repos being exported
+        artifacts (django.db.models.Artifacts): Set of artifacts in all repos being exported
 
     Raises:
         ValidationError: When path is not in the ALLOWED_EXPORT_PATHS setting
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -401,7 +401,7 @@ def _do_export(pulp_exporter, tar, the_export):
     starting_versions = _get_starting_versions(do_incremental, pulp_exporter, the_export)
     vers_match = _version_match(ending_versions, starting_versions)
     # Gather up versions and artifacts
-    artifacts = []
+    artifacts = set()
     for version in ending_versions:
         # Check version-content to make sure we're not being asked to export
         # an on_demand repo
@@ -413,7 +413,9 @@ def _do_export(pulp_exporter, tar, the_export):
             vers_artifacts = version.artifacts.difference(vers_match[version].artifacts).all()
         else:
             vers_artifacts = version.artifacts.all()
-        artifacts.extend(vers_artifacts)
+
+        artifacts.update(vers_artifacts)
+
     # export plugin-version-info
     export_versions(the_export, plugin_version_info)
     # Export the top-level entities (artifacts and repositories)


Test patch:
Repository: pulp/pulpcore

Issue:
Also set the compression level when not using chunked exports
We didn't use the `compresslevel` argument because it didn't work in streaming mode, so we went for a monkeypatch.  But the monkeypatch doesn't apply to non-streaming mode (oversight on my part)... which we also use, in the non-chunked path.

We don't actually need a monkeypatch on the other side because the argument `compresslevel=1` is supported for non-streaming mode, but we do still have to pass that arg, so let's do that.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -3,6 +3,7 @@
 import logging
 import os
 import os.path
+import sys
 import subprocess
 import tarfile
 
@@ -414,8 +415,15 @@ def pulp_export(exporter_pk, params):
                 stdin=subprocess.PIPE,
             ) as split_process:
                 try:
-                    with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
-                        _do_export(pulp_exporter, tar, the_export)
+                    # on Python < 3.12 we have a monkeypatch which enables compression levels
+                    if sys.version_info.major == 3 and sys.version_info.minor < 12:
+                        with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
+                    else:
+                        with tarfile.open(
+                            tarfile_fp, "w|gz", fileobj=split_process.stdin, compresslevel=1
+                        ) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
                 except Exception:
                     # no matter what went wrong, we can't trust the files we (may have) created.
                     # Delete the ones we can find and pass the problem up.
@@ -433,7 +441,7 @@ def pulp_export(exporter_pk, params):
         else:
             # write into the file
             try:
-                with tarfile.open(tarfile_fp, "w:gz") as tar:
+                with tarfile.open(tarfile_fp, "w:gz", compresslevel=1) as tar:
                     _do_export(pulp_exporter, tar, the_export)
             except Exception:
                 # no matter what went wrong, we can't trust the file we created.


Test patch:
Repository: pulp/pulpcore

Issue:
Also set the compression level when not using chunked exports
We didn't use the `compresslevel` argument because it didn't work in streaming mode, so we went for a monkeypatch.  But the monkeypatch doesn't apply to non-streaming mode (oversight on my part)... which we also use, in the non-chunked path.

We don't actually need a monkeypatch on the other side because the argument `compresslevel=1` is supported for non-streaming mode, but we do still have to pass that arg, so let's do that.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -3,6 +3,7 @@
 import logging
 import os
 import os.path
+import sys
 import subprocess
 import tarfile
 
@@ -311,8 +312,15 @@ def pulp_export(exporter_pk, params):
                 stdin=subprocess.PIPE,
             ) as split_process:
                 try:
-                    with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
-                        _do_export(pulp_exporter, tar, the_export)
+                    # on Python < 3.12 we have a monkeypatch which enables compression levels
+                    if sys.version_info.major == 3 and sys.version_info.minor < 12:
+                        with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
+                    else:
+                        with tarfile.open(
+                            tarfile_fp, "w|gz", fileobj=split_process.stdin, compresslevel=1
+                        ) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
                 except Exception:
                     # no matter what went wrong, we can't trust the files we (may have) created.
                     # Delete the ones we can find and pass the problem up.
@@ -330,7 +338,7 @@ def pulp_export(exporter_pk, params):
         else:
             # write into the file
             try:
-                with tarfile.open(tarfile_fp, "w:gz") as tar:
+                with tarfile.open(tarfile_fp, "w:gz", compresslevel=1) as tar:
                     _do_export(pulp_exporter, tar, the_export)
             except Exception:
                 # no matter what went wrong, we can't trust the file we created.


Test patch: