internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
Also set the compression level when not using chunked exports
We didn't use the `compresslevel` argument because it didn't work in streaming mode, so we went for a monkeypatch.  But the monkeypatch doesn't apply to non-streaming mode (oversight on my part)... which we also use, in the non-chunked path.

We don't actually need a monkeypatch on the other side because the argument `compresslevel=1` is supported for non-streaming mode, but we do still have to pass that arg, so let's do that.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -3,6 +3,7 @@
 import logging
 import os
 import os.path
+import sys
 import subprocess
 import tarfile
 
@@ -409,8 +410,15 @@ def pulp_export(exporter_pk, params):
                 stdin=subprocess.PIPE,
             ) as split_process:
                 try:
-                    with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
-                        _do_export(pulp_exporter, tar, the_export)
+                    # on Python < 3.12 we have a monkeypatch which enables compression levels
+                    if sys.version_info.major == 3 and sys.version_info.minor < 12:
+                        with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
+                    else:
+                        with tarfile.open(
+                            tarfile_fp, "w|gz", fileobj=split_process.stdin, compresslevel=1
+                        ) as tar:
+                            _do_export(pulp_exporter, tar, the_export)
                 except Exception:
                     # no matter what went wrong, we can't trust the files we (may have) created.
                     # Delete the ones we can find and pass the problem up.
@@ -428,7 +436,7 @@ def pulp_export(exporter_pk, params):
         else:
             # write into the file
             try:
-                with tarfile.open(tarfile_fp, "w:gz") as tar:
+                with tarfile.open(tarfile_fp, "w:gz", compresslevel=1) as tar:
                     _do_export(pulp_exporter, tar, the_export)
             except Exception:
                 # no matter what went wrong, we can't trust the file we created.


Test patch:
Repository: pulp/pulpcore

Issue:
Export artifact-dedup fix from #4210 incomplete
Version
main, 18, 21, 22, 23, 28, 29

Describe the bug
The change for https://github.com/pulp/pulpcore/issues/4210 failed to completely fix the problem uncovered in pulp_rpm.

`django-importexport.export()` apparently **really** wants either a QuerySet or a **list** (specifically).

Fix and backport to the above versions.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/importexport.py b/pulpcore/app/importexport.py
--- a/pulpcore/app/importexport.py
+++ b/pulpcore/app/importexport.py
@@ -94,7 +94,7 @@ def export_artifacts(export, artifacts):
 
     Args:
         export (django.db.models.PulpExport): export instance that's doing the export
-        artifacts (django.db.models.Artifacts): Set of artifacts in all repos being exported
+        artifacts (django.db.models.Artifacts): List of artifacts in all repos being exported
 
     Raises:
         ValidationError: When path is not in the ALLOWED_EXPORT_PATHS setting
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -413,14 +413,13 @@ def _do_export(pulp_exporter, tar, the_export):
             vers_artifacts = version.artifacts.difference(vers_match[version].artifacts).all()
         else:
             vers_artifacts = version.artifacts.all()
-
         artifacts.update(vers_artifacts)
 
     # export plugin-version-info
     export_versions(the_export, plugin_version_info)
     # Export the top-level entities (artifacts and repositories)
     # Note: we've already handled "what about incrementals" when building the 'artifacts' list
-    export_artifacts(the_export, artifacts)
+    export_artifacts(the_export, list(artifacts))
     # Export the repository-version data, per-version
     for version in ending_versions:
         export_content(the_export, version)


Test patch:
Repository: pulp/pulpcore

Issue:
As a user I can list repos whose name match a regex
I'd like to be able to hit the `/pulp/api/v3/repositories/` endpoint with a filter like `name__regex` or `name__iregex` to list repos whose name match a particular regex. We have hundreds of repos so this would be a much welcome feature.

https://docs.djangoproject.com/en/4.2/ref/models/querysets/#regex
https://docs.djangoproject.com/en/4.2/ref/models/querysets/#iregex


Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/base.py b/pulpcore/app/viewsets/base.py
--- a/pulpcore/app/viewsets/base.py
+++ b/pulpcore/app/viewsets/base.py
@@ -39,6 +39,8 @@
     "icontains",
     "startswith",
     "istartswith",
+    "regex",
+    "iregex",
 ]
 # e.g.
 # /?name=foo


Test patch:
diff --git a/pulpcore/tests/functional/api/test_repos.py b/pulpcore/tests/functional/api/test_repos.py
--- a/pulpcore/tests/functional/api/test_repos.py
+++ b/pulpcore/tests/functional/api/test_repos.py
@@ -48,3 +48,22 @@ def test_repository_content_filters(
     # but not in its latest version anymore
     results = file_repository_api_client.list(latest_with_content=content.pulp_href).results
     assert results == []
+
+
+@pytest.mark.parallel
+def test_repository_name_regex_filters(file_repository_factory, file_repository_api_client):
+    """Test repository's name regex filters."""
+    uuid = uuid4()
+    repo = file_repository_factory(name=f"{uuid}-regex-test-repo")
+    pattern = f"^{uuid}-regex-test.*$"
+
+    results = file_repository_api_client.list(name__regex=pattern).results
+    assert results == [repo]
+
+    # upper case pattern
+    results = file_repository_api_client.list(name__regex=pattern.upper()).results
+    assert repo not in results
+
+    # upper case pattern with iregex
+    results = file_repository_api_client.list(name__iregex=pattern.upper()).results
+    assert results == [repo]
Repository: pulp/pulpcore

Issue:
Allow whitespace and comments in db key file
It would be useful if the db key file parsed generously around additional whitespace.

Also in order to help guide an admin while rotating keys, we should allow to add comments.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/fields.py b/pulpcore/app/models/fields.py
--- a/pulpcore/app/models/fields.py
+++ b/pulpcore/app/models/fields.py
@@ -1,17 +1,15 @@
 import json
 import logging
 import os
-from gettext import gettext as _
 from functools import lru_cache
+from gettext import gettext as _
 
 from cryptography.fernet import Fernet, MultiFernet
 from django.conf import settings
 from django.core.exceptions import ImproperlyConfigured
-from django.db.models import Lookup, FileField, JSONField
+from django.db.models import FileField, JSONField, Lookup
 from django.db.models.fields import Field, TextField
 from django.utils.encoding import force_bytes, force_str
-
-
 from pulpcore.app.files import TemporaryDownloadedFile
 from pulpcore.app.loggers import deprecation_logger
 
@@ -23,7 +21,13 @@ def _fernet():
     # Cache the enryption keys once per application.
     _logger.debug(f"Loading encryption key from {settings.DB_ENCRYPTION_KEY}")
     with open(settings.DB_ENCRYPTION_KEY, "rb") as key_file:
-        return MultiFernet([Fernet(key) for key in key_file.readlines()])
+        return MultiFernet(
+            [
+                Fernet(key.strip())
+                for key in key_file.readlines()
+                if not key.startswith(b"#") and key.strip() != b""
+            ]
+        )
 
 
 class ArtifactFileField(FileField):


Test patch:
Repository: pulp/pulpcore

Issue:
Cache Content qs results during ContentArtifactResource import
**Version**
Please provide the versions of the pulpcore and plugin packages in use, and how they are installed.  If you are using Pulp via Katello, please provide the Katello version.

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/modelresource.py b/pulpcore/app/modelresource.py
--- a/pulpcore/app/modelresource.py
+++ b/pulpcore/app/modelresource.py
@@ -75,9 +75,12 @@ class ContentArtifactResource(QueryModelResource):
     artifact = fields.Field(
         column_name="artifact", attribute="artifact", widget=ForeignKeyWidget(Artifact, "sha256")
     )
+    linked_content = {}
 
     def __init__(self, repo_version=None, content_mapping=None):
         self.content_mapping = content_mapping
+        if not ContentArtifactResource.linked_content:
+            ContentArtifactResource.linked_content = self.fetch_linked_content()
         super().__init__(repo_version)
 
     def before_import_row(self, row, **kwargs):
@@ -96,8 +99,7 @@ def before_import_row(self, row, **kwargs):
             (tablib.Dataset row): row that now points to the new downstream uuid for its content.
         """
 
-        linked_content = Content.objects.get(upstream_id=row["content"])
-        row["content"] = str(linked_content.pulp_id)
+        row["content"] = self.linked_content[row["content"]]
 
     def set_up_queryset(self):
         content_pks = set(self.repo_version.content.values_list("pk", flat=True))
@@ -115,6 +117,14 @@ def set_up_queryset(self):
     def dehydrate_content(self, content_artifact):
         return str(content_artifact.content_id)
 
+    def fetch_linked_content(self):
+        linked_content = {}
+        c_qs = Content.objects.filter(upstream_id__isnull=False).values("upstream_id", "pulp_id")
+        for c in c_qs.iterator():
+            linked_content[str(c["upstream_id"])] = str(c["pulp_id"])
+
+        return linked_content
+
     class Meta:
         model = ContentArtifact
         import_id_fields = (


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -850,7 +850,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -760,7 +760,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -791,7 +791,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -812,7 +812,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch: