internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -817,7 +817,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -827,7 +827,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Regression on performance improvements in rv.get_content()
**Version**
foreman-3.5.3, katello-4.7.6, python39-pulpcore-3.21.17+, and postgresql-12.15

**Describe the bug**
Users are experiencing a slowdown in their installations.

**To Reproduce**
Sync the content.


**Additional context**
https://community.theforeman.org/t/slow-content-proxy-sync/35162/26
https://github.com/pulp/pulpcore/blob/bc9ca511dc13fbae7c49b19f1ebe8b527c67484a/pulpcore/app/models/repository.py#L853
https://github.com/pulp/pulpcore/pull/4275

The Postgres version might be one of the culprits. All the experiments in the attached PR were performed on Postgres 13.


Discussion:
The reason for believing Postgresql might be the culprit is that we cannot reproduce thus far on Postgresql 13, and the Postgresql 13 changelog contains an entry which near perfectly matches this situation.

> Allow [hash aggregation](https://www.postgresql.org/docs/13/runtime-config-query.html#GUC-ENABLE-HASHAGG) to use disk storage for large aggregation result sets (Jeff Davis)

> Previously, hash aggregation was avoided if it was expected to use more than [work_mem](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-WORK-MEM) memory. Now, a hash aggregation plan can be chosen despite that. The hash table will be spilled to disk if it exceeds work_mem times [hash_mem_multiplier](https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-HASH-MEM-MULTIPLIER).

> This behavior is normally preferable to the old behavior, in which once hash aggregation had been chosen, the hash table would be kept in memory no matter how large it got — which could be very large if the planner had misestimated. If necessary, behavior similar to that can be obtained by increasing hash_mem_multiplier.

https://www.postgresql.org/docs/release/13.0/
At minimum we could probably recommend PG13+ in our docs.
Should we recommend the increase of the `hash_mem_multiplier` for pulp installations, or at least for katello installations?

We shouldn't recommend anything as a workaround until we've confirmed that it works for a user or have verified it ourselves.  We need to reproduce the issue.

Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -835,7 +835,9 @@ def add_content(self, content):
             raise ResourceImmutableError(self)
 
         repo_content = []
-        to_add = set(content.exclude(pk__in=self.content).values_list("pk", flat=True))
+        to_add = set(content.values_list("pk", flat=True)) - set(
+            self.content.values_list("pk", flat=True)
+        )
 
         # Normalize representation if content has already been removed in this version and
         # is re-added: Undo removal by setting version_removed to None.


Test patch:
Repository: pulp/pulpcore

Issue:
Remove eval fallback from EncryptedJSONField
#4359


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/fields.py b/pulpcore/app/models/fields.py
--- a/pulpcore/app/models/fields.py
+++ b/pulpcore/app/models/fields.py
@@ -11,7 +11,6 @@
 from django.db.models.fields import Field, TextField
 from django.utils.encoding import force_bytes, force_str
 from pulpcore.app.files import TemporaryDownloadedFile
-from pulpcore.app.loggers import deprecation_logger
 
 _logger = logging.getLogger(__name__)
 
@@ -141,15 +140,7 @@ def decrypt(self, value):
             return [self.decrypt(v) for v in value]
 
         dec_value = force_str(_fernet().decrypt(force_bytes(value)))
-        try:
-            return json.loads(dec_value, cls=self.decoder)
-        except json.JSONDecodeError:
-            deprecation_logger.info(
-                "Failed to decode json in an EncryptedJSONField. Falling back to eval. "
-                "Please run pulpcore-manager rotate-db-key to repair."
-                "This is deprecated and will be removed in pulpcore 3.40."
-            )
-            return eval(dec_value)
+        return json.loads(dec_value, cls=self.decoder)
 
     def get_prep_value(self, value):
         if value is not None:


Test patch:
Repository: pulp/pulpcore

Issue:
Expose batch_qs() to the plugin API
If there are no concerns, I would like to have `batch_qs` from https://github.com/pulp/pulpcore/blob/main/pulpcore/app/util.py#L182

Made available to plugins here: https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/util.py#L15

The reason, is that I need this helper function to fix https://github.com/pulp/pulp_deb/issues/921 and I would prefer not to duplicate it.

See also: https://discourse.pulpproject.org/t/exposing-new-things-to-the-plugin-api/1021

Note: I am fine with duplicating the pulpcore function for any pulp_deb backports, and switching to the pulpcore version with whatever pulpcore release first exposes it. So there is no need to backport the pulpcore change anywhere.


Discussion:
Ok, so my request may have been premature, the draft PR I added would allow me to reuse more pulpcore code than exposing `batch_qs` would. However, it needs to be analyzed carefully to ensure it is performance neutral. If not, I am back to my plan of writing my own version of pulpcore's `remove_duplicates` in pulp_deb, thus making use of `batch_qs`.
The approach from my draft PR turned out to be a complete dead end, so I am back to needing `batch_qs` exposed.
In fact I now have a working (draft PR) in pulp_deb that uses it: https://github.com/pulp/pulp_deb/pull/922

Resolving patch:
diff --git a/pulpcore/app/util.py b/pulpcore/app/util.py
--- a/pulpcore/app/util.py
+++ b/pulpcore/app/util.py
@@ -180,15 +180,21 @@ def get_view_name_for_model(model_obj, view_action):
 
 
 def batch_qs(qs, batch_size=1000):
-    """
-    Returns a queryset batch in the given queryset.
-
-    Usage:
-        # Make sure to order your querset
-        article_qs = Article.objects.order_by('id')
-        for qs in batch_qs(article_qs):
-            for article in qs:
-                print article.body
+    """Returns a queryset batch from the given queryset.
+
+    Make sure to order the queryset.
+
+    Args:
+       qs: The queryset we want to iterate over in batches.
+       batch_size: Defaults to 1000.
+
+    Example:
+        To iterate over a queryset while retrieving records from the DB in batches, use::
+
+            article_qs = Article.objects.order_by('id')
+            for qs in batch_qs(article_qs):
+                for article in qs:
+                    print article.body
     """
     total = qs.count()
     for start in range(0, total, batch_size):
diff --git a/pulpcore/plugin/util.py b/pulpcore/plugin/util.py
--- a/pulpcore/plugin/util.py
+++ b/pulpcore/plugin/util.py
@@ -13,6 +13,7 @@
 )
 
 from pulpcore.app.util import (  # noqa: F401
+    batch_qs,
     extract_pk,
     get_artifact_url,
     get_url,


Test patch:
Repository: pulp/pulpcore

Issue:
Generate a Response for artifacts returned from a content handler
The problem when defining a `content_handler` to serve an artifact is trying to generate the response which has to account for things like object storage. The content `Handler` class has helpful methods such as `_serve_content_artifact` and `_stream_content_artifact` but they are methods and they require `self`. This change would make it easier for plugin writers to return a `ContentArtifact` so that the plugin writer does not have to generate the response.


Discussion:

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -573,11 +573,22 @@ async def _match_and_stream(self, path, request):
         if not ends_in_slash:
             rel_path = f"{rel_path}/"
 
+        headers = self.response_headers(original_rel_path, distro)
+
         content_handler_result = await sync_to_async(distro.content_handler)(original_rel_path)
         if content_handler_result is not None:
-            return content_handler_result
-
-        headers = self.response_headers(original_rel_path, distro)
+            if isinstance(content_handler_result, ContentArtifact):
+                if content_handler_result.artifact:
+                    return await self._serve_content_artifact(
+                        content_handler_result, headers, request
+                    )
+                else:
+                    return await self._stream_content_artifact(
+                        request, StreamResponse(headers=headers), content_handler_result
+                    )
+            else:
+                # the result is a response so just return it
+                return content_handler_result
 
         repository = distro.repository
         publication = distro.publication


Test patch:
Repository: pulp/pulpcore

Issue:
pulp_file version is set to 3.40.0.dev 
**Version**
pulpcore 3.40.0

**Describe the bug**
Status API reports pulp_file version as 3.40.0.dev


Discussion:

Resolving patch:
diff --git a/pulp_file/app/__init__.py b/pulp_file/app/__init__.py
--- a/pulp_file/app/__init__.py
+++ b/pulp_file/app/__init__.py
@@ -8,6 +8,6 @@ class PulpFilePluginAppConfig(PulpPluginAppConfig):
 
     name = "pulp_file.app"
     label = "file"
-    version = "3.40.0.dev"
+    version = "3.41.0.dev"
     python_package_name = "pulp_file"  # TODO Add python_module_name
     domain_compatible = True


Test patch:
Repository: pulp/pulpcore

Issue:
serializer. get_or_create can still raise objectdoesnot exist
**Version**
Please provide the versions of the pulpcore and plugin packages in use, and how they are installed.  If you are using Pulp via Katello, please provide the Katello version.

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/base.py b/pulpcore/app/serializers/base.py
--- a/pulpcore/app/serializers/base.py
+++ b/pulpcore/app/serializers/base.py
@@ -358,6 +358,7 @@ class GetOrCreateSerializerMixin:
 
     @classmethod
     def get_or_create(cls, natural_key, default_values=None):
+        result = None
         try:
             result = cls.Meta.model.objects.get(**natural_key)
         except ObjectDoesNotExist:
@@ -368,8 +369,21 @@ def get_or_create(cls, natural_key, default_values=None):
             serializer = cls(data=data)
             try:
                 serializer.is_valid(raise_exception=True)
-                result = serializer.create(serializer.validated_data)
-            except (IntegrityError, serializers.ValidationError):
+            except serializers.ValidationError as e:
+                returned_codes = e.get_codes().values()
+                error_codes = set(["required", "invalid"])
+                all_codes = []
+                for c in returned_codes:
+                    all_codes.extend(c)
+                if error_codes.intersection(all_codes):
+                    # validation failed because fields are invalid, missing
+                    raise e
+                # recover from a race condition, where another thread just created the object
+                # validation failed with 400 'unique' error code only
+                result = cls.Meta.model.objects.get(**natural_key)
+            try:
+                result = result or serializer.create(serializer.validated_data)
+            except IntegrityError:
                 # recover from a race condition, where another thread just created the object
                 result = cls.Meta.model.objects.get(**natural_key)
         return result


Test patch:
Repository: pulp/pulpcore

Issue:
Remove the ability to call non-pulp-related functions as task targets



Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/tasks.py b/pulpcore/tasking/tasks.py
--- a/pulpcore/tasking/tasks.py
+++ b/pulpcore/tasking/tasks.py
@@ -11,7 +11,6 @@
 from django.db.models import Model
 from django_guid import get_guid
 from pulpcore.app.apps import MODULE_PLUGIN_VERSIONS
-from pulpcore.app.loggers import deprecation_logger
 from pulpcore.app.models import Task
 from pulpcore.app.util import current_task, get_domain, get_url
 from pulpcore.constants import TASK_FINAL_STATES, TASK_INCOMPLETE_STATES, TASK_STATES
@@ -129,17 +128,7 @@ def dispatch(
         function_name = func
 
     if versions is None:
-        try:
-            versions = MODULE_PLUGIN_VERSIONS[function_name.split(".", maxsplit=1)[0]]
-        except KeyError:
-            deprecation_logger.warn(
-                _(
-                    "Using functions outside of pulp components as tasks is not supported and will "
-                    "result in runtime errors with pulpcore>=3.40."
-                )
-            )
-            # The best we can do now...
-            versions = MODULE_PLUGIN_VERSIONS["pulpcore"]
+        versions = MODULE_PLUGIN_VERSIONS[function_name.split(".", maxsplit=1)[0]]
 
     if exclusive_resources is None:
         exclusive_resources = []


Test patch:
Repository: pulp/pulpcore

Issue:
app entrypoint no longer supports --max-requests-jitter
**Version**
3.39

**Describe the bug**
--max-requests-jitter is not recognized

**To Reproduce**
Run the pulpcore-api entrypoint with --max-requests-jitter

**Expected behavior**
Accepts the argument

**Additional context**
Requested for Katello.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/entrypoint.py b/pulpcore/app/entrypoint.py
--- a/pulpcore/app/entrypoint.py
+++ b/pulpcore/app/entrypoint.py
@@ -118,6 +118,7 @@ def load(self):
 @click.option("--limit-request-fields", type=int)
 @click.option("--limit-request-field-size", type=int)
 @click.option("--max-requests", type=int)
+@click.option("--max-requests-jitter", type=int)
 @click.option("--access-logfile", "accesslog")
 @click.option(
     "--access-logformat",
diff --git a/pulpcore/content/entrypoint.py b/pulpcore/content/entrypoint.py
--- a/pulpcore/content/entrypoint.py
+++ b/pulpcore/content/entrypoint.py
@@ -33,6 +33,7 @@ def load(self):
 @click.option("--limit-request-fields", type=int)
 @click.option("--limit-request-field-size", type=int)
 @click.option("--max-requests", type=int)
+@click.option("--max-requests-jitter", type=int)
 @click.option("--access-logfile", "accesslog")
 @click.option("--access-logformat", "access_log_format")
 @click.option("--error-logfile", "--log-file", "errorlog")


Test patch:
Repository: pulp/pulpcore

Issue:
file:// sync deletes files from directory
**Version**
Pulpcore 3.39

**Describe the bug**
When syncing file:// repositories, files are disappearing after the sync.

**To Reproduce**
1) Copy these two repositories to the FS:
  - https://github.com/Katello/katello/tree/master/test/fixtures/test_repos/file1
  - https://github.com/Katello/katello/tree/master/test/fixtures/test_repos/file2
2) Sync one, then the other
3) See that some files disappeared.
  - In my case, file2 lost every file except PULP_MANIFEST


**Expected behavior**
No files disappear.

**Additional context**
This also occurred with RPM content type files.



Discussion:
Likely related to https://github.com/pulp/pulpcore/pull/4428
A repeat of https://pulp.plan.io/issues/9146 ?

Resolving patch:
diff --git a/pulpcore/download/file.py b/pulpcore/download/file.py
--- a/pulpcore/download/file.py
+++ b/pulpcore/download/file.py
@@ -57,7 +57,7 @@ async def _run(self, extra_data=None):
                     break  # the reading is done
                 await self.handle_data(chunk)
             return DownloadResult(
-                path=self._path,
+                path=self.path,
                 artifact_attributes=self.artifact_attributes,
                 url=self.url,
                 headers=None,


Test patch:
diff --git a/pulpcore/tests/functional/api/pulp_file/test_sync.py b/pulpcore/tests/functional/api/pulp_file/test_sync.py
--- a/pulpcore/tests/functional/api/pulp_file/test_sync.py
+++ b/pulpcore/tests/functional/api/pulp_file/test_sync.py
@@ -1,4 +1,6 @@
 """Tests that sync file plugin repositories."""
+
+import os
 import uuid
 
 import pytest
@@ -22,17 +24,20 @@ def test_sync_file_protocol_handler(
 ):
     """Test syncing from a file repository with the file:// protocol handler"""
     wget_recursive_download_on_host(urljoin(fixtures_cfg.remote_fixtures_origin, "file/"), "/tmp")
-
     remote_kwargs = {
         "url": "file:///tmp/file/PULP_MANIFEST",
         "policy": "immediate",
         "name": str(uuid.uuid4()),
     }
     remote = gen_object_with_cleanup(file_remote_api_client, remote_kwargs)
+    files = set(os.listdir("/tmp/file/"))
 
     body = RepositorySyncURL(remote=remote.pulp_href)
     monitor_task(file_repository_api_client.sync(file_repo.pulp_href, body).task)
 
+    # test that all the files are still present
+    assert set(os.listdir("/tmp/file/")) == files
+
     file_repo = file_repository_api_client.read(file_repo.pulp_href)
     assert file_repo.latest_version_href.endswith("/versions/1/")