internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
The import procedure incorrectly counts the number of imported content
https://github.com/pulp/pulpcore/blob/90142202f0ecc2e47dafbc2c517691fdc60b0eef/pulpcore/app/tasks/importer.py#L269-L278

We are overwriting the content_count variable in a loop over and over again. We should instead accumulate the number of imported content and then save it to a progress bar.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -265,20 +265,22 @@ def import_repository_version(importer_pk, destination_repo_pk, source_repo_name
                 with open(os.path.join(temp_dir, mapping_path), "r") as mapping_file:
                     mapping = json.load(mapping_file)
 
+        content_count = 0
         if mapping:
             # use the content mapping to map content to repos
             for repo_name, content_ids in mapping.items():
                 repo = _destination_repo(importer, repo_name)
                 content = Content.objects.filter(upstream_id__in=content_ids)
+                content_count += content.count()
                 with repo.new_version() as new_version:
                     new_version.set_content(content)
         else:
             # just map all the content to our destination repo
             content = Content.objects.filter(pk__in=resulting_content_ids)
+            content_count += content.count()
             with dest_repo.new_version() as new_version:
                 new_version.set_content(content)
 
-        content_count = content.count()
         pb.total = content_count
         pb.done = content_count
         pb.state = TASK_STATES.COMPLETED


Test patch:
Repository: pulp/pulpcore

Issue:
Export the EncryptedTextField in the plugin API



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/models/__init__.py b/pulpcore/plugin/models/__init__.py
--- a/pulpcore/plugin/models/__init__.py
+++ b/pulpcore/plugin/models/__init__.py
@@ -41,3 +41,6 @@
     Upload,
     UploadChunk,
 )
+
+
+from pulpcore.app.models.fields import EncryptedTextField  # noqa


Test patch:
Repository: pulp/pulpcore

Issue:
content_stages deadlock - Once More Unto The Breach
version: core/3.16+

content-stage deadlock still occurs in QE test case - see https://bugzilla.redhat.com/show_bug.cgi?id=2082209#c27


Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/stages/content_stages.py b/pulpcore/plugin/stages/content_stages.py
--- a/pulpcore/plugin/stages/content_stages.py
+++ b/pulpcore/plugin/stages/content_stages.py
@@ -183,7 +183,10 @@ def process_batch():
                         .order_by("pulp_id")
                         .select_for_update()
                     )
-                    len(subq.values_list())
+                    # NOTE: it might look like you can "safely" make this request
+                    # "more efficient". You'd be wrong, and would only be removing an
+                    # ordering-guardrail preventing deadlock. Don't touch.
+                    len(ContentArtifact.objects.filter(pk__in=subq).values_list())
                     ContentArtifact.objects.bulk_update(to_update_ca_bulk, ["artifact"])
 
                     # To avoid a similar deadlock issue when calling get_or_create, we sort the


Test patch:
Repository: pulp/pulpcore

Issue:
`UnexportableArtifactException` does not expect any arguments during the initialization
`UnexportableArtifactException.__init__() takes 1 positional argument but 2 were given)'`

https://github.com/pulp/pulpcore/blob/351bd8653820195ffd3b16c5fbab0f14fd0061fe/pulpcore/app/tasks/export.py#L41-L59


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/export.py b/pulpcore/app/tasks/export.py
--- a/pulpcore/app/tasks/export.py
+++ b/pulpcore/app/tasks/export.py
@@ -54,9 +54,7 @@ def _validate_fs_export(content_artifacts):
         RuntimeError: If Artifacts are not downloaded or when trying to link non-fs files
     """
     if content_artifacts.filter(artifact=None).exists():
-        raise UnexportableArtifactException(
-            _("Cannot export artifacts that haven't been downloaded.")
-        )
+        raise UnexportableArtifactException()
 
 
 def _export_to_file_system(path, relative_paths_to_artifacts, method=FS_EXPORT_METHODS.WRITE):


Test patch:
Repository: pulp/pulpcore

Issue:
CVE-2007-4559 fix tar.extractall() use in pulp-import
**Version**
all

**Describe the bug**
python's tar.extractall() allows for malicious file overwriting on the executing system. See CVE-2007-4559 for details. PulpImport uses extractall() (although the workflow is such that the attack-window is small and requires social-engineering access to admins of importing systems).

**Additional context**
TrellixVulnTeam submitted PR #3287 to address - that work needs to be picked up and moved through the Pulp submission process.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -447,7 +447,26 @@ def validate_and_assemble(toc_filename):
 
     with tempfile.TemporaryDirectory(dir=".") as temp_dir:
         with tarfile.open(path, "r:gz") as tar:
-            tar.extractall(path=temp_dir)
+
+            def is_within_directory(directory, target):
+
+                abs_directory = os.path.abspath(directory)
+                abs_target = os.path.abspath(target)
+
+                prefix = os.path.commonprefix([abs_directory, abs_target])
+
+                return prefix == abs_directory
+
+            def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
+
+                for member in tar.getmembers():
+                    member_path = os.path.join(path, member.name)
+                    if not is_within_directory(path, member_path):
+                        raise Exception("Attempted Path Traversal in Tar File")
+
+                tar.extractall(path, members, numeric_owner=numeric_owner)
+
+            safe_extract(tar, path=temp_dir)
 
         # Check version info
         with open(os.path.join(temp_dir, VERSIONS_FILE)) as version_file:


Test patch:
Repository: pulp/pulpcore

Issue:
Export is not locking on the exported repositories
SSIA


Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/exporter.py b/pulpcore/app/viewsets/exporter.py
--- a/pulpcore/app/viewsets/exporter.py
+++ b/pulpcore/app/viewsets/exporter.py
@@ -146,6 +146,7 @@ def create(self, request, exporter_pk):
         task = dispatch(
             pulp_export,
             exclusive_resources=[exporter],
+            shared_resources=exporter.repositories.all(),
             kwargs={"exporter_pk": str(exporter.pk), "params": request.data},
         )
 


Test patch:
Repository: pulp/pulpcore

Issue:
fields and exclude_fields querystring parameters should apecify array-of-strings in openapi

`pulp debug openapi operation --id tasks_read | jq '.operation.parameters[]|select(.name=="fields")'`

current behaviour:
```json
{
  "in": "query",
  "name": "fields",
  "schema": {
    "type": "string"
  },
  "description": "A list of fields to include in the response."
}
```

expected behaviour:
```json
{
  "in": "query",
  "name": "fields",
  "schema": {
    "type": "array",
    "items": {
      "type": "string"
    }
  },
  "description": "A list of fields to include in the response."
}
```


Discussion:

Resolving patch:
diff --git a/pulpcore/openapi/__init__.py b/pulpcore/openapi/__init__.py
--- a/pulpcore/openapi/__init__.py
+++ b/pulpcore/openapi/__init__.py
@@ -428,20 +428,20 @@ def parse(self, input_request, public):
 
             # Adding query parameters
             if "parameters" in operation and schema.method.lower() == "get":
-                fields_paramenter = build_parameter_type(
+                fields_parameter = build_parameter_type(
                     name="fields",
-                    schema={"type": "string"},
+                    schema={"type": "array", "items": {"type": "string"}},
                     location=OpenApiParameter.QUERY,
                     description="A list of fields to include in the response.",
                 )
-                operation["parameters"].append(fields_paramenter)
-                not_fields_paramenter = build_parameter_type(
+                operation["parameters"].append(fields_parameter)
+                exclude_fields_parameter = build_parameter_type(
                     name="exclude_fields",
-                    schema={"type": "string"},
+                    schema={"type": "array", "items": {"type": "string"}},
                     location=OpenApiParameter.QUERY,
                     description="A list of fields to exclude from the response.",
                 )
-                operation["parameters"].append(not_fields_paramenter)
+                operation["parameters"].append(exclude_fields_parameter)
 
             # Normalise path for any provided mount url.
             if path.startswith("/"):


Test patch:
Repository: pulp/pulpcore

Issue:
Expose "get_url" via the plugin interface



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/util.py b/pulpcore/plugin/util.py
--- a/pulpcore/plugin/util.py
+++ b/pulpcore/plugin/util.py
@@ -12,4 +12,4 @@
     remove_role,
 )
 
-from pulpcore.app.util import get_artifact_url, gpg_verify, verify_signature  # noqa
+from pulpcore.app.util import get_artifact_url, get_url, gpg_verify, verify_signature  # noqa


Test patch:
Repository: pulp/pulpcore

Issue:
Correlation ID should be available to signing cli
The Correlation ID / guid is used throughout Pulp to correlate logs with a particular request, however it is not made available to the signing cli script. It should be, in case that script wants to log things too.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/content.py b/pulpcore/app/models/content.py
--- a/pulpcore/app/models/content.py
+++ b/pulpcore/app/models/content.py
@@ -20,6 +20,7 @@
 from django.db import IntegrityError, models, transaction
 from django.forms.models import model_to_dict
 from django.utils.timezone import now
+from django_guid import get_guid
 from django_lifecycle import BEFORE_UPDATE, BEFORE_SAVE, hook
 
 from pulpcore.constants import ALL_KNOWN_CONTENT_CHECKSUMS
@@ -734,6 +735,16 @@ class SigningService(BaseModel):
     pubkey_fingerprint = models.TextField()
     script = models.TextField()
 
+    def _env_variables(self, env_vars=None):
+        guid = get_guid()
+        env = {
+            "PULP_SIGNING_KEY_FINGERPRINT": self.pubkey_fingerprint,
+            "CORRELATION_ID": guid if guid else "",
+        }
+        if env_vars:
+            env.update(env_vars)
+        return env
+
     def sign(self, filename, env_vars=None):
         """
         Signs the file provided via 'filename' by invoking an external script (or executable).
@@ -752,12 +763,9 @@ def sign(self, filename, env_vars=None):
         Returns:
             A dictionary as validated by the validate() method.
         """
-        env = {"PULP_SIGNING_KEY_FINGERPRINT": self.pubkey_fingerprint}
-        if env_vars:
-            env.update(env_vars)
         completed_process = subprocess.run(
             [self.script, filename],
-            env=env,
+            env=self._env_variables(env_vars),
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
         )
@@ -774,13 +782,10 @@ def sign(self, filename, env_vars=None):
 
     async def asign(self, filename, env_vars=None):
         """Async version of sign."""
-        env = {"PULP_SIGNING_KEY_FINGERPRINT": self.pubkey_fingerprint}
-        if env_vars:
-            env.update(env_vars)
         process = await asyncio.create_subprocess_exec(
             self.script,
             filename,
-            env=env,
+            env=self._env_variables(env_vars),
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,
         )


Test patch:
Repository: pulp/pulpcore

Issue:
Improve the logic in the  ACSHandleStage
**Version**
main

**Describe the bug**
If there are ACS that point to the same content, pick the first RA instead of last

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/stages/artifact_stages.py b/pulpcore/plugin/stages/artifact_stages.py
--- a/pulpcore/plugin/stages/artifact_stages.py
+++ b/pulpcore/plugin/stages/artifact_stages.py
@@ -465,7 +465,8 @@ async def run(self):
                 for ra in existing_ras:
                     for c_type in Artifact.COMMON_DIGEST_FIELDS:
                         checksum = await sync_to_async(getattr)(ra, c_type)
-                        if checksum:
+                        # pick the first occurence of RA from ACS
+                        if checksum and checksum not in existing_ras_dict:
                             existing_ras_dict[checksum] = {
                                 "remote": ra.remote,
                                 "url": ra.url,


Test patch:
Repository: pulp/pulpcore

Issue:
Aioredis is no longer supported, replace the dependency with redis-py (which we are already using)
**Version**

3.16+

**Describe the bug**

pulpcore depends on aioredis, which is no longer supported, and has been subsumed by the standard redis library which we are already using in synchronous contexts

https://github.com/aio-libs/aioredis-py#-aioredis-is-now-in-redis-py-420rc1-


Discussion:

Resolving patch:
diff --git a/pulpcore/app/redis_connection.py b/pulpcore/app/redis_connection.py
--- a/pulpcore/app/redis_connection.py
+++ b/pulpcore/app/redis_connection.py
@@ -1,5 +1,5 @@
 from redis import Redis
-from aioredis import Redis as aRedis
+from redis.asyncio import Redis as aRedis
 
 from pulpcore.app.settings import settings
 
diff --git a/pulpcore/cache/cache.py b/pulpcore/cache/cache.py
--- a/pulpcore/cache/cache.py
+++ b/pulpcore/cache/cache.py
@@ -11,7 +11,7 @@
 from aiohttp.web_exceptions import HTTPFound
 
 from redis import ConnectionError
-from aioredis import ConnectionError as AConnectionError
+from redis.asyncio import ConnectionError as AConnectionError
 
 from pulpcore.app.settings import settings
 from pulpcore.app.redis_connection import (


Test patch:
Repository: pulp/pulpcore

Issue:
LabelFilter is not exported the plugin directory.
**Version**
latest

**Describe the bug**
LabelSelectFilter is exported via the plugin/viewsets directory, but has been marked as deprecated by LabelFIlter ... which is not exported via the plugin/viewsets directory.

**To Reproduce**
```from pulpcore.plugin.viewsets import LabelFilter``` produces an import error.

**Expected behavior**
```from pulpcore.plugin.viewsets import LabelFilter``` does not produce an import error.

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/viewsets/__init__.py b/pulpcore/plugin/viewsets/__init__.py
--- a/pulpcore/plugin/viewsets/__init__.py
+++ b/pulpcore/plugin/viewsets/__init__.py
@@ -36,6 +36,7 @@
 
 from pulpcore.app.viewsets.custom_filters import (  # noqa
     CharInFilter,
+    LabelFilter,
     LabelSelectFilter,
     RepositoryVersionFilter,
 )


Test patch:
Repository: pulp/pulpcore

Issue:
openapi spec does not specify integer format leading to unusable apis
**Version**
3.22.0

**Describe the bug**
Currently the openapi spec just specifies an integer for the type, without adding the 'format'.  (https://swagger.io/docs/specification/data-models/data-types/)
This works fine for most things, but for things related to filesize or storage, this tends to break when using lower level languages like go:

For example the status api errors when using generated go bindings:
```
json: cannot unmarshal number 1022488477696 into Go struct field StatusResponseStorage.storage.total of type int32
```

I'm sure there are other areas where this could be an issue, such as a file of size greater than ~4GB, reporting its filesize would also hit this problem.

**To Reproduce**
Generate Go bindings, see that int32 is used for things like storage amount in the status api.  

**Expected behavior**
format: int64 should be used for values that could be large (file sizes, disk sizes, anything else?)

**Additional context**
https://swagger.io/docs/specification/data-models/data-types/



Discussion:

Resolving patch:
diff --git a/pulpcore/openapi/__init__.py b/pulpcore/openapi/__init__.py
--- a/pulpcore/openapi/__init__.py
+++ b/pulpcore/openapi/__init__.py
@@ -20,13 +20,17 @@
 )
 from drf_spectacular.settings import spectacular_settings
 from drf_spectacular.types import OpenApiTypes
-from drf_spectacular.utils import OpenApiParameter
+from drf_spectacular.utils import OpenApiParameter, extend_schema_field
 from rest_framework import mixins, serializers
 from rest_framework.schemas.utils import get_pk_description
 
 from pulpcore.app.apps import pulp_plugin_configs
 
 
+# Python does not distinguish integer sizes. The safest assumtion is that they are large.
+extend_schema_field(OpenApiTypes.INT64)(serializers.IntegerField)
+
+
 class PulpAutoSchema(AutoSchema):
     """Pulp Auto Schema."""
 


Test patch:
Repository: pulp/pulpcore

Issue:
Expose functions for parsing URIs and validating content units to plugin writers



Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/util.py b/pulpcore/plugin/util.py
--- a/pulpcore/plugin/util.py
+++ b/pulpcore/plugin/util.py
@@ -12,4 +12,11 @@
     remove_role,
 )
 
-from pulpcore.app.util import get_artifact_url, get_url, gpg_verify, verify_signature  # noqa
+from pulpcore.app.util import (  # noqa
+    extract_pk,
+    get_artifact_url,
+    get_url,
+    gpg_verify,
+    raise_for_unknown_content_units,
+    verify_signature,
+)


Test patch:
Repository: pulp/pulpcore

Issue:
Allow customizing the field name on LabelFilter
**Is your feature request related to a problem? Please describe.**
As a plugin writer I would like to be able to customize the field that LabelFilter filters the queryset on.

**Describe the solution you'd like**
LabelFilter should accept a parameter that allows it to search on a custom field name.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/custom_filters.py b/pulpcore/app/viewsets/custom_filters.py
--- a/pulpcore/app/viewsets/custom_filters.py
+++ b/pulpcore/app/viewsets/custom_filters.py
@@ -279,6 +279,10 @@ class LabelFilter(Filter):
 
     def __init__(self, *args, **kwargs):
         kwargs.setdefault("help_text", _("Filter labels by search string"))
+        if "label_field_name" in kwargs:
+            self.label_field_name = kwargs.pop("label_field_name")
+        else:
+            self.label_field_name = "pulp_labels"
         super().__init__(*args, **kwargs)
 
     def filter(self, qs, value):
@@ -293,6 +297,11 @@ def filter(self, qs, value):
         Raises:
             rest_framework.exceptions.ValidationError: on invalid search string
         """
+
+        # NOTE: can't use self.field_name because the default for that is the name
+        # of the method on the filter class (which is pulp_label_select on all of
+        # the pulp filtersets)
+        field_name = self.label_field_name
         if value is None:
             # user didn't supply a value
             return qs
@@ -307,17 +316,19 @@ def filter(self, qs, value):
                 raise DRFValidationError(_("Cannot use an operator with '{}'.").format(key))
 
             if op == "=":
-                qs = qs.filter(**{f"pulp_labels__{key}": val})
+                qs = qs.filter(**{f"{field_name}__{key}": val})
             elif op == "!=":
-                qs = qs.filter(pulp_labels__has_key=key).exclude(**{f"pulp_labels__{key}": val})
+                qs = qs.filter(**{f"{field_name}__has_key": key}).exclude(
+                    **{f"{field_name}__{key}": val}
+                )
             elif op == "~":
-                qs = qs.filter(**{f"pulp_labels__{key}__icontains": val})
+                qs = qs.filter(**{f"{field_name}__{key}__icontains": val})
             else:
                 # 'foo', '!foo'
                 if key.startswith("!"):
-                    qs = qs.exclude(pulp_labels__has_key=key[1:])
+                    qs = qs.exclude(**{f"{field_name}__has_key": key[1:]})
                 else:
-                    qs = qs.filter(pulp_labels__has_key=key)
+                    qs = qs.filter(**{f"{field_name}__has_key": key})
 
         return qs
 


Test patch:
Repository: pulp/pulpcore

Issue:
Content endpoints return duplicates when using a non-superuser
**Version**
The main branch; parallelized pulp_rpm.tests.functional.api.test_rbac_crud

**Describe the bug**
After querying an endpoint hosting content referenced by multiple repositories, while logged in a non-superuser, Pulp returns duplicated content.

**To Reproduce**
1. Create a user with `view_content` permissions.
2. Create/Sync two repositories with the same content.
3. Send a GET request with the query parameter `repository_version=one_of_the_repos` to the endpoint hosting the content.
4. Watch the returned duplicates (same pulp_hrefs).

**Expected behaviour**
No duplicates should be returned from the endpoint.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/content.py b/pulpcore/app/viewsets/content.py
--- a/pulpcore/app/viewsets/content.py
+++ b/pulpcore/app/viewsets/content.py
@@ -121,7 +121,12 @@ def scope_queryset(self, qs):
                 repo_viewset = get_viewset_for_model(repo)()
                 setattr(repo_viewset, "request", self.request)
                 scoped_repos.extend(repo_viewset.get_queryset().values_list("pk", flat=True))
-            return qs.filter(repositories__in=scoped_repos)
+
+            # calling the distinct clause at end of the query ensures that no duplicates from
+            # joined tables will be returned to the end-user; this behaviour is documented at
+            # https://docs.djangoproject.com/en/3.2/topics/db/queries, in the section Spanning
+            # multi-valued relationships
+            return qs.filter(repositories__in=scoped_repos).distinct()
         return qs
 
 


Test patch:
Repository: pulp/pulpcore

Issue:
Refactor templatetags app to use force_str instead of deprecated force_text
Since we're testing and moving to Django 4.x(expect to stay on 4.2 LTS when it's released) we expect to find some code marked as deprecated as we move on.

This task is about the moving from the deprecated `force_text` to the new `force_str` function usage on our codebase.
This was spotted here -> https://github.com/pulp/pulpcore/pull/3618#issuecomment-1452483429


Discussion:

Resolving patch:
diff --git a/pulpcore/app/templatetags/pulp_urls.py b/pulpcore/app/templatetags/pulp_urls.py
--- a/pulpcore/app/templatetags/pulp_urls.py
+++ b/pulpcore/app/templatetags/pulp_urls.py
@@ -2,7 +2,7 @@
 
 from django import template
 from django.conf import settings
-from django.utils.encoding import force_text
+from django.utils.encoding import force_str
 from django.utils.html import escape
 from django.utils.safestring import SafeData, mark_safe
 
@@ -35,7 +35,7 @@ def trim_url(x, limit=trim_url_limit):
         return limit is not None and (len(x) > limit and ("%s..." % x[: max(0, limit - 3)])) or x
 
     safe_input = isinstance(text, SafeData)
-    words = word_split_re.split(force_text(text))
+    words = word_split_re.split(force_str(text))
     for i, word in enumerate(words):
         if settings.V3_API_ROOT in word:
             # Deal with punctuation.


Test patch: