internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
The validation of input parameters for the repair endpoint is omitted
```
curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' -d '[]' http://localhost:5001/pulp/api/v3/repair/
```

```
pulp [804a07335b9f4417ad0c71dde478634e]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repair/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/src/pulpcore/pulpcore/app/views/repair.py", line 27, in post
    verify_checksums = serializer.validated_data["verify_checksums"]
KeyError: 'verify_checksums'
```


Discussion:

Resolving patch:
diff --git a/pulpcore/app/views/repair.py b/pulpcore/app/views/repair.py
--- a/pulpcore/app/views/repair.py
+++ b/pulpcore/app/views/repair.py
@@ -22,7 +22,7 @@ def post(self, request):
         Repair artifacts.
         """
         serializer = RepairSerializer(data=request.data)
-        serializer.is_valid()
+        serializer.is_valid(raise_exception=True)
 
         verify_checksums = serializer.validated_data["verify_checksums"]
 


Test patch:
Repository: pulp/pulpcore

Issue:
The search bar issue in sphinx docs persists
https://github.com/pulp/pulpcore/issues/3569

The search feature bug in docs was not resolved.

https://docs.pulpproject.org/pulpcore


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -37,7 +37,8 @@
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = ['sphinx.ext.extlinks', 'sphinx.ext.autodoc', 'sphinx.ext.autosummary',
-              'napoleon_django', 'sphinx.ext.napoleon', 'sphinxcontrib.openapi']
+              'napoleon_django', 'sphinx.ext.napoleon', 'sphinxcontrib.openapi',
+              'sphinxcontrib.jquery']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -119,7 +120,7 @@
 #html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
-html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] if sphinx_rtd_theme else []
+#html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] if sphinx_rtd_theme else []
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".


Test patch:
Repository: pulp/pulpcore

Issue:
fields and exclude_fields querystring parameters should apecify array-of-strings in openapi

`pulp debug openapi operation --id tasks_read | jq '.operation.parameters[]|select(.name=="fields")'`

current behaviour:
```json
{
  "in": "query",
  "name": "fields",
  "schema": {
    "type": "string"
  },
  "description": "A list of fields to include in the response."
}
```

expected behaviour:
```json
{
  "in": "query",
  "name": "fields",
  "schema": {
    "type": "array",
    "items": {
      "type": "string"
    }
  },
  "description": "A list of fields to include in the response."
}
```


Discussion:

Resolving patch:
diff --git a/pulpcore/openapi/__init__.py b/pulpcore/openapi/__init__.py
--- a/pulpcore/openapi/__init__.py
+++ b/pulpcore/openapi/__init__.py
@@ -426,20 +426,20 @@ def parse(self, input_request, public):
 
             # Adding query parameters
             if "parameters" in operation and schema.method.lower() == "get":
-                fields_paramenter = build_parameter_type(
+                fields_parameter = build_parameter_type(
                     name="fields",
-                    schema={"type": "string"},
+                    schema={"type": "array", "items": {"type": "string"}},
                     location=OpenApiParameter.QUERY,
                     description="A list of fields to include in the response.",
                 )
-                operation["parameters"].append(fields_paramenter)
-                not_fields_paramenter = build_parameter_type(
+                operation["parameters"].append(fields_parameter)
+                exclude_fields_parameter = build_parameter_type(
                     name="exclude_fields",
-                    schema={"type": "string"},
+                    schema={"type": "array", "items": {"type": "string"}},
                     location=OpenApiParameter.QUERY,
                     description="A list of fields to exclude from the response.",
                 )
-                operation["parameters"].append(not_fields_paramenter)
+                operation["parameters"].append(exclude_fields_parameter)
 
             # Normalise path for any provided mount url.
             if path.startswith("/"):


Test patch:
Repository: pulp/pulpcore

Issue:
Download machinery should attempt to keep the file extension of the downloaded URL intact, if one exists
**Version**

All

**Describe the bug**

Sometimes libraries will get upset when you pass into them a file without the file extension they expect for the type of file they are handling.  It is best to keep the file extension intact to avoid those kinds of issues.

See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

There is a deeper issue there with the `file` / `libmagic` library, but until that is fixed and shipped the best we can do is help it out by keeping the original filenames.

I'm sure it will also help with debugging on occasion.

**To Reproduce**
See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

**Additional context**
BZ https://bugzilla.redhat.com/show_bug.cgi?id=2176870



Discussion:

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -5,6 +5,7 @@
 import logging
 import os
 import tempfile
+from urllib.parse import urlsplit
 
 from pulpcore.app import pulp_hashlib
 from pulpcore.app.models import Artifact
@@ -116,7 +117,17 @@ def _ensure_writer_has_open_file(self):
         allowing plugin writers to instantiate many downloaders in memory.
         """
         if not self._writer:
-            self._writer = tempfile.NamedTemporaryFile(dir=".", delete=False)
+            filename = urlsplit(self.url).path.split("/")[-1]
+            # linux allows any character except NUL or / in a filename and has a length limit of
+            # 255. Making it urlencoding-aware would be nice, but not critical, because urlencoded
+            # paths should be OK
+            is_legal_filename = filename and (len(filename) <= 243)  # 255 - prefix length
+            # if the filename isn't legal then we just fall back to no suffix (random name)
+            suffix = "-" + filename if is_legal_filename else None
+            # write the file to the current working directory with a random prefix and the
+            # desired suffix. we always want the random prefix as it is possible to download
+            # the same filename from two different URLs, and the files may not be the same.
+            self._writer = tempfile.NamedTemporaryFile(dir=".", suffix=suffix, delete=False)
             self.path = self._writer.name
             self._digests = {n: pulp_hashlib.new(n) for n in Artifact.DIGEST_FIELDS}
             self._size = 0


Test patch:
Repository: pulp/pulpcore

Issue:
Download machinery should attempt to keep the file extension of the downloaded URL intact, if one exists
**Version**

All

**Describe the bug**

Sometimes libraries will get upset when you pass into them a file without the file extension they expect for the type of file they are handling.  It is best to keep the file extension intact to avoid those kinds of issues.

See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

There is a deeper issue there with the `file` / `libmagic` library, but until that is fixed and shipped the best we can do is help it out by keeping the original filenames.

I'm sure it will also help with debugging on occasion.

**To Reproduce**
See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

**Additional context**
BZ https://bugzilla.redhat.com/show_bug.cgi?id=2176870



Discussion:

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -5,6 +5,7 @@
 import logging
 import os
 import tempfile
+from urllib.parse import urlsplit
 
 from pulpcore.app import pulp_hashlib
 from pulpcore.app.loggers import deprecation_logger
@@ -129,7 +130,17 @@ def _ensure_writer_has_open_file(self):
         allowing plugin writers to instantiate many downloaders in memory.
         """
         if not self._writer:
-            self._writer = tempfile.NamedTemporaryFile(dir=".", delete=False)
+            filename = urlsplit(self.url).path.split("/")[-1]
+            # linux allows any character except NUL or / in a filename and has a length limit of
+            # 255. Making it urlencoding-aware would be nice, but not critical, because urlencoded
+            # paths should be OK
+            is_legal_filename = filename and (len(filename) <= 243)  # 255 - prefix length
+            # if the filename isn't legal then we just fall back to no suffix (random name)
+            suffix = "-" + filename if is_legal_filename else None
+            # write the file to the current working directory with a random prefix and the
+            # desired suffix. we always want the random prefix as it is possible to download
+            # the same filename from two different URLs, and the files may not be the same.
+            self._writer = tempfile.NamedTemporaryFile(dir=".", suffix=suffix, delete=False)
             self.path = self._writer.name
             self._digests = {n: pulp_hashlib.new(n) for n in Artifact.DIGEST_FIELDS}
             self._size = 0


Test patch:
Repository: pulp/pulpcore

Issue:
Download machinery should attempt to keep the file extension of the downloaded URL intact, if one exists
**Version**

All

**Describe the bug**

Sometimes libraries will get upset when you pass into them a file without the file extension they expect for the type of file they are handling.  It is best to keep the file extension intact to avoid those kinds of issues.

See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

There is a deeper issue there with the `file` / `libmagic` library, but until that is fixed and shipped the best we can do is help it out by keeping the original filenames.

I'm sure it will also help with debugging on occasion.

**To Reproduce**
See https://bugzilla.redhat.com/show_bug.cgi?id=2176870

**Additional context**
BZ https://bugzilla.redhat.com/show_bug.cgi?id=2176870



Discussion:

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -5,6 +5,7 @@
 import logging
 import os
 import tempfile
+from urllib.parse import urlsplit
 
 from pulpcore.app import pulp_hashlib
 from pulpcore.app.models import Artifact
@@ -122,7 +123,17 @@ def _ensure_writer_has_open_file(self):
         allowing plugin writers to instantiate many downloaders in memory.
         """
         if not self._writer:
-            self._writer = tempfile.NamedTemporaryFile(dir=".", delete=False)
+            filename = urlsplit(self.url).path.split("/")[-1]
+            # linux allows any character except NUL or / in a filename and has a length limit of
+            # 255. Making it urlencoding-aware would be nice, but not critical, because urlencoded
+            # paths should be OK
+            is_legal_filename = filename and (len(filename) <= 243)  # 255 - prefix length
+            # if the filename isn't legal then we just fall back to no suffix (random name)
+            suffix = "-" + filename if is_legal_filename else None
+            # write the file to the current working directory with a random prefix and the
+            # desired suffix. we always want the random prefix as it is possible to download
+            # the same filename from two different URLs, and the files may not be the same.
+            self._writer = tempfile.NamedTemporaryFile(dir=".", suffix=suffix, delete=False)
             self.path = self._writer.name
             self._digests = {n: pulp_hashlib.new(n) for n in Artifact.DIGEST_FIELDS}
             self._size = 0


Test patch:
Repository: pulp/pulpcore

Issue:
The order of authentication classes causes Pulp to return 403 instead of 401 on failed authentication
https://github.com/pulp/pulpcore/blob/50bdd1ba0d2ce7ae4d6eef15a2dd7c569d49c03a/pulpcore/app/settings.py#L150

In pulp_container, we have a view that should handle insufficient, invalid, or valid credentials provided by users. If the users pass invalid credentials to that view, Pulp returns 403 instead of 401.

Changing the order of authentication classes could resolve the problem according to https://www.django-rest-framework.org/api-guide/authentication/#unauthorized-and-forbidden-responses.

> The first authentication class set on the view is used when determining the type of response.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/settings.py b/pulpcore/app/settings.py
--- a/pulpcore/app/settings.py
+++ b/pulpcore/app/settings.py
@@ -148,8 +148,8 @@
     "PAGE_SIZE": 100,
     "DEFAULT_PERMISSION_CLASSES": ("pulpcore.plugin.access_policy.AccessPolicyFromDB",),
     "DEFAULT_AUTHENTICATION_CLASSES": (
-        "rest_framework.authentication.SessionAuthentication",
         "rest_framework.authentication.BasicAuthentication",
+        "rest_framework.authentication.SessionAuthentication",
     ),
     "UPLOADED_FILES_USE_URL": False,
     "DEFAULT_VERSIONING_CLASS": "rest_framework.versioning.URLPathVersioning",


Test patch:
diff --git a/pulpcore/tests/functional/api/test_auth.py b/pulpcore/tests/functional/api/test_auth.py
--- a/pulpcore/tests/functional/api/test_auth.py
+++ b/pulpcore/tests/functional/api/test_auth.py
@@ -32,7 +32,7 @@ def test_base_auth_failure(artifacts_api_client, invalid_user):
         with pytest.raises(ApiException) as e:
             artifacts_api_client.list()
 
-    assert e.value.status == 403
+    assert e.value.status == 401
     response = json.loads(e.value.body)
     response_detail = response["detail"].lower()
     for key in ("invalid", "username", "password"):
@@ -49,7 +49,7 @@ def test_base_auth_required(artifacts_api_client, anonymous_user):
         with pytest.raises(ApiException) as e:
             artifacts_api_client.list()
 
-    assert e.value.status == 403
+    assert e.value.status == 401
     response = json.loads(e.value.body)
     response_detail = response["detail"].lower()
     for key in ("authentication", "credentials", "provided"):
Repository: pulp/pulpcore

Issue:
There's a race when creating the same content in multiple processes
`Task 90df818f-3645-4b7c-8aaa-fae400a10ae0 failed (duplicate key value violates unique constraint "file_filecontent_relative_path_digest__pu_b4bae2c2_uniq"
DETAIL:  Key (relative_path, digest, _pulp_domain_id)=(B, df7e70e5021544f4834bbee64a9e3789febc4be81470df629cad6ddb03320a5c, be9b7087-f318-48ef-81ce-f3141524c659) already exists.
)`


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/content.py b/pulpcore/app/serializers/content.py
--- a/pulpcore/app/serializers/content.py
+++ b/pulpcore/app/serializers/content.py
@@ -1,6 +1,6 @@
 from gettext import gettext as _
 
-from django.db import transaction
+from django.db import transaction, IntegrityError
 from rest_framework import serializers
 from rest_framework.validators import UniqueValidator
 
@@ -45,7 +45,6 @@ def __init__(self, *args, **kwargs):
         if hasattr(self.Meta.model, "relative_path") and "relative_path" in self.fields:
             self.fields["relative_path"].write_only = False
 
-    @transaction.atomic
     def create(self, validated_data):
         """
         Create the content and associate it with its Artifact, or retrieve the existing content.
@@ -63,10 +62,16 @@ def create(self, validated_data):
                 relative_path = validated_data.pop("relative_path")
             else:
                 relative_path = validated_data.get("relative_path")
-            content = self.Meta.model.objects.create(**validated_data)
-            models.ContentArtifact.objects.create(
-                artifact=artifact, content=content, relative_path=relative_path
-            )
+            try:
+                with transaction.atomic():
+                    content = self.Meta.model.objects.create(**validated_data)
+                    models.ContentArtifact.objects.create(
+                        artifact=artifact, content=content, relative_path=relative_path
+                    )
+            except IntegrityError:
+                content = self.retrieve(validated_data)
+                if content is None:
+                    raise
 
         return content
 


Test patch:
Repository: pulp/pulpcore

Issue:
Remove verify_signature() in favour of gpg_verify()



Discussion:
This is a follow-up task to remove deprecated functions/methods that should no longer be available for users as of v3.25.

Resolving patch:
diff --git a/pulpcore/app/util.py b/pulpcore/app/util.py
--- a/pulpcore/app/util.py
+++ b/pulpcore/app/util.py
@@ -288,33 +288,6 @@ def get_request_without_query_params(context):
     return request
 
 
-def verify_signature(filepath, public_key, detached_data=None):
-    """
-    Check whether the provided file can be verified with the particular public key.
-
-    When dealing with a detached signature (referenced by the 'filepath' argument), one have to pass
-    the reference to a data file that was signed by that signature.
-    """
-    deprecation_logger.warning(
-        "verify_signature() is deprecated and will be removed in pulpcore==3.25; use gpg_verify()."
-    )
-
-    with tempfile.TemporaryDirectory(dir=settings.WORKING_DIRECTORY) as temp_directory_name:
-        gpg = gnupg.GPG(gnupghome=temp_directory_name)
-        gpg.import_keys(public_key)
-        imported_keys = gpg.list_keys()
-
-        if len(imported_keys) != 1:
-            raise RuntimeError("Exactly one key must be imported.")
-
-        with open(filepath, "rb") as signature:
-            verified = gpg.verify_file(signature, detached_data)
-            if not verified.valid:
-                raise InvalidSignatureError(
-                    f"The file '{filepath}' does not contain a valid signature."
-                )
-
-
 def gpg_verify(public_keys, signature, detached_data=None):
     """
     Check whether the provided gnupg signature is valid for one of the provided public keys.
diff --git a/pulpcore/plugin/util.py b/pulpcore/plugin/util.py
--- a/pulpcore/plugin/util.py
+++ b/pulpcore/plugin/util.py
@@ -18,7 +18,6 @@
     get_url,
     gpg_verify,
     raise_for_unknown_content_units,
-    verify_signature,
     get_default_domain,
     get_domain,
     get_domain_pk,


Test patch:
Repository: pulp/pulpcore

Issue:
Remove `BaseDistribution` from the plugin API
We tried to do this once before but there were migration issues perhaps?

edit: See comment below
> Maybe we just update the docstring with info that it can't be removed until Pulp4. and use this issue for that .misc change.


Discussion:
Posting some matrix convo that happened some time ago:

x9c4
Hey folks, do you think we are ready to remove the BaseDistribution model from the database? Or is there any reason not to do so?
^
dalley maybe you remember?
dralley
I don't remember the exact reason but I think it had to do with migrations.  Basically the historical migration record has lots of references to this BaseDistribution model
hopefully we wrote down the reason somewhere
x9c4
not in the code at least.
dralley
x9c4: https://pulp.plan.io/issues/8386#note-5
dralley
I think the TL;DR is:  the migrations happen all at once for a given app, be it pulpcore or a plugin, so if you do a speedrun through all the pulpcore migrations and remove the basedistribution model entirely, then the early migrations for some plugin will fail
in a new installation, where they haven't already been applied
x9c4
So we cannot reall remove it, because we do not controll all plugins migrations. I'l stop thinking about it again for a year.
x9c4
I was once looking into squashing migrations, but maybe for the same reasons of plugin interdependence we cannot really do this in Pulp.
dralley
Pulp 4 (tm)

We could at the least remove it from the plugin API and it would just live in the DB and the model definition inside of `pulpcore.app.models`.
Actually it already is removed from the plugin API https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/models/__init__.py Maybe we just update the docstring with info that it can't be removed until Pulp4. and use this issue for that .misc change.
@bmbouter i agree
... until we squashed enough migrations we cannot remove it.

Resolving patch:
diff --git a/pulpcore/app/models/publication.py b/pulpcore/app/models/publication.py
--- a/pulpcore/app/models/publication.py
+++ b/pulpcore/app/models/publication.py
@@ -469,6 +469,11 @@ class BaseDistribution(MasterModel):
     of ``a/path/foo`` existed, you could not make a second Distribution with a ``base_path`` of
     ``a/path`` or ``a`` because both are subpaths of ``a/path/foo``.
 
+    Note:
+        This class is no longer supported and cannot be removed from Pulp 3 due to possible
+        problems with old migrations for plugins. Until the migrations are squashed, this class
+        should be preserved.
+
     Fields:
         name (models.TextField): The name of the distribution. Examples: "rawhide" and "stable".
         base_path (models.TextField): The base (relative) path component of the published url.


Test patch: