internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
Artifacts storage association does not honor transaction semantics.
**Version**
At least up to 3.24

**Describe the bug**
When deleting an Artifact in a transaction that may subsequently fail and not be commited, the file associated with the artifact is already removed from the storage. This should not happen.

**To Reproduce**
See listing below.

**Expected behavior**
When not committing the transaction after deleting, the artifact should not be mutilated.

**Additional context**
```
In [6]: Artifact.objects.all()
Out[6]: <BulkTouchQuerySet [<Artifact: pk=0188014a-120f-7e05-9e65-3de72d45182e>]>

In [7]: a = Artifact.objects.first()

In [8]: with transaction.atomic():
   ...:     a.delete()
   ...:     raise Exception()
   ...: 
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[8], line 3
      1 with transaction.atomic():
      2     a.delete()
----> 3     raise Exception()

Exception: 

In [9]: a
Out[9]: <Artifact: pk=None>

In [10]: a = Artifact.objects.first()

In [11]: a
Out[11]: <Artifact: pk=0188014a-120f-7e05-9e65-3de72d45182e>

In [12]: a.file
Out[12]: <FieldFile: artifact/99/322da247546beac65c2f5b7d164e94dcaf050b85a6076a00cf4fec5e21a6ed>

In [13]: a.file.read()
---------------------------------------------------------------------------
FileNotFoundError
```



Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/content.py b/pulpcore/app/models/content.py
--- a/pulpcore/app/models/content.py
+++ b/pulpcore/app/models/content.py
@@ -11,7 +11,7 @@
 import subprocess
 
 from collections import defaultdict
-from functools import lru_cache
+from functools import lru_cache, partial
 from itertools import chain
 
 from django.conf import settings
@@ -162,7 +162,8 @@ def delete(self, *args, **kwargs):
             kwargs (dict): dictionary of keyword arguments to pass to Model.delete()
         """
         super().delete(*args, **kwargs)
-        self.file.delete(save=False)
+        # In case of rollback, we want the artifact to stay connected with it's file.
+        transaction.on_commit(partial(self.file.delete, save=False))
 
 
 class ArtifactManager(BulkCreateManager):


Test patch:
Repository: pulp/pulpcore

Issue:
Global tasks like repair or reclaim should probably not run in parallel
**Version**
Please provide the versions of the pulpcore and plugin packages in use, and how they are installed.  If you are using Pulp via Katello, please provide the Katello version.

**Describe the bug**
Global tasks that involve all repos like repair or reclaim should run one at a time, like orphan cleanup https://github.com/pulp/pulpcore/blob/main/pulpcore/app/viewsets/orphans.py#L29

**To Reproduce**
Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.  Please provide links to any previous discussions via Discourse or Bugzilla.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/views/repair.py b/pulpcore/app/views/repair.py
--- a/pulpcore/app/views/repair.py
+++ b/pulpcore/app/views/repair.py
@@ -1,4 +1,5 @@
 from drf_spectacular.utils import extend_schema
+from django.conf import settings
 from rest_framework.views import APIView
 
 from pulpcore.app.response import OperationPostponedResponse
@@ -26,6 +27,9 @@ def post(self, request):
 
         verify_checksums = serializer.validated_data["verify_checksums"]
 
-        task = dispatch(repair_all_artifacts, args=[verify_checksums])
+        uri = "/api/v3/repair/"
+        if settings.DOMAIN_ENABLED:
+            uri = f"/{request.pulp_domain.name}{uri}"
+        task = dispatch(repair_all_artifacts, exclusive_resources=[uri], args=[verify_checksums])
 
         return OperationPostponedResponse(task, request)
diff --git a/pulpcore/app/viewsets/reclaim.py b/pulpcore/app/viewsets/reclaim.py
--- a/pulpcore/app/viewsets/reclaim.py
+++ b/pulpcore/app/viewsets/reclaim.py
@@ -1,4 +1,5 @@
 from drf_spectacular.utils import extend_schema
+from django.conf import settings
 from rest_framework.viewsets import ViewSet
 
 from pulpcore.app.response import OperationPostponedResponse
@@ -35,8 +36,17 @@ def reclaim(self, request):
             repos.append(rv.repository)
             keeplist_rv_pks.append(rv.pk)
 
+        if repos:
+            exclusive_resources = None
+        else:
+            uri = "/api/v3/repositories/reclaim_space/"
+            if settings.DOMAIN_ENABLED:
+                uri = f"/{request.pulp_domain.name}{uri}"
+            exclusive_resources = [uri]
+
         task = dispatch(
             reclaim_space,
+            exclusive_resources=exclusive_resources,
             shared_resources=repos,
             kwargs={
                 "repo_pks": reclaim_repo_pks,


Test patch:
Repository: pulp/pulpcore

Issue:
The repository of a repository version should be pre-selected when looking up the distribution
https://github.com/pulp/pulpcore/blob/0a50a368b5090135076b608c61db01514350d46d/pulpcore/content/handler.py#L291


Discussion:

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -289,6 +289,7 @@ def _match_distribution(cls, path):
                 .select_related(
                     "repository",
                     "repository_version",
+                    "repository_version__repository",
                     "publication",
                     "remote",
                     "pulp_domain",


Test patch:
Repository: pulp/pulpcore

Issue:
Make it  possible for plugins to specify headers for content-app responses.
See https://github.com/pulp/pulp_rpm/issues/2947 for the need that drives this.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/publication.py b/pulpcore/app/models/publication.py
--- a/pulpcore/app/models/publication.py
+++ b/pulpcore/app/models/publication.py
@@ -574,6 +574,17 @@ def content_handler_list_directory(self, rel_path):
         """
         return set()
 
+    def content_headers_for(self, path):
+        """
+        Opportunity for Distribution to specify response-headers for a specific path
+
+        Args:
+            path (str): The path being requested
+        Returns:
+            Empty dictionary, or a dictionary with HTTP Response header/value pairs.
+        """
+        return {}
+
     @hook(BEFORE_DELETE)
     @hook(
         AFTER_UPDATE,
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -1,5 +1,6 @@
 import asyncio
 import logging
+from multidict import CIMultiDict
 import os
 import re
 from gettext import gettext as _
@@ -345,20 +346,29 @@ def _permit(request, distribution):
         return True
 
     @staticmethod
-    def response_headers(path):
+    def response_headers(path, distribution=None):
         """
         Get the Content-Type and Encoding-Type headers for the requested `path`.
 
         Args:
             path (str): The relative path that was requested.
-
+            distribution(Distribution) : Distribution detail that might want to add headers for path
         Returns:
             headers (dict): A dictionary of response headers.
         """
+        # headers are case-insensitive
+        headers = CIMultiDict({})
+
+        # Determine a content-type from mime_types and set.
+        # Note: plugin-Distribution can override this.
         content_type = mime_types.get_type(path)
-        headers = {}
         if content_type:
             headers["Content-Type"] = content_type
+
+        # Let plugin-Distribution set headers for this path if it wants.
+        if distribution:
+            headers.update(distribution.content_headers_for(path))
+
         return headers
 
     @staticmethod
@@ -537,7 +547,7 @@ async def _match_and_stream(self, path, request):
         if content_handler_result is not None:
             return content_handler_result
 
-        headers = self.response_headers(rel_path)
+        headers = self.response_headers(rel_path, distro)
 
         repository = distro.repository
         publication = distro.publication
@@ -569,7 +579,7 @@ async def _match_and_stream(self, path, request):
                     await publication.published_artifact.aget(relative_path=index_path)
 
                     rel_path = index_path
-                    headers = self.response_headers(rel_path)
+                    headers = self.response_headers(rel_path, distro)
                 except ObjectDoesNotExist:
                     dir_list, dates, sizes = await self.list_directory(None, publication, rel_path)
                     dir_list.update(


Test patch:
Repository: pulp/pulpcore

Issue:
Pull through requesting existing content produces a bad call to contentartifact_set
**Version**
3.28, as this code was not released earlier.

**Describe the bug**
```
File "/usr/local/lib/python3.8/site-packages/pulpcore/content/handler.py", line 847, in _save_artifact
  cas = list(content.contentartifact_set().select_related("artifact"))
TypeError: __call__() missing 1 required keyword-only argument: 'manager'
```


Discussion:

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -844,7 +844,7 @@ def _save_artifact(self, download_result, remote_artifact, request=None):
                     # There is already content saved
                     content = c_type.objects.get(content.q())
                     created_artifact_digests = {rp: a.sha256 for rp, a in artifacts.items() if a}
-                    cas = list(content.contentartifact_set().select_related("artifact"))
+                    cas = list(content.contentartifact_set.select_related("artifact"))
                     found_artifact_digests = {
                         ca.relative_path: ca.artifact.sha256 for ca in cas if ca.artifact
                     }


Test patch:
Repository: pulp/pulpcore

Issue:
RESTAPI document fix for Upstream Pulp Replication API
**Version**
Pulp installed through the Python modules.
"core:3.28.0"
"certguard:3.28.0"
"file:3.28.0"
"python:3.28.0"
"rpm:3.28.0"

**Describe the bug**
Why the attributes of **upstream_pulps_create**/**update**  is mentioned again in the **upstream_pulps_replicate" document? Are those attributes (base_url, api_root, domain,...) used at time making an API request "https://PULP-SERVER/pulp/api/v3/upstream_pulps/{object_id}/replicate/"?

**To Reproduce**
None.

**Expected behavior**
A fix is required in the REST API document.

**Additional context**
Create Upstream Pulp API document: https://docs.pulpproject.org/pulpcore/restapi.html#tag/Upstream-Pulps/operation/upstream_pulps_create
Upstream Replication API document: https://docs.pulpproject.org/pulpcore/restapi.html#tag/Upstream-Pulps/operation/upstream_pulps_replicate




Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/replica.py b/pulpcore/app/viewsets/replica.py
--- a/pulpcore/app/viewsets/replica.py
+++ b/pulpcore/app/viewsets/replica.py
@@ -33,6 +33,7 @@ class UpstreamPulpViewSet(
         summary="Replicate",
         description="Trigger an asynchronous repository replication task group. This API is "
         "provided as a tech preview.",
+        request=None,
         responses={202: AsyncOperationResponseSerializer},
     )
     @action(detail=True, methods=["post"])


Test patch:
Repository: pulp/pulpcore

Issue:
BaseDownloader.fetch passes coroutine to asyncio.wait which is forbidden in python 3.11
Python 3.8 deprecated passing coroutines to `asyncio.wait` and Python 3.11 will now [raise an error](https://github.com/python/cpython/blob/a6313d78f21f79ca64dedd38e637509dc530a1b6/Lib/asyncio/tasks.py#L414C13-L414C13). This causes the BaseDownloader.fetch call to fail on Python 3.11 https://github.com/pulp/pulpcore/blob/9dbcc8810f97f53297a933df2e1b74cdc324a8ea/pulpcore/download/base.py#L185 .

Python provides the solution in the error message:  "Passing coroutines is forbidden, use tasks explicitly."

I believe this can be fixed by explicitly converting the coroutine to a task using asyncio's `create_task`


Discussion:
Honestly why is it wrapped inside `asyncio.wait` anyway?
Also please read the python docs carefully about when things got added. There was confusion about the availability of create_task vs. ensure_future depending on the python version before.
We need to maintain at least python 3.8 compatibility.

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -182,8 +182,8 @@ def fetch(self):
         Raises:
             Exception: Any fatal exception emitted during downloading
         """
-        done, _ = asyncio.get_event_loop().run_until_complete(asyncio.wait([self.run()]))
-        return done.pop().result()
+        result = asyncio.get_event_loop().run_until_complete(self.run())
+        return result
 
     def _record_size_and_digests_for_data(self, data):
         """


Test patch:
Repository: pulp/pulpcore

Issue:
BaseDownloader.fetch passes coroutine to asyncio.wait which is forbidden in python 3.11
Python 3.8 deprecated passing coroutines to `asyncio.wait` and Python 3.11 will now [raise an error](https://github.com/python/cpython/blob/a6313d78f21f79ca64dedd38e637509dc530a1b6/Lib/asyncio/tasks.py#L414C13-L414C13). This causes the BaseDownloader.fetch call to fail on Python 3.11 https://github.com/pulp/pulpcore/blob/9dbcc8810f97f53297a933df2e1b74cdc324a8ea/pulpcore/download/base.py#L185 .

Python provides the solution in the error message:  "Passing coroutines is forbidden, use tasks explicitly."

I believe this can be fixed by explicitly converting the coroutine to a task using asyncio's `create_task`


Discussion:
Honestly why is it wrapped inside `asyncio.wait` anyway?
Also please read the python docs carefully about when things got added. There was confusion about the availability of create_task vs. ensure_future depending on the python version before.
We need to maintain at least python 3.8 compatibility.

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -187,8 +187,8 @@ def fetch(self):
         Raises:
             Exception: Any fatal exception emitted during downloading
         """
-        done, _ = asyncio.get_event_loop().run_until_complete(asyncio.wait([self.run()]))
-        return done.pop().result()
+        result = asyncio.get_event_loop().run_until_complete(self.run())
+        return result
 
     def _record_size_and_digests_for_data(self, data):
         """


Test patch:
Repository: pulp/pulpcore

Issue:
BaseDownloader.fetch passes coroutine to asyncio.wait which is forbidden in python 3.11
Python 3.8 deprecated passing coroutines to `asyncio.wait` and Python 3.11 will now [raise an error](https://github.com/python/cpython/blob/a6313d78f21f79ca64dedd38e637509dc530a1b6/Lib/asyncio/tasks.py#L414C13-L414C13). This causes the BaseDownloader.fetch call to fail on Python 3.11 https://github.com/pulp/pulpcore/blob/9dbcc8810f97f53297a933df2e1b74cdc324a8ea/pulpcore/download/base.py#L185 .

Python provides the solution in the error message:  "Passing coroutines is forbidden, use tasks explicitly."

I believe this can be fixed by explicitly converting the coroutine to a task using asyncio's `create_task`


Discussion:
Honestly why is it wrapped inside `asyncio.wait` anyway?
Also please read the python docs carefully about when things got added. There was confusion about the availability of create_task vs. ensure_future depending on the python version before.
We need to maintain at least python 3.8 compatibility.

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -194,8 +194,8 @@ def fetch(self):
         Raises:
             Exception: Any fatal exception emitted during downloading
         """
-        done, _ = asyncio.get_event_loop().run_until_complete(asyncio.wait([self.run()]))
-        return done.pop().result()
+        result = asyncio.get_event_loop().run_until_complete(self.run())
+        return result
 
     def _record_size_and_digests_for_data(self, data):
         """


Test patch:
Repository: pulp/pulpcore

Issue:
Reclaim space for repository fails with Cannot delete some instances of model 'Artifact' because they are referenced through protected foreign keys: 'ContentArtifact.artifact'."
**Version**
3.16, but probably all versions

**Describe the bug**

- Reclaim space for repository fails with the following error.

  ~~~
  Task paused with error: "("Cannot delete some instances of model 'Artifact' because they are referenced through protected foreign keys: 'ContentArtifact.artifact'.", {<ContentArtifact: pk=452959ad-c045-4e85-bf9f-6651ba37f57d>})"
  ~~~

**To Reproduce**
See BZ

**Additional context**
https://bugzilla.redhat.com/show_bug.cgi?id=2169322



Discussion:
Isn't that the expected behaviour? When an artifact is used by another content then reclaim must not delete it.
OK, maybe there is a check missing that we do not even attempt to delete it in that case.
The issue is that it shouldn't perform invalid actions in the first place, not that the database catches them.
use this repo to reproduce `Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server`

Resolving patch:
diff --git a/pulpcore/app/tasks/reclaim_space.py b/pulpcore/app/tasks/reclaim_space.py
--- a/pulpcore/app/tasks/reclaim_space.py
+++ b/pulpcore/app/tasks/reclaim_space.py
@@ -1,3 +1,7 @@
+from logging import getLogger
+
+from django.db.models.deletion import ProtectedError
+
 from pulpcore.app.models import (
     Artifact,
     Content,
@@ -9,6 +13,8 @@
 )
 from pulpcore.app.util import get_domain
 
+log = getLogger(__name__)
+
 
 def reclaim_space(repo_pks, keeplist_rv_pks=None, force=False):
     """
@@ -76,10 +82,16 @@ def reclaim_space(repo_pks, keeplist_rv_pks=None, force=False):
     counter = 0
     interval = 100
     for artifact in artifacts_to_delete.iterator():
-        # we need to manually call delete() because it cleans up the file on the filesystem
-        artifact.delete()
-        progress_bar.done += 1
-        counter += 1
+        try:
+            # we need to manually call delete() because it cleans up the file on the filesystem
+            artifact.delete()
+        except ProtectedError as e:
+            # Rarely artifact could be shared between to different content units.
+            # Just log and skip the artifact deletion in this case
+            log.info(e)
+        else:
+            progress_bar.done += 1
+            counter += 1
 
         if counter >= interval:
             progress_bar.save()


Test patch:
Repository: pulp/pulpcore

Issue:
Reclaim space for repository fails with Cannot delete some instances of model 'Artifact' because they are referenced through protected foreign keys: 'ContentArtifact.artifact'."
**Version**
3.16, but probably all versions

**Describe the bug**

- Reclaim space for repository fails with the following error.

  ~~~
  Task paused with error: "("Cannot delete some instances of model 'Artifact' because they are referenced through protected foreign keys: 'ContentArtifact.artifact'.", {<ContentArtifact: pk=452959ad-c045-4e85-bf9f-6651ba37f57d>})"
  ~~~

**To Reproduce**
See BZ

**Additional context**
https://bugzilla.redhat.com/show_bug.cgi?id=2169322



Discussion:
Isn't that the expected behaviour? When an artifact is used by another content then reclaim must not delete it.
OK, maybe there is a check missing that we do not even attempt to delete it in that case.
The issue is that it shouldn't perform invalid actions in the first place, not that the database catches them.
use this repo to reproduce `Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server`

Resolving patch:
diff --git a/pulpcore/app/tasks/reclaim_space.py b/pulpcore/app/tasks/reclaim_space.py
--- a/pulpcore/app/tasks/reclaim_space.py
+++ b/pulpcore/app/tasks/reclaim_space.py
@@ -1,3 +1,7 @@
+from logging import getLogger
+
+from django.db.models.deletion import ProtectedError
+
 from pulpcore.app.models import (
     Artifact,
     Content,
@@ -8,6 +12,8 @@
     RepositoryVersion,
 )
 
+log = getLogger(__name__)
+
 
 def reclaim_space(repo_pks, keeplist_rv_pks=None, force=False):
     """
@@ -74,10 +80,16 @@ def reclaim_space(repo_pks, keeplist_rv_pks=None, force=False):
     counter = 0
     interval = 100
     for artifact in artifacts_to_delete.iterator():
-        # we need to manually call delete() because it cleans up the file on the filesystem
-        artifact.delete()
-        progress_bar.done += 1
-        counter += 1
+        try:
+            # we need to manually call delete() because it cleans up the file on the filesystem
+            artifact.delete()
+        except ProtectedError as e:
+            # Rarely artifact could be shared between to different content units.
+            # Just log and skip the artifact deletion in this case
+            log.info(e)
+        else:
+            progress_bar.done += 1
+            counter += 1
 
         if counter >= interval:
             progress_bar.save()


Test patch:
Repository: pulp/pulpcore

Issue:
Enable passing extra data to the fetch method of BaseDownloader
This will enable plugins to issue head requests if needed.

```
        try:
            response = downloader.fetch(
                extra_data={"headers": V2_ACCEPT_HEADERS, "http_method": "head"}
            )
        except ClientResponseError:
            raise ManifestNotFound(reference=pk)
```


Discussion:

Resolving patch:
diff --git a/pulpcore/download/base.py b/pulpcore/download/base.py
--- a/pulpcore/download/base.py
+++ b/pulpcore/download/base.py
@@ -172,7 +172,7 @@ async def finalize(self):
         self.validate_size()
         log.debug(f"Downloaded file from {self.url}")
 
-    def fetch(self):
+    def fetch(self, extra_data=None):
         """
         Run the download synchronously and return the `DownloadResult`.
 
@@ -182,7 +182,7 @@ def fetch(self):
         Raises:
             Exception: Any fatal exception emitted during downloading
         """
-        result = asyncio.get_event_loop().run_until_complete(self.run())
+        result = asyncio.get_event_loop().run_until_complete(self.run(extra_data=extra_data))
         return result
 
     def _record_size_and_digests_for_data(self, data):


Test patch: