internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
file:// sync deletes files from directory
**Version**
Pulpcore 3.39

**Describe the bug**
When syncing file:// repositories, files are disappearing after the sync.

**To Reproduce**
1) Copy these two repositories to the FS:
  - https://github.com/Katello/katello/tree/master/test/fixtures/test_repos/file1
  - https://github.com/Katello/katello/tree/master/test/fixtures/test_repos/file2
2) Sync one, then the other
3) See that some files disappeared.
  - In my case, file2 lost every file except PULP_MANIFEST


**Expected behavior**
No files disappear.

**Additional context**
This also occurred with RPM content type files.



Discussion:
Likely related to https://github.com/pulp/pulpcore/pull/4428
A repeat of https://pulp.plan.io/issues/9146 ?

Resolving patch:
diff --git a/pulpcore/download/file.py b/pulpcore/download/file.py
--- a/pulpcore/download/file.py
+++ b/pulpcore/download/file.py
@@ -57,7 +57,7 @@ async def _run(self, extra_data=None):
                     break  # the reading is done
                 await self.handle_data(chunk)
             return DownloadResult(
-                path=self._path,
+                path=self.path,
                 artifact_attributes=self.artifact_attributes,
                 url=self.url,
                 headers=None,


Test patch:
Repository: pulp/pulpcore

Issue:
pulp file python package reporting wrongly
Starting with pulpcore 3.40 the pulp_file plugins python package started reporting as pulp_file instead of pulp-file.


Discussion:

Resolving patch:
diff --git a/pulp_file/app/__init__.py b/pulp_file/app/__init__.py
--- a/pulp_file/app/__init__.py
+++ b/pulp_file/app/__init__.py
@@ -9,5 +9,5 @@ class PulpFilePluginAppConfig(PulpPluginAppConfig):
     name = "pulp_file.app"
     label = "file"
     version = "3.42.0.dev"
-    python_package_name = "pulp_file"  # TODO Add python_module_name
+    python_package_name = "pulp-file"  # TODO Add python_module_name
     domain_compatible = True


Test patch:
Repository: pulp/pulpcore

Issue:
pulp file python package reporting wrongly
Starting with pulpcore 3.40 the pulp_file plugins python package started reporting as pulp_file instead of pulp-file.


Discussion:

Resolving patch:
diff --git a/pulp_file/app/__init__.py b/pulp_file/app/__init__.py
--- a/pulp_file/app/__init__.py
+++ b/pulp_file/app/__init__.py
@@ -9,5 +9,5 @@ class PulpFilePluginAppConfig(PulpPluginAppConfig):
     name = "pulp_file.app"
     label = "file"
     version = "3.41.1.dev"
-    python_package_name = "pulp_file"  # TODO Add python_module_name
+    python_package_name = "pulp-file"  # TODO Add python_module_name
     domain_compatible = True


Test patch:
Repository: pulp/pulpcore

Issue:
HTTPs Proxies not working.
**Version**
tfm-pulpcore-python3-pulpcore-3.17.7-1.el7.noarch
tfm-pulpcore-python3-aiohttp-3.8.1-2.el7.x86_64

**Describe the bug**
There was HTTPS proxy tunneling support added to the aiohttp library for 3.8.1 => https://github.com/aio-libs/aiohttp/pull/5992
However pulpcore does not 

1. Have the necessary bindings to get this mode to run.
2. Expects the CA Cert of the proxy to be concatenated to the repo's remote CA cert instead of expecting it in the default trust store (as it used to do in pulp2). 

**To Reproduce**
Steps to reproduce the behavior:
- Setup a https proxy or ping me about one
- Add the proxy's cacert to the default trust store.
- Add a repo remote with the feed and proxy urls
- Sync

**Expected behavior**
Clean Sync

**Actual behavior**
SSL error while trying to connect to the proxy.




Discussion:
Bugzilla 
https://bugzilla.redhat.com/show_bug.cgi?id=1993917
This works with Python 3.11. Closing the issue since Pulp is not in the position to update Python for users. 
@dkliban Does this work *immediately* with Python 3.11 or are other patches required?  Because I see Partha's PR which monkeypatches the internal flag that enables this support, but that's not all it does, it does other things too.

Will we need to adopt some or all of those other changes in order for this to work?  If so then this issue ought to be reopened because we won't get this resolved for free by upgrading the Python runtime.
This should just work with Python 3.11. There are no code changes in Pulp required. 
Reopening - while python/3.11 and recent aiohttp will (finally) allow this, Pulp still needs to load the system-allowed certstore to be able to trust an HTTPS proxy's CA.  

Resolving patch:
diff --git a/pulpcore/download/factory.py b/pulpcore/download/factory.py
--- a/pulpcore/download/factory.py
+++ b/pulpcore/download/factory.py
@@ -120,6 +120,8 @@ def _make_aiohttp_session_from_remote(self):
             sslcontext.verify_mode = ssl.CERT_NONE
         if sslcontext:
             tcp_conn_opts["ssl_context"] = sslcontext
+            # Trust the system-known CA certs, not just the end-remote CA
+            sslcontext.load_default_certs()
 
         headers = MultiDict({"User-Agent": DownloaderFactory.user_agent()})
         if self._remote.headers is not None:


Test patch:
Repository: pulp/pulpcore

Issue:
Unawaited cache coroutine throws warnings
We're seeing the following error on pulpcore 3.45.0:

```
/venv/lib/python3.9/site-packages/pulpcore/cache/cache.py:386: RuntimeWarning: coroutine 'AsyncCache.delete' was never awaited
```


Discussion:

Resolving patch:
diff --git a/pulpcore/cache/cache.py b/pulpcore/cache/cache.py
--- a/pulpcore/cache/cache.py
+++ b/pulpcore/cache/cache.py
@@ -383,7 +383,7 @@ async def make_response(self, key, base_key):
             expires and expires < time.time()
         ):
             # Bad entry, delete from cache
-            self.delete(key, base_key)
+            await self.delete(key, base_key)
             return None
         response = self.RESPONSE_TYPES[response_type](**entry)
         response.headers.update({"X-PULP-CACHE": "HIT"})


Test patch:
Repository: pulp/pulpcore

Issue:
Unawaited cache coroutine throws warnings
We're seeing the following error on pulpcore 3.45.0:

```
/venv/lib/python3.9/site-packages/pulpcore/cache/cache.py:386: RuntimeWarning: coroutine 'AsyncCache.delete' was never awaited
```


Discussion:

Resolving patch:
diff --git a/pulpcore/cache/cache.py b/pulpcore/cache/cache.py
--- a/pulpcore/cache/cache.py
+++ b/pulpcore/cache/cache.py
@@ -364,7 +364,7 @@ async def make_response(self, key, base_key):
         response_type = entry.pop("type", None)
         if not response_type or response_type not in self.RESPONSE_TYPES:
             # Bad entry, delete from cache
-            self.delete(key, base_key)
+            await self.delete(key, base_key)
             return None
         response = self.RESPONSE_TYPES[response_type](**entry)
         response.headers.update({"X-PULP-CACHE": "HIT"})


Test patch:
Repository: pulp/pulpcore

Issue:
Unawaited cache coroutine throws warnings
We're seeing the following error on pulpcore 3.45.0:

```
/venv/lib/python3.9/site-packages/pulpcore/cache/cache.py:386: RuntimeWarning: coroutine 'AsyncCache.delete' was never awaited
```


Discussion:

Resolving patch:
diff --git a/pulpcore/cache/cache.py b/pulpcore/cache/cache.py
--- a/pulpcore/cache/cache.py
+++ b/pulpcore/cache/cache.py
@@ -366,7 +366,7 @@ async def make_response(self, key, base_key):
         response_type = entry.pop("type", None)
         if not response_type or response_type not in self.RESPONSE_TYPES:
             # Bad entry, delete from cache
-            self.delete(key, base_key)
+            await self.delete(key, base_key)
             return None
         response = self.RESPONSE_TYPES[response_type](**entry)
         response.headers.update({"X-PULP-CACHE": "HIT"})


Test patch:
Repository: pulp/pulpcore

Issue:
Plugin writers should have access to `TimeoutException` exception
This exception may be raised from the downloader and cannot be caught properly by the plugin writers.

Required for https://github.com/pulp/pulp_container/issues/1499.


Discussion:

Resolving patch:
diff --git a/pulpcore/plugin/exceptions.py b/pulpcore/plugin/exceptions.py
--- a/pulpcore/plugin/exceptions.py
+++ b/pulpcore/plugin/exceptions.py
@@ -4,5 +4,6 @@
     PulpException,
     SizeValidationError,
     MissingDigestValidationError,
+    TimeoutException,
     UnsupportedDigestValidationError,
 )


Test patch:
Repository: pulp/pulpcore

Issue:
OTEL: pulp_disk_usage_Bytes reports name instead of domain_name
**Version**
3.49.0

**Describe the bug**
The `pulp_disk_usage_Bytes` metric emitted by Pulp contains a `name` attribute that represents a domain name. It would be better if the attribute was called `domain_name` instead of `name`. Here is what I am currently seeing in Prometheus:

```
pulp_disk_usage_Bytes{endpoint="otel-8889", exported_job="pulp-api", instance="10.129.9.102:8889", job="pulp-otel-collector-svc", name="25c87df8", namespace="pulp-stage", pod="pulp-otel-collector-5dd77b5668-pq8pv", pulp_href="/api/pulp/default/api/v3/domains/018d1440-d486-7722-9319-ac0364cc5408/", service="pulp-otel-collector-svc"}
```


Discussion:

Resolving patch:
diff --git a/pulpcore/app/models/domain.py b/pulpcore/app/models/domain.py
--- a/pulpcore/app/models/domain.py
+++ b/pulpcore/app/models/domain.py
@@ -95,5 +95,5 @@ def disk_usage_callback(domain):
         distinct_artifacts = Artifact.objects.filter(pulp_domain=domain).distinct()
         total_size = distinct_artifacts.aggregate(size=models.Sum("size", default=0))["size"]
         options = yield [  # noqa
-            Observation(total_size, {"pulp_href": get_url(domain), "name": domain.name})
+            Observation(total_size, {"pulp_href": get_url(domain), "domain_name": domain.name})
         ]
diff --git a/pulpcore/app/util.py b/pulpcore/app/util.py
--- a/pulpcore/app/util.py
+++ b/pulpcore/app/util.py
@@ -547,7 +547,8 @@ def _disk_usage_callback(self):
                 total_size = artifacts.aggregate(size=Sum("size", default=0))["size"]
                 options = yield [  # noqa
                     metrics.Observation(
-                        total_size, {"pulp_href": get_url(self.domain), "name": self.domain.name}
+                        total_size,
+                        {"pulp_href": get_url(self.domain), "domain_name": self.domain.name},
                     )
                 ]
 


Test patch:
Repository: pulp/pulpcore

Issue:
Task cleanup must not delete content nor artifacts
Deleting content or artifacts outside of orphan cleanup is breaking the rules.
And no, we cannot get away with that.



Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/_util.py b/pulpcore/tasking/_util.py
--- a/pulpcore/tasking/_util.py
+++ b/pulpcore/tasking/_util.py
@@ -15,7 +15,7 @@
 from django.utils import timezone
 from django_guid import set_guid
 from django_guid.utils import generate_guid
-from pulpcore.app.models import Task, TaskSchedule
+from pulpcore.app.models import Artifact, Content, Task, TaskSchedule
 from pulpcore.app.role_util import get_users_with_perms
 from pulpcore.app.util import (
     set_current_user,
@@ -73,6 +73,8 @@ def delete_incomplete_resources(task):
     if task.state != TASK_STATES.CANCELING:
         raise RuntimeError(_("Task must be canceling."))
     for model in (r.content_object for r in task.created_resources.all()):
+        if isinstance(model, (Artifact, Content)):
+            continue
         try:
             if model.complete:
                 continue


Test patch:
Repository: pulp/pulpcore

Issue:
Add indices to task table to speed up worker queries.
Specifically dispatching with `immediate=True` yields a query like:
`SELECT 1 AS "a" FROM "core_task" WHERE ("core_task"."pulp_created" < '2024-05-13 23:29:28.153293+00:00'::timestamptz AND "core_task"."state" IN ('waiting', 'running', 'canceling') AND "core_task"."reserved_resources_record" && (ARRAY['/pulp/api/v3/domains/018f5c78-b36b-7b26-9b62-46d8a412c1a3/', '/api/v3/distributions/', 'shared:/api/v3/distributions/'])::text[]) LIMIT 1;`

It is not entirely clear if an index on `state` only or a combined index with `state`, `pulp_created` and `unblocked_at` will perform best.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/migrations/0118_task_core_task_unblock_2276a4_idx_and_more.py b/pulpcore/app/migrations/0118_task_core_task_unblock_2276a4_idx_and_more.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/migrations/0118_task_core_task_unblock_2276a4_idx_and_more.py
@@ -0,0 +1,25 @@
+# Generated by Django 4.2.11 on 2024-05-14 08:07
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("core", "0117_task_unblocked_at"),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name="task",
+            index=models.Index(fields=["unblocked_at"], name="core_task_unblock_2276a4_idx"),
+        ),
+        migrations.AddIndex(
+            model_name="task",
+            index=models.Index(fields=["state"], name="core_task_state_61f0ca_idx"),
+        ),
+        migrations.AddIndex(
+            model_name="task",
+            index=models.Index(fields=["state", "pulp_created"], name="core_task_state_3742f2_idx"),
+        ),
+    ]
diff --git a/pulpcore/app/models/task.py b/pulpcore/app/models/task.py
--- a/pulpcore/app/models/task.py
+++ b/pulpcore/app/models/task.py
@@ -278,7 +278,12 @@ def refresh_from_db(self, using=None, fields=None, **kwargs):
         super().refresh_from_db(using, fields, **kwargs)
 
     class Meta:
-        indexes = [models.Index(fields=["pulp_created"])]
+        indexes = [
+            models.Index(fields=["pulp_created"]),
+            models.Index(fields=["unblocked_at"]),
+            models.Index(fields=["state"]),
+            models.Index(fields=["state", "pulp_created"]),
+        ]
         permissions = [
             ("manage_roles_task", "Can manage role assignments on task"),
         ]


Test patch:
Repository: pulp/pulpcore

Issue:
object_id on user/group_roles tables should be indexed



Discussion:

Resolving patch:
diff --git a/pulpcore/app/migrations/0119_grouprole_core_groupr_object__250e22_idx_and_more.py b/pulpcore/app/migrations/0119_grouprole_core_groupr_object__250e22_idx_and_more.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/migrations/0119_grouprole_core_groupr_object__250e22_idx_and_more.py
@@ -0,0 +1,21 @@
+# Generated by Django 4.2.11 on 2024-05-14 08:30
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("core", "0118_task_core_task_unblock_2276a4_idx_and_more"),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name="grouprole",
+            index=models.Index(fields=["object_id"], name="core_groupr_object__250e22_idx"),
+        ),
+        migrations.AddIndex(
+            model_name="userrole",
+            index=models.Index(fields=["object_id"], name="core_userro_object__adb04a_idx"),
+        ),
+    ]
diff --git a/pulpcore/app/models/role.py b/pulpcore/app/models/role.py
--- a/pulpcore/app/models/role.py
+++ b/pulpcore/app/models/role.py
@@ -51,7 +51,10 @@ class UserRole(BaseModel):
 
     class Meta:
         unique_together = (("user", "role", "content_type", "object_id", "domain"),)
-        indexes = [models.Index(fields=["content_type", "object_id"])]
+        indexes = [
+            models.Index(fields=["object_id"]),
+            models.Index(fields=["content_type", "object_id"]),
+        ]
 
 
 class GroupRole(BaseModel):
@@ -76,4 +79,7 @@ class GroupRole(BaseModel):
 
     class Meta:
         unique_together = (("group", "role", "content_type", "object_id", "domain"),)
-        indexes = [models.Index(fields=["content_type", "object_id"])]
+        indexes = [
+            models.Index(fields=["object_id"]),
+            models.Index(fields=["content_type", "object_id"]),
+        ]


Test patch:
Repository: pulp/pulpcore

Issue:
Task cleanup must not delete content nor artifacts
Deleting content or artifacts outside of orphan cleanup is breaking the rules.
And no, we cannot get away with that.



Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/_util.py b/pulpcore/tasking/_util.py
--- a/pulpcore/tasking/_util.py
+++ b/pulpcore/tasking/_util.py
@@ -15,7 +15,7 @@
 from django.utils import timezone
 from django_guid import set_guid
 from django_guid.utils import generate_guid
-from pulpcore.app.models import Task, TaskSchedule
+from pulpcore.app.models import Artifact, Content, Task, TaskSchedule
 from pulpcore.app.role_util import get_users_with_perms
 from pulpcore.app.util import set_current_user, set_domain, configure_analytics, configure_cleanup
 from pulpcore.constants import TASK_FINAL_STATES, TASK_STATES, VAR_TMP_PULP
@@ -68,6 +68,8 @@ def delete_incomplete_resources(task):
     if task.state != TASK_STATES.CANCELING:
         raise RuntimeError(_("Task must be canceling."))
     for model in (r.content_object for r in task.created_resources.all()):
+        if isinstance(model, (Artifact, Content)):
+            continue
         try:
             if model.complete:
                 continue


Test patch:
Repository: pulp/pulpcore

Issue:
Task cleanup must not delete content nor artifacts
Deleting content or artifacts outside of orphan cleanup is breaking the rules.
And no, we cannot get away with that.



Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/util.py b/pulpcore/tasking/util.py
--- a/pulpcore/tasking/util.py
+++ b/pulpcore/tasking/util.py
@@ -4,7 +4,7 @@
 from django.db import transaction
 from django.db import connection
 
-from pulpcore.app.models import Task
+from pulpcore.app.models import Artifact, Content, Task
 from pulpcore.constants import TASK_FINAL_STATES, TASK_STATES
 
 _logger = logging.getLogger(__name__)
@@ -55,6 +55,8 @@ def _delete_incomplete_resources(task):
     if task.state != TASK_STATES.CANCELING:
         raise RuntimeError(_("Task must be canceling."))
     for model in (r.content_object for r in task.created_resources.all()):
+        if isinstance(model, (Artifact, Content)):
+            continue
         try:
             if model.complete:
                 continue


Test patch:
Repository: pulp/pulpcore

Issue:
Task cleanup must not delete content nor artifacts
Deleting content or artifacts outside of orphan cleanup is breaking the rules.
And no, we cannot get away with that.



Discussion:

Resolving patch:
diff --git a/pulpcore/tasking/util.py b/pulpcore/tasking/util.py
--- a/pulpcore/tasking/util.py
+++ b/pulpcore/tasking/util.py
@@ -4,7 +4,7 @@
 from django.db import transaction
 from django.db import connection
 
-from pulpcore.app.models import Task
+from pulpcore.app.models import Artifact, Content, Task
 from pulpcore.constants import TASK_FINAL_STATES, TASK_INCOMPLETE_STATES, TASK_STATES
 
 _logger = logging.getLogger(__name__)
@@ -60,6 +60,8 @@ def _delete_incomplete_resources(task):
     if task.state not in [TASK_STATES.CANCELED, TASK_STATES.CANCELING]:
         raise RuntimeError(_("Task must be canceled."))
     for model in (r.content_object for r in task.created_resources.all()):
+        if isinstance(model, (Artifact, Content)):
+            continue
         try:
             if model.complete:
                 continue


Test patch:
Repository: pulp/pulpcore

Issue:
Replication assumes every plugin supports Publications
The ```Replicator``` class, which is being subclassed inside plugins to support the replication feature, assumes that every plugin supports publications as it tries to access ```upstream_distribution["publication"]``` inside one of its methods (for some plugins, the dictionary simply doesn't contain the "publication" key so an exception gets raised). This forces certain subclasses of ```Replicator``` to create workarounds or rewrite the given method.

I propose making the method more general, removing such assumptions.

Relevant code: https://github.com/pulp/pulpcore/blob/c31a39c186fd734e9e0409321735d7d6509a12ee/pulpcore/app/replica.py#L97C9-L97C96



Discussion:

Resolving patch:
diff --git a/pulpcore/app/replica.py b/pulpcore/app/replica.py
--- a/pulpcore/app/replica.py
+++ b/pulpcore/app/replica.py
@@ -94,7 +94,9 @@ def remote_extra_fields(self, upstream_distribution):
         return {}
 
     def create_or_update_remote(self, upstream_distribution):
-        if not upstream_distribution["repository"] and not upstream_distribution["publication"]:
+        if not upstream_distribution.get("repository") and not upstream_distribution.get(
+            "publication"
+        ):
             return None
         url = self.url(upstream_distribution)
         remote_fields_dict = {"url": url}


Test patch:
Repository: pulp/pulpcore

Issue:
Replication assumes every plugin supports Publications
The ```Replicator``` class, which is being subclassed inside plugins to support the replication feature, assumes that every plugin supports publications as it tries to access ```upstream_distribution["publication"]``` inside one of its methods (for some plugins, the dictionary simply doesn't contain the "publication" key so an exception gets raised). This forces certain subclasses of ```Replicator``` to create workarounds or rewrite the given method.

I propose making the method more general, removing such assumptions.

Relevant code: https://github.com/pulp/pulpcore/blob/c31a39c186fd734e9e0409321735d7d6509a12ee/pulpcore/app/replica.py#L97C9-L97C96



Discussion:

Resolving patch:
diff --git a/pulpcore/app/replica.py b/pulpcore/app/replica.py
--- a/pulpcore/app/replica.py
+++ b/pulpcore/app/replica.py
@@ -99,7 +99,9 @@ def remote_extra_fields(self, upstream_distribution):
         return {}
 
     def create_or_update_remote(self, upstream_distribution):
-        if not upstream_distribution["repository"] and not upstream_distribution["publication"]:
+        if not upstream_distribution.get("repository") and not upstream_distribution.get(
+            "publication"
+        ):
             return None
         url = self.url(upstream_distribution)
         remote_fields_dict = {"url": url}


Test patch:
Repository: getsentry/snuba

Issue:
Support Redis Authentication
I'm trying to install Snuba on my Kubernetes instance alongside Sentry.
Sentry's Helm chart installs Redis with a password (It generates a secret), and there was no option for me to specify that password for Snuba.

I opened up the source code and it looks like a simple solution: 
Another setting (REDIS_PASSWORD) that would be passed to startup_nodes and to StrictRedis' constructor on the snuba/redis.py module.


Discussion:

Resolving patch:
diff --git a/snuba/redis.py b/snuba/redis.py
--- a/snuba/redis.py
+++ b/snuba/redis.py
@@ -37,11 +37,13 @@ def execute_command(self, *args, **kwargs):
     redis_client = RetryingStrictRedisCluster(
         startup_nodes=startup_nodes,
         socket_keepalive=True,
+        password=settings.REDIS_PASSWORD,
     )
 else:
     redis_client = StrictRedis(
         host=settings.REDIS_HOST,
         port=settings.REDIS_PORT,
+        password=settings.REDIS_PASSWORD,
         db=settings.REDIS_DB,
         socket_keepalive=True,
     )
diff --git a/snuba/settings_base.py b/snuba/settings_base.py
--- a/snuba/settings_base.py
+++ b/snuba/settings_base.py
@@ -28,6 +28,7 @@
 REDIS_CLUSTER_STARTUP_NODES = None
 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost')
 REDIS_PORT = 6379
+REDIS_PASSWORD = None
 REDIS_DB = 1
 
 # Query Recording Options
diff --git a/snuba/settings_docker.py b/snuba/settings_docker.py
--- a/snuba/settings_docker.py
+++ b/snuba/settings_docker.py
@@ -9,5 +9,6 @@
 
 REDIS_HOST = env('REDIS_HOST', 'localhost')
 REDIS_PORT = int(env('REDIS_PORT', 6379))
+REDIS_PASSWORD = env('REDIS_PASSWORD')
 REDIS_DB = int(env('REDIS_DB', 1))
 USE_REDIS_CLUSTER = False


Test patch: