Repository: pulp/pulpcore

Issue:
Remove orphans: Repository matching query does not exist.
**Version**
```
$ rpm -qa | grep tfm-rubygem-pulp
tfm-rubygem-pulp_certguard_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.13.1-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.18.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.17.4-1.el7.noarch
tfm-rubygem-pulp_python_client-3.6.0-1.el7.noarch
tfm-rubygem-pulp_ostree_client-2.0.0-0.1.a1.el7.noarch
tfm-rubygem-pulpcore_client-3.18.5-1.el7.noarch
tfm-rubygem-pulp_container_client-2.10.3-1.el7.noarch
tfm-rubygem-pulp_file_client-1.10.0-1.el7.noarch

$ rpm -qa | grep katello-4
tfm-rubygem-katello-4.5.0-1.el7.noarch
katello-4.5.0-1.el7.noarch
```

**Describe the bug**
Running `foreman-rake katello:delete_orphaned_content RAILS_ENV=production` creates two tasks.
One of the tasks gets stuck in _paused_ with the following error: `Repository matching query does not exist.`


**To Reproduce**

- Run 
  `foreman-rake katello:delete_orphaned_content RAILS_ENV=production`.

- After a few minutes run 
  `hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused"'`

**Expected behavior**
We expected both tasks to run successfully.

**Additional context**
We had to manually apply this patch: https://github.com/pulp/pulp_rpm/pull/2619 - before that, we had his same issue: https://github.com/pulp/pulp_rpm/issues/2459 .

```
$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" '
[
  {
    "ID": "a8aae19c-7c62-4a60-ab12-935687e3723e",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-21 15:29:02 UTC",
    "Ended at": null,
    "Duration": "00:02:29.56437",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]
```
```
$ http_pulp /pulp/api/v3/tasks/e7df82c7-cb38-4fac-a28a-edd8842b3b69/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 3335
Content-Type: application/json
Correlation-ID: e1f2e082da6e401196007424799a05b5
Date: Thu, 21 Jul 2022 17:10:58 GMT
Keep-Alive: timeout=15, max=100
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept,Cookie
Via: 1.1 updates.eurotux.com
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "child_tasks": [],
    "created_resources": [],
    "error": {
        "description": "Repository matching query does not exist.",
        "traceback": "  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 410, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/orphan.py\", line 66, in orphan_cleanup\n    c.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 340, in cleanup_subrepos\n    Variant.objects.filter(repository=subrepo).delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 746, in delete\n    deleted, _rows_count = collector.delete()\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/deletion.py\", line 435, in delete\n    signals.post_delete.send(\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 180, in send\n    return [\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/dispatch/dispatcher.py\", line 181, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
    },
    "finished_at": "2022-07-21T15:30:11.075104Z",
    "logging_cid": "ac4070d63c764578bb854f97873ba88a",
    "name": "pulpcore.app.tasks.orphan.orphan_cleanup",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "clean-up.content",
            "done": 2000,
            "message": "Clean up orphan Content",
            "state": "running",
            "suffix": null,
            "total": 17098
        }
    ],
    "pulp_created": "2022-07-21T15:29:58.868921Z",
    "pulp_href": "/pulp/api/v3/tasks/e7df82c7-cb38-4fac-a28a-edd8842b3b69/",
    "reserved_resources_record": [],
    "started_at": "2022-07-21T15:29:59.335110Z",
    "state": "failed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/6ff07070-3e14-49b3-b508-a0c300683f28/"
}
```

edit: related to https://community.theforeman.org/t/delete-orphaned-content-fails/29494/6

https://bugzilla.redhat.com/show_bug.cgi?id=2115881




Discussion:
Based o nthe traceback this seems to be a pulp-rpm issue, transferring.
`
File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/models/distribution.py\", line 338, in cleanup_subrepos\n    subrepo = instance.repository\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 187, in __get__\n    rel_obj = self.get_object(instance)\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py\", line 154, in get_object\n    return qs.get(self.field.get_reverse_related_filter(instance))\n  File \"/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py\", line 435, in get\n    raise self.model.DoesNotExist(\n"
`
You said there are two tasks created and listed one, what's the other task?  Is it two orphan cleanup tasks?  It could be a race condition

It does not appear that we use exclusive resources to prevent more than one instance of orphan cleanup running simultaneously.
Hello @dralley,

Yes, it's another orphan cleanup task.

```
$ foreman-rake katello:delete_orphaned_content RAILS_ENV=production -n -v
** Invoke katello:delete_orphaned_content (first_time)
** Invoke dynflow:client (first_time)
** Invoke environment (first_time)
** Execute (dry run) environment
** Execute (dry run) dynflow:client
** Execute (dry run) katello:delete_orphaned_content

$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "running" or state = "paused" '
[
  {
    "ID": "078a72dc-8002-46bd-a988-7383f9604c42",
    "Action": "Remove orphans ",
    "State": "running",
    "Result": "pending",
    "Started at": "2022-07-26 16:35:51 UTC",
    "Ended at": null,
    "Duration": "00:00:09.569835",
    "Owner": "foreman_admin",
    "Task errors": [

    ]
  },
  {
    "ID": "9fb6a3ee-5428-4c5e-b42f-22de91545a4d",
    "Action": "Remove orphans ",
    "State": "running",
    "Result": "pending",
    "Started at": "2022-07-26 16:35:50 UTC",
    "Ended at": null,
    "Duration": "00:00:09.974835",
    "Owner": "foreman_admin",
    "Task errors": [

    ]
  }
]
```
A few minutes later:

```
$ hammer --no-headers --output json task list  --order 'started_at DESC' --search 'state = "paused" or state = "running" '
[
  {
    "ID": "9fb6a3ee-5428-4c5e-b42f-22de91545a4d",
    "Action": "Remove orphans ",
    "State": "paused",
    "Result": "error",
    "Started at": "2022-07-26 16:35:50 UTC",
    "Ended at": null,
    "Duration": "00:02:28.169247",
    "Owner": "foreman_admin",
    "Task errors": [
      "Repository matching query does not exist."
    ]
  }
]
```
Apparently the task that was first launched got stuck.

So you're probably right and there's a race condition.

I also tried to run only one task at a time:

```
$ foreman-rake katello:delete_orphaned_content RAILS_ENV=production -v --jobs 1
```
But I still have the same issue.
Any ideas on how to avoid (or debug) this possible race condition?

Resolving patch:
diff --git a/pulpcore/app/views/orphans.py b/pulpcore/app/views/orphans.py
--- a/pulpcore/app/views/orphans.py
+++ b/pulpcore/app/views/orphans.py
@@ -25,6 +25,6 @@ def delete(self, request, format=None):
             "`POST /pulp/api/v3/orphans/cleanup/` instead."
         )
 
-        task = dispatch(orphan_cleanup)
+        task = dispatch(orphan_cleanup, exclusive_resources=["/pulp/api/v3/orphans/cleanup/"])
 
         return OperationPostponedResponse(task, request)
diff --git a/pulpcore/app/viewsets/orphans.py b/pulpcore/app/viewsets/orphans.py
--- a/pulpcore/app/viewsets/orphans.py
+++ b/pulpcore/app/viewsets/orphans.py
@@ -26,6 +26,7 @@ def cleanup(self, request):
 
         task = dispatch(
             orphan_cleanup,
+            exclusive_resources=["/pulp/api/v3/orphans/cleanup/"],
             kwargs={"content_pks": content_pks, "orphan_protection_time": orphan_protection_time},
         )
 


Test patch: