Repository: pulp/pulpcore

Issue:
As a user, I want to know which repositories some content is in 
**Is your feature request related to a problem? Please describe.**
Content in pulp is distributed through repositories and distributions. If a client knows the name or HREF for some content, there's currently no way of know which repositories it's available in without looking in each repository. 

**Describe the solution you'd like**
Add a `?content_in_latest=` filter to the distributions and repositories APIs that allow users to specify a content HREF and get a list of repositories or distributions that contain the selected content.

**Describe alternatives you've considered**
Are there any existing ways to do this in pulp?

**Additional context**
This is needed for repository management in galaxy_ng



Discussion:
as of today it is possible to list all repo_versions(and publications)  that contain a certain content or list of content
https://docs.pulpproject.org/pulpcore/restapi.html#tag/Repository_Versions
```
GET http :/pulp/api/v3/repository_versions/?content=<content_href>

[ 
  {
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/07c41c5f-59e4-4371-942a-b6a006a6d2cf/versions/1/",
    "pulp_created": "2021-03-18T19:23:31.661940Z",
    "repository_href": "/pulp/api/v3/repositories/rpm/rpm/07c41c5f-59e4-4371-942a-b6a006a6d2cf/",
    "version": 1,
  }
  ... 
]

GET http :/pulp/api/v3/publications/?content=<content_href>

[ 
  {
    "pulp_href": "/pulp/api/v3/publications/rpm/rpm/4d5bb614-4318-4408-8c18-ab8b6b4c016f/",
    "pulp_created": "2021-03-17T12:24:31.661940Z",
    "repository_version_href": "/pulp/api/v3/repositories/rpm/rpm/07c41c5f-59e4-4371-942a-b6a006a6d2cf/versions/1/",
  }
  ... 
]
```
oh neat, I didn't realize the repository version api existed. I was looking at /<repo_href>/repository-versions/ 
yeah our bad - we have no docs on this :/
with the existing implementation we could add a query param ?version=latest which would filter the latest repo version and show it instead of showing all of them.

Distribution search can be based of the repo_version search.
Adding a `?version=latest` flag to `repository_versions` could be sufficient. The potential issues I see with this are:
- The `repository_versions` API is not quite as data rich as the repositories API. You can't see names, remotes etc. This could be partially address with #2828
- This will make developing UIs more challenging and the end user experience worse overall. Lets say I have a "Repositories" view in my UI. If I want to filter that list of repositories by content. I have to make an api call to a different API endpoint and map the data so that the UI for repositories can interpret it. Without #2828, I would also have to make a bunch of additional API calls to get information about the repository itself. I also won't be able to use the content filter with any other repository filters such as repo labels, names or created dates.
I think our users would more or less appreciate all of these options, including:

(a) Adding the `?version=latest` flag to  the repository_versions API
(b) Adding similar filter capability to the repositories endpoint
(c) Adding similar filtering capability to the distributions endpoint

The distributions endpoint (c) could be a little trickier due to some distributions have publications, and some don't, so for this to be implemented in pulpcore it would have to support both types. Totally do-able, just a bit more to it.

I want (a) because it's really easy, but then it requires the PoC Related Fields, which is a non-trivial piece of work. I think we should avoid tying this need up in that, so I believe the ideal path is to implement (b) and (c) as separate tickets. This ticket could be for one of them, and we'd need another ticket for the other.

For (b) it would be nice to have the ability to search against latest or not. For (c) it shouldn't take a "latest" option since the distribution already encapsulates that feature. Distributions can point to a repo which implies "latest" or a specific repo_version or publication, which implies "just this one".

Given all ^,  I'm happy to defer to @newswangerd for whichever one you want.

A short aside on (a), maybe positioning it as `latest_only=True/False` (with default to False), would be good because I don't think semantically having the option accept mixed types, e.g. `latest` or in other cases "a url to a repo version" is a positive user experience.
At the pulpcore meeting, we want to pursue using this ticket for implementing (b) and @newswangerd will file another ticket for (c) and @newswangerd or galaxy_ng will implement both?
I'll file another ticket for distributions. We can implement both
The reason we have implemented (a) and not (b) because the content's presence and distribution comes from a repo version specifically.
Your story sounds like "As a user, I want to know which repos have content X in it's latest version" which is not as generic as "As a user, I want to know which repo versions  have content X" . Not every distribution serves latest version, as it was pointed out earlier in the comment, but what the (b) will show in case I don't want to search against latest? The api will give you the repo result like this but will not tell in which version specifically the content is in.
```
"results": [
        {
            "description": null,
            "latest_version_href": "/pulp/api/v3/repositories/container/container/6d0d0d25-b3c0-49a2-a798-358fbe6f5031/versions/0/",
            "name": "lala",
            "pulp_created": "2022-06-29T11:38:58.610462Z",
            "pulp_href": "/pulp/api/v3/repositories/container/container/6d0d0d25-b3c0-49a2-a798-358fbe6f5031/",
            "pulp_labels": {},
            "remote": null,
            "retain_repo_versions": null,
            "versions_href": "/pulp/api/v3/repositories/container/container/6d0d0d25-b3c0-49a2-a798-358fbe6f5031/versions/"
        }
    ]
```

And even if you're interested only  in the  latest repo version, there is no guarantee that by the time the user will decide to consume/use/copy/etc content from that repo 1) that repo will not have other versions created so the latest won't be latest you searched against anymore 2) the new latest might not have that content anymore; that's why it is probably safer to search through repo versions because they are immutable and content will be there until the repo version exists.
I agree on other points of repo versions api not being as rich as repo api and that different endpoints would need to be called but i am hoping that https://github.com/pulp/pulpcore/pull/2828 will help.
@ipanova thanks for such an informative post. Given that is the recommendation to focus on developing https://github.com/pulp/pulpcore/pull/2828 ?

I still think from a usability perspective users want to search from repos and distributions directly, but given the concerns you raise a clear path to doing that is not immediately clear to me. I guess one of the concerns is that if repo endpoints start returning serialized objects that aren't repos (for example) the endpoint is now polymorphic from an openAPI perspective. That being said it kind of would be anyway if we implement https://github.com/pulp/pulpcore/pull/2828  Just some rambling thoughts on this. More conversation is welcome.
@ipanova Let me see if I can summarize your objections to this:

1. Distributions can point to any repository version, so searching latest doesn't work
2. The repository serializer only links to the latest repo version, so searching for content in any other version doesn't make sense
3. Content is ultimately distributed from versions, so searching the versions directly is more reliable.
4. The repo version pointers for for distributions and repositories can be updated in-between searching for content and requesting content, so they don't accurately represent if the content is actually there.

1, 2 and 3 are great points. Since the content can be distributed from any version in the repository it doesn't make sense to limit your search to just the latest on the repositories endpoint.

I'm not 100% certain 3 matters as much. The best any REST API can do is communicate to the client what the state of the system is in right now. If I make a followup call to the system to request another piece of data there's no guarantee that the object still exists, that the object is the same as it was before, the the server is running, the authentication credentials are still valid, etc. The current endpoint for repository versions has this problem too. The repo versions are immutable, but they're also deletable. There's no guarantee the version exists if I make a followup call to grab my content from it.

With all this in mind, lets revisit some use cases from for this:

- U1: As an admin, I want to know who has access to download some content
- U2: As a content consumer, I want to know where to go to download some content

As you pointed out, (b) doesn't make any sense for U1. The content can be distributed from any version of the repository, so knowing if it's in latest doesn't help you make that determination anymore, so it would be better to make a call to the repo versions api endpoint.

U2 is a little more complicated. In the ansible world, clients can only download content from a distribution. If we just implement (a), to determine where my content is available, I would need to request the list of repository versions, and then figure out which repository versions are part of a distribution I have access to.  Since distributions are just pointers to repository versions, it seems like it would still make sense to provide a `?contains_content` filter to the distributions endpoint.

Providing a `?contains_content` on the repository APIs would also be very helpful from a usability perspective, even if it means that you have to make a followup request to get the list of versions that the content is actually in. Going back to U1, let's say an admin is trying to assess which clients may have downloaded some malicious content. If the content is in 3000 versions of repo `foo`, 200 versions of `bar` and 1 version of `foobar`, then the admin has to page through 3201 repository versions to find out that there are 3 repos that demand their attention. It would be easier to get list of 3 repositories in one API call and then go through the list of repo versions for each repo to perform cleanup or whatever they need to do.

I guess this is a long winded way of saying I really want us to implement (c). Limiting your search to just the latest version on a repo doesn't make as much sense, so (a) and (b) might not be necessary, but I would still love to have some content filtering capabilities on the repository endpoint.

cc @bmbouter 
Is the question about content in a Distribution controversial at this point?
I think the question of what is currently presented in a distribution is rather clear. Should we break this out as a separate issue?
Ah, maybe I misunderstood the comment. I just added https://github.com/pulp/pulpcore/issues/2952 to track the distribution filter.
I don't think I will be able to attend in time our today's meeting, so let me leave here some of my thoughts.

> @ipanova Let me see if I can summarize your objections to this:
> 
>     1. Distributions can point to any repository version, so searching latest doesn't work
> 
>     2. The repository serializer only links to the latest repo version, so searching for content in any other version doesn't make sense
> 
>     3. Content is ultimately distributed from versions, so searching the versions directly is more reliable.
> 
>     4. The repo version pointers for for distributions and repositories can be updated in-between searching for content and requesting content, so they don't accurately represent if the content is actually there.
> 
> 
> 1, 2 and 3 are great points. Since the content can be distributed from any version in the repository it doesn't make sense to limit your search to just the latest on the repositories endpoint.
> 
> I'm not 100% certain 3 matters as much. The best any REST API can do is communicate to the client what the state of the system is in right now. If I make a followup call to the system to request another piece of data there's no guarantee that the object still exists, that the object is the same as it was before, the the server is running, the authentication credentials are still valid, etc. The current endpoint for repository versions has this problem too. The repo versions are immutable, but they're also deletable. There's no guarantee the version exists if I make a followup call to grab my content from it.
> 
> With all this in mind, lets revisit some use cases from for this:
> 
>     * U1: As an admin, I want to know who has access to download some content
> 
>     * U2: As a content consumer, I want to know where to go to download some content
> 
> 
> As you pointed out, (b) doesn't make any sense for U1. The content can be distributed from any version of the repository, so knowing if it's in latest doesn't help you make that determination anymore, so it would be better to make a call to the repo versions api endpoint.
> 
> U2 is a little more complicated. In the ansible world, clients can only download content from a distribution. If we just implement (a), to determine where my content is available, I would need to request the list of repository versions, and then figure out which repository versions are part of a distribution I have access to. Since distributions are just pointers to repository versions, it seems like it would still make sense to provide a `?contains_content` filter to the distributions eddpoint.

In my previous 2 comments I was mostly expressing my concerns over (b) as the main API endpoint of reference for repo content search. (a) and (c) make sense to me.
> 
> Providing a `?contains_content` on the repository APIs would also be very helpful from a usability perspective, even if it means that you have to make a followup request to get the list of versions that the content is actually in. Going back to U1, let's say an admin is trying to assess which clients may have downloaded some malicious content. If the content is in 3000 versions of repo `foo`, 200 versions of `bar` and 1 version of `foobar`, then the admin has to page through 3201 repository versions to find out that there are 3 repos that demand their attention. It would be easier to get list of 3 repositories in one API call and then go through the list of repo versions for each repo to perform cleanup or whatever they need to do.

I am still not sure I am sold to this. If I have a malicious content I plan to remove it.
1) I will make a call to repo_versions API endpoint  with `?latest_only=True` to identify the href  for  foo, bar and foobar and then 
2) have 3 repo API endpoint calls to each repos to remove the malicious content
3) Since repo versions are immutable I can just delete them so won't I perform 3201 DELETE calls on repo_versions? 

I am not very much against providing `?contains_content` on the repository APIs I just don't see any added value to it because the result of the API call will tell me that one of the 3000 repo_versions from repo foo contain content X. Well, thanks and what's next? Next, I am going perform (1) with`?latest_only=False` so I can find all affected repo_versions and then (3) to delete them.

EDIT: well, I do see value for the sake of convenience and generic informative call, to not scroll through the X number of pages of all repo_version results.
> 
> I guess this is a long winded way of saying I really want us to implement (c). Limiting your search to just the latest version on a repo doesn't make as much sense, so (a) and (b) might not be necessary, but I would still love to have some content filtering capabilities on the repository endpoint.

yep +1 on (c)

> 
> cc @bmbouter


We would really like to have this feature. And we'd prefer to filter repositories (as opposed to repo versions) by content as we don't expose repo versions to our users. Our use of Pulp is that users simply create, publish, and distribute repos so they have no concept of repo versions.

Resolving patch:
diff --git a/pulpcore/app/viewsets/repository.py b/pulpcore/app/viewsets/repository.py
--- a/pulpcore/app/viewsets/repository.py
+++ b/pulpcore/app/viewsets/repository.py
@@ -16,6 +16,7 @@
     Content,
     Remote,
     Repository,
+    RepositoryContent,
     RepositoryVersion,
 )
 from pulpcore.app.response import OperationPostponedResponse
@@ -41,9 +42,51 @@
 from pulpcore.filters import HyperlinkRelatedFilter
 
 
+class RepositoryContentFilter(Filter):
+    """
+    Filter used to filter repositories which have a piece of content
+    """
+
+    def __init__(self, *args, **kwargs):
+        kwargs.setdefault("help_text", _("Content Unit referenced by HREF"))
+        self.latest = kwargs.pop("latest", False)
+        super().__init__(*args, **kwargs)
+
+    def filter(self, qs, value):
+        """
+        Args:
+            qs (django.db.models.query.QuerySet): The Repository Queryset
+            value (string): of content href to filter
+
+        Returns:
+            Queryset of the Repository containing the specified content
+        """
+
+        if value is None:
+            # user didn't supply a value
+            return qs
+
+        if not value:
+            raise serializers.ValidationError(detail=_("No value supplied for content filter"))
+
+        # Get the content object from the content_href
+        content = NamedModelViewSet.get_resource(value, Content)
+
+        if self.latest:
+            return qs.filter(
+                pk__in=RepositoryContent.objects.filter(
+                    version_removed=None, content__pk=content.pk
+                ).values_list("repository__pk", flat=True)
+            )
+        else:
+            return qs.filter(content__pk=content.pk)
+
+
 class RepositoryFilter(BaseFilterSet):
     pulp_label_select = LabelFilter()
     remote = HyperlinkRelatedFilter(allow_null=True)
+    with_content = RepositoryContentFilter()
+    latest_with_content = RepositoryContentFilter(latest=True)
 
     class Meta:
         model = Repository


Test patch:
diff --git a/pulpcore/tests/functional/api/test_repos.py b/pulpcore/tests/functional/api/test_repos.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/tests/functional/api/test_repos.py
@@ -0,0 +1,50 @@
+"""Tests that CRUD repositories."""
+from uuid import uuid4
+
+import pytest
+
+from pulpcore.client.pulp_file import RepositorySyncURL
+
+
+@pytest.mark.parallel
+def test_repository_content_filters(
+    file_content_api_client,
+    file_repository_api_client,
+    file_repository_factory,
+    file_remote_factory,
+    gen_object_with_cleanup,
+    write_3_iso_file_fixture_data_factory,
+    monitor_task,
+):
+    """Test repository's content filters."""
+    # generate a repo with some content
+    repo = file_repository_factory()
+    repo_manifest_path = write_3_iso_file_fixture_data_factory(str(uuid4()))
+    remote = file_remote_factory(manifest_path=repo_manifest_path, policy="on_demand")
+    body = RepositorySyncURL(remote=remote.pulp_href)
+    task_response = file_repository_api_client.sync(repo.pulp_href, body).task
+    version_href = monitor_task(task_response).created_resources[0]
+    content = file_content_api_client.list(repository_version_added=version_href).results[0]
+    repo = file_repository_api_client.read(repo.pulp_href)
+
+    # filter repo by the content
+    results = file_repository_api_client.list(with_content=content.pulp_href).results
+    assert results == [repo]
+    results = file_repository_api_client.list(latest_with_content=content.pulp_href).results
+    assert results == [repo]
+
+    # remove the content
+    response = file_repository_api_client.modify(
+        repo.pulp_href,
+        {"remove_content_units": [content.pulp_href]},
+    )
+    monitor_task(response.task)
+    repo = file_repository_api_client.read(repo.pulp_href)
+
+    # the repo still has the content unit
+    results = file_repository_api_client.list(with_content=content.pulp_href).results
+    assert results == [repo]
+
+    # but not in its latest version anymore
+    results = file_repository_api_client.list(latest_with_content=content.pulp_href).results
+    assert results == []