Repository: pulp/pulpcore

Issue:
Allow OR filters
**Is your feature request related to a problem? Please describe.**
We would like to include more advanced filtering options in the Automation Hub UI that allow for users to filter API results using OR lookups, for example I should be able to filter results using expressions such as `name__icontains=foo OR name__icontains=bar`.

**Describe the solution you'd like**
There are a number of possible options to consider for this:

- ORing multiple params with the same name (eg `?name=foo&name=bar`). This would break backwards compatibility with the ruby and python bindings, as the open api spec type field would change to a list type.
- Provide a character delimited list (eg `?name=foo,bar`, `?name=foo|bar`). `|` may make more sense, as `,` are used for list values in other parts of the API.
- Adding `__in` to the list of default lookup expressions (eg `?name__in=foo,bar`). This may not work with more complex lookups like `name__icontains__in=`.

**Additional context**
The AAP team is work on unifying the UI experience across the ansible platform, and this is one capability supported by the other platform components (such as ansible controller), but is currently missing in Automation Hub



Discussion:
**Example**

Querying for name contains james or name contains john and status equals online or status equals offline.

`( name = john | james ) & ( status = online | offline )`

That would result in:
- James Online
- John Offline

Example of how PatternFly displays the filter criteria:
![Screenshot 2023-06-08 at 1 44 54 PM](https://github.com/pulp/pulpcore/assets/6277895/5b676957-91c1-4892-bce4-cf71231db1c0)

Attaching https://github.com/pulp/pulpcore/issues/2480 for completeness.
adding `icontains__in` to the filter expressions doesn't work.

```
pulp_1          | django_filters.exceptions.FieldLookupError: Unsupported lookup 'icontains__in' for field 'core.Domain.name'.
```
`OR`ing same name might not work either - collection version search by tag does allow `?tags=cloud&tags=security` meaning `AND` - and that seems like something we'd want to keep working
**Background** 
Ansible is moving to have more of a unified look, experience, and feel.

A big part of the user experience is the filter capabilities of the different tables with the UI.  
AWX and EDA backends already support the filtering needed for the experience UXD has designed.

Example: Show jobs of job type (playbook run or workflow job) and with status (running or failed or error)
![Screenshot 2023-07-31 at 11 14 28 AM](https://github.com/pulp/pulpcore/assets/6277895/7710c67a-7e9f-4180-8122-037cd4063455)

AWX QueryString example:
```
?or__type=system_job&or__type=workflow_job&or__status=running&or__status=failed&or__status=error
```

HUB needs to support the same functionality. For the backend, HUB uses Pulp, thus we need this support in Pulp.

Note: There are some use cases such as the one with labels where a customer could use an advanced filter to filter using an AND operation. Example: Instead of records with a tag of (tag1 OR tag2) we would also still need to have a way through the API to query for records with tags of (tag1 AND tag2). That would not be the default filtering, but would be available in advanced filters. The Ansible AWX backend already supports advanced filters.

We are moving forward with the work on the unified Ansible experience, with the assumption that Pulp we have this support. If that is not that case, we would love to hear about it sooner than later.

Resolving patch:
diff --git a/pulpcore/app/viewsets/base.py b/pulpcore/app/viewsets/base.py
--- a/pulpcore/app/viewsets/base.py
+++ b/pulpcore/app/viewsets/base.py
@@ -25,7 +25,15 @@
 from pulpcore.tasking.tasks import dispatch
 
 # These should be used to prevent duplication and keep things consistent
-NAME_FILTER_OPTIONS = ["exact", "in", "icontains", "contains", "startswith"]
+NAME_FILTER_OPTIONS = [
+    "exact",
+    "iexact",
+    "in",
+    "contains",
+    "icontains",
+    "startswith",
+    "istartswith",
+]
 # e.g.
 # /?name=foo
 # /?name__in=foo,bar
diff --git a/pulpcore/app/viewsets/task.py b/pulpcore/app/viewsets/task.py
--- a/pulpcore/app/viewsets/task.py
+++ b/pulpcore/app/viewsets/task.py
@@ -60,9 +60,9 @@ class TaskFilter(BaseFilterSet):
     class Meta:
         model = Task
         fields = {
-            "state": ["exact", "in"],
-            "worker": ["exact", "in"],
-            "name": ["exact", "contains", "in"],
+            "state": ["exact", "in", "ne"],
+            "worker": ["exact", "in", "isnull"],
+            "name": ["exact", "contains", "in", "ne"],
             "logging_cid": ["exact", "contains"],
             "started_at": DATETIME_FILTER_OPTIONS,
             "finished_at": DATETIME_FILTER_OPTIONS,
diff --git a/pulpcore/filters.py b/pulpcore/filters.py
--- a/pulpcore/filters.py
+++ b/pulpcore/filters.py
@@ -1,8 +1,11 @@
 from gettext import gettext as _
 
-from functools import lru_cache
+import pyparsing as pp
+
+from functools import lru_cache, partial
 from urllib.parse import urlparse
 from uuid import UUID
+from django import forms
 from django.db import models
 from django.forms.utils import ErrorList
 from django.urls import Resolver404, resolve
@@ -128,7 +131,7 @@ class HREFInFilter(BaseInFilter, filters.CharFilter):
     pass
 
 
-class PulpTypeInFilter(BaseInFilter, filters.ChoiceFilter):
+class PulpTypeFilter(filters.ChoiceFilter):
     """Special pulp_type filter only added to generic list endpoints."""
 
     def __init__(self, *args, **kwargs):
@@ -150,6 +153,110 @@ def pulp_type_choices(model):
         return choices
 
 
+class PulpTypeInFilter(BaseInFilter, PulpTypeFilter):
+    """Special pulp_type filter only added to generic list endpoints."""
+
+
+class ExpressionFilterField(forms.CharField):
+    class _FilterAction:
+        def __init__(self, filterset, tokens):
+            key = tokens[0].key
+            value = tokens[0].value
+            self.filter = filterset.filters.get(key)
+            if self.filter is None:
+                raise forms.ValidationError(_("Filter '{key}' does not exist.").format(key=key))
+            if isinstance(self.filter, ExpressionFilter):
+                raise forms.ValidationError(
+                    _("You cannot use '{key}' in complex filtering.").format(key=key)
+                )
+            if isinstance(self.filter, filters.OrderingFilter):
+                raise forms.ValidationError(
+                    _("An ordering filter cannot be used in complex filtering.")
+                )
+            form = filterset.form.__class__({key: value})
+            if not form.is_valid():
+                raise forms.ValidationError(form.errors.as_json())
+            self.value = form.cleaned_data[key]
+            self.complexity = 1
+
+        def evaluate(self, qs):
+            return self.filter.filter(qs, self.value)
+
+    class _NotAction:
+        def __init__(self, tokens):
+            self.expr = tokens[0][0]
+            self.complexity = self.expr.complexity + 1
+
+        def evaluate(self, qs):
+            return qs.difference(self.expr.evaluate(qs))
+
+    class _AndAction:
+        def __init__(self, tokens):
+            self.exprs = tokens[0]
+            self.complexity = sum((expr.complexity for expr in self.exprs)) + 1
+
+        def evaluate(self, qs):
+            return (
+                self.exprs[0]
+                .evaluate(qs)
+                .intersection(*[expr.evaluate(qs) for expr in self.exprs[1:]])
+            )
+
+    class _OrAction:
+        def __init__(self, tokens):
+            self.exprs = tokens[0]
+            self.complexity = sum((expr.complexity for expr in self.exprs)) + 1
+
+        def evaluate(self, qs):
+            return self.exprs[0].evaluate(qs).union(*[expr.evaluate(qs) for expr in self.exprs[1:]])
+
+    def __init__(self, *args, **kwargs):
+        self.filterset = kwargs.pop("filter").parent
+        super().__init__(*args, **kwargs)
+
+    def clean(self, value):
+        value = super().clean(value)
+        if value not in EMPTY_VALUES:
+            slug = pp.Word(pp.alphas, pp.alphanums + "_")
+            word = pp.Word(pp.alphanums + pp.alphas8bit + ".,_-*")
+            rhs = word | pp.quoted_string.set_parse_action(pp.remove_quotes)
+            group = pp.Group(
+                slug.set_results_name("key")
+                + pp.Suppress(pp.Literal("="))
+                + rhs.set_results_name("value")
+            ).set_parse_action(partial(self._FilterAction, self.filterset))
+
+            expr = pp.infix_notation(
+                group,
+                [
+                    (pp.Suppress(pp.Keyword("NOT")), 1, pp.opAssoc.RIGHT, self._NotAction),
+                    (pp.Suppress(pp.Keyword("AND")), 2, pp.opAssoc.LEFT, self._AndAction),
+                    (pp.Suppress(pp.Keyword("OR")), 2, pp.opAssoc.LEFT, self._OrAction),
+                ],
+            )
+            try:
+                result = expr.parse_string(value, parse_all=True)[0]
+            except pp.ParseException:
+                raise forms.ValidationError(_("Syntax error in expression."))
+            if result.complexity > 8:
+                raise forms.ValidationError(_("Filter expression exceeds allowed complexity."))
+            return result
+        return None
+
+
+class ExpressionFilter(filters.CharFilter):
+    field_class = ExpressionFilterField
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.extra["filter"] = self
+
+    def filter(self, qs, value):
+        if value is not None:
+            qs = value.evaluate(qs)
+        return qs
+
+
 class BaseFilterSet(filterset.FilterSet):
     """
     Class to override django_filter's FilterSet and provide a way to set help text
@@ -165,6 +272,7 @@ class BaseFilterSet(filterset.FilterSet):
     help_text = {}
     pulp_id__in = IdInFilter(field_name="pk", lookup_expr="in")
     pulp_href__in = HREFInFilter(field_name="pk", method="filter_pulp_href")
+    q = ExpressionFilter()
 
     FILTER_DEFAULTS = {
         **filterset.FilterSet.FILTER_DEFAULTS,
@@ -296,6 +404,7 @@ def get_filterset_class(self, view, queryset=None):
             if hasattr(view, "is_master_viewset") and view.is_master_viewset():
 
                 class PulpTypeFilterSet(filterset_class):
+                    pulp_type = PulpTypeFilter(field_name="pulp_type", model=queryset.model)
                     pulp_type__in = PulpTypeInFilter(field_name="pulp_type", model=queryset.model)
 
                 return PulpTypeFilterSet


Test patch:
diff --git a/pulpcore/tests/functional/api/test_filter.py b/pulpcore/tests/functional/api/test_filter.py
--- a/pulpcore/tests/functional/api/test_filter.py
+++ b/pulpcore/tests/functional/api/test_filter.py
@@ -1,130 +1,231 @@
 import pytest
-from random import sample
-from uuid import uuid4
+import random
+import uuid
 
 from pulpcore.client.pulpcore.exceptions import ApiException, ApiTypeError
 
 
+# Warning: Do not use HEX digits here!
+NAMES = (
+    "GGGG",
+    "GGHH",
+    "gggg",
+    "GgHh",
+    "áàâãäæÁÀÂÃÄçÇéèêëíìĩïóòôõöúùûũüßþ",
+)
+
+
 def extract_pk(href):
     """Simulate what pulpcore.app.util.extract_pk does."""
     parts = href.split("/")
     return parts[-2]
 
 
-@pytest.fixture
-def gen_rbac_and_redirect_guards(
+@pytest.fixture(scope="class")
+def rbac_and_redirect_guards(
     redirect_contentguard_api_client,
     rbac_contentguard_api_client,
     gen_object_with_cleanup,
 ):
-    def _generate(number=5):
-        rbacs = []
-        redis = []
-        for _ in range(number):
-            rbacs.append(
-                gen_object_with_cleanup(rbac_contentguard_api_client, {"name": str(uuid4())})
-            )
-            redis.append(
-                gen_object_with_cleanup(redirect_contentguard_api_client, {"name": str(uuid4())})
-            )
-        return rbacs, redis
-
-    return _generate
-
-
-@pytest.mark.parallel
-@pytest.mark.parametrize(
-    "filter,f,exception_message",
-    [("pulp_id__in", extract_pk, "Enter a valid UUID"), ("pulp_href__in", str, "URI not valid")],
-)
-def test_pulp_id_href_filter(
-    filter,
-    f,
-    exception_message,
-    content_guards_api_client,
-    redirect_contentguard_api_client,
-    rbac_contentguard_api_client,
-    gen_rbac_and_redirect_guards,
-):
-    """Tests pulp_href__in and pulp_id__in filters."""
-    rbacs, redis = gen_rbac_and_redirect_guards()
-    rbac_extracted = [f(cg.pulp_href) for cg in rbacs]
-    redi_extracted = [f(cg.pulp_href) for cg in redis]
-
-    rbac_sample = sample(rbac_extracted, 3)
-    redi_sample = sample(redi_extracted, 3)
-
-    rbac_results = rbac_contentguard_api_client.list(**{filter: rbac_sample})
-    assert rbac_results.count == 3
-    assert set(rbac_sample) == {f(cg.pulp_href) for cg in rbac_results.results}
-
-    redi_results = redirect_contentguard_api_client.list(**{filter: redi_sample})
-    assert redi_results.count == 3
-    assert set(redi_sample) == {f(cg.pulp_href) for cg in redi_results.results}
-
-    # Test that generic endpoint can return both
-    results = content_guards_api_client.list(**{filter: rbac_sample + redi_sample})
-    assert results.count == 6
-    assert set(redi_sample + rbac_sample) == {f(cg.pulp_href) for cg in results.results}
-
-    # Test swapping rbac & redirect return 0
-    rbac_results = rbac_contentguard_api_client.list(**{filter: redi_sample})
-    assert rbac_results.count == 0
-
-    redi_results = redirect_contentguard_api_client.list(**{filter: rbac_sample})
-    assert redi_results.count == 0
-
-    # Test that filter fails when not a valid type
-    with pytest.raises(ApiException) as exc:
-        content_guards_api_client.list(**{filter: ["hello"]})
-
-    assert exc.value.status == 400
-    assert exception_message in exc.value.body
-
-
-@pytest.mark.parallel
-def test_pulp_type_filter(
-    content_guards_api_client,
-    redirect_contentguard_api_client,
-    rbac_contentguard_api_client,
-    gen_rbac_and_redirect_guards,
-):
-    """Tests the pulp_type__in filter."""
-    gen_rbac_and_redirect_guards()
-
-    # Test filtering by one pulp_type
-    rbac_result = content_guards_api_client.list(pulp_type__in=["core.rbac"])
-    assert rbac_result.count >= 5
-    for c in rbac_result.results:
-        assert "core/rbac" in c.pulp_href
-
-    redirect_result = content_guards_api_client.list(pulp_type__in=["core.content_redirect"])
-    assert redirect_result.count >= 5
-    for c in redirect_result.results:
-        assert "core/content_redirect" in c.pulp_href
-
-    # Test filtering by multiple pulp_types
-    together_result = content_guards_api_client.list(
-        pulp_type__in=["core.rbac", "core.content_redirect"]
+    prefix = str(uuid.uuid4())
+    rbacs = []
+    redis = []
+    for name in NAMES:
+        rbacs.append(
+            gen_object_with_cleanup(rbac_contentguard_api_client, {"name": prefix + "-" + name})
+        )
+        redis.append(
+            gen_object_with_cleanup(redirect_contentguard_api_client, {"name": prefix + "+" + name})
+        )
+    return prefix, rbacs, redis
+
+
+class TestFilter:
+    @pytest.mark.parallel
+    @pytest.mark.parametrize(
+        "filter,f,exception_message",
+        [
+            ("pulp_id__in", extract_pk, "Enter a valid UUID"),
+            ("pulp_href__in", str, "URI not valid"),
+        ],
     )
-    assert together_result.count >= 10
-    for c in together_result.results:
-        assert "core/rbac" in c.pulp_href or "core/content_redirect" in c.pulp_href
-
-    # Test filtering by invalid pulp_type
-    with pytest.raises(ApiException) as exc:
-        content_guards_api_client.list(pulp_type__in=["i.invalid"])
-
-    assert exc.value.status == 400
-    assert "Select a valid choice. i.invalid is not one of the available choices." in exc.value.body
-
-    # Test filter does not exist on child viewsets
-    with pytest.raises(ApiTypeError) as exc:
-        rbac_contentguard_api_client.list(pulp_type__in=["core.rbac"])
-
-    assert "Got an unexpected keyword argument 'pulp_type__in'" in str(exc.value)
-
-    with pytest.raises(ApiTypeError) as exc:
-        redirect_contentguard_api_client.list(pulp_type__in=["core.content_redirect"])
-
-    assert "Got an unexpected keyword argument 'pulp_type__in'" in str(exc.value)
+    def test_pulp_id_href_filter(
+        self,
+        filter,
+        f,
+        exception_message,
+        content_guards_api_client,
+        redirect_contentguard_api_client,
+        rbac_contentguard_api_client,
+        rbac_and_redirect_guards,
+    ):
+        """Tests pulp_href__in and pulp_id__in filters."""
+        prefix, rbacs, redis = rbac_and_redirect_guards
+        rbac_extracted = [f(cg.pulp_href) for cg in rbacs]
+        redi_extracted = [f(cg.pulp_href) for cg in redis]
+
+        rbac_sample = random.sample(rbac_extracted, 3)
+        redi_sample = random.sample(redi_extracted, 3)
+
+        rbac_results = rbac_contentguard_api_client.list(**{filter: rbac_sample})
+        assert rbac_results.count == 3
+        assert set(rbac_sample) == {f(cg.pulp_href) for cg in rbac_results.results}
+
+        redi_results = redirect_contentguard_api_client.list(**{filter: redi_sample})
+        assert redi_results.count == 3
+        assert set(redi_sample) == {f(cg.pulp_href) for cg in redi_results.results}
+
+        # Test that generic endpoint can return both
+        results = content_guards_api_client.list(**{filter: rbac_sample + redi_sample})
+        assert results.count == 6
+        assert set(redi_sample + rbac_sample) == {f(cg.pulp_href) for cg in results.results}
+
+        # Test swapping rbac & redirect return 0
+        rbac_results = rbac_contentguard_api_client.list(**{filter: redi_sample})
+        assert rbac_results.count == 0
+
+        redi_results = redirect_contentguard_api_client.list(**{filter: rbac_sample})
+        assert redi_results.count == 0
+
+        # Test that filter fails when not a valid type
+        with pytest.raises(ApiException) as exc:
+            content_guards_api_client.list(**{filter: ["hello"]})
+
+        assert exc.value.status == 400
+        assert exception_message in exc.value.body
+
+    @pytest.mark.parallel
+    def test_pulp_type_filter(
+        self,
+        content_guards_api_client,
+        redirect_contentguard_api_client,
+        rbac_contentguard_api_client,
+        rbac_and_redirect_guards,
+    ):
+        """Tests the pulp_type__in filter."""
+        prefix = rbac_and_redirect_guards[0]
+        # Test filtering by one pulp_type
+        rbac_result = content_guards_api_client.list(
+            name__startswith=prefix, pulp_type__in=["core.rbac"]
+        )
+        assert rbac_result.count == 5
+        for c in rbac_result.results:
+            assert "core/rbac" in c.pulp_href
+
+        redirect_result = content_guards_api_client.list(
+            name__startswith=prefix, pulp_type__in=["core.content_redirect"]
+        )
+        assert redirect_result.count == 5
+        for c in redirect_result.results:
+            assert "core/content_redirect" in c.pulp_href
+
+        # Test filtering by multiple pulp_types
+        together_result = content_guards_api_client.list(
+            name__startswith=prefix, pulp_type__in=["core.rbac", "core.content_redirect"]
+        )
+        assert together_result.count == 10
+        for c in together_result.results:
+            assert "core/rbac" in c.pulp_href or "core/content_redirect" in c.pulp_href
+
+        # Test filtering by invalid pulp_type
+        with pytest.raises(ApiException) as exc:
+            content_guards_api_client.list(pulp_type__in=["i.invalid"])
+
+        assert exc.value.status == 400
+        assert (
+            "Select a valid choice. i.invalid is not one of the available choices."
+            in exc.value.body
+        )
+
+        # Test filter does not exist on child viewsets
+        with pytest.raises(ApiTypeError) as exc:
+            rbac_contentguard_api_client.list(pulp_type__in=["core.rbac"])
+
+        assert "Got an unexpected keyword argument 'pulp_type__in'" in str(exc.value)
+
+        with pytest.raises(ApiTypeError) as exc:
+            redirect_contentguard_api_client.list(pulp_type__in=["core.content_redirect"])
+
+        assert "Got an unexpected keyword argument 'pulp_type__in'" in str(exc.value)
+
+    @pytest.mark.parallel
+    @pytest.mark.parametrize(
+        "q,count",
+        [
+            pytest.param(*data, id=data[0])
+            for data in [
+                ("pulp_type__in='core.rbac'", 5),
+                ('NOT pulp_type="core.rbac"', 5),
+                ("pulp_type__in=core.rbac,core.content_redirect", 10),
+                ('pulp_type="core.rbac" OR pulp_type="core.content_redirect"', 10),
+                ('pulp_type="core.rbac" AND name__contains=GGGG', 1),
+                ('pulp_type="core.rbac" AND name__iexact={prefix}-gGgG', 2),
+                ('pulp_type="core.rbac" AND name__icontains=gg AND NOT name__contains=HH', 3),
+                ('NOT (pulp_type="core.rbac" AND name__icontains=gGgG)', 8),
+                ('pulp_type="core.rbac" AND name__contains="{4}"', 1),
+                ('pulp_type="core.rbac" AND NOT name__contains="{4}"', 4),
+                ('pulp_type="core.rbac" AND(   name__icontains=gh OR name__contains="{4}")', 3),
+                ('pulp_type="core.rbac" OR name__icontains=gh OR name__contains="{4}"', 8),
+            ]
+        ],
+    )
+    def test_q_filter_valid(
+        self,
+        q,
+        count,
+        content_guards_api_client,
+        rbac_and_redirect_guards,
+    ):
+        """Tests the "q" filter."""
+        prefix = rbac_and_redirect_guards[0]
+
+        result = content_guards_api_client.list(
+            name__startswith=prefix, q=q.format(*NAMES, prefix=prefix)
+        )
+        assert result.count == count
+
+    @pytest.mark.parallel
+    @pytest.mark.parametrize(
+        "q,exception_message",
+        [
+            pytest.param(*data, id=data[0])
+            for data in [
+                ('pulp_type_in="core.rbac"', '{"q":["Filter \'pulp_type_in\' does not exist."]}'),
+                (
+                    'pulp_type="core.foo"',
+                    '{"q":["{\\"pulp_type\\": [{\\"message\\": \\"Select a valid choice. core.foo is not one of the available choices.\\", \\"code\\": \\"invalid_choice\\"}]}"]}',  # noqa
+                ),
+                (
+                    'pulp_type__in="core.rbac,core.foo"',
+                    '{"q":["{\\"pulp_type__in\\": [{\\"message\\": \\"Select a valid choice. core.foo is not one of the available choices.\\", \\"code\\": \\"invalid_choice\\"}]}"]}',  # noqa
+                ),
+                ('name="test" and', '{"q":["Syntax error in expression."]}'),
+                ('name="test" AND', '{"q":["Syntax error in expression."]}'),
+                ('name="test" AND (name="test2"', '{"q":["Syntax error in expression."]}'),
+                (
+                    'name="test1" OR name="test2" OR name="test3" OR name="test4" OR name="test5" OR name="test6" OR name="test7" OR name="test8"',  # noqa
+                    '{"q":["Filter expression exceeds allowed complexity."]}',
+                ),
+                (
+                    '(name="test1" OR name="test2") OR name="test3" OR name="test4" OR name="test5" OR name="test6" OR name="test7"',  # noqa
+                    '{"q":["Filter expression exceeds allowed complexity."]}',
+                ),
+                (
+                    '(name="test1" OR NOT name="test2") OR name="test3" OR name="test4" OR name="test5" OR name="test6"',  # noqa
+                    '{"q":["Filter expression exceeds allowed complexity."]}',
+                ),
+            ]
+        ],
+    )
+    def test_q_filter_invalid(
+        self,
+        q,
+        exception_message,
+        content_guards_api_client,
+    ):
+        """Tests the "q" filter with invalid expressions."""
+
+        with pytest.raises(ApiException) as exc_info:
+            content_guards_api_client.list(q=q.format(*NAMES))
+        assert exc_info.value.status == 400
+        assert exc_info.value.body == exception_message