Repository: encode/django-rest-framework

Issue:
SearchFilter time grows exponentially by # of search terms
## Checklist

- [x] I have verified that that issue exists against the `master` branch of Django REST framework.
- [x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [x] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [x] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](http://www.django-rest-framework.org/topics/third-party-resources/#about-third-party-packages) where possible.)
- [x] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

## Steps to reproduce

Use `filters.SearchFilter` and include a seach_field which is a many to many lookup.

```
    filter_backends = (filters.SearchFilter)
    search_fields = ('many__to__many__field')
```
Make a query against this view with several search terms.

## Expected behavior

The search time would increase somewhat linearly with the # of terms.

## Actual behavior

The search grows exponentially with each added term. In our application several words (3) resulted in a 30 sec query against a model that only had several hundred entries. It would take several minutes for another term and so on.

## Summary

I was able to change a single block in drf and the performance became linear as I would expect. The problem and (a potential) solution are known. I wanted to bring them to your attention.

## The culprit

Chaining filters in `django` on querysets doesn't behave as one would expect when dealing with ManyToMany relations. If you look at the gist below, you'll see that the second bit of sql is quite different from the first bit because of this difference.

https://gist.github.com/cdosborn/cb4bdfd0467feaf987476f4aefdf7ee5

From looking at the sql, you'll notice the first bit generated a bunch of unnecessary joins. These joins result in a multiplicative factor on the number of rows that the query contains. Notice how the bottom query doesn't have the redundant joins. So what we can conclude is that chaining filters can produce unnecessary joins which can dramatically effect the performance.

So there is a bit of code in drf, which chains `filter` for each term in the search query. This explodes whenever the `search_fields` contains a ManyToMany. 

## A solution

Rather than chaining filters in `SearchFilter` we build up a query first, and call `filter` once.
```diff
diff --git a/rest_framework/filters.py b/rest_framework/filters.py
index 531531e..0e7329b 100644
--- a/rest_framework/filters.py
+++ b/rest_framework/filters.py
@@ -144,13 +144,15 @@ class SearchFilter(BaseFilterBackend):
         ]
 
         base = queryset
+        conditions = []
         for search_term in search_terms:
             queries = [
                 models.Q(**{orm_lookup: search_term})
                 for orm_lookup in orm_lookups
             ]
-            queryset = queryset.filter(reduce(operator.or_, queries))
+            conditions.append(reduce(operator.or_, queries))
 
+        queryset = queryset.filter(reduce(operator.and_, conditions))
         if self.must_call_distinct(queryset, search_fields):
             # Filtering against a many-to-many field requires us to
             # call queryset.distinct() in order to avoid duplicate items
```

This may not be the fix you want. My guess is that the `must_call_distinct` was trying to fix this problem, but it's not sufficient. My impression is that this is a pretty serious issue that django needs to resolve.




Discussion:
These [docs](https://docs.djangoproject.com/en/1.10/topics/db/queries/#spanning-multi-valued-relationships) are relevant here. At the end of the day, you're getting two different queries that return two completely different sets of results. Regardless of performance, I'd argue that the proposed changes are more correct. 

Would you like me to submit a PR? 

Some thoughts:
Can self.must_call_distinct be removed?
Should probably search for similar uses of filter.
How should we address that this is something django should fix or provide a workaround for (normally I'd inline a comment including a link to the [discussion](https://code.djangoproject.com/ticket/27303)).

> How should we address that this is something django should fix or provide a workaround for 

If you believe this represents an issue in Django core then raise a ticket on Trac. It'd be worth reviewing what happens in the admin, and if this is replicable in the the search there too. I'd be surprised if the issue hadn't already come up before _if_ that's the case.

> Can self.must_call_distinct be removed?

Start by seeing what tests fail if you do remove it. We can then take the conversation from there.

In the past we have similar problem with a pure django project (not using django rest framework at all). We used django-tagging and searched in the tags (which are many to many to the object). We used MySQL for database engine and when query string in our form contained a lot of words then  MySQL raised that it can not join more than 40 tables (or 41 I can't remember exactly).

We fixed that by using Q objects and `or`-ing them instead of the querysets.

@rpkilby yes at the end you have two different SQL queries but you still have the same results set because you are using `or` and not `and`. In the django docs that you linked they write about using `and` with many to many relations.

@cdosborn 
About `self.must_call_distinct` it should not be removed. Everytime when you perform query against reverse relation (many to many is de facto reverse relation from both ends) you should call `distinct()` or you will have duplicates. The only exception is when the reverse relation is one to one.
> @rpkilby yes at the end you have two different SQL queries but you still have the same results set because you are using `or` and not `and`.

@vstoykov - The search *fields* per term are grouped together with `or`, however each group is `and`ed together. For example, take `ordering_fields = ('name', 'groups__name')` and this query:
```http
GET https://localhost/api/users?search=bob,joe
```
With the existing implementation, we should get a queryset equivalent to the following:
```python
User.objects \
    .filter(Q(name__icontains='bob') | Q(groups__name__icontains='bob') \
    .filter(Q(name__icontains='joe') | Q(groups__name__icontains='joe')
```

The proposed changes would result in this query:

```python

User.objects.filter((Q(name__icontains='bob') | Q(groups__name__icontains='bob'))
                  & (Q(name__icontains='joe') | Q(groups__name__icontains='joe')))
```

I'd have to double check, but this seems to fall under the caveats described in the docs.
@rpkilby Sorry I totally missed  `operator.and_` in:
```
queryset = queryset.filter(reduce(operator.and_, conditions))
```

This will make the situation complex. On one hand the search need to return as many as possible matching results, on other hand it should not DOS the application.

Probably there should be something that can configure this (`SearchFilter`'s argument, or separate class which developers can use) and mentioning in documentation what are the differences and then each project developer will decide which variant to use.

From one point of view, the current behavior is a bug w.r.t to handling m2m. From the [docs](http://www.django-rest-framework.org/api-guide/filtering/#searchfilter):

> If multiple search terms are used then objects will be returned in the list only if **all the provided terms are matched**.

As you mentioned, if we went ahead with the changes, then applications would see fewer results.
Hey folks, 
Wanted to inquire if these changes have been tested by anyone? 
I wanted to override the SearchFilter class and add the changes to speed up M2M searches. 
@vimarshc You may use [this](https://github.com/cyverse/atmosphere/blob/6ef13abb9e20a241bf5c7e68e790e02f4f1f761c/api/v2/views/image.py#L12-L43) as a reference. We monkey-patched the search filter for the mean time. 
If anyone wants to progress this issue, I'd suggest making a pull request so we can look at the effects of this change on the current test suite, which would help highlight any problems it might have.

Resolving patch:
diff --git a/rest_framework/filters.py b/rest_framework/filters.py
--- a/rest_framework/filters.py
+++ b/rest_framework/filters.py
@@ -140,12 +140,14 @@ def filter_queryset(self, request, queryset, view):
         ]
 
         base = queryset
+        conditions = []
         for search_term in search_terms:
             queries = [
                 models.Q(**{orm_lookup: search_term})
                 for orm_lookup in orm_lookups
             ]
-            queryset = queryset.filter(reduce(operator.or_, queries))
+            conditions.append(reduce(operator.or_, queries))
+        queryset = queryset.filter(reduce(operator.and_, conditions))
 
         if self.must_call_distinct(queryset, search_fields):
             # Filtering against a many-to-many field requires us to


Test patch:
diff --git a/tests/test_filters.py b/tests/test_filters.py
--- a/tests/test_filters.py
+++ b/tests/test_filters.py
@@ -5,6 +5,7 @@
 import warnings
 from decimal import Decimal
 
+import django
 import pytest
 from django.conf.urls import url
 from django.core.exceptions import ImproperlyConfigured
@@ -645,6 +646,51 @@ def test_must_call_distinct(self):
             )
 
 
+class Blog(models.Model):
+    name = models.CharField(max_length=20)
+
+
+class Entry(models.Model):
+    blog = models.ForeignKey(Blog, on_delete=models.CASCADE)
+    headline = models.CharField(max_length=120)
+    pub_date = models.DateField(null=True)
+
+
+class BlogSerializer(serializers.ModelSerializer):
+    class Meta:
+        model = Blog
+        fields = '__all__'
+
+
+class SearchFilterToManyTests(TestCase):
+
+    @classmethod
+    def setUpTestData(cls):
+        b1 = Blog.objects.create(name='Blog 1')
+        b2 = Blog.objects.create(name='Blog 2')
+
+        # Multiple entries on Lennon published in 1979 - distinct should deduplicate
+        Entry.objects.create(blog=b1, headline='Something about Lennon', pub_date=datetime.date(1979, 1, 1))
+        Entry.objects.create(blog=b1, headline='Another thing about Lennon', pub_date=datetime.date(1979, 6, 1))
+
+        # Entry on Lennon *and* a separate entry in 1979 - should not match
+        Entry.objects.create(blog=b2, headline='Something unrelated', pub_date=datetime.date(1979, 1, 1))
+        Entry.objects.create(blog=b2, headline='Retrospective on Lennon', pub_date=datetime.date(1990, 6, 1))
+
+    @unittest.skipIf(django.VERSION < (1, 9), "Django 1.8 does not support transforms")
+    def test_multiple_filter_conditions(self):
+        class SearchListView(generics.ListAPIView):
+            queryset = Blog.objects.all()
+            serializer_class = BlogSerializer
+            filter_backends = (filters.SearchFilter,)
+            search_fields = ('=name', 'entry__headline', '=entry__pub_date__year')
+
+        view = SearchListView.as_view()
+        request = factory.get('/', {'search': 'Lennon,1979'})
+        response = view(request)
+        assert len(response.data) == 1
+
+
 class OrderingFilterModel(models.Model):
     title = models.CharField(max_length=20, verbose_name='verbose title')
     text = models.CharField(max_length=100)