Repository: carltongibson/django-filter

Issue:
Fix MultipleChoiceFilter for non-required fields
MultipleChoiceFilter skips filtering if the number of choices of the filter are equal to the number of choices of the field. The reason behind this is probably that if all choices are allowed then filtering doesn't make any sense. That is correct of the field is required to have a value, but not if the field can be empty. In that case it will fail to filter out the records that don't have any value at all. This patch fixes MultipleChoiceFilter so that it only optimises the case were a value is required.

Multiple choice filtering
Hi,

This change fixes the behaviour of a MultipleChoiceField when the submitted choices are the same as the available choices.

Currently the code short circuits and returns the queryset as-is, without applying the choices. This is a bug because:
- The database field may allow NULL values, in which case the queryset returns all rows including rows where the value was NULL.
- The filter field may have been configured with fewer choices than are available on the underlying database field.

In both cases the the filter isn't applied, and the queryset may included unwanted records. Test including demonstrating how the filters aren't applied.



Discussion:
I have run into this same issue. Is this patch likely to be accepted and included in a future release?

@bobsilverberg I suppose it won't get much far while it makes tests fail. Also #128 is a duplicate of this one.

Hi @steffann — all the tests are failing. Can you please update this to the latest `develop` — just pull, check it still works and push to your fork again — and we'll see if that's still the case. 

A test case covering the bad behaviour would be great too :-)

The short circuit logic in MultipleChoiceFilter was added as a fix to https://github.com/alex/django-filter/issues/20 . The original complaint was that the query being generated was taking too long to execute in a situation where a simple 'is null' would have been sufficient.

There is also a related bug when the choices are filtered and a users selects all the presented choices. Because of the short circuit no filtering will be done, rather than correctly filtering by the users's selection.

I would like to propose a different fix for these issues. The original issue was about performance, and the way MultipleChoiceFilter currently filters generates some hugely inefficient SQL. It generates the equivalent of qs.filter(Q(book__author__email='anne')|Q(book__author__email='bob')|Q(book_author__email='charlie'))
where we would typically write qs.filter(book__author__email__in=('anne', 'bob', 'charlie')).

Both forms are logically identical, but the first form will require the database to join the book and author tables three times (once for each option, it gets worse the more options you choose), the second form only once.

By using the preferred form we could eliminate the short circuit code and all the bugs that it introduced and use straight forward code with no edge cases that still performs well.

I have added a test case for the related issue, and a fix removing the short circuit logic on this branch: 
https://github.com/jimpot/django-filter/tree/short-circuit-bug

This solution should also fix the issue raised here.

Actually I have to take some of that back, the Q or'ing doesn't cause multiple joins.

However, my failing test case is still valid and I still think it might be a good idea to remove the short circuit logic as it is causing a few bugs and is so rarely applicable.

HI @jimpot — thanks for the input. It's on my list to clear up the remaining issues/pull requests on Django Filter — including this one. 

Can you make a pull request from you branch? Much easier to review that way. Thanks! 

Related to #132, #128, #176 and #20.

Hi @davidwtbuxton — Thanks for the pull. I see you noted the related issues. I'm looking to do a release with a fix included this week. 


Resolving patch:
diff --git a/django_filters/filters.py b/django_filters/filters.py
--- a/django_filters/filters.py
+++ b/django_filters/filters.py
@@ -100,13 +100,42 @@ class ChoiceFilter(Filter):
 class MultipleChoiceFilter(Filter):
     """
     This filter preforms an OR query on the selected options.
+
+    Advanced Use
+    ------------
+    Depending on your application logic, when all or no choices are selected, filtering may be a noop. In this case you may wish to avoid the filtering overhead, particularly of the `distinct` call.
+
+    Set `always_filter` to False after instantiation to enable the default `is_noop` test.
+
+    Override `is_noop` if you require a different test for your application.
     """
     field_class = forms.MultipleChoiceField
 
+    always_filter = True
+
+    def is_noop(self, qs, value):
+        """
+        Return True to short-circuit unnecessary and potentially slow filtering.
+        """
+        if self.always_filter:
+            return False
+
+        # A reasonable default for being a noop...
+        if self.required and len(value) == len(self.field.choices):
+            return True
+
+        return False
+
     def filter(self, qs, value):
-        value = value or ()
-        if len(value) == len(self.field.choices):
+        value = value or () # Make sure we have an iterable
+
+        if self.is_noop(qs, value):
             return qs
+
+        # Even though not a noop, no point filtering if empty
+        if not value:
+            return qs
+
         q = Q()
         for v in value:
             q |= Q(**{self.name: v})


Test patch:
diff --git a/tests/models.py b/tests/models.py
--- a/tests/models.py
+++ b/tests/models.py
@@ -80,6 +80,11 @@ class Article(models.Model):
     published = models.DateTimeField()
     author = models.ForeignKey(User, null=True)
 
+    def __str__(self):
+        if self.author_id:
+            return "%s on %s" % (self.author, self.published)
+        return "Anonymous on %s" % self.published
+
 
 @python_2_unicode_compatible
 class Book(models.Model):
diff --git a/tests/test_filtering.py b/tests/test_filtering.py
--- a/tests/test_filtering.py
+++ b/tests/test_filtering.py
@@ -23,6 +23,7 @@
 # from django_filters.filters import DateTimeFilter
 from django_filters.filters import MethodFilter
 from django_filters.filters import MultipleChoiceFilter
+from django_filters.filters import ModelMultipleChoiceFilter
 from django_filters.filters import NumberFilter
 from django_filters.filters import RangeFilter
 # from django_filters.widgets import LinkWidget
@@ -304,6 +305,8 @@ def setUp(self):
         alex.favorite_books = [b1, b2]
         aaron.favorite_books = [b1, b3]
 
+        self.alex = alex
+
     def test_filtering(self):
         class F(FilterSet):
             class Meta:
@@ -342,6 +345,60 @@ class Meta:
         f = F({'favorite_books': ['4']}, queryset=qs)
         self.assertQuerysetEqual(f.qs, [], lambda o: o.username)
 
+    def test_filtering_on_all_of_subset_of_choices(self):
+        class F(FilterSet):
+            class Meta:
+                model = User
+                fields = ['favorite_books']
+
+            def __init__(self, *args, **kwargs):
+                super(F, self).__init__(*args, **kwargs)
+                # This filter has a limited number of choices.
+                self.filters['favorite_books'].extra.update({
+                    'queryset': Book.objects.filter(id__in=[1, 2])
+                })
+
+                self.filters['favorite_books'].required = True
+
+        qs = User.objects.all().order_by('username')
+
+        # Select all the given choices.
+        f = F({'favorite_books': ['1', '2']}, queryset=qs)
+
+        # The results should only include matching users - not Jacob.
+        self.assertQuerysetEqual(f.qs, ['aaron', 'alex'], lambda o: o.username)
+
+    def test_filtering_on_non_required_fields(self):
+        # See issue #132 - filtering with all options on a non-required
+        # field should exclude any results where the field is null.
+        class F(FilterSet):
+            author = ModelMultipleChoiceFilter(queryset=User.objects.all())
+
+            class Meta:
+                model = Article
+                fields = ['author']
+
+        published = now()
+        Article.objects.create(published=published, author=self.alex)
+        Article.objects.create(published=published, author=self.alex)
+        Article.objects.create(published=published)
+
+        qs = Article.objects.all()
+
+        # Select all authors.
+        authors = [
+            str(user.id)
+            for user in User.objects.all()
+        ]
+        f = F({'author': authors}, queryset=qs)
+
+        # The results should not include anonymous articles
+        self.assertEqual(
+            set(f.qs),
+            set(Article.objects.exclude(author__isnull=True)),
+        )
+
+
 class NumberFilterTests(TestCase):
 
     def setUp(self):
diff --git a/tests/test_filters.py b/tests/test_filters.py
--- a/tests/test_filters.py
+++ b/tests/test_filters.py
@@ -254,9 +254,10 @@ def test_filtering(self):
             qs.filter.assert_called_once_with(mockQ1.__ior__.return_value)
             qs.filter.return_value.distinct.assert_called_once_with()
 
-    def test_filtering_skipped_when_len_of_value_is_len_of_field_choices(self):
+    def test_filtering_on_required_skipped_when_len_of_value_is_len_of_field_choices(self):
         qs = mock.Mock(spec=[])
-        f = MultipleChoiceFilter(name='somefield')
+        f = MultipleChoiceFilter(name='somefield', required=True)
+        f.always_filter = False
         result = f.filter(qs, [])
         self.assertEqual(len(f.field.choices), 0)
         self.assertEqual(qs, result)