Repository: carltongibson/django-filter

Issue:
`isnull` for `ForeignObjectRel` fields
First some setup.  I have the following models defined:

```python
class Task(models.Model):
    # ...

class Domain(models.Model):
    # ...

class DomainTask(models.Model):
    task = models.ForeignKey('Task')
    domain = models.ForeignKey('Domain')
    # ...
```

To get all `Task`s not referenced by a `DomainTask`, Django allows: `Task.objects.filter(domaintask__isnull=True)`.

Using the `'django_filters.rest_framework.DjangoFilterBackend'` as a default filter, I have the following view:

```python
from rest_framework import mixins
from rest_framework import viewsets

class Task(mixins.ListModelMixin,
           # ...
           viewsets.GenericViewSet):
    queryset = models.Task.objects.all()
    filter_fields = {
        'domaintask': ['exact', 'isnull'],
        'domaintask__domain': ['exact'],
        # ...
    }
```

The following request doesn't seem to apply the `isnull` filter.

```
GET http://localhost:8000/api/tasks/?domaintask__isnull=true
```

I looked into the filter generation and it seems as though [`BaseFilterSet.get_filters` doesn't create filters for the lookup expressions](https://github.com/carltongibson/django-filter/blob/a88c2bc98d28686b3b0f2ba5ae9528f38c8bb5a6/django_filters/filterset.py#L290-L293).

I hacked together a [proof-of-concept for supporting `isnull` here](https://github.com/jacktrades/django-filter/pull/1).  I'm curious if there's a reason `isnull` isn't supported, or if there is an existing way to achieve this that I'm simply overlooking?


Discussion:
Try this: 

```
GET http://localhost:8000/api/tasks/?domaintask=None
```

Does this get converted to `isnull` under the hood? (It should I think.) 
@carltongibson I don't believe it does.  Both of these queries return no results for me (they should):

```
GET http://localhost:8000/api/tasks/?domaintask=None
GET http://localhost:8000/api/tasks/?domaintask=null
```

Though, I don't think I'd be able to do the opposite query (`Task.objects.filter(domaintask__isnull=False)`) using that method either.
After a bit more investigation, it looks like `django.forms.models.ModelMultipleChoiceField._check_values` is raising a `ValidationError` which is being swallowed somewhere (though I'm not sure where).

It appears that the code intends to ["filter out the null value"](https://github.com/carltongibson/django-filter/blob/a88c2bc98d28686b3b0f2ba5ae9528f38c8bb5a6/django_filters/fields.py#L287-L289), but I don't think it's succeeding in my case.  I added the following print statements to `django_filters.fields.ModelMultipleChoiceField._check_values`:

```python
class ModelMultipleChoiceField(ChoiceIteratorMixin, forms.ModelMultipleChoiceField):
    iterator = ModelChoiceIterator

    def _check_values(self, value):
        print 'self.null_label', self.null_label
        print 'self.null_value', self.null_value
        print 'self.null_label is not None: ', self.null_label is not None
        print 'value: ', value
        print 'self.null_value in value: ', self.null_value in value

        null = self.null_label is not None and value and self.null_value in value
        if null:  # remove the null value and any potential duplicates

            print 'We never get here because self.null_label is None'
            value = [v for v in value if v != self.null_value]

        result = list(super(ModelMultipleChoiceField, self)._check_values(value))
        print 'We never get here because the superclass raises a ValidationError on `[u'null']`.'

        result += [self.null_value] if null else []
        return result
```

This is the output for `GET http://localhost:8000/api/tasks/?domaintask=null`:

```
self.null_label: None
self.null_value: null
self.null_label is not None:  False
value:  [u'null']
self.null_value in value:  True
```

And this is the output for `GET http://localhost:8000/api/tasks/?domaintask=None`:

```
self.null_label: None
self.null_value: null
self.null_label is not None:  False
value:  [u'None']
self.null_value in value:  False
```

EDIT: This does filter properly if I change the null check to: `null = value and self.null_value in value`
> `GET http://localhost:8000/api/tasks/?domaintask=None`
>
> Does this get converted to isnull under the hood? (It should I think.)

This works if `settings.FILTERS_NULL_CHOICE_LABEL` is set to a truthy value (like `'None'`).  So I can at least achieve the goal in my example of filtering for `Task`s not referenced by a `DomainTask`.

Though there are at least 2 outstanding issues with this approach:

1) I don't believe this solves for the case of filtering for `Tasks.objects.filter(domaintask__isnull=False)`.

2) `FILTERS_NULL_CHOICE_LABEL` seems kind-of orthogonal to whether a NULL value should be allowed in a multiple choice field.  Though I can see why the presence of a label would be a good indicator that you want to accept NULL values, setting a label to enable NULL values isn't entirely intuitive (especially if you're not using any of the generated filter forms).

@carltongibson if you're interested, I'd be happy to whip up a *real* PR for:

1) Allowing `isnull` (as demonstrated in https://github.com/jacktrades/django-filter/pull/1 but in a much less hacky way and with some actual tests).

and/or

2) Removing gating on the presence of `FILTERS_NULL_CHOICE_LABEL` for allowing a NULL value to be passed to `ModelMultipleChoiceField`.  Or possibly allowing an alternative setting that's unrelated to the label, (for a likely horrible name off-the-top-of-my-head, maybe `FILTERS_ALLOW_NULL_CHOICE`).  Something like:

    `null = (self.null_label is not None or settings.FILTER_ALLOW_NULL_CHOICE) and value and self.null_value in value`

Resolving patch:
diff --git a/django_filters/filterset.py b/django_filters/filterset.py
--- a/django_filters/filterset.py
+++ b/django_filters/filterset.py
@@ -4,7 +4,11 @@
 from django import forms
 from django.db import models
 from django.db.models.constants import LOOKUP_SEP
-from django.db.models.fields.related import ForeignObjectRel
+from django.db.models.fields.related import (
+    ManyToManyRel,
+    ManyToOneRel,
+    OneToOneRel
+)
 
 from .conf import settings
 from .constants import ALL_FIELDS
@@ -33,9 +37,17 @@
 
 
 def remote_queryset(field):
-    model = field.remote_field.model
-    limit_choices_to = field.get_limit_choices_to()
+    """
+    Get the queryset for the other side of a relationship. This works
+    for both `RelatedField`s and `ForignObjectRel`s.
+    """
+    model = field.related_model
 
+    # Reverse relationships do not have choice limits
+    if not hasattr(field, 'get_limit_choices_to'):
+        return model._default_manager.all()
+
+    limit_choices_to = field.get_limit_choices_to()
     return model._default_manager.complex_filter(limit_choices_to)
 
 
@@ -58,6 +70,14 @@ def __new__(cls, name, bases, attrs):
         new_class._meta = FilterSetOptions(getattr(new_class, 'Meta', None))
         new_class.base_filters = new_class.get_filters()
 
+        # TODO: remove assertion in 2.1
+        assert not hasattr(new_class, 'filter_for_reverse_field'), (
+            "`%(cls)s.filter_for_reverse_field` has been removed. "
+            "`%(cls)s.filter_for_field` now generates filters for reverse fields. "
+            "See: https://django-filter.readthedocs.io/en/master/guide/migration.html"
+            % {'cls': new_class.__name__}
+        )
+
         return new_class
 
     @classmethod
@@ -110,6 +130,8 @@ def get_declared_filters(cls, bases, attrs):
     models.GenericIPAddressField:       {'filter_class': CharFilter},
     models.CommaSeparatedIntegerField:  {'filter_class': CharFilter},
     models.UUIDField:                   {'filter_class': UUIDFilter},
+
+    # Forward relationships
     models.OneToOneField: {
         'filter_class': ModelChoiceFilter,
         'extra': lambda f: {
@@ -132,6 +154,27 @@ def get_declared_filters(cls, bases, attrs):
             'queryset': remote_queryset(f),
         }
     },
+
+    # Reverse relationships
+    OneToOneRel: {
+        'filter_class': ModelChoiceFilter,
+        'extra': lambda f: {
+            'queryset': remote_queryset(f),
+            'null_label': settings.NULL_CHOICE_LABEL if f.null else None,
+        }
+    },
+    ManyToOneRel: {
+        'filter_class': ModelMultipleChoiceFilter,
+        'extra': lambda f: {
+            'queryset': remote_queryset(f),
+        }
+    },
+    ManyToManyRel: {
+        'filter_class': ModelMultipleChoiceFilter,
+        'extra': lambda f: {
+            'queryset': remote_queryset(f),
+        }
+    },
 }
 
 
@@ -287,11 +330,6 @@ def get_filters(cls):
             if field is None:
                 undefined.append(field_name)
 
-            # ForeignObjectRel does not support non-exact lookups
-            if isinstance(field, ForeignObjectRel):
-                filters[field_name] = cls.filter_for_reverse_field(field, field_name)
-                continue
-
             for lookup_expr in lookups:
                 filter_name = cls.get_filter_name(field_name, lookup_expr)
 
@@ -337,19 +375,6 @@ def filter_for_field(cls, field, field_name, lookup_expr='exact'):
 
         return filter_class(**default)
 
-    @classmethod
-    def filter_for_reverse_field(cls, field, field_name):
-        rel = field.field.remote_field
-        queryset = field.field.model._default_manager.all()
-        default = {
-            'field_name': field_name,
-            'queryset': queryset,
-        }
-        if rel.multiple:
-            return ModelMultipleChoiceFilter(**default)
-        else:
-            return ModelChoiceFilter(**default)
-
     @classmethod
     def filter_for_lookup(cls, field, lookup_type):
         DEFAULTS = dict(cls.FILTER_DEFAULTS)
@@ -365,7 +390,7 @@ def filter_for_lookup(cls, field, lookup_type):
             return None, {}
 
         # perform lookup specific checks
-        if lookup_type == 'exact' and field.choices:
+        if lookup_type == 'exact' and getattr(field, 'choices', None):
             return ChoiceFilter, {'choices': field.choices}
 
         if lookup_type == 'isnull':


Test patch:
diff --git a/tests/test_filterset.py b/tests/test_filterset.py
--- a/tests/test_filterset.py
+++ b/tests/test_filterset.py
@@ -242,11 +242,12 @@ class Meta:
         self.assertEqual(params['widget'], BooleanWidget)
 
 
-class FilterSetFilterForReverseFieldTests(TestCase):
+class ReverseFilterSetFilterForFieldTests(TestCase):
+    # Test reverse relationships for `filter_for_field`
 
     def test_reverse_o2o_relationship(self):
         f = Account._meta.get_field('profile')
-        result = FilterSet.filter_for_reverse_field(f, 'profile')
+        result = FilterSet.filter_for_field(f, 'profile')
         self.assertIsInstance(result, ModelChoiceFilter)
         self.assertEqual(result.field_name, 'profile')
         self.assertTrue('queryset' in result.extra)
@@ -255,7 +256,7 @@ def test_reverse_o2o_relationship(self):
 
     def test_reverse_fk_relationship(self):
         f = User._meta.get_field('comments')
-        result = FilterSet.filter_for_reverse_field(f, 'comments')
+        result = FilterSet.filter_for_field(f, 'comments')
         self.assertIsInstance(result, ModelMultipleChoiceFilter)
         self.assertEqual(result.field_name, 'comments')
         self.assertTrue('queryset' in result.extra)
@@ -264,7 +265,7 @@ def test_reverse_fk_relationship(self):
 
     def test_reverse_m2m_relationship(self):
         f = Book._meta.get_field('lovers')
-        result = FilterSet.filter_for_reverse_field(f, 'lovers')
+        result = FilterSet.filter_for_field(f, 'lovers')
         self.assertIsInstance(result, ModelMultipleChoiceFilter)
         self.assertEqual(result.field_name, 'lovers')
         self.assertTrue('queryset' in result.extra)
@@ -273,7 +274,7 @@ def test_reverse_m2m_relationship(self):
 
     def test_reverse_non_symmetrical_selfref_m2m_field(self):
         f = DirectedNode._meta.get_field('inbound_nodes')
-        result = FilterSet.filter_for_reverse_field(f, 'inbound_nodes')
+        result = FilterSet.filter_for_field(f, 'inbound_nodes')
         self.assertIsInstance(result, ModelMultipleChoiceFilter)
         self.assertEqual(result.field_name, 'inbound_nodes')
         self.assertTrue('queryset' in result.extra)
@@ -282,13 +283,33 @@ def test_reverse_non_symmetrical_selfref_m2m_field(self):
 
     def test_reverse_m2m_field_with_through_model(self):
         f = Worker._meta.get_field('employers')
-        result = FilterSet.filter_for_reverse_field(f, 'employers')
+        result = FilterSet.filter_for_field(f, 'employers')
         self.assertIsInstance(result, ModelMultipleChoiceFilter)
         self.assertEqual(result.field_name, 'employers')
         self.assertTrue('queryset' in result.extra)
         self.assertIsNotNone(result.extra['queryset'])
         self.assertEqual(result.extra['queryset'].model, Business)
 
+    def test_reverse_relationship_lookup_expr(self):
+        f = Book._meta.get_field('lovers')
+        result = FilterSet.filter_for_field(f, 'lovers', 'isnull')
+        self.assertIsInstance(result, BooleanFilter)
+        self.assertEqual(result.field_name, 'lovers')
+        self.assertEqual(result.lookup_expr, 'isnull')
+
+
+class FilterSetFilterForReverseFieldTests(TestCase):
+
+    def test_method_raises_assertion(self):
+        msg = ("`F.filter_for_reverse_field` has been removed. "
+               "`F.filter_for_field` now generates filters for reverse fields.")
+
+        with self.assertRaisesMessage(AssertionError, msg):
+            class F(FilterSet):
+                @classmethod
+                def filter_for_reverse_field(cls, field, field_name):
+                    pass
+
 
 class FilterSetClassCreationTests(TestCase):