Repository: carltongibson/django-filter

Issue:
Deprecate autogeneration of filters for empty Meta.fields in favor of '__all__'
I was somewhat surprised to find that filters are automatically generated for all model fields for the following filterset:

``` py
class UserFilter(FilterSet):
    username = filter.CharFilter(lookup_expr='iexact')

    class Meta:
        model = User

>>> UserFilter.base_filters.keys()
['username', 'first_name', 'last_name', 'status', 'is_active', 'favorite_books']
```

I would have expected just the one `username` filter. It makes sense to me to deprecate this behavior in favor of using `fields = '__all__'` - similar to DRF's serializers and Django's ModelForms.
- Aside from API consistency, there are all of the [stated reasons](https://docs.djangoproject.com/en/1.9/topics/forms/modelforms/#selecting-the-fields-to-use) for requiring a declared value. The security risk is smaller since filters can't alter data, but users could still unintentionally expose information about the data model or the data itself. e.g., you may not be able to view an employee's salary, but you could accidentally allow users to filter by it.
- The implementation to support `Meta.fields = None` requires ignoring unrecognized field types, such as `tests.models.SubnetMaskField`. This prevents us from providing meaningful errors to the user. e.g.,
  
  ``` py
  class NetworkSettingFilter(FilterSet):
      class Meta:
          model = NetworkSetting
          fields = ['mask']
  ```
  
  We should be able to inform the user that the field type is unrecognized, and that they need to use `filter_overrides` or alter the `DEFAULTS`. Currently, users get:
  
  ``` py
  AttributeError: 'NoneType' object has no attribute 'model'
  ```

Thoughts?



Discussion:
OK yes. I'm in favour of this at first glance. 

I've got a couple of API changed in mind. I also need to merge you outstanding PRs

I like the roadmap to a 1.0 we've discussed. Target for all of this is August, when I'll have the time to give. i.e. 1.0 before September. I guess this would go there. 

Incidentally, avoiding automatic creation of (potentially dangerous) filters is why I'm not keen on the automatic relationship spanning of DRF filters.

> Incidentally, avoiding automatic creation of (potentially dangerous) filters is why I'm not keen on the automatic relationship spanning of DRF filters.

You're absolutely right. There's been some recent discussion on spanning to-many relationships that has had me thinking about this. The short version is that a viewset typically has a get_queryset implementation that will provide default view permissions filtering. eg, for a Post resource, you may allow the logged-in user to view all of his/her posts and any _published_ posts for all other authors. With drf-filters, there's no access to the request context, so you can't do any meaningful filtering - most users will just use the default queryset. So, while I may not be able to view another author's unpublished posts, I could try to divine what they're talking about. eg,

``` http
/api/authors?post__is_published=False&post__body__contains=juicy%20story%20details
```

Not _super_ terrible, but eek :/ 
The Good News is that I've got some ideas on how to solve this in a dryish, user-friendly manner.


Resolving patch:
diff --git a/django_filters/filterset.py b/django_filters/filterset.py
--- a/django_filters/filterset.py
+++ b/django_filters/filterset.py
@@ -3,7 +3,6 @@
 
 import copy
 import re
-import warnings
 from collections import OrderedDict
 
 from django import forms
@@ -21,18 +20,12 @@
                       ChoiceFilter, DateFilter, DateTimeFilter, TimeFilter, ModelChoiceFilter,
                       ModelMultipleChoiceFilter, NumberFilter, UUIDFilter,
                       DurationFilter)
-from .utils import try_dbfield, get_model_field, resolve_field
+from .utils import try_dbfield, get_all_model_fields, get_model_field, resolve_field, deprecate
 
 
 ORDER_BY_FIELD = 'o'
 
 
-def deprecate(msg):
-    warnings.warn(
-        "%s See: https://django-filter.readthedocs.io/en/latest/migration.html" % msg,
-        DeprecationWarning, stacklevel=3)
-
-
 class STRICTNESS(object):
     """
     Values of False & True chosen for backward compatability reasons.
@@ -68,13 +61,15 @@ def get_declared_filters(bases, attrs, with_base_filters=True):
 def filters_for_model(model, fields=None, exclude=None, filter_for_field=None,
                       filter_for_reverse_field=None):
     field_dict = OrderedDict()
-    opts = model._meta
-    if fields is None:
-        fields = [
-            f.name for f in sorted(opts.fields + opts.many_to_many)
-            if not isinstance(f, models.AutoField) and
-            not (getattr(remote_field(f), 'parent_link', False))
-        ]
+
+    # Setting exclude with no fields implies all other fields.
+    if exclude is not None and fields is None:
+        fields = '__all__'
+
+    # All implies all db fields associated with a filter_class.
+    if fields == '__all__':
+        fields = get_all_model_fields(model)
+
     # Loop through the list of fields.
     for f in fields:
         # Skip the field if excluded.
@@ -139,6 +134,19 @@ def all_valid(fieldset):
 
 class FilterSetOptions(object):
     def __init__(self, options=None):
+        if getattr(options, 'model', None) is not None:
+            if not hasattr(options, 'fields') and not hasattr(options, 'exclude'):
+                deprecate(
+                    "Not setting Meta.fields with Meta.model is undocumented behavior "
+                    "and may result in unintentionally exposing filter fields. This has "
+                    "been deprecated in favor of setting Meta.fields = '__all__' or by "
+                    "setting the Meta.exclude attribute.", 1)
+
+            elif getattr(options, 'fields', -1) is None:
+                deprecate(
+                    "Setting 'Meta.fields = None' is undocumented behavior and has been "
+                    "deprecated in favor of Meta.fields = '__all__'.", 1)
+
         self.model = getattr(options, 'model', None)
         self.fields = getattr(options, 'fields', None)
         self.exclude = getattr(options, 'exclude', None)
@@ -166,6 +174,9 @@ def __new__(cls, name, bases, attrs):
 
         opts = new_class._meta = FilterSetOptions(
             getattr(new_class, 'Meta', None))
+
+        # TODO: replace with deprecations
+        # if opts.model and opts.fields:
         if opts.model:
             filters = new_class.filters_for_model(opts.model, opts)
             filters.update(declared_filters)
@@ -432,8 +443,15 @@ def get_order_by(self, order_choice):
 
     @classmethod
     def filters_for_model(cls, model, opts):
+        # TODO: remove with deprecations - this emulates the old behavior
+        fields = opts.fields
+        if fields is None:
+            DEFAULTS = dict(FILTER_FOR_DBFIELD_DEFAULTS)
+            DEFAULTS.update(cls.filter_overrides)
+            fields = get_all_model_fields(model, field_types=DEFAULTS.keys())
+
         return filters_for_model(
-            model, opts.fields, opts.exclude,
+            model, fields, opts.exclude,
             cls.filter_for_field,
             cls.filter_for_reverse_field
         )
@@ -451,8 +469,13 @@ def filter_for_field(cls, f, name, lookup_expr='exact'):
         filter_class, params = cls.filter_for_lookup(f, lookup_type)
         default.update(params)
 
-        if filter_class is not None:
-            return filter_class(**default)
+        assert filter_class is not None, (
+            "%s resolved field '%s' with '%s' lookup to an unrecognized field "
+            "type %s. Try adding an override to 'filter_overrides'. See: "
+            "https://django-filter.readthedocs.io/en/latest/usage.html#overriding-default-filters"
+        ) % (cls.__name__, name, lookup_expr, f.__class__.__name__)
+
+        return filter_class(**default)
 
     @classmethod
     def filter_for_reverse_field(cls, f, name):
@@ -543,7 +566,7 @@ class FilterSet(six.with_metaclass(FilterSetMetaclass, BaseFilterSet)):
 
 
 def filterset_factory(model):
-    meta = type(str('Meta'), (object,), {'model': model})
+    meta = type(str('Meta'), (object,), {'model': model, 'fields': '__all__'})
     filterset = type(str('%sFilterSet' % model._meta.object_name),
                      (FilterSet,), {'Meta': meta})
     return filterset
diff --git a/django_filters/utils.py b/django_filters/utils.py
--- a/django_filters/utils.py
+++ b/django_filters/utils.py
@@ -1,3 +1,5 @@
+import warnings
+
 from django.conf import settings
 from django.core.exceptions import FieldError
 from django.db import models
@@ -7,10 +9,16 @@
 from django.db.models.fields.related import ForeignObjectRel
 from django.utils import six, timezone
 
-from .compat import remote_model
+from .compat import remote_field, remote_model
 from .exceptions import FieldLookupError
 
 
+def deprecate(msg, level_modifier=0):
+    warnings.warn(
+        "%s See: https://django-filter.readthedocs.io/en/latest/migration.html" % msg,
+        DeprecationWarning, stacklevel=3 + level_modifier)
+
+
 def try_dbfield(fn, field_class):
     """
     Try ``fn`` with the DB ``field_class`` by walking its
@@ -31,6 +39,25 @@ def try_dbfield(fn, field_class):
             return data
 
 
+# TODO: remove field_types arg with deprecations
+def get_all_model_fields(model, field_types=None):
+    opts = model._meta
+
+    if field_types is not None:
+        return [
+            f.name for f in sorted(opts.fields + opts.many_to_many)
+            if not isinstance(f, models.AutoField) and
+            not (getattr(remote_field(f), 'parent_link', False)) and
+            f.__class__ in field_types
+        ]
+
+    return [
+        f.name for f in sorted(opts.fields + opts.many_to_many)
+        if not isinstance(f, models.AutoField) and
+        not (getattr(remote_field(f), 'parent_link', False))
+    ]
+
+
 def get_model_field(model, field_name):
     """
     Get a ``model`` field, traversing relationships


Test patch:
diff --git a/tests/test_deprecations.py b/tests/test_deprecations.py
--- a/tests/test_deprecations.py
+++ b/tests/test_deprecations.py
@@ -3,12 +3,16 @@
 from django.test import TestCase
 
 from django_filters import FilterSet
+from django_filters.filters import CharFilter
 from .models import User
+from .models import NetworkSetting
+from .models import SubnetMaskField
 
 
 class UserFilter(FilterSet):
     class Meta:
         model = User
+        fields = '__all__'
 
 
 class FilterSetContainerDeprecationTests(TestCase):
@@ -47,3 +51,58 @@ def test__count__notification(self):
             UserFilter().count()
 
             self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
+
+
+class FilterSetMetaDeprecationTests(TestCase):
+    def test_fields_not_set(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+
+            class F(FilterSet):
+                class Meta:
+                    model = User
+
+            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
+            self.assertIn("Not setting Meta.fields with Meta.model is undocumented behavior", str(w[-1].message))
+
+    def test_fields_is_none(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+
+            class F(FilterSet):
+                class Meta:
+                    model = User
+                    fields = None
+
+            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
+            self.assertIn("Setting 'Meta.fields = None' is undocumented behavior", str(w[-1].message))
+
+    def test_fields_not_set_ignore_unknown(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+
+            class F(FilterSet):
+                class Meta:
+                    model = NetworkSetting
+
+            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
+            self.assertIn("Not setting Meta.fields with Meta.model is undocumented behavior", str(w[-1].message))
+
+        self.assertNotIn('mask', F.base_filters.keys())
+
+    def test_fields_not_set_with_override(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+
+            class F(FilterSet):
+                filter_overrides = {
+                    SubnetMaskField: {'filter_class': CharFilter}
+                }
+
+                class Meta:
+                    model = NetworkSetting
+
+            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
+            self.assertIn("Not setting Meta.fields with Meta.model is undocumented behavior", str(w[-1].message))
+
+        self.assertEqual(list(F.base_filters.keys()), ['ip', 'mask'])
diff --git a/tests/test_filtering.py b/tests/test_filtering.py
--- a/tests/test_filtering.py
+++ b/tests/test_filtering.py
@@ -390,6 +390,7 @@ class F(FilterSet):
 
             class Meta:
                 model = SpacewalkRecord
+                fields = '__all__'
 
         qs = SpacewalkRecord.objects.order_by('duration')
 
diff --git a/tests/test_filterset.py b/tests/test_filterset.py
--- a/tests/test_filterset.py
+++ b/tests/test_filterset.py
@@ -153,6 +153,17 @@ def test_field_that_is_subclassed(self):
         result = FilterSet.filter_for_field(f, 'first_name')
         self.assertIsInstance(result, CharFilter)
 
+    def test_unknown_field_type_error(self):
+        f = NetworkSetting._meta.get_field('mask')
+
+        with self.assertRaises(AssertionError) as excinfo:
+            FilterSet.filter_for_field(f, 'mask')
+
+        self.assertIn(
+            "FilterSet resolved field 'mask' with 'exact' lookup "
+            "to an unrecognized field type SubnetMaskField",
+            excinfo.exception.args[0])
+
     def test_symmetrical_selfref_m2m_field(self):
         f = Node._meta.get_field('adjacents')
         result = FilterSet.filter_for_field(f, 'adjacents')
@@ -301,6 +312,7 @@ def test_model_derived(self):
         class F(FilterSet):
             class Meta:
                 model = Book
+                fields = '__all__'
 
         self.assertEqual(len(F.declared_filters), 0)
         self.assertEqual(len(F.base_filters), 3)
@@ -313,6 +325,7 @@ class F(FilterSet):
 
             class Meta:
                 model = Book
+                fields = '__all__'
 
         self.assertEqual(len(F.declared_filters), 1)
         self.assertEqual(len(F.base_filters), 4)
@@ -421,10 +434,22 @@ class Meta:
         self.assertListEqual(list(F.base_filters),
                              ['username', 'price'])
 
+    def test_meta_exlude_with_no_fields(self):
+        class F(FilterSet):
+            class Meta:
+                model = Book
+                exclude = ('price', )
+
+        self.assertEqual(len(F.declared_filters), 0)
+        self.assertEqual(len(F.base_filters), 2)
+        self.assertListEqual(list(F.base_filters),
+                             ['title', 'average_rating'])
+
     def test_filterset_class_inheritance(self):
         class F(FilterSet):
             class Meta:
                 model = Book
+                fields = '__all__'
 
         class G(F):
             pass
@@ -435,6 +460,7 @@ class F(FilterSet):
 
             class Meta:
                 model = Book
+                fields = '__all__'
 
         class G(F):
             pass
@@ -444,6 +470,7 @@ def test_abstract_model_inheritance(self):
         class F(FilterSet):
             class Meta:
                 model = Restaurant
+                fields = '__all__'
 
         self.assertEqual(set(F.base_filters), set(['name', 'serves_pizza']))
 
@@ -454,13 +481,6 @@ class Meta:
 
         self.assertEqual(set(F.base_filters), set(['name', 'serves_pizza']))
 
-    def test_custom_field_ignored(self):
-        class F(FilterSet):
-            class Meta:
-                model = NetworkSetting
-
-        self.assertEqual(list(F.base_filters.keys()), ['ip'])
-
     def test_custom_field_gets_filter_from_override(self):
         class F(FilterSet):
             filter_overrides = {
@@ -468,6 +488,7 @@ class F(FilterSet):
 
             class Meta:
                 model = NetworkSetting
+                fields = '__all__'
 
         self.assertEqual(list(F.base_filters.keys()), ['ip', 'mask'])
 
@@ -475,10 +496,12 @@ def test_filterset_for_proxy_model(self):
         class F(FilterSet):
             class Meta:
                 model = User
+                fields = '__all__'
 
         class ProxyF(FilterSet):
             class Meta:
                 model = AdminUser
+                fields = '__all__'
 
         self.assertEqual(list(F.base_filters), list(ProxyF.base_filters))
 
@@ -486,10 +509,12 @@ def test_filterset_for_mti_model(self):
         class F(FilterSet):
             class Meta:
                 model = Account
+                fields = '__all__'
 
         class FtiF(FilterSet):
             class Meta:
                 model = BankAccount
+                fields = '__all__'
 
         # fails due to 'account_ptr' getting picked up
         self.assertEqual(
@@ -760,7 +785,7 @@ class Meta:
         self.assertQuerysetEqual(f.qs, [self.alex.pk], lambda o: o.pk)
 
 
-@unittest.skip('remove when relevant deprecations have been completed')
+@unittest.skip('TODO: remove when relevant deprecations have been completed')
 class MiscFilterSetTests(TestCase):
 
     def test_no__getitem__(self):
diff --git a/tests/test_forms.py b/tests/test_forms.py
--- a/tests/test_forms.py
+++ b/tests/test_forms.py
@@ -39,6 +39,7 @@ class MyForm(forms.Form):
         class F(FilterSet):
             class Meta:
                 model = Book
+                fields = '__all__'
                 form = MyForm
 
         f = F().form