Repository: wagtail/wagtail

Issue:
System check for missing core page fields in search_fields
### Issue Summary

It's possible to accidentally set a page model's `search_fields` to a list that is missing the core Page fields such as `content_type` (either by forgetting the `search_fields = Page.search_fields + ...` incantation, or by not being careful enough with mixin classes). This causes an error when one of those results is returned in the Wagtail admin search, because it performs a facet query on `content_type_id` and does not catch instances of `None` in the output.

Having discussed with @kaedroho, we think the best remedy is a system check that produces a warning if any core search_fields (at least title, id, content_type, path, depth) are missing on any (non-abstract) subclass of Page. (This is a better option than a hard error because missing fields _might_ be valid on an extensively customised Wagtail installation; and adding error checking on the search view isn't desirable because it'll end up hiding a legitimate issue that's producing worse search results.)

### Steps to Reproduce

1. Set up a project with Elasticsearch
2. Define a page model with a `search_fields` definition that does not extend `Page.search_fields`:

        class BlogPage(Page):
            body = RichTextField()
            content_panels = Page.content_panels + [FieldPanel('body')]

            search_fields = [index.SearchField('body')]

3. Create an instance of that page, and ensure it's indexed by Elasticsearch
4. From the wagtail admin search bar, search for a term that's in the 'body' field

This throws the error:


    Traceback:

    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
      34.             response = get_response(request)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
      126.                 response = self.process_exception_by_middleware(e, request)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
      124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
      44.         response = view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/urls/__init__.py" in wrapper
      102.             return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/decorators.py" in decorated_view
      34.             return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/views/decorators/vary.py" in inner_func
      20.             response = func(*args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/utils.py" in wrapped_view_func
      147.                 return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/views/pages.py" in search
      943.                     for content_type_id, count in all_pages.facet('content_type_id').items()
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/views/pages.py" in <listcomp>
      943.                     for content_type_id, count in all_pages.facet('content_type_id').items()
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/db/models/manager.py" in manager_method
      82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/db/models/query.py" in get
      399.                 self.model._meta.object_name

    Exception Type: DoesNotExist at /admin/pages/search/
    Exception Value: ContentType matching query does not exist.

* I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: no


### Technical details

* Python version: 3.6.1
* Django version: 2.1.3
* Wagtail version: 2.3
* Elasticsearch 5.3.3 (client library version 5.5.3)

System check for missing core page fields in search_fields
### Issue Summary

It's possible to accidentally set a page model's `search_fields` to a list that is missing the core Page fields such as `content_type` (either by forgetting the `search_fields = Page.search_fields + ...` incantation, or by not being careful enough with mixin classes). This causes an error when one of those results is returned in the Wagtail admin search, because it performs a facet query on `content_type_id` and does not catch instances of `None` in the output.

Having discussed with @kaedroho, we think the best remedy is a system check that produces a warning if any core search_fields (at least title, id, content_type, path, depth) are missing on any (non-abstract) subclass of Page. (This is a better option than a hard error because missing fields _might_ be valid on an extensively customised Wagtail installation; and adding error checking on the search view isn't desirable because it'll end up hiding a legitimate issue that's producing worse search results.)

### Steps to Reproduce

1. Set up a project with Elasticsearch
2. Define a page model with a `search_fields` definition that does not extend `Page.search_fields`:

        class BlogPage(Page):
            body = RichTextField()
            content_panels = Page.content_panels + [FieldPanel('body')]

            search_fields = [index.SearchField('body')]

3. Create an instance of that page, and ensure it's indexed by Elasticsearch
4. From the wagtail admin search bar, search for a term that's in the 'body' field

This throws the error:


    Traceback:

    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
      34.             response = get_response(request)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
      126.                 response = self.process_exception_by_middleware(e, request)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
      124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
      44.         response = view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/urls/__init__.py" in wrapper
      102.             return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/decorators.py" in decorated_view
      34.             return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/views/decorators/vary.py" in inner_func
      20.             response = func(*args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/utils.py" in wrapped_view_func
      147.                 return view_func(request, *args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/views/pages.py" in search
      943.                     for content_type_id, count in all_pages.facet('content_type_id').items()
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/wagtail/admin/views/pages.py" in <listcomp>
      943.                     for content_type_id, count in all_pages.facet('content_type_id').items()
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/db/models/manager.py" in manager_method
      82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)
    File "/home/vagrant/.virtualenvs/myproject/lib/python3.6/site-packages/django/db/models/query.py" in get
      399.                 self.model._meta.object_name

    Exception Type: DoesNotExist at /admin/pages/search/
    Exception Value: ContentType matching query does not exist.

* I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: no


### Technical details

* Python version: 3.6.1
* Django version: 2.1.3
* Wagtail version: 2.3
* Elasticsearch 5.3.3 (client library version 5.5.3)



Discussion:

Resolving patch:
diff --git a/wagtail/search/apps.py b/wagtail/search/apps.py
--- a/wagtail/search/apps.py
+++ b/wagtail/search/apps.py
@@ -5,6 +5,8 @@
 
 from wagtail.search.signal_handlers import register_signal_handlers
 
+from . import checks  # NOQA
+
 
 class WagtailSearchAppConfig(AppConfig):
     name = 'wagtail.search'
diff --git a/wagtail/search/checks.py b/wagtail/search/checks.py
new file mode 100644
--- /dev/null
+++ b/wagtail/search/checks.py
@@ -0,0 +1,26 @@
+from django.core.checks import Warning, register
+
+
+@register('search')
+def page_search_fields_check(app_configs, **kwargs):
+    """Checks each page model with search_fields to core fields are included"""
+    from wagtail.core.models import Page, get_page_models
+
+    page_models = get_page_models()
+    errors = []
+
+    for cls in page_models:
+        # Only checks an initial subset of fields as only need to check some are missing to show the warning
+        if not all(field in cls.search_fields for field in Page.search_fields[:10]):
+            errors.append(
+                Warning(
+                    'Core Page fields missing in `search_fields`',
+                    hint=' '.join([
+                        'Ensure that {} extends the Page model search fields',
+                        '`search_fields = Page.search_fields + [...]`'
+                    ]).format(cls.__name__),
+                    obj=cls,
+                    id='wagtailsearch.W001'
+                ))
+
+    return errors


Test patch:
diff --git a/wagtail/search/tests/test_indexed_class.py b/wagtail/search/tests/test_indexed_class.py
--- a/wagtail/search/tests/test_indexed_class.py
+++ b/wagtail/search/tests/test_indexed_class.py
@@ -3,8 +3,10 @@
 from django.core import checks
 from django.test import TestCase
 
+from wagtail.core.models import Page
 from wagtail.search import index
 from wagtail.tests.search import models
+from wagtail.tests.testapp.models import EventPage, SingleEventPage
 
 
 @contextmanager
@@ -33,6 +35,15 @@ class TestSearchFields(TestCase):
     def make_dummy_type(self, search_fields):
         return type(str('DummyType'), (index.Indexed, ), dict(search_fields=search_fields))
 
+    def get_checks_result(warning_id=None):
+        """Run Django checks on any with the 'search' tag used when registering the check"""
+        checks_result = checks.run_checks()
+        if warning_id:
+            return [
+                warning for warning in
+                checks_result if warning.id == warning_id]
+        return checks_result
+
     def test_basic(self):
         cls = self.make_dummy_type([
             index.SearchField('test', boost=100, partial_match=False),
@@ -92,3 +103,23 @@ def test_checking_search_fields(self):
             ]
             errors = models.Book.check()
             self.assertEqual(errors, expected_errors)
+
+    def test_checking_core_page_fields_are_indexed(self):
+        """Run checks to ensure that when core page fields are missing we get a warning"""
+
+        # first confirm that errors show as EventPage (in test models) has no Page.search_fields
+        errors = [error for error in checks.run_checks() if error.id == 'wagtailsearch.W001']
+
+        # should only ever get this warning on the sub-classes of the page model
+        self.assertEqual([EventPage, SingleEventPage], [error.obj for error in errors])
+
+        for error in errors:
+            self.assertEqual(error.msg, 'Core Page fields missing in `search_fields`', )
+            self.assertIn(
+                'Page model search fields `search_fields = Page.search_fields + [...]`',
+                error.hint)
+
+        # second check that we get no errors when setting up the models correctly
+        with patch_search_fields(EventPage, Page.search_fields + EventPage.search_fields):
+            errors = [error for error in checks.run_checks() if error.id == 'wagtailsearch.W001']
+            self.assertEqual([], errors)