Repository: wagtail/wagtail

Issue:
Error "unknown output_field" after updating Django to 2.2.*
### Issue Summary

With Wagtail 2.4, 2.5 and 2.6.1, and Django 2.2.*, every save of a page in the wagtail interface, returns a 500 error.

> Cannot resolve expression type, unknown output_field

Also, running `manage.py update_index` raises the same error.

```
[...]
  File "[...]/venv/lib/python3.5/site-packages/wagtail/search/management/commands/update_index.py", line 87, in update_backend
    index.add_items(model, chunk)
  File "[...]/venv/lib/python3.5/site-packages/wagtail/contrib/postgres_search/backend.py", line 180, in add_items
    update_method(content_type_pk, objs)
  File "[...]/venv/lib/python3.5/site-packages/wagtail/contrib/postgres_search/backend.py", line 156, in add_items_update_then_create
    .update(autocomplete=obj._autocomplete_, body=obj._body_)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/query.py", line 741, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1429, in execute_sql
    cursor = super().execute_sql(result_type)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1087, in execute_sql
    sql, params = self.as_sql()
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1404, in as_sql
    sql, params = self.compile(val)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 405, in compile
    sql, params = node.as_sql(self, self.connection)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/expressions.py", line 435, in as_sql
    sql, params = compiler.compile(self.lhs)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 405, in compile
    sql, params = node.as_sql(self, self.connection)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/expressions.py", line 435, in as_sql
    sql, params = compiler.compile(self.lhs)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 405, in compile
    sql, params = node.as_sql(self, self.connection)
  File "[...]/venv/lib/python3.5/site-packages/django/contrib/postgres/search.py", line 78, in as_sql
    ) for expression in clone.get_source_expressions()
  File "[...]/venv/lib/python3.5/site-packages/django/contrib/postgres/search.py", line 78, in <listcomp>
    ) for expression in clone.get_source_expressions()
  File "[...]/venv/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "[...]/venv/lib/python3.5/site-packages/django/db/models/expressions.py", line 261, in output_field
    raise FieldError('Cannot resolve expression type, unknown output_field')
django.core.exceptions.FieldError: Cannot resolve expression type, unknown output_field
```

The bug occurs only when using Django >= 2.2. No problem on 2.1.7 or 2.1.12.


### Steps to Reproduce

1. Have the versions specified below (they are documented as compatible)
2. Install the postgres search backend
3. Run `manage.py update_index`, or hit "Save preview" or "Publish" in a Wagtail page edition.


### Technical details

* Python version: 3.5.3
* Requirements:
```
beautifulsoup4==4.6.0
certifi==2019.6.16
chardet==3.0.4
dj-database-url==0.5.0
dj-email-url==0.2.0
django-basicauth==0.5.2
django-modelcluster==4.4
django-taggit==0.24.0
django-treebeard==4.3
django==2.2.5
djangorestframework==3.10.2
draftjs-exporter==2.1.6
html5lib==1.0.1
idna==2.8
mailchimp3==3.0.7
pillow==6.1.0
psycopg2-binary==2.8.3
pytz==2019.2
requests==2.22.0
six==1.12.0
sqlparse==0.3.0
unidecode==1.1.1
urllib3==1.25.3
wagtail==2.6.1
webencodings==0.5.1
```
* Postgres version: 9.6.15


Discussion:
Hi @T4m - thanks for the report.

I'm unable to replicate this with the following steps:

* start from a clean python 3.5 virtualenv
* ```
    pip install wagtail==2.6.1
    wagtail start issue5547
    cd issue5547
    ```
* Replace the contents of requirements.txt with the list above
* `pip install -r requirements.txt`
* Edit issue5547/settings/base.py:
  * Add `'wagtail.contrib.postgres_search',` to installed apps
  * Change the `DATABASES` setting to:
    ```
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': 'issue5547',
        }
    }
  * Add the `WAGTAILSEARCH_BACKENDS` setting:
    ```
    WAGTAILSEARCH_BACKENDS = {
        'default': {
            'BACKEND': 'wagtail.contrib.postgres_search.backend',
        },
    }
* ```
    createdb issue5547
    ./manage.py migrate
    ./manage.py update_index
  ```

Can you confirm whether these steps reproduce the issue for you? The only differences in my setup are that I'm on Python 3.5.2 and Postgresql 9.5.19, and it doesn't look like the Postgres version makes a difference as the error occurs while Django is building the query.
Closing this as unable to replicate - happy to reopen if you can identify steps that will reproduce this on a clean Wagtail install.
Hi @gasman,

Thanks for taking the time to investigate the issue. I wasn't able to reproduce it either, but then I tried using Postgresql 9.4 and I could safely reproduce the issue with the steps you mentioned above. For the record I installed the `postgresql-9.4` package using this source: https://www.postgresql.org/download/linux/debian/.

I know Postgresql 9.4 is quite old, but [since it's supported by Django 2.2](https://docs.djangoproject.com/en/2.2/ref/databases/#postgresql-notes) I think Wagtail should also support it.

Could you please try reproducing the issue with Postgresql 9.4 and reopen the issue?
Thanks for the confirmation @sephii! I can confirm that I'm now seeing this with Postgres 9.4.
Looks like this error was triggered by this commit in Django 2.2.1: https://github.com/django/django/commit/88bf635c356b4d3a47e88dc4142b90060ce3c2ef
The reason we're only seeing this on Postgres 9.4 turns out to be fairly simple - we have a different code path for >=9.5...

https://github.com/wagtail/wagtail/blob/92aeaf00ff2275b67558e46803c887dfacd96516/wagtail/contrib/postgres_search/backend.py#L176-L180

The following code snippet replicates the error (presumably on any Postgres version):

    from wagtail.contrib.postgres_search.models import IndexEntry
    from django.contrib.postgres.search import SearchVector
    from django.db.models import Value

    IndexEntry.objects.all().update(body=SearchVector(Value('')))

Will rewrite this to a non-Wagtail-specific test case and open a Django bug report shortly.
On closer inspection, I'm not sure if this is a Django bug - from the [Django docs on `Value()`](https://docs.djangoproject.com/en/2.2/ref/models/expressions/#value-expressions), it's not clear in which circumstances it's valid to omit the `output_field` parameter, so it may be that it only worked "accidentally" prior to Django 2.2.1. Any thoughts, @BertrandBordage?
I'm not sure either, I already saw some weird behaviour around `output_field` requirements. When in doubt, we just have to add it, so `output_field=TextField())` in our case.
This is covered by the existing issue https://code.djangoproject.com/ticket/30446 - I've queried whether this qualifies as a bug, given that it's a regression in 2.2.1.

Resolving patch:
diff --git a/wagtail/contrib/postgres_search/backend.py b/wagtail/contrib/postgres_search/backend.py
--- a/wagtail/contrib/postgres_search/backend.py
+++ b/wagtail/contrib/postgres_search/backend.py
@@ -19,7 +19,7 @@
     get_content_type_pk, get_descendants_content_types_pks, get_postgresql_connections,
     get_sql_weights, get_weight)
 
-EMPTY_VECTOR = SearchVector(Value(''))
+EMPTY_VECTOR = SearchVector(Value('', output_field=TextField()))
 
 
 class Index:
@@ -32,6 +32,10 @@ def __init__(self, backend, db_alias=None):
             raise NotSupportedError(
                 'You must select a PostgreSQL database '
                 'to use PostgreSQL search.')
+
+        # Whether to allow adding items via the faster upsert method available in Postgres >=9.5
+        self._enable_upsert = (self.connection.pg_version >= 90500)
+
         self.entries = IndexEntry._default_manager.using(self.db_alias)
 
     def add_model(self, model):
@@ -137,11 +141,11 @@ def add_items_update_then_create(self, content_type_pk, objs):
         ids_and_objs = {}
         for obj in objs:
             obj._autocomplete_ = (
-                ADD([SearchVector(Value(text), weight=weight, config=config)
+                ADD([SearchVector(Value(text, output_field=TextField()), weight=weight, config=config)
                      for text, weight in obj._autocomplete_])
                 if obj._autocomplete_ else EMPTY_VECTOR)
             obj._body_ = (
-                ADD([SearchVector(Value(text), weight=weight, config=config)
+                ADD([SearchVector(Value(text, output_field=TextField()), weight=weight, config=config)
                      for text, weight in obj._body_])
                 if obj._body_ else EMPTY_VECTOR)
             ids_and_objs[obj._object_id_] = obj
@@ -173,9 +177,9 @@ def add_items(self, model, objs):
         # TODO: Delete unindexed objects while dealing with proxy models.
         if objs:
             content_type_pk = get_content_type_pk(model)
-            # Use a faster method for PostgreSQL >= 9.5
+
             update_method = (
-                self.add_items_upsert if self.connection.pg_version >= 90500
+                self.add_items_upsert if self._enable_upsert
                 else self.add_items_update_then_create)
             update_method(content_type_pk, objs)
 


Test patch:
diff --git a/wagtail/contrib/postgres_search/tests/test_backend.py b/wagtail/contrib/postgres_search/tests/test_backend.py
--- a/wagtail/contrib/postgres_search/tests/test_backend.py
+++ b/wagtail/contrib/postgres_search/tests/test_backend.py
@@ -135,3 +135,17 @@ def test_autocomplete_tsquery_chars(self):
         # Now the phrase operator.
         results = self.backend.autocomplete("first <-> second", models.Book)
         self.assertUnsortedListEqual([r.title for r in results], [])
+
+    def test_index_without_upsert(self):
+        # Test the add_items code path for Postgres 9.4, where upsert is not available
+        self.backend.reset_index()
+
+        index = self.backend.get_index_for_model(models.Book)
+        index._enable_upsert = False
+        index.add_items(models.Book, models.Book.objects.all())
+
+        results = self.backend.search("JavaScript", models.Book)
+        self.assertUnsortedListEqual([r.title for r in results], [
+            "JavaScript: The good parts",
+            "JavaScript: The Definitive Guide"
+        ])