Repository: wagtail/wagtail

Issue:
MySQL search test failures on Django 4.1
MySQL search tests are currently failing against Django 4.1:

https://github.com/wagtail/wagtail/runs/7674979114?check_suite_focus=true

This wasn't caught earlier because our tests against the Django main branch were only running against Postgres and/or Sqlite :-(

Bisecting shows that https://github.com/django/django/commit/407fe95cb116599adeb4b9ed01df5673aa5cb1db was the commit where the failures started.


Discussion:

Resolving patch:
diff --git a/wagtail/search/backends/database/mysql/mysql.py b/wagtail/search/backends/database/mysql/mysql.py
--- a/wagtail/search/backends/database/mysql/mysql.py
+++ b/wagtail/search/backends/database/mysql/mysql.py
@@ -2,6 +2,7 @@
 from collections import OrderedDict
 
 from django.db import DEFAULT_DB_ALIAS, NotSupportedError, connections, transaction
+from django.db.models import Case, When
 from django.db.models.aggregates import Avg, Count
 from django.db.models.constants import LOOKUP_SEP
 from django.db.models.expressions import F
@@ -452,6 +453,32 @@ def search(self, config, start, stop, score_field=None):
         match_expression = MatchExpression(
             search_query, columns=["title", "body"], output_field=BooleanField()
         )  # For example: MATCH (`title`, `body`) AGAINST ('+query' IN BOOLEAN MODE)
+
+        # In Django 4.0 the above match expression would produce this SQL WHERE clause:
+        #
+        # WHERE ... MATCH (`title`, `body`) AGAINST (query IN BOOLEAN MODE)
+        #
+        # In Django 4.1, this behavior was changed:
+        #
+        # https://code.djangoproject.com/ticket/32691
+        # https://github.com/django/django/commit/407fe95cb116599adeb4b9ed01df5673aa5cb1db
+        #
+        # so that instead this SQL WHERE clause is generated, explicitly filtering
+        # against "= True":
+        #
+        # WHERE ... MATCH (`title`, `body`) AGAINST (query IN BOOLEAN MODE) = True
+        #
+        # This no longer works properly because MATCH returns a floating point score
+        # as a measurement of the match quality, not a boolean value:
+        #
+        # https://dev.mysql.com/doc/refman/8.0/en/fulltext-boolean.html
+        #
+        # In order for filtering on "= True" to work, we change the match expression
+        # SQL to be:
+        #
+        # WHERE ... CASE WHEN MATCH (`title`, `body`) AGAINST (query IN BOOLEAN MODE) THEN True ELSE False END = True
+        match_expression = Case(When(match_expression, then=True), default=False)
+
         score_expression = MatchExpression(
             search_query, columns=["title"], output_field=FloatField()
         ) * F("title_norm") + MatchExpression(


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -27,6 +27,7 @@ concurrency:
 # - django 4.0, python 3.10, postgres, USE_EMAIL_USER_MODEL=yes
 # - django 4.1, python 3.10, postgres, DISABLE_TIMEZONE=yes
 # - django stable/4.1.x, python 3.10, postgres (allow failures)
+# - django stable/4.1.x, python 3.10, mysql (allow failures)
 # - django main, python 3.10, postgres (allow failures)
 # - elasticsearch 5, django 3.2, python 3.7, sqlite
 # - elasticsearch 6, django 3.2, python 3.7, postgres
@@ -131,6 +132,9 @@ jobs:
           - python: '3.9'
             django: 'Django>=4.1,<4.2'
             experimental: false
+          - python: '3.10'
+            django: 'git+https://github.com/django/django.git@stable/4.1.x#egg=Django'
+            experimental: true
 
     services:
       mysql: