Repository: wagtail/wagtail

Issue:
Search no longer respects `fields` parameter
### Issue Summary

As shown [here](https://docs.wagtail.org/en/stable/topics/search/searching.html#specifying-the-fields-to-search), the `search` function lets the user specify which fields to search on. 

If any `boost` parameter is specified in `index.SearchField`, all fields are searched even if you specify a limited subset. This seems to be a regression after [this PR](https://github.com/wagtail/wagtail/pull/10653). The query compiler adds a [combined field](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/copy-to.html) with a name similar to `_all_text_boost_2` to each query, even if the `fields` parameter was used. This boosted field contains data from every field, so in effect all fields are searched.

An example of what the compiled query looks like exists in the tests [here](https://github.com/wagtail/wagtail/blob/5cf37f3a9dd586f1a1a86e831bae0a6f1eff2b3b/wagtail/search/tests/test_elasticsearch7_backend.py#L350). I'm not sure that `_all_text_boost_2_0^2.0` and `_all_text_boost_10_0^10.0` should be in this test.

I'm not sure what the intended behavior is. If the goal is only to have `boost` apply when searching all fields, you could only add these boosted fields when `fields` is not specified. Otherwise I think a different way of adding the boost is necessary ([this issue](https://github.com/wagtail/wagtail/issues/5422#issuecomment-645435567)).

I found this issue using the ElasicSearch7 backed. I'm not sure if it applies to database backends. I did not have this issue after removing all `boost` parameters from the models.

### Steps to Reproduce

1. Start a new project and setup ElasicSearch as the search backend. Create a model `Model` with 2 searchable fields, `title` and `author`.
2. Create a dummy entry, such as `Model(title="Title", author="Author"). 
3. Index the data
4. Run the following searches. This is the correct behavior.
```python
backend.search("Title", qs)  # 1 result
backend.search("Author", qs)  # 1 result
backend.search("Title", qs, fields=["author"])  # 0 results
backend.search("Author", qs, fields=["title"])  # 0 results
```
5. Add a `boost` parameter to each searchable field in `Model`
6. Re-index the data
7. Run the following code. This behavior is incorrect.
```python
backend.search("Title", qs)  # 1 result
backend.search("Author", qs)  # 1 result
backend.search("Title", qs, fields=["author"])  # 1 result, should be 0
backend.search("Author", qs, fields=["title"])  # 1 result, should be 0
```

### Technical details

-   Python version: 3.10
-   Django version: 4.2.7
-   Wagtail version: 5.2
-   Browser version: N/A


Discussion:
@scott-8 Thanks for the report and the detailed investigation! Can confirm I'm seeing the same issue.
FWIW, the database backends are even more limited in their support for `fields` - they incorporate a special case for fields named 'title', and (I suspect) only support `fields=["title"]` and no other cases. https://github.com/wagtail/wagtail/blob/2d5dc4d8f5c18e5aa1e174215e9d2f119b7110ee/wagtail/search/backends/database/postgres/postgres.py#L119-L133

Resolving patch:
diff --git a/wagtail/search/backends/elasticsearch7.py b/wagtail/search/backends/elasticsearch7.py
--- a/wagtail/search/backends/elasticsearch7.py
+++ b/wagtail/search/backends/elasticsearch7.py
@@ -38,6 +38,13 @@ def __init__(self, field_name, boost=1):
         self.field_name = field_name
         self.boost = boost
 
+    @property
+    def field_name_with_boost(self):
+        if self.boost == 1:
+            return self.field_name
+        else:
+            return f"{self.field_name}^{self.boost}"
+
 
 class Elasticsearch7Mapping:
     all_field_name = "_all_text"
@@ -458,42 +465,31 @@ def _remap_fields(self, fields):
         if fields:
             searchable_fields = {f.field_name: f for f in self.get_searchable_fields()}
             for field_name in fields:
-                if field_name in searchable_fields:
-                    field_name = self.mapping.get_field_column_name(
-                        searchable_fields[field_name]
-                    )
-
-                remapped_fields.append(Field(field_name))
+                field = searchable_fields.get(field_name)
+                if field:
+                    field_name = self.mapping.get_field_column_name(field)
+                    remapped_fields.append(Field(field_name, field.boost or 1))
         else:
             remapped_fields.append(Field(self.mapping.all_field_name))
 
-        models = get_indexed_models()
-        unique_boosts = set()
-        for model in models:
-            for field in model.get_searchable_search_fields():
-                if field.boost:
-                    unique_boosts.add(float(field.boost))
-
-        remapped_fields.extend(
-            [
-                Field(self.mapping.get_boost_field_name(boost), boost)
-                for boost in unique_boosts
-            ]
-        )
+            models = get_indexed_models()
+            unique_boosts = set()
+            for model in models:
+                if not issubclass(model, self.queryset.model):
+                    continue
+                for field in model.get_searchable_search_fields():
+                    if field.boost:
+                        unique_boosts.add(float(field.boost))
+
+            remapped_fields.extend(
+                [
+                    Field(self.mapping.get_boost_field_name(boost), boost)
+                    for boost in unique_boosts
+                ]
+            )
 
         return remapped_fields
 
-    def get_boosted_fields(self, fields):
-        boosted_fields = []
-        if not isinstance(fields, list):
-            fields = [fields]
-        for field in fields:
-            if field.boost != 1:
-                boosted_fields.append(f"{field.field_name}^{field.boost}")
-            else:
-                boosted_fields.append(field.field_name)
-        return boosted_fields
-
     def _process_lookup(self, field, lookup, value):
         column_name = self.mapping.get_field_column_name(field)
 
@@ -584,49 +580,53 @@ def _connect_filters(self, filters, connector, negated):
             return filter_out
 
     def _compile_plaintext_query(self, query, fields, boost=1.0):
-        fields = self.get_boosted_fields(fields)
         match_query = {"query": query.query_string}
 
         if query.operator != "or":
             match_query["operator"] = query.operator
 
-        if boost != 1.0:
-            match_query["boost"] = boost
-
         if len(fields) == 1:
-            return {"match": {fields[0]: match_query}}
+            if boost != 1.0 or fields[0].boost != 1.0:
+                match_query["boost"] = boost * fields[0].boost
+            return {"match": {fields[0].field_name: match_query}}
         else:
-            match_query["fields"] = fields
+            if boost != 1.0:
+                match_query["boost"] = boost
+            match_query["fields"] = [field.field_name_with_boost for field in fields]
 
             return {"multi_match": match_query}
 
     def _compile_fuzzy_query(self, query, fields):
-        if len(fields) == 1:
-            return {
-                "match": {
-                    fields[0]: {
-                        "query": query.query_string,
-                        "fuzziness": "AUTO",
-                    }
-                }
-            }
-        return {
-            "multi_match": {
-                "query": query.query_string,
-                "fields": self.get_boosted_fields(fields),
-                "fuzziness": "AUTO",
-            }
+        match_query = {
+            "query": query.query_string,
+            "fuzziness": "AUTO",
         }
+        if len(fields) == 1:
+            if fields[0].boost != 1.0:
+                match_query["boost"] = fields[0].boost
+            return {"match": {fields[0].field_name: match_query}}
+        else:
+            match_query["fields"] = [field.field_name_with_boost for field in fields]
+            return {"multi_match": match_query}
 
     def _compile_phrase_query(self, query, fields):
-        fields = self.get_boosted_fields(fields)
         if len(fields) == 1:
-            return {"match_phrase": {fields[0]: query.query_string}}
+            if fields[0].boost != 1.0:
+                return {
+                    "match_phrase": {
+                        fields[0].field_name: {
+                            "query": query.query_string,
+                            "boost": fields[0].boost,
+                        }
+                    }
+                }
+            else:
+                return {"match_phrase": {fields[0].field_name: query.query_string}}
         else:
             return {
                 "multi_match": {
                     "query": query.query_string,
-                    "fields": fields,
+                    "fields": [field.field_name_with_boost for field in fields],
                     "type": "phrase",
                 }
             }


Test patch:
diff --git a/wagtail/search/tests/test_elasticsearch7_backend.py b/wagtail/search/tests/test_elasticsearch7_backend.py
--- a/wagtail/search/tests/test_elasticsearch7_backend.py
+++ b/wagtail/search/tests/test_elasticsearch7_backend.py
@@ -67,7 +67,6 @@ def test_simple(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -125,7 +124,6 @@ def test_and_operator(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                         "operator": "and",
@@ -153,7 +151,6 @@ def test_filter(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -190,7 +187,6 @@ def test_and_filter(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -242,7 +238,6 @@ def test_or_filter(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -276,7 +271,6 @@ def test_negated_filter(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -296,13 +290,11 @@ def test_fields(self):
             "bool": {
                 "filter": {"match": {"content_type": "searchtests.Book"}},
                 "must": {
-                    "multi_match": {
-                        "fields": [
-                            "title",
-                            "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
-                        ],
-                        "query": "Hello",
+                    "match": {
+                        "title": {
+                            "query": "Hello",
+                            "boost": 2.0,
+                        },
                     },
                 },
             }
@@ -320,14 +312,12 @@ def test_fields_with_and_operator(self):
             "bool": {
                 "filter": {"match": {"content_type": "searchtests.Book"}},
                 "must": {
-                    "multi_match": {
-                        "fields": [
-                            "title",
-                            "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
-                        ],
-                        "query": "Hello",
-                        "operator": "and",
+                    "match": {
+                        "title": {
+                            "query": "Hello",
+                            "boost": 2.0,
+                            "operator": "and",
+                        }
                     },
                 },
             }
@@ -337,7 +327,7 @@ def test_fields_with_and_operator(self):
     def test_multiple_fields(self):
         # Create a query
         query = self.query_compiler_class(
-            models.Book.objects.all(), "Hello", fields=["title", "content"]
+            models.Book.objects.all(), "Hello", fields=["title", "summary"]
         )
 
         # Check it
@@ -347,10 +337,8 @@ def test_multiple_fields(self):
                 "must": {
                     "multi_match": {
                         "fields": [
-                            "title",
-                            "content",
-                            "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
+                            "title^2.0",
+                            "summary",
                         ],
                         "query": "Hello",
                     }
@@ -364,7 +352,7 @@ def test_multiple_fields_with_and_operator(self):
         query = self.query_compiler_class(
             models.Book.objects.all(),
             "Hello",
-            fields=["title", "content"],
+            fields=["title", "summary"],
             operator="and",
         )
 
@@ -375,10 +363,8 @@ def test_multiple_fields_with_and_operator(self):
                 "must": {
                     "multi_match": {
                         "fields": [
-                            "title",
-                            "content",
-                            "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
+                            "title^2.0",
+                            "summary",
                         ],
                         "query": "Hello",
                         "operator": "and",
@@ -406,7 +392,6 @@ def test_exact_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -433,7 +418,6 @@ def test_none_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -460,7 +444,6 @@ def test_isnull_true_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -487,7 +470,6 @@ def test_isnull_false_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -514,7 +496,6 @@ def test_startswith_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -546,7 +527,6 @@ def test_gt_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -576,7 +556,6 @@ def test_lt_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -606,7 +585,6 @@ def test_gte_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -636,7 +614,6 @@ def test_lte_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -674,7 +651,6 @@ def test_range_lookup(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     },
@@ -734,7 +710,6 @@ def test_phrase_query(self):
                 "fields": [
                     "_all_text",
                     "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
                 ],
                 "query": "Hello world",
                 "type": "phrase",
@@ -747,7 +722,7 @@ def test_phrase_query_multiple_fields(self):
         query_compiler = self.query_compiler_class(
             models.Book.objects.all(),
             Phrase("Hello world"),
-            fields=["title", "content"],
+            fields=["title", "summary"],
         )
 
         # Check it
@@ -755,10 +730,8 @@ def test_phrase_query_multiple_fields(self):
             "multi_match": {
                 "query": "Hello world",
                 "fields": [
-                    "title",
-                    "content",
-                    "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
+                    "title^2.0",
+                    "summary",
                 ],
                 "type": "phrase",
             }
@@ -773,14 +746,11 @@ def test_phrase_query_single_field(self):
 
         # Check it
         expected_result = {
-            "multi_match": {
-                "fields": [
-                    "title",
-                    "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
-                ],
-                "query": "Hello world",
-                "type": "phrase",
+            "match_phrase": {
+                "title": {
+                    "query": "Hello world",
+                    "boost": 2.0,
+                },
             },
         }
         self.assertDictEqual(query_compiler.get_inner_query(), expected_result)
@@ -798,7 +768,6 @@ def test_fuzzy_query(self):
                 "fields": [
                     "_all_text",
                     "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
                 ],
                 "query": "Hello world",
                 "fuzziness": "AUTO",
@@ -816,14 +785,12 @@ def test_fuzzy_query_single_field(self):
 
         # Check it
         expected_result = {
-            "multi_match": {
-                "fields": [
-                    "title",
-                    "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
-                ],
-                "query": "Hello world",
-                "fuzziness": "AUTO",
+            "match": {
+                "title": {
+                    "query": "Hello world",
+                    "fuzziness": "AUTO",
+                    "boost": 2.0,
+                },
             }
         }
         self.assertDictEqual(query_compiler.get_inner_query(), expected_result)
@@ -833,16 +800,14 @@ def test_fuzzy_query_multiple_fields(self):
         query_compiler = self.query_compiler_class(
             models.Book.objects.all(),
             Fuzzy("Hello world"),
-            fields=["title", "body"],
+            fields=["title", "summary"],
         )
 
         expected_result = {
             "multi_match": {
                 "fields": [
-                    "title",
-                    "body",
-                    "_all_text_boost_2_0^2.0",
-                    "_all_text_boost_10_0^10.0",
+                    "title^2.0",
+                    "summary",
                 ],
                 "query": "Hello world",
                 "fuzziness": "AUTO",
@@ -868,7 +833,6 @@ def test_year_filter(self):
                         "fields": [
                             "_all_text",
                             "_all_text_boost_2_0^2.0",
-                            "_all_text_boost_10_0^10.0",
                         ],
                         "query": "Hello",
                     }
@@ -1117,6 +1081,7 @@ def test_get_mapping(self):
                 },
                 "authors_filter": {"type": "integer"},
                 "publication_date_filter": {"type": "date"},
+                "summary": {"copy_to": "_all_text", "type": "text"},
                 "number_of_pages_filter": {"type": "integer"},
                 "tags": {
                     "type": "nested",
@@ -1162,6 +1127,7 @@ def test_get_document(self):
             ],
             "authors_filter": [2],
             "publication_date_filter": datetime.date(1954, 7, 29),
+            "summary": "",
             "number_of_pages_filter": 423,
             "tags": [],
             "tags_filter": [],
@@ -1265,6 +1231,7 @@ def test_get_mapping(self):
                 "authors_filter": {"type": "integer"},
                 "publication_date_filter": {"type": "date"},
                 "number_of_pages_filter": {"type": "integer"},
+                "summary": {"copy_to": "_all_text", "type": "text"},
                 "tags": {
                     "type": "nested",
                     "properties": {
@@ -1333,6 +1300,7 @@ def test_get_document(self):
             "authors_filter": [2],
             "publication_date_filter": datetime.date(1954, 7, 29),
             "number_of_pages_filter": 423,
+            "summary": "",
             "tags": [],
             "tags_filter": [],
         }
diff --git a/wagtail/test/search/migrations/0003_book_summary.py b/wagtail/test/search/migrations/0003_book_summary.py
new file mode 100644
--- /dev/null
+++ b/wagtail/test/search/migrations/0003_book_summary.py
@@ -0,0 +1,18 @@
+# Generated by Django 4.0.10 on 2023-11-10 01:12
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("searchtests", "0002_bookunindexed"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="book",
+            name="summary",
+            field=models.TextField(blank=True),
+        ),
+    ]
diff --git a/wagtail/test/search/models.py b/wagtail/test/search/models.py
--- a/wagtail/test/search/models.py
+++ b/wagtail/test/search/models.py
@@ -20,6 +20,7 @@ def __str__(self):
 
 class Book(index.Indexed, models.Model):
     title = models.CharField(max_length=255)
+    summary = models.TextField(blank=True)
     authors = models.ManyToManyField(Author, related_name="books")
     publication_date = models.DateField()
     number_of_pages = models.IntegerField()
@@ -27,6 +28,7 @@ class Book(index.Indexed, models.Model):
 
     search_fields = [
         index.SearchField("title", boost=2.0),
+        index.SearchField("summary"),
         index.AutocompleteField("title"),
         index.FilterField("title"),
         index.FilterField("authors"),