Repository: openfun/richie

Issue:
Manage textual search query analysis language
## Feature Request

**Is your feature request related to a problem or unsupported use case? Please describe.**
This is somewhat between a bug report and a feature request.

Up to this point, we're indexing content using analyzers matching its language. Then, when a textual query hits, we're searching in all languages for fields we're searching in.

This can cause unwanted results, most notably because eg. a French query will be analyzed by an English analyzer to search in the English version of the field (in addition to being analyzed by a French analyzer to search in the French version of the same field). The bug here happens when there are stop words in the query: they are removed by French analysis but kept in by the English analysis, which matches a lot more documents than we'd like.

This is especially problematic because we have our own sorting algorithm for courses so the irrelevant results generated by this issue can be at the top of the search results.

**Describe the solution you'd like**
Searching in all languages for a field is not just an oversight. We actually think it can support more use cases and be an affordance that helps users find what they're looking for (if we think of a user that landed on the wrong language for the portal, or a user from a country where more than one language is routinely used).

**This means we do not want to just force the site language onto the query analysis, or add an API parameter to control it.**

Instead, we can use NLP to try and guess the language in the query and use the appropriate analyzer if we have one and if the guess is conclusive enough. We can also use the current website language to guide this guess and serve as a default of last resort.

Practically, this means doing this guessing work in the request cycle and specifying the analyzer on the query instead of relying on the one listed in our mappings.

As a bonus, we might look into weighing results for languages depending on our best guess for a language and the current site language.

**Describe alternatives you've considered**
See above for discussion on other strategies.

**Discovery, Documentation, Adoption, Migration Strategy**
This should be transparent for users, resolving an issue around cases where they get bogus results due to stop word analysis across languages, and generally providing the best results we reasonably can.
This does not touch indices so the upgrade is transparent there too.

**Do you want to work on it through a Pull Request?**
Currently working on it 👍



Discussion:
@2018D I'm not sure you already encountered this issue, but as you operate a Canadian portal, if you have some feedback regarding users' query language, it could be of interest here.

Would Québecois use both French and English words in the same query? Would they try both languages regardless of the UI language? Would they just write their query in the language of the UI?
What if the UI does not match their preferred language?

Resolving patch:
diff --git a/src/richie/apps/search/defaults.py b/src/richie/apps/search/defaults.py
--- a/src/richie/apps/search/defaults.py
+++ b/src/richie/apps/search/defaults.py
@@ -12,6 +12,11 @@
 # Use a lazy to enable easier testing by not defining the value at bootstrap time
 ES_INDICES_PREFIX = lazy(lambda: settings.RICHIE_ES_INDICES_PREFIX)()
 
+# Define which analyzer should be used for each language
+QUERY_ANALYZERS = getattr(
+    settings, "RICHIE_QUERY_ANALYZERS", {"en": "english", "fr": "french"}
+)
+
 # Define the scoring boost (in ElasticSearch) related value names receive when using
 # full-text search.
 # For example, when a user searches for "Science" in full-text, it should match any
diff --git a/src/richie/apps/search/forms.py b/src/richie/apps/search/forms.py
--- a/src/richie/apps/search/forms.py
+++ b/src/richie/apps/search/forms.py
@@ -6,10 +6,11 @@
 
 from django import forms
 from django.conf import settings
+from django.utils.translation import get_language
 
 import arrow
 
-from .defaults import RELATED_CONTENT_BOOST
+from .defaults import QUERY_ANALYZERS, RELATED_CONTENT_BOOST
 from .filter_definitions import FILTERS, AvailabilityFilterDefinition
 
 # Instantiate filter fields for each filter defined in settings
@@ -192,48 +193,26 @@ def get_queries(self):
         # Full text search is a regular (multilingual) match query
         full_text = self.cleaned_data.get("query")
         if full_text:
+            related_content_boost = getattr(
+                settings, "RICHIE_RELATED_CONTENT_BOOST", RELATED_CONTENT_BOOST
+            )
+            lang = get_language()
             queries.append(
                 {
                     "key": "query",
                     "fragment": [
                         {
-                            "bool": {
-                                "should": [
-                                    {
-                                        "multi_match": {
-                                            "fields": ["description.*", "title.*"],
-                                            "query": full_text,
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                    {
-                                        "multi_match": {
-                                            "boost": getattr(
-                                                settings,
-                                                "RICHIE_RELATED_CONTENT_BOOST",
-                                                RELATED_CONTENT_BOOST,
-                                            ),
-                                            "fields": [
-                                                "categories_names.*",
-                                                "organizations_names.*",
-                                            ],
-                                            "query": full_text,
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                    {
-                                        "multi_match": {
-                                            "boost": getattr(
-                                                settings,
-                                                "RICHIE_RELATED_CONTENT_BOOST",
-                                                RELATED_CONTENT_BOOST,
-                                            ),
-                                            "fields": ["persons_names.*"],
-                                            "query": full_text,
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                ]
+                            "multi_match": {
+                                "analyzer": QUERY_ANALYZERS[lang],
+                                "fields": [
+                                    "description.*",
+                                    "title.*",
+                                    f"categories_names.*^{related_content_boost}",
+                                    f"organizations_names.*^{related_content_boost}",
+                                    f"persons_names.*^{related_content_boost}",
+                                ],
+                                "query": full_text,
+                                "type": "cross_fields",
                             }
                         }
                     ],
@@ -334,7 +313,15 @@ def build_es_query(self, kind=None):
         # Add a match on the name field if it was handed by the client
         full_text = self.cleaned_data.get("query")
         if full_text:
-            clauses.append({"multi_match": {"query": full_text, "fields": ["title.*"]}})
+            clauses.append(
+                {
+                    "multi_match": {
+                        "analyzer": QUERY_ANALYZERS[get_language()],
+                        "fields": ["title.*"],
+                        "query": full_text,
+                    }
+                }
+            )
 
         # Build the query around the clauses if there are any
         if clauses:


Test patch:
diff --git a/tests/apps/search/test_forms_search_courses.py b/tests/apps/search/test_forms_search_courses.py
--- a/tests/apps/search/test_forms_search_courses.py
+++ b/tests/apps/search/test_forms_search_courses.py
@@ -116,9 +116,19 @@ def test_forms_courses_single_values_in_querystring(self, *_):
         form = CourseSearchForm(
             data=QueryDict(
                 query_string=(
-                    "availability=coming_soon&levels=1&levels_include=.*&languages=fr&limit=9&"
-                    "offset=3&organizations=10&organizations_include=.*&query=maths&new=new&"
-                    "scope=objects&subjects=1&subjects_include=.*"
+                    "availability=coming_soon"
+                    "&languages=fr"
+                    "&levels=1"
+                    "&levels_include=.*"
+                    "&limit=9"
+                    "&new=new"
+                    "&offset=3"
+                    "&organizations=10"
+                    "&organizations_include=.*"
+                    "&query=maths"
+                    "&scope=objects"
+                    "&subjects=1"
+                    "&subjects_include=.*"
                 )
             )
         )
@@ -152,10 +162,25 @@ def test_forms_courses_multi_values_in_querystring(self, *_):
         form = CourseSearchForm(
             data=QueryDict(
                 query_string=(
-                    "availability=coming_soon&availability=ongoing&levels=1&levels=2&"
-                    "languages=fr&languages=en&limit=9&limit=11&offset=3&offset=17&"
-                    "organizations=10&organizations=11&query=maths&query=physics&new=new&"
-                    "scope=objects&scope=filters&subjects=1&subjects=2"
+                    "availability=coming_soon"
+                    "&availability=ongoing"
+                    "&languages=fr"
+                    "&languages=en"
+                    "&levels=1"
+                    "&levels=2"
+                    "&limit=9"
+                    "&limit=11"
+                    "&new=new"
+                    "&offset=3"
+                    "&offset=17"
+                    "&organizations=10"
+                    "&organizations=11"
+                    "&query=maths"
+                    "&query=physics"
+                    "&scope=objects"
+                    "&scope=filters"
+                    "&subjects=1"
+                    "&subjects=2"
                 )
             )
         )
@@ -187,7 +212,7 @@ def test_forms_courses_build_es_query_search_by_match_text(self, *_):
         Happy path: build a query that filters courses by matching text
         """
         form = CourseSearchForm(
-            data=QueryDict(query_string="query=some%20phrase%20terms&limit=2&offset=20")
+            data=QueryDict(query_string="limit=2&offset=20&query=some%20phrase%20terms")
         )
         self.assertTrue(form.is_valid())
         self.assertEqual(
@@ -196,35 +221,17 @@ def test_forms_courses_build_es_query_search_by_match_text(self, *_):
                 "bool": {
                     "must": [
                         {
-                            "bool": {
-                                "should": [
-                                    {
-                                        "multi_match": {
-                                            "fields": ["description.*", "title.*"],
-                                            "query": "some phrase terms",
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                    {
-                                        "multi_match": {
-                                            "boost": 0.05,
-                                            "fields": [
-                                                "categories_names.*",
-                                                "organizations_names.*",
-                                            ],
-                                            "query": "some phrase terms",
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                    {
-                                        "multi_match": {
-                                            "boost": 0.05,
-                                            "fields": ["persons_names.*"],
-                                            "query": "some phrase " "terms",
-                                            "type": "cross_fields",
-                                        }
-                                    },
-                                ]
+                            "multi_match": {
+                                "analyzer": "english",
+                                "fields": [
+                                    "description.*",
+                                    "title.*",
+                                    "categories_names.*^0.05",
+                                    "organizations_names.*^0.05",
+                                    "persons_names.*^0.05",
+                                ],
+                                "query": "some phrase terms",
+                                "type": "cross_fields",
                             }
                         }
                     ]
diff --git a/tests/apps/search/test_forms_search_items.py b/tests/apps/search/test_forms_search_items.py
--- a/tests/apps/search/test_forms_search_items.py
+++ b/tests/apps/search/test_forms_search_items.py
@@ -93,7 +93,7 @@ def test_forms_items_query_between_3_and_100_characters_long(self, *_):
     def test_forms_items_single_values_in_querystring(self, *_):
         """
         The fields from filter definitions should be normalized as lists. The fields defined
-        on the form should be single values (limit, offset and query)
+        on the form should be single values (limit, offset and query).
         """
         form = ItemSearchForm(
             data=QueryDict(query_string=("limit=9&offset=3&query=maths"))
@@ -108,7 +108,7 @@ def test_forms_items_build_es_query_search_by_match_text(self, *_):
         Happy path: build a query that filters items by matching text
         """
         form = ItemSearchForm(
-            data=QueryDict(query_string="query=some%20phrase%20terms&limit=20&offset=2")
+            data=QueryDict(query_string="limit=20&offset=2&query=some%20phrase%20terms")
         )
         self.assertTrue(form.is_valid())
         self.assertEqual(
@@ -122,6 +122,7 @@ def test_forms_items_build_es_query_search_by_match_text(self, *_):
                             "must": [
                                 {
                                     "multi_match": {
+                                        "analyzer": "english",
                                         "fields": ["title.*"],
                                         "query": "some phrase " "terms",
                                     }
@@ -139,7 +140,7 @@ def test_forms_items_build_es_query_by_match_text_with_kind(self, *_):
         as argument.
         """
         form = ItemSearchForm(
-            data=QueryDict(query_string="query=some%20phrase%20terms&limit=20&offset=2")
+            data=QueryDict(query_string="limit=20&offset=2&query=some%20phrase%20terms")
         )
         self.assertTrue(form.is_valid())
         self.assertEqual(
@@ -154,6 +155,7 @@ def test_forms_items_build_es_query_by_match_text_with_kind(self, *_):
                                 {"term": {"kind": "subjects"}},
                                 {
                                     "multi_match": {
+                                        "analyzer": "english",
                                         "fields": ["title.*"],
                                         "query": "some phrase " "terms",
                                     }
diff --git a/tests/apps/search/test_viewsets_categories.py b/tests/apps/search/test_viewsets_categories.py
--- a/tests/apps/search/test_viewsets_categories.py
+++ b/tests/apps/search/test_viewsets_categories.py
@@ -148,8 +148,9 @@ def test_viewsets_categories_search(self, mock_search):
                             {"term": {"kind": "subjects"}},
                             {
                                 "multi_match": {
-                                    "query": "Science",
+                                    "analyzer": "english",
                                     "fields": ["title.*"],
+                                    "query": "Science",
                                 }
                             },
                         ]
diff --git a/tests/apps/search/test_viewsets_organizations.py b/tests/apps/search/test_viewsets_organizations.py
--- a/tests/apps/search/test_viewsets_organizations.py
+++ b/tests/apps/search/test_viewsets_organizations.py
@@ -53,10 +53,6 @@ def test_viewsets_organizations_retrieve_unknown(self):
         self.assertEqual(response.status_code, 404)
 
     @override_settings(RICHIE_ES_INDICES_PREFIX="richie")
-    @mock.patch(
-        "richie.apps.search.forms.ItemSearchForm.build_es_query",
-        lambda x: (2, 0, {"query": "something"}),
-    )
     @mock.patch.object(ES_CLIENT, "search")
     def test_viewsets_organizations_search(self, mock_search):
         """
@@ -101,7 +97,21 @@ def test_viewsets_organizations_search(self, mock_search):
         # The ES connector was called with a query that matches the client's request
         mock_search.assert_called_with(
             _source=["absolute_url", "logo", "title.*"],
-            body={"query": "something"},
+            body={
+                "query": {
+                    "bool": {
+                        "must": [
+                            {
+                                "multi_match": {
+                                    "analyzer": "english",
+                                    "fields": ["title.*"],
+                                    "query": "Université",
+                                }
+                            }
+                        ]
+                    }
+                }
+            },
             doc_type="organization",
             from_=0,
             index="richie_organizations",
diff --git a/tests/apps/search/test_viewsets_persons.py b/tests/apps/search/test_viewsets_persons.py
--- a/tests/apps/search/test_viewsets_persons.py
+++ b/tests/apps/search/test_viewsets_persons.py
@@ -53,10 +53,6 @@ def test_viewsets_persons_retrieve_unknown(self):
         self.assertEqual(response.status_code, 404)
 
     @override_settings(RICHIE_ES_INDICES_PREFIX="richie")
-    @mock.patch(
-        "richie.apps.search.forms.ItemSearchForm.build_es_query",
-        lambda x: (2, 0, {"query": "something"}),
-    )
     @mock.patch.object(ES_CLIENT, "search")
     def test_viewsets_persons_search(self, mock_search):
         """
@@ -109,7 +105,21 @@ def test_viewsets_persons_search(self, mock_search):
         # The ES connector was called with a query that matches the client's request
         mock_search.assert_called_with(
             _source=["absolute_url", "portrait", "title.*"],
-            body={"query": "something"},
+            body={
+                "query": {
+                    "bool": {
+                        "must": [
+                            {
+                                "multi_match": {
+                                    "analyzer": "english",
+                                    "fields": ["title.*"],
+                                    "query": "Michel",
+                                }
+                            }
+                        ]
+                    }
+                }
+            },
             doc_type="person",
             from_=0,
             index="richie_persons",