Repository: searxng/searxng

Issue:
Search query got emphasized even in the middle of another, unrelated word
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG**

2023.01.09-afd71a6c

**How did you install SearXNG?**

Installed using docker with clone, docker build and run.

**What happened?**

Query letters got emphasized even in the middle of another unrelated word.

**How To Reproduce**

Do some search using alphabetic, English words with all language flag, such as `the :all`, `java :all`, `master :all`. 

**Expected behavior**

Emphasization should occur on queries found in standalone words only, such as `java` query only emphasize '**java**' instead of '**java**script', or `master` only in '**Master**' or '**master**' instead of 'grand**master**'.

**Screenshots & Logs**

| ![screenshot-localhost_8080-2023 01 10-21_55_30](https://user-images.githubusercontent.com/22837764/211661070-d639abf1-0a7d-4840-945e-ef11199c516c.png) |
| --- |
| In the word 'o**the**rwise' |
 
| ![screenshot-localhost_8080-2023 01 10-21_55_08](https://user-images.githubusercontent.com/22837764/211661185-4cbe4f57-4694-4d4d-b148-b05cf2fdacdd.png) |
| --- |
| In the word '**The**saurus' and '**the**ir' |

| ![screenshot-localhost_8080-2023 01 10-22_02_31](https://user-images.githubusercontent.com/22837764/211662283-32044b5b-447e-43d6-9428-4d28fc81e36e.png) |
| --- |
| In the word '**master**ful', '**master**s', 'grand**master**' |

**Additional context**

Likely happened because the regex being used does not isolate the query for occurrences in standalone words and instead it looks for all occurrences in the whole text without requiring the presence of spaces before or after it. This regex actually works well for the emphasization of queries in Chinese characters, for example:

| ![screenshot-localhost_8080-2023 01 10-22_09_43](https://user-images.githubusercontent.com/22837764/211663203-add6aabe-4fa9-428f-9bcc-0f646b0c5a88.png) |
| --- |
| Query used: ’村上春樹’ |




Discussion:

Resolving patch:
diff --git a/searx/webutils.py b/searx/webutils.py
--- a/searx/webutils.py
+++ b/searx/webutils.py
@@ -113,31 +113,68 @@ def prettify_url(url, max_length=74):
         return url
 
 
+def contains_cjko(s: str) -> bool:
+    """This function check whether or not a string contains Chinese, Japanese,
+    or Korean characters. It employs regex and uses the u escape sequence to
+    match any character in a set of Unicode ranges.
+
+    Args:
+        s (str): string to be checked.
+
+    Returns:
+        bool: True if the input s contains the characters and False otherwise.
+    """
+    unicode_ranges = (
+        '\u4e00-\u9fff'  # Chinese characters
+        '\u3040-\u309f'  # Japanese hiragana
+        '\u30a0-\u30ff'  # Japanese katakana
+        '\u4e00-\u9faf'  # Japanese kanji
+        '\uac00-\ud7af'  # Korean hangul syllables
+        '\u1100-\u11ff'  # Korean hangul jamo
+    )
+    return bool(re.search(fr'[{unicode_ranges}]', s))
+
+
+def regex_highlight_cjk(word: str) -> str:
+    """Generate the regex pattern to match for a given word according
+    to whether or not the word contains CJK characters or not.
+    If the word is and/or contains CJK character, the regex pattern
+    will match standalone word by taking into account the presence
+    of whitespace before and after it; if not, it will match any presence
+    of the word throughout the text, ignoring the whitespace.
+
+    Args:
+        word (str): the word to be matched with regex pattern.
+
+    Returns:
+        str: the regex pattern for the word.
+    """
+    rword = re.escape(word)
+    if contains_cjko(rword):
+        return fr'({rword})'
+    else:
+        return fr'\b({rword})(?!\w)'
+
+
 def highlight_content(content, query):
 
     if not content:
         return None
+
     # ignoring html contents
     # TODO better html content detection
     if content.find('<') != -1:
         return content
 
-    if content.lower().find(query.lower()) > -1:
-        query_regex = '({0})'.format(re.escape(query))
-        content = re.sub(query_regex, '<span class="highlight">\\1</span>', content, flags=re.I | re.U)
-    else:
-        regex_parts = []
-        for chunk in query.split():
-            chunk = chunk.replace('"', '')
-            if len(chunk) == 0:
-                continue
-            elif len(chunk) == 1:
-                regex_parts.append('\\W+{0}\\W+'.format(re.escape(chunk)))
-            else:
-                regex_parts.append('{0}'.format(re.escape(chunk)))
-        query_regex = '({0})'.format('|'.join(regex_parts))
-        content = re.sub(query_regex, '<span class="highlight">\\1</span>', content, flags=re.I | re.U)
-
+    querysplit = query.split()
+    queries = []
+    for qs in querysplit:
+        qs = qs.replace("'", "").replace('"', '').replace(" ", "")
+        if len(qs) > 0:
+            queries.extend(re.findall(regex_highlight_cjk(qs), content, flags=re.I | re.U))
+    if len(queries) > 0:
+        for q in set(queries):
+            content = re.sub(regex_highlight_cjk(q), f'<span class="highlight">{q}</span>', content)
     return content
 
 


Test patch:
diff --git a/tests/unit/test_webutils.py b/tests/unit/test_webutils.py
--- a/tests/unit/test_webutils.py
+++ b/tests/unit/test_webutils.py
@@ -28,32 +28,33 @@ def test_highlight_content(self):
 
         content = 'a'
         query = 'test'
-        self.assertEqual(webutils.highlight_content(content, query), content)
+        self.assertEqual(webutils.highlight_content(content, query), 'a')
         query = 'a test'
-        self.assertEqual(webutils.highlight_content(content, query), content)
+        self.assertEqual(webutils.highlight_content(content, query), '<span class="highlight">a</span>')
 
         data = (
             ('" test "', 'a test string', 'a <span class="highlight">test</span> string'),
-            ('"a"', 'this is a test string', 'this is<span class="highlight"> a </span>test string'),
+            ('"a"', 'this is a test string', 'this is <span class="highlight">a</span> test string'),
             (
                 'a test',
                 'this is a test string that matches entire query',
-                'this is <span class="highlight">a test</span> string that matches entire query',
+                'this is <span class="highlight">a</span> <span class="highlight">test</span> string that matches entire query',
             ),
             (
                 'this a test',
                 'this is a string to test.',
                 (
-                    '<span class="highlight">this</span> is<span class="highlight"> a </span>'
-                    'string to <span class="highlight">test</span>.'
+                    '<span class="highlight">this</span> is <span class="highlight">a</span> string to <span class="highlight">test</span>.'
                 ),
             ),
             (
                 'match this "exact phrase"',
                 'this string contains the exact phrase we want to match',
-                (
-                    '<span class="highlight">this</span> string contains the <span class="highlight">exact</span>'
-                    ' <span class="highlight">phrase</span> we want to <span class="highlight">match</span>'
+                ''.join(
+                    [
+                        '<span class="highlight">this</span> string contains the <span class="highlight">exact</span> ',
+                        '<span class="highlight">phrase</span> we want to <span class="highlight">match</span>',
+                    ]
                 ),
             ),
         )