Repository: matrix-org/synapse

Issue:
Search user does not work with some specific Vietnamese letters
### Description

If you search for a user having some special letters with accents in its name (like "á") then the suggestions become empty as soon as you type the subsequent letter after the special character.

See also the attached video demonstrating the problem.

### Steps to reproduce

Please refer to the attached video.

### Homeserver

matrix.org

### Synapse Version

{"server_version":"1.66.0rc1 (b=matrix-org-hotfixes,ce8f7d118c)","python_version":"3.8.12"}

### Installation Method

_No response_

### Platform

app.element.io as webclient, matrix.org as homeserver.

https://user-images.githubusercontent.com/66737707/187195948-2c1f1aae-b72b-4686-a716-3df89cea020f.mp4



### Relevant log output

```shell
We have also reproduced the problem on our own debug server, the following log lines are the relevant part of the synapse server log:

2022-08-29 14:03:19,983 - synapse.storage.txn - 795 - DEBUG - expire_url_cache_data-763 - [TXN END] {get_url_cache_media_before-17c4} 0.001260 sec
2022-08-29 14:03:19,983 - synapse.rest.media.v1.preview_url_resource - 840 - DEBUG - expire_url_cache_data-763 - No media removed from url preview cache
2022-08-29 14:03:19,990 - synapse.storage.TIME - 602 - DEBUG - sentinel - Total database time: 0.092% {_prune_old_user_ips(2): 0.038%, _update_client_ips_batch(1): 0.031%, get_url_cache_media_before(1): 0.013%}
2022-08-29 14:03:21,645 - synapse.access.http.8008 - 405 - DEBUG - GET-2345 - ::ffff:127.0.0.1 - 8008 - Received request: GET /health
2022-08-29 14:03:21,646 - synapse.access.http.8008 - 450 - DEBUG - GET-2345 - ::ffff:127.0.0.1 - 8008 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 2B 200 "GET /health HTTP/1.1" "curl/7.74.0" [0 dbevts]
2022-08-29 14:03:23,330 - synapse.http.site - 533 - WARNING - sentinel - forwarded request lacks an x-forwarded-proto header: assuming https
2022-08-29 14:03:23,330 - synapse.access.http.8008 - 405 - DEBUG - OPTIONS-2346 - 185.150.4.97 - 8008 - Received request: OPTIONS /_matrix/client/r0/user_directory/search
2022-08-29 14:03:23,331 - synapse.access.http.8008 - 450 - DEBUG - OPTIONS-2346 - 185.150.4.97 - 8008 - {None} Processed request: 0.000sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 0B 204 "OPTIONS /_matrix/client/r0/user_directory/search HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" [0 dbevts]
2022-08-29 14:03:23,337 - synapse.http.site - 533 - WARNING - sentinel - forwarded request lacks an x-forwarded-proto header: assuming https
2022-08-29 14:03:23,337 - synapse.access.http.8008 - 405 - DEBUG - POST-2347 - 185.150.4.97 - 8008 - Received request: POST /_matrix/client/r0/user_directory/search
2022-08-29 14:03:23,338 - synapse.storage.txn - 691 - DEBUG - POST-2347 - [TXN START] {search_user_dir-17c5}
2022-08-29 14:03:23,338 - synapse.storage.SQL - 409 - DEBUG - POST-2347 - [SQL] {search_user_dir-17c5} SELECT d.user_id AS user_id, display_name, avatar_url FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) WHERE user_id != ? AND vector @@ to_tsquery('simple', ?) ORDER BY (CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END) * (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END) * (CASE WHEN avatar_url IS NOT NULL THEN 1.2 ELSE 1.0 END) * ( 3 * ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) + ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) ) DESC, display_name IS NULL, avatar_url IS NULL LIMIT ?
2022-08-29 14:03:23,339 - synapse.storage.SQL - 417 - DEBUG - POST-2347 - [SQL values] {search_user_dir-17c5} ('@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com', '(Gi:* | Gi)', 'Gi', 'Gi:*', 11)
2022-08-29 14:03:23,341 - synapse.storage.SQL - 438 - DEBUG - POST-2347 - [SQL time] {search_user_dir-17c5} 0.002775 sec
2022-08-29 14:03:23,342 - synapse.storage.txn - 795 - DEBUG - POST-2347 - [TXN END] {search_user_dir-17c5} 0.003450 sec
2022-08-29 14:03:23,343 - synapse.access.http.8008 - 450 - INFO - POST-2347 - 185.150.4.97 - 8008 - {@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com} Processed request: 0.005sec/0.001sec (0.002sec, 0.000sec) (0.000sec/0.003sec/1) 155B 200 "POST /_matrix/client/r0/user_directory/search HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" [0 dbevts]
2022-08-29 14:03:23,414 - synapse.metrics._gc - 118 - DEBUG - sentinel - Collecting gc 0
2022-08-29 14:03:24,654 - synapse.storage.txn - 691 - DEBUG - prune_old_user_ips-1528 - [TXN START] {_prune_old_user_ips-17c6}
2022-08-29 14:03:24,654 - synapse.storage.SQL - 409 - DEBUG - prune_old_user_ips-1528 - [SQL] {_prune_old_user_ips-17c6} DELETE FROM user_ips WHERE last_seen <= ( SELECT COALESCE(MAX(last_seen), -1) FROM ( SELECT last_seen FROM user_ips WHERE last_seen <= ? ORDER BY last_seen ASC LIMIT 5000 ) AS u )
2022-08-29 14:03:24,655 - synapse.storage.SQL - 417 - DEBUG - prune_old_user_ips-1528 - [SQL values] {_prune_old_user_ips-17c6} (1659355404653,)
2022-08-29 14:03:24,655 - synapse.storage.SQL - 438 - DEBUG - prune_old_user_ips-1528 - [SQL time] {_prune_old_user_ips-17c6} 0.000770 sec
2022-08-29 14:03:24,656 - synapse.storage.txn - 795 - DEBUG - prune_old_user_ips-1528 - [TXN END] {_prune_old_user_ips-17c6} 0.001809 sec
2022-08-29 14:03:24,757 - synapse.http.site - 533 - WARNING - sentinel - forwarded request lacks an x-forwarded-proto header: assuming https
2022-08-29 14:03:24,758 - synapse.access.http.8008 - 405 - DEBUG - POST-2348 - 185.150.4.97 - 8008 - Received request: POST /_matrix/client/r0/user_directory/search
2022-08-29 14:03:24,759 - synapse.storage.txn - 691 - DEBUG - POST-2348 - [TXN START] {search_user_dir-17c7}
2022-08-29 14:03:24,759 - synapse.storage.SQL - 409 - DEBUG - POST-2348 - [SQL] {search_user_dir-17c7} SELECT d.user_id AS user_id, display_name, avatar_url FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) WHERE user_id != ? AND vector @@ to_tsquery('simple', ?) ORDER BY (CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END) * (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END) * (CASE WHEN avatar_url IS NOT NULL THEN 1.2 ELSE 1.0 END) * ( 3 * ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) + ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) ) DESC, display_name IS NULL, avatar_url IS NULL LIMIT ?
2022-08-29 14:03:24,759 - synapse.storage.SQL - 417 - DEBUG - POST-2348 - [SQL values] {search_user_dir-17c7} ('@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com', '(Gia:* | Gia)', 'Gia', 'Gia:*', 11)
2022-08-29 14:03:24,762 - synapse.storage.SQL - 438 - DEBUG - POST-2348 - [SQL time] {search_user_dir-17c7} 0.002633 sec
2022-08-29 14:03:24,762 - synapse.storage.txn - 795 - DEBUG - POST-2348 - [TXN END] {search_user_dir-17c7} 0.003215 sec
2022-08-29 14:03:24,763 - synapse.access.http.8008 - 450 - INFO - POST-2348 - 185.150.4.97 - 8008 - {@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com} Processed request: 0.005sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.003sec/1) 155B 200 "POST /_matrix/client/r0/user_directory/search HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" [0 dbevts]
2022-08-29 14:03:24,908 - synapse.handlers.typing - 100 - DEBUG - typing._handle_timeouts-1528 - Checking for typing timeouts
2022-08-29 14:03:24,908 - synapse.handlers.presence - 901 - DEBUG - handle_presence_timeouts-1522 - Handling presence timeouts
2022-08-29 14:03:24,908 - synapse.util.metrics - 163 - DEBUG - handle_presence_timeouts-1522 - Entering block presence_update_states
2022-08-29 14:03:24,909 - synapse.util.metrics - 176 - DEBUG - handle_presence_timeouts-1522 - Exiting block presence_update_states
2022-08-29 14:03:26,325 - synapse.http.site - 533 - WARNING - sentinel - forwarded request lacks an x-forwarded-proto header: assuming https
2022-08-29 14:03:26,326 - synapse.access.http.8008 - 405 - DEBUG - POST-2349 - 185.150.4.97 - 8008 - Received request: POST /_matrix/client/r0/user_directory/search
2022-08-29 14:03:26,327 - synapse.storage.txn - 691 - DEBUG - POST-2349 - [TXN START] {search_user_dir-17c8}
2022-08-29 14:03:26,327 - synapse.storage.SQL - 409 - DEBUG - POST-2349 - [SQL] {search_user_dir-17c8} SELECT d.user_id AS user_id, display_name, avatar_url FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) WHERE user_id != ? AND vector @@ to_tsquery('simple', ?) ORDER BY (CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END) * (CASE WHEN display_name IS NOT NULL THEN 1.2 ELSE 1.0 END) * (CASE WHEN avatar_url IS NOT NULL THEN 1.2 ELSE 1.0 END) * ( 3 * ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) + ts_rank_cd( '{0.1, 0.1, 0.9, 1.0}', vector, to_tsquery('simple', ?), 8 ) ) DESC, display_name IS NULL, avatar_url IS NULL LIMIT ?
2022-08-29 14:03:26,328 - synapse.storage.SQL - 417 - DEBUG - POST-2349 - [SQL values] {search_user_dir-17c8} ('@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com', '(Gia:* | Gia) & (o:* | o)', 'Gia & o', 'Gia:* & o:*', 11)
2022-08-29 14:03:26,329 - synapse.storage.SQL - 438 - DEBUG - POST-2349 - [SQL time] {search_user_dir-17c8} 0.001021 sec
2022-08-29 14:03:26,329 - synapse.storage.txn - 795 - DEBUG - POST-2349 - [TXN END] {search_user_dir-17c8} 0.001580 sec
2022-08-29 14:03:26,330 - synapse.access.http.8008 - 450 - INFO - POST-2349 - 185.150.4.97 - 8008 - {@2_cb874bcb1f1b5219:anconnect-server-dev107.aarenet.com} Processed request: 0.004sec/0.001sec (0.002sec, 0.000sec) (0.001sec/0.002sec/1) 30B 200 "POST /_matrix/client/r0/user_directory/search HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" [0 dbevts]
2022-08-29 14:03:28,370 - synapse.storage.txn - 691 - DEBUG - _get_stats_for_federation_staging-254 - [TXN START] {_get_stats_for_federation_staging-17c9}
2022-08-29 14:03:28,370 - synapse.storage.SQL - 409 - DEBUG - _get_stats_for_federation_staging-254 - [SQL] {_get_stats_for_federation_staging-17c9} SELECT count(*) FROM federation_inbound_events_staging
2022-08-29 14:03:28,371 - synapse.storage.SQL - 438 - DEBUG - _get_stats_for_federation_staging-254 - [SQL time] {_get_stats_for_federation_staging-17c9} 0.000456 sec
```


### Anything else that would be useful to know?

_No response_


Discussion:
~~Possibly related to #3116?~~ Ah no, this is user directory search.

Can you please confirm what database you're using? If postgres, what is the synapse database's locale and encoding? (`SHOW lc_ctype; SHOW lc_collate; SHOW server_encoding;`)
Hi, 

We have reported the problem for matrix.org, because the behaviour there is the same.  But originally encountered it on our own server which has a postgres DB.
The results on our server for your queries are as follows:

```
postgres=# SHOW lc_ctype;
 lc_ctype
----------
 C
(1 row)

postgres=# SHOW lc_collate;
 lc_collate
------------
 C
(1 row)

postgres=# SHOW server_encoding;
 server_encoding
-----------------
 UTF8
(1 row)
```


It's worth noting that the `á` is a lowercase `a` followed by a U+0301 COMBINING ACUTE ACCENT.

`_parse_query_postgres` notably tries to split the search term into words but gets it wrong:
https://github.com/matrix-org/synapse/blob/888a29f4127723a8d048ce47cff37ee8a7a6f1b9/synapse/storage/databases/main/user_directory.py#L910-L924

```py
>>> from synapse.storage.databases.main.user_directory import _parse_query_postgres
>>> _parse_query_postgres("Gá")
('(Ga:* | Ga)', 'Ga', 'Ga:*')
>>> _parse_query_postgres("Gáo")
('(Ga:* | Ga) & (o:* | o)', 'Ga & o', 'Ga:* & o:*')
```

Line 918 probably needs to accept `\p{Mark}` code points, except that syntax isn't supported by the default `re` library in Python. The [`regex`](https://pypi.org/project/regex/) package does, or `re` could be given code point ranges to match.

There are probably other places in the code where `\w` is used with the same intent too.
We have a similar problem is with all Russian characters too - case-insensitive search does not work, here is the issue about this: https://github.com/matrix-org/synapse/issues/3116 so I guess the solution could be the same.
related:
* https://github.com/matrix-org/synapse/issues/1523
> Line 918 probably needs to accept `\p{Mark}` code points, except that syntax isn't supported by the default `re` library in Python. The [`regex`](https://pypi.org/project/regex/) package does, or `re` could be given code point ranges to match.
> 
> There are probably other places in the code where `\w` is used with the same intent too.

To expand on this: the quick and dirty proposal is to use something like `regex.findall(r"\b\w.*?\b", search_term, regex.WORD)` instead of `re.findall(r"([\w\-]+)", search_term, re.UNICODE)` to identify whole words.

This will fix exact matches not working, but will not resolve #1523, where `Gao` or `Gáo` (a with acute accent) will not match `Gáo` (a followed by combining acute accent). The latter may or may not already work depending on what postgres does.

---

Note that this still performs poorly. There are languages whose words consist of a variable number of `\w` code points that do not have spaces between them. A much better solution is to integrate libicu's word boundaries (https://unicode-org.github.io/icu/userguide/boundaryanalysis/#word-boundary), [which is what chromium supposedly uses](https://stackoverflow.com/questions/8492763/thai-line-breaking-how-to-break-thai-text-effectively/8950895#8950895). In the comparison below, it can be seen that only icu does something vaguely reasonable for Japanese.

<details>
<summary>Test code to compare re, regex and icu</summary>

```py
#!/usr/bin/env python3

import re
import regex
import icu

test_cases = [
    "It's a nice day outside.",
    "Received foo.png!",
    "Gáo",
    "C++20",
    "3.14159. 3.",
    "あなたはそれを行うべきではありません",
]

for text in test_cases:
    re1_output = re.findall(r"([\w\-]+)", text, re.UNICODE)
    re2_output = re.findall(r"\b\w.*?\b", text, re.UNICODE)
    regex_output = regex.findall(r"\b\w.*?\b", text, regex.WORD)
    icu_output = []
    breaker = icu.BreakIterator.createWordInstance(icu.Locale.getDefault())
    breaker.setText(text)
    i = 0
    while True:
        j = breaker.nextBoundary()
        if j < 0:
            break
        icu_output.append(text[i:j])
        i = j

    print(f"Text: {text!r}")
    print(f"    re.findall(r\"([\\w\\-]+)\"):    {re1_output!r}")
    print(f"    re.findall(r\"\\b\\w.*?\\b\"):    {re2_output!r}")
    print(f"    regex.findall(r\"\\b\\w.*?\\b\"): {regex_output!r}")
    print(f"    icu:                         {icu_output!r}")
```
</details>

```
Text: "It's a nice day outside."
    re.findall(r"([\w\-]+)"):    ['It', 's', 'a', 'nice', 'day', 'outside']
    re.findall(r"\b\w.*?\b"):    ['It', 's', 'a', 'nice', 'day', 'outside']
    regex.findall(r"\b\w.*?\b"): ["It's", 'a', 'nice', 'day', 'outside']
    icu:                         ["It's", ' ', 'a', ' ', 'nice', ' ', 'day', ' ', 'outside', '.']
Text: 'Received foo.png!'
    re.findall(r"([\w\-]+)"):    ['Received', 'foo', 'png']
    re.findall(r"\b\w.*?\b"):    ['Received', 'foo', 'png']
    regex.findall(r"\b\w.*?\b"): ['Received', 'foo.png']
    icu:                         ['Received', ' ', 'foo.png', '!']
Text: 'Gáo'
    re.findall(r"([\w\-]+)"):    ['Ga', 'o']
    re.findall(r"\b\w.*?\b"):    ['Ga', 'o']
    regex.findall(r"\b\w.*?\b"): ['Gáo']
    icu:                         ['Gáo']
Text: 'C++20'
    re.findall(r"([\w\-]+)"):    ['C', '20']
    re.findall(r"\b\w.*?\b"):    ['C', '20']
    regex.findall(r"\b\w.*?\b"): ['C', '20']
    icu:                         ['C', '+', '+', '20']
Text: '3.14159. 3.'
    re.findall(r"([\w\-]+)"):    ['3', '14159', '3']
    re.findall(r"\b\w.*?\b"):    ['3', '14159', '3']
    regex.findall(r"\b\w.*?\b"): ['3.14159', '3']
    icu:                         ['3.14159', '.', ' ', '3', '.']
Text: 'あなたはそれを行うべきではありません'
    re.findall(r"([\w\-]+)"):    ['あなたはそれを行うべきではありません']
    re.findall(r"\b\w.*?\b"):    ['あなたはそれを行うべきではありません']
    regex.findall(r"\b\w.*?\b"): ['あ', 'な', 'た', 'は', 'そ', 'れ', 'を', '行', 'う', 'べ', 'き', 'で', 'は', 'あ', 'り', 'ま', 'せ', 'ん']
    icu:                         ['あなた', 'は', 'それ', 'を', '行う', 'べ', 'き', 'では', 'ありま', 'せん']
```

This issue only concerns word boundaries, and not any sort of normalization, stemming or case/accent folding for searching. And we will have to ensure that the words in the postgres index are the same as the words we search for if/when we change the logic.

@reivilibre's view is that it would be best if we can find a way to have postgres or some library handle all this for us.

> @reivilibre's view is that it would be best if we can find a way to have postgres or some library handle all this for us.

Or even some external full-text search database. Lucene or something that uses it?
> > Line 918 probably needs to accept `\p{Mark}` code points, except that syntax isn't supported by the default `re` library in Python. The [`regex`](https://pypi.org/project/regex/) package does, or `re` could be given code point ranges to match.
> > There are probably other places in the code where `\w` is used with the same intent too.
> 
> To expand on this: the quick and dirty proposal is to use something like `regex.findall(r"\b\w.*?\b", search_term, regex.WORD)` instead of `re.findall(r"([\w\-]+)", search_term, re.UNICODE)` to identify whole words.
> 
> This will fix exact matches not working, but will not resolve #1523, where `Gao` or `Gáo` (a with acute accent) will not match `Gáo` (a followed by combining acute accent). The latter may or may not already work depending on what postgres does.
> 
> Note that this still performs poorly. There are languages whose words consist of a variable number of `\w` code points that do not have spaces between them. A much better solution is to integrate libicu's word boundaries (https://unicode-org.github.io/icu/userguide/boundaryanalysis/#word-boundary), [which is what chromium supposedly uses](https://stackoverflow.com/questions/8492763/thai-line-breaking-how-to-break-thai-text-effectively/8950895#8950895). In the comparison below, it can be seen that only icu does something vaguely reasonable for Japanese.
> 
> Test code to compare re, regex and icu
> ```
> Text: "It's a nice day outside."
>     re.findall(r"([\w\-]+)"):    ['It', 's', 'a', 'nice', 'day', 'outside']
>     re.findall(r"\b\w.*?\b"):    ['It', 's', 'a', 'nice', 'day', 'outside']
>     regex.findall(r"\b\w.*?\b"): ["It's", 'a', 'nice', 'day', 'outside']
>     icu:                         ["It's", ' ', 'a', ' ', 'nice', ' ', 'day', ' ', 'outside', '.']
> Text: 'Received foo.png!'
>     re.findall(r"([\w\-]+)"):    ['Received', 'foo', 'png']
>     re.findall(r"\b\w.*?\b"):    ['Received', 'foo', 'png']
>     regex.findall(r"\b\w.*?\b"): ['Received', 'foo.png']
>     icu:                         ['Received', ' ', 'foo.png', '!']
> Text: 'Gáo'
>     re.findall(r"([\w\-]+)"):    ['Ga', 'o']
>     re.findall(r"\b\w.*?\b"):    ['Ga', 'o']
>     regex.findall(r"\b\w.*?\b"): ['Gáo']
>     icu:                         ['Gáo']

We have tried this approach, but unfortunately it didn't help.
Hey, just wondering, what would the timeline be for integrating such a library or some external full-text search database?
Customer tried to use the fix proposed by @squahtx, but was unsuccessful.

Resolving patch:
diff --git a/synapse/storage/databases/main/user_directory.py b/synapse/storage/databases/main/user_directory.py
--- a/synapse/storage/databases/main/user_directory.py
+++ b/synapse/storage/databases/main/user_directory.py
@@ -26,6 +26,14 @@
     cast,
 )
 
+try:
+    # Figure out if ICU support is available for searching users.
+    import icu
+
+    USE_ICU = True
+except ModuleNotFoundError:
+    USE_ICU = False
+
 from typing_extensions import TypedDict
 
 from synapse.api.errors import StoreError
@@ -900,7 +908,7 @@ def _parse_query_sqlite(search_term: str) -> str:
     """
 
     # Pull out the individual words, discarding any non-word characters.
-    results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
+    results = _parse_words(search_term)
     return " & ".join("(%s* OR %s)" % (result, result) for result in results)
 
 
@@ -910,12 +918,63 @@ def _parse_query_postgres(search_term: str) -> Tuple[str, str, str]:
     We use this so that we can add prefix matching, which isn't something
     that is supported by default.
     """
-
-    # Pull out the individual words, discarding any non-word characters.
-    results = re.findall(r"([\w\-]+)", search_term, re.UNICODE)
+    results = _parse_words(search_term)
 
     both = " & ".join("(%s:* | %s)" % (result, result) for result in results)
     exact = " & ".join("%s" % (result,) for result in results)
     prefix = " & ".join("%s:*" % (result,) for result in results)
 
     return both, exact, prefix
+
+
+def _parse_words(search_term: str) -> List[str]:
+    """Split the provided search string into a list of its words.
+
+    If support for ICU (International Components for Unicode) is available, use it.
+    Otherwise, fall back to using a regex to detect word boundaries. This latter
+    solution works well enough for most latin-based languages, but doesn't work as well
+    with other languages.
+
+    Args:
+        search_term: The search string.
+
+    Returns:
+        A list of the words in the search string.
+    """
+    if USE_ICU:
+        return _parse_words_with_icu(search_term)
+
+    return re.findall(r"([\w\-]+)", search_term, re.UNICODE)
+
+
+def _parse_words_with_icu(search_term: str) -> List[str]:
+    """Break down the provided search string into its individual words using ICU
+    (International Components for Unicode).
+
+    Args:
+        search_term: The search string.
+
+    Returns:
+        A list of the words in the search string.
+    """
+    results = []
+    breaker = icu.BreakIterator.createWordInstance(icu.Locale.getDefault())
+    breaker.setText(search_term)
+    i = 0
+    while True:
+        j = breaker.nextBoundary()
+        if j < 0:
+            break
+
+        result = search_term[i:j]
+
+        # libicu considers spaces and punctuation between words as words, but we don't
+        # want to include those in results as they would result in syntax errors in SQL
+        # queries (e.g. "foo bar" would result in the search query including "foo &  &
+        # bar").
+        if len(re.findall(r"([\w\-]+)", result, re.UNICODE)):
+            results.append(result)
+
+        i = j
+
+    return results


Test patch:
diff --git a/tests/storage/test_user_directory.py b/tests/storage/test_user_directory.py
--- a/tests/storage/test_user_directory.py
+++ b/tests/storage/test_user_directory.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import re
 from typing import Any, Dict, Set, Tuple
 from unittest import mock
 from unittest.mock import Mock, patch
@@ -30,6 +31,12 @@
 from tests.test_utils.event_injection import inject_member_event
 from tests.unittest import HomeserverTestCase, override_config
 
+try:
+    import icu
+except ImportError:
+    icu = None  # type: ignore
+
+
 ALICE = "@alice:a"
 BOB = "@bob:b"
 BOBBY = "@bobby:a"
@@ -467,3 +474,39 @@ def test_search_user_dir_stop_words(self) -> None:
             r["results"][0],
             {"user_id": BELA, "display_name": "Bela", "avatar_url": None},
         )
+
+
+class UserDirectoryICUTestCase(HomeserverTestCase):
+    if not icu:
+        skip = "Requires PyICU"
+
+    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
+        self.store = hs.get_datastores().main
+        self.user_dir_helper = GetUserDirectoryTables(self.store)
+
+    def test_icu_word_boundary(self) -> None:
+        """Tests that we correctly detect word boundaries when ICU (International
+        Components for Unicode) support is available.
+        """
+
+        display_name = "Gáo"
+
+        # This word is not broken down correctly by Python's regular expressions,
+        # likely because á is actually a lowercase a followed by a U+0301 combining
+        # acute accent. This is specifically something that ICU support fixes.
+        matches = re.findall(r"([\w\-]+)", display_name, re.UNICODE)
+        self.assertEqual(len(matches), 2)
+
+        self.get_success(
+            self.store.update_profile_in_user_dir(ALICE, display_name, None)
+        )
+        self.get_success(self.store.add_users_in_public_rooms("!room:id", (ALICE,)))
+
+        # Check that searching for this user yields the correct result.
+        r = self.get_success(self.store.search_user_dir(BOB, display_name, 10))
+        self.assertFalse(r["limited"])
+        self.assertEqual(len(r["results"]), 1)
+        self.assertDictEqual(
+            r["results"][0],
+            {"user_id": ALICE, "display_name": display_name, "avatar_url": None},
+        )