Repository: zulip/zulip-flutter (language: dart)

Issue:
autocomplete: In @-mention results, match insensitively to diacritics
If there's a user with `Étienne` in their name, we should still include it in the results for the query `etienne`.

The matching is already case-insensitive, but it should be insensitive to diacritics too.
For emoji autocomplete, match insensitively to diacritics
For emoji autocomplete, if the emoji's name has `é` where the user's query has `e` or vice versa, we should treat that as a match. The semantics should match what the web app does.

This doesn't much arise for Unicode emoji, because for all the (small number of) Unicode emoji where a name has a diacritic, we also give the emoji an alternate name by removing the diacritics. It might come up for custom emoji in some realms.

### Related issues
* Similar but for @-mentions: #237
* Prerequisite: #670



Discussion hints:
(In principle this should also apply to user-group mentions, not just user mentions. User-group mention autocomplete hasn't been implemented yet: #233.)
Can we use something like [diacritic](https://github.com/agilord/diacritic)?
BTW I'm new here (and new to open source in general) so what is the policy when it comes to packages and dependencies?
It looks like that package consists of a hand-maintained set of mappings:
https://github.com/agilord/diacritic/blob/8cc02fb838f69bfffd25a7629787267814669b28/lib/src/replacement_map.dart#L59-L60
with no particular documentation of where the list comes from. So that isn't something I'd be eager to rely on.

The Zulip web client and the legacy zulip-mobile app have functionality like this. So let's implement the same behavior they have.

That behavior has been stable for a long time and I'm not aware of any bug reports we've gotten about it. If we later learn about ways the behavior can be improved, we can make the improvement for web too at the same time.

> The Zulip web client and the legacy zulip-mobile app have functionality like this. So let's implement the same behavior they have.

After taking a look back at the mobile app [implementation for this feature](https://github.com/zulip/zulip-mobile/pull/5292/files) I found that it depends on [@zulip/shared](https://www.npmjs.com/package/@zulip/shared) to handle this. check:
file: [zulip-mobile/src/users/userHelpers.js](https://github.com/Fingel/zulip-mobile/blob/d73e7644f9a2cf3b61b827f10c9e10a38d35c9ae/src/users/userHelpers.js)
```js
import * as typeahead from '@zulip/shared/js/typeahead';

const loweredFilter = filter.toLowerCase();
const isAscii = /^[a-z]+$/.test(loweredFilter);
return users.filter(user => {
  const full_name = isAscii ? typeahead.remove_diacritics(user.full_name) : user.full_name;
  return user.user_id !== ownUserId && full_name.toLowerCase().startsWith(loweredFilter);
});
```

Which uses the convenient Js String built in method `normalize`

file: [@zulip/shared/lib/typeahead.js](https://www.npmjs.com/package/@zulip/shared?activeTab=code)
```js
const unicode_marks = /\p{M}/gu;
export function remove_diacritics(s) {
    return s.normalize("NFKD").replace(unicode_marks, "");
}
```
Which unfortunately dart does not have *for now at least* so we might have to consider third party packages, I am still searching and evaluating the options but I found several candidates:
* [unorm-dart](https://pub.dev/packages/unorm_dart)
* [normalizer](https://pub.dev/packages/normalizer)

So I will do some research to be able make an educated suggestion
Updating this thread with a conclusion from [chat](https://chat.zulip.org/#narrow/channel/48-mobile/topic/deps.3A.20Add.20new.20package.20to.20handle.20diacritics/near/1770345) and the PR thread https://github.com/zulip/zulip-flutter/pull/588#issuecomment-2080022746 :

We'd use [`package:unorm_dart`](https://pub.dev/packages/unorm_dart) for this.

When making the search "fuzzy" like this, ideally we'd prioritize non-fuzzy matches over fuzzy matches. (zulip/zulip#32634 gave me the idea to comment here; we can split this to a new issue if helpful.)
This came up in early post-launch feedback today: #1623
@chrisbobbe just pinging this issue. We're a paying customer and this is disrespectful to some of our colleagues. Please don't mess with peoples names.

I also find it disrespectful to be asked to upvote to prioritise - this is a very basic feature. "Zulip is not able to mention people by name proper" is a clear defect, not a future feature.
As part of the same work, let's also handle the possibility that the emoji's name has an upper-case letter where the query has lower case.

This one doesn't come up at all for Unicode emoji: all our names are in lower case. But again it might come up for custom emoji in some realms.

When making the search "fuzzy" like this, ideally we'd prioritize non-fuzzy matches over fuzzy matches. (zulip/zulip#32634 gave me the idea to comment here; we can split this to a new issue if helpful.)
Hello @gnprice ,
This would be solution for the issue right?

https://github.com/user-attachments/assets/28b02c43-ba54-48a8-b40a-40a0d13e94c1
From the snippet in that demo, that looks like broadly the right sort of behavior. To tell whether it's an accurate solution would require more detail, like an implementation and its tests.

This is a post-launch issue, though — in fact it's in the M7 Future milestone, beyond M6 Post-launch. So we won't be able to spend time giving feedback on PRs for it until after the app has been launched.

To get clear idea of implementation I had created a PR.

Pull request: Match insensitively to diacritics (autocomplete and new-DMs)
Fixes #237.
Fixes #1067.

Test command: flutter test test/model/autocomplete_test.dart test/model/emoji_test.dart test/widgets/new_dm_sheet_test.dart --reporter expanded 2>&1

Fix patch:
diff --git a/lib/model/autocomplete.dart b/lib/model/autocomplete.dart
index 0b17f50b27..cf5c6b86e0 100644
--- a/lib/model/autocomplete.dart
+++ b/lib/model/autocomplete.dart
@@ -2,6 +2,7 @@ import 'dart:math';
 
 import 'package:flutter/foundation.dart';
 import 'package:flutter/services.dart';
+import 'package:unorm_dart/unorm_dart.dart' as unorm;
 
 import '../api/model/events.dart';
 import '../api/model/model.dart';
@@ -643,9 +644,9 @@ class MentionAutocompleteView extends AutocompleteView<MentionAutocompleteQuery,
   static int compareGroupsByAlphabeticalOrder(UserGroup userGroupA, UserGroup userGroupB,
       {required PerAccountStore store}) {
     final groupAName = store.autocompleteViewManager.autocompleteDataCache
-      .lowercaseNameForUserGroup(userGroupA);
+      .normalizedNameForUserGroup(userGroupA);
     final groupBName = store.autocompleteViewManager.autocompleteDataCache
-      .lowercaseNameForUserGroup(userGroupB);
+      .normalizedNameForUserGroup(userGroupB);
     return groupAName.compareTo(groupBName); // TODO(i18n): add locale-aware sorting
   }
 
@@ -732,42 +733,53 @@ class MentionAutocompleteView extends AutocompleteView<MentionAutocompleteQuery,
 /// An [AutocompleteQuery] object stores the user's actual query string
 /// as [raw].
 /// It may also store processed forms of the query
-/// (for example, converted to lowercase or split on whitespace)
+/// (for example, normalized by case and diacritics, or split on whitespace)
 /// to prepare for whatever particular form of searching will be done
 /// for the given type of autocomplete interaction.
 abstract class AutocompleteQuery {
   AutocompleteQuery(this.raw) {
-    _lowercase = raw.toLowerCase();
-    _lowercaseWords = _lowercase.split(' ');
+    _normalized = lowercaseAndStripDiacritics(raw);
+    // TODO(#1805) split on space characters that the user is actually using
+    //   (e.g. U+3000 IDEOGRAPHIC SPACE);
+    //   could check active keyboard or just split on all kinds of spaces
+    _normalizedWords = _normalized.split(' ');
   }
 
   /// The actual string the user entered.
   final String raw;
 
-  late final String _lowercase;
+  late final String _normalized;
 
-  late final List<String> _lowercaseWords;
+  late final List<String> _normalizedWords;
+
+  static final RegExp _regExpStripMarkCharacters = RegExp(r'\p{M}', unicode: true);
+
+  static String lowercaseAndStripDiacritics(String input) {
+    // Anders reports that this is what web does; see discussion:
+    //   https://chat.zulip.org/#narrow/channel/48-mobile/topic/deps.3A.20Add.20new.20package.20to.20handle.20diacritics/near/2244487
+    final lowercase = input.toLowerCase();
+    final compatibilityNormalized = unorm.nfkd(lowercase);
+    return compatibilityNormalized.replaceAll(_regExpStripMarkCharacters, '');
+  }
 
   /// Whether all of this query's words have matches in [words],
-  /// modulo case, that appear in order.
+  /// insensitively to case and diacritics, that appear in order.
   ///
   /// A "match" means the word in [words] starts with the query word.
   ///
-  /// [words] must all be lowercased.
+  /// [words] must all have been passed through [lowercaseAndStripDiacritics].
   bool _testContainsQueryWords(List<String> words) {
-    // TODO(#237) test with diacritics stripped, where appropriate,
-    //   and update dartdoc's summary line and its restriction about [words].
     int wordsIndex = 0;
     int queryWordsIndex = 0;
     while (true) {
-      if (queryWordsIndex == _lowercaseWords.length) {
+      if (queryWordsIndex == _normalizedWords.length) {
         return true;
       }
       if (wordsIndex == words.length) {
         return false;
       }
 
-      if (words[wordsIndex].startsWith(_lowercaseWords[queryWordsIndex])) {
+      if (words[wordsIndex].startsWith(_normalizedWords[queryWordsIndex])) {
         queryWordsIndex++;
       }
       wordsIndex++;
@@ -825,9 +837,9 @@ class MentionAutocompleteQuery extends ComposeAutocompleteQuery {
 
   WildcardMentionAutocompleteResult? testWildcardOption(WildcardMentionOption wildcardOption, {
       required ZulipLocalizations localizations}) {
-    // TODO(#237): match insensitively to diacritics
-    final matches = wildcardOption.canonicalString.contains(_lowercase)
-      || wildcardOption.localizedCanonicalString(localizations).contains(_lowercase);
+    final localized = wildcardOption.localizedCanonicalString(localizations);
+    final matches = wildcardOption.canonicalString.contains(_normalized)
+      || AutocompleteQuery.lowercaseAndStripDiacritics(localized).contains(_normalized);
     if (!matches) return null;
     return WildcardMentionAutocompleteResult(
       wildcardOption: wildcardOption, rank: _rankWildcardResult);
@@ -857,8 +869,8 @@ class MentionAutocompleteQuery extends ComposeAutocompleteQuery {
     required String normalizedName,
     required List<String> normalizedNameWords,
   }) {
-    if (normalizedName.startsWith(_lowercase)) {
-      if (normalizedName.length == _lowercase.length) {
+    if (normalizedName.startsWith(_normalized)) {
+      if (normalizedName.length == _normalized.length) {
         return NameMatchQuality.exact;
       } else {
         return NameMatchQuality.totalPrefix;
@@ -873,17 +885,17 @@ class MentionAutocompleteQuery extends ComposeAutocompleteQuery {
   }
 
   bool _matchEmail(User user, AutocompleteDataCache cache) {
-    final lowercaseEmail = cache.normalizedEmailForUser(user);
-    if (lowercaseEmail == null) return false; // Email not known
-    return lowercaseEmail.startsWith(_lowercase);
+    final normalizedEmail = cache.normalizedEmailForUser(user);
+    if (normalizedEmail == null) return false; // Email not known
+    return normalizedEmail.startsWith(_normalized);
   }
 
   MentionAutocompleteResult? testUserGroup(UserGroup userGroup, PerAccountStore store) {
     final cache = store.autocompleteViewManager.autocompleteDataCache;
 
     final nameMatchQuality = _matchName(
-      normalizedName: cache.lowercaseNameForUserGroup(userGroup),
-      normalizedNameWords: cache.lowercaseNameWordsForUserGroup(userGroup));
+      normalizedName: cache.normalizedNameForUserGroup(userGroup),
+      normalizedNameWords: cache.normalizedNameWordsForUserGroup(userGroup));
 
     if (nameMatchQuality == null) return null;
 
@@ -970,12 +982,16 @@ extension WildcardMentionOptionExtension on WildcardMentionOption {
 /// but kept around in between autocomplete interactions.
 ///
 /// An instance of this class is managed by [AutocompleteViewManager].
+// TODO(#1805) when splitting words, split on space characters that are likely
+//   to be used (e.g. U+3000 IDEOGRAPHIC SPACE);
+//   could check server language or just split on all kinds of spaces
 class AutocompleteDataCache {
   final Map<int, String> _normalizedNamesByUser = {};
 
   /// The normalized `fullName` of [user].
   String normalizedNameForUser(User user) {
-    return _normalizedNamesByUser[user.userId] ??= user.fullName.toLowerCase();
+    return _normalizedNamesByUser[user.userId]
+      ??= AutocompleteQuery.lowercaseAndStripDiacritics(user.fullName);
   }
 
   final Map<int, List<String>> _normalizedNameWordsByUser = {};
@@ -989,21 +1005,25 @@ class AutocompleteDataCache {
 
   /// The normalized `deliveryEmail` of [user], or null if that's null.
   String? normalizedEmailForUser(User user) {
-    return _normalizedEmailsByUser[user.userId] ??= user.deliveryEmail?.toLowerCase();
+    return _normalizedEmailsByUser[user.userId]
+      ??= (user.deliveryEmail != null
+            ? AutocompleteQuery.lowercaseAndStripDiacritics(user.deliveryEmail!)
+            : null);
   }
 
-  final Map<int, String> _lowercaseNamesByUserGroup = {};
+  final Map<int, String> _normalizedNamesByUserGroup = {};
 
   /// The normalized `name` of [userGroup].
-  String lowercaseNameForUserGroup(UserGroup userGroup) {
-    return _lowercaseNamesByUserGroup[userGroup.id] ??= userGroup.name.toLowerCase();
+  String normalizedNameForUserGroup(UserGroup userGroup) {
+    return _normalizedNamesByUserGroup[userGroup.id]
+      ??= AutocompleteQuery.lowercaseAndStripDiacritics(userGroup.name);
   }
 
-  final Map<int, List<String>> _lowercaseNameWordsByUserGroup = {};
+  final Map<int, List<String>> _normalizedNameWordsByUserGroup = {};
 
-  List<String> lowercaseNameWordsForUserGroup(UserGroup userGroup) {
-    return _lowercaseNameWordsByUserGroup[userGroup.id]
-      ??= lowercaseNameForUserGroup(userGroup).split(' ');
+  List<String> normalizedNameWordsForUserGroup(UserGroup userGroup) {
+    return _normalizedNameWordsByUserGroup[userGroup.id]
+      ??= normalizedNameForUserGroup(userGroup).split(' ');
   }
 
   void invalidateUser(int userId) {
@@ -1013,8 +1033,8 @@ class AutocompleteDataCache {
   }
 
   void invalidateUserGroup(int id) {
-    _lowercaseNamesByUserGroup.remove(id);
-    _lowercaseNameWordsByUserGroup.remove(id);
+    _normalizedNamesByUserGroup.remove(id);
+    _normalizedNameWordsByUserGroup.remove(id);
   }
 }
 
@@ -1196,11 +1216,11 @@ class TopicAutocompleteQuery extends AutocompleteQuery {
     // TODO(#881): Sort by match relevance, like web does.
 
     if (topic.displayName == null) {
-      return store.realmEmptyTopicDisplayName.toLowerCase()
-        .contains(raw.toLowerCase());
+      return AutocompleteQuery.lowercaseAndStripDiacritics(store.realmEmptyTopicDisplayName)
+        .contains(_normalized);
     }
     return topic.displayName != raw
-      && topic.displayName!.toLowerCase().contains(raw.toLowerCase());
+      && AutocompleteQuery.lowercaseAndStripDiacritics(topic.displayName!).contains(_normalized);
   }
 
   @override
diff --git a/lib/model/emoji.dart b/lib/model/emoji.dart
index 12424104bf..8d6a1cab0e 100644
--- a/lib/model/emoji.dart
+++ b/lib/model/emoji.dart
@@ -74,11 +74,23 @@ final class EmojiCandidate {
   /// This might not be the only name this emoji has; see [aliases].
   final String emojiName;
 
+  /// [emojiName], but via [AutocompleteQuery.lowercaseAndStripDiacritics]
+  /// to support fuzzy matching.
+  String get normalizedEmojiName => _normalizedEmojiName
+    ??= AutocompleteQuery.lowercaseAndStripDiacritics(emojiName);
+  String? _normalizedEmojiName;
+
   /// Additional Zulip "emoji name" values for this emoji,
   /// to show in the emoji picker UI.
   Iterable<String> get aliases => _aliases ?? const [];
   final List<String>? _aliases;
 
+  /// [aliases], but via [AutocompleteQuery.lowercaseAndStripDiacritics]
+  /// to support fuzzy matching.
+  Iterable<String> get normalizedAliases => _normalizedAliases
+    ??= aliases.map((alias) => AutocompleteQuery.lowercaseAndStripDiacritics(alias));
+  Iterable<String>? _normalizedAliases;
+
   final EmojiDisplay emojiDisplay;
 
   EmojiCandidate({
@@ -501,9 +513,8 @@ class EmojiAutocompleteQuery extends ComposeAutocompleteQuery {
 
   static const _separator = '_';
 
-  static String _adjustQuery(String raw) {
-    return raw.toLowerCase().replaceAll(' ', '_'); // TODO(#1067) remove diacritics too
-  }
+  static String _adjustQuery(String raw) =>
+    AutocompleteQuery.lowercaseAndStripDiacritics(raw.replaceAll(' ', '_'));
 
   @override
   EmojiAutocompleteView initViewModel({
@@ -533,29 +544,28 @@ class EmojiAutocompleteQuery extends ComposeAutocompleteQuery {
       }
     }
 
-    EmojiMatchQuality? result = _matchName(candidate.emojiName);
-    for (final alias in candidate.aliases) {
+    EmojiMatchQuality? result = _matchName(candidate.normalizedEmojiName);
+    for (final normalizedAlias in candidate.normalizedAliases) {
       if (result == EmojiMatchQuality.best) return result;
-      result = EmojiMatchQuality.bestOf(result, _matchName(alias));
+      result = EmojiMatchQuality.bestOf(result, _matchName(normalizedAlias));
     }
     return result;
   }
 
-  EmojiMatchQuality? _matchName(String emojiName) {
+  EmojiMatchQuality? _matchName(String normalizedName) {
     // Compare query_matches_string_in_order in Zulip web:shared/src/typeahead.ts
     // for a Boolean version of this logic (match vs. no match),
     // and triage_raw in the same file web:shared/src/typeahead.ts
     // for the finer distinctions.
     // See also commentary in [_rankResult].
 
-    // TODO(#1067) this assumes emojiName is already lower-case (and no diacritics)
-    if (emojiName == _adjusted)           return EmojiMatchQuality.exact;
-    if (emojiName.startsWith(_adjusted))  return EmojiMatchQuality.prefix;
-    if (emojiName.contains(_sepAdjusted)) return EmojiMatchQuality.wordAligned;
+    if (normalizedName == _adjusted)           return EmojiMatchQuality.exact;
+    if (normalizedName.startsWith(_adjusted))  return EmojiMatchQuality.prefix;
+    if (normalizedName.contains(_sepAdjusted)) return EmojiMatchQuality.wordAligned;
     if (!_adjusted.contains(_separator)) {
       // If the query is a single token (doesn't contain a separator),
       // allow a match anywhere in the string, too.
-      if (emojiName.contains(_adjusted))  return EmojiMatchQuality.other;
+      if (normalizedName.contains(_adjusted))  return EmojiMatchQuality.other;
     } else {
       // Otherwise, require at least a word-aligned match.
     }
diff --git a/lib/widgets/autocomplete.dart b/lib/widgets/autocomplete.dart
index 5c853d2da0..e345969189 100644
--- a/lib/widgets/autocomplete.dart
+++ b/lib/widgets/autocomplete.dart
@@ -208,7 +208,7 @@ class ComposeAutocomplete extends AutocompleteField<ComposeAutocompleteQuery, Co
           // and losing data for the user.
           return;
         }
-        // TODO(i18n) language-appropriate space character; check active keyboard?
+        // TODO(#1805) language-appropriate space character; check active keyboard?
         //   (maybe handle centrally in `controller`)
         replacementString = '${userMention(user, silent: query.silent, users: store)} ';
       case WildcardMentionAutocompleteResult(:var wildcardOption):
@@ -223,7 +223,7 @@ class ComposeAutocomplete extends AutocompleteField<ComposeAutocompleteQuery, Co
           // and losing data for the group.
           return;
         }
-        // TODO(i18n) language-appropriate space character; check active keyboard?
+        // TODO(#1805) language-appropriate space character; check active keyboard?
         //   (maybe handle centrally in `controller`)
         replacementString = '${userGroupMention(userGroup.name, silent: query.silent)} ';
     }
diff --git a/lib/widgets/new_dm_sheet.dart b/lib/widgets/new_dm_sheet.dart
index 05e682e849..0ab0dc41b5 100644
--- a/lib/widgets/new_dm_sheet.dart
+++ b/lib/widgets/new_dm_sheet.dart
@@ -87,12 +87,14 @@ class _NewDmPickerState extends State<NewDmPicker> with PerAccountStoreAwareStat
   void _updateFilteredUsers(PerAccountStore store) {
     final excludeSelfUser = selectedUserIds.isNotEmpty
       && !selectedUserIds.contains(store.selfUserId);
-    final searchTextLower = searchController.text.toLowerCase();
+    final normalizedQuery =
+      AutocompleteQuery.lowercaseAndStripDiacritics(searchController.text);
 
     final result = <User>[];
     for (final user in sortedUsers) {
       if (excludeSelfUser && user.userId == store.selfUserId) continue;
-      if (user.fullName.toLowerCase().contains(searchTextLower)) {
+      final normalizedName = AutocompleteQuery.lowercaseAndStripDiacritics(user.fullName);
+      if (normalizedName.contains(normalizedQuery)) {
         result.add(user);
       }
     }
diff --git a/pubspec.lock b/pubspec.lock
index c52f5e6c1e..bc3cb33f00 100644
--- a/pubspec.lock
+++ b/pubspec.lock
@@ -1115,6 +1115,14 @@ packages:
       url: "https://pub.dev"
     source: hosted
     version: "1.4.0"
+  unorm_dart:
+    dependency: "direct main"
+    description:
+      name: unorm_dart
+      sha256: "8e3870a1caa60bde8352f9597dd3535d8068613269444f8e35ea8925ec84c1f5"
+      url: "https://pub.dev"
+    source: hosted
+    version: "0.3.1+1"
   url_launcher:
     dependency: "direct main"
     description:
diff --git a/pubspec.yaml b/pubspec.yaml
index 9b1c3566c5..64b5a6b111 100644
--- a/pubspec.yaml
+++ b/pubspec.yaml
@@ -65,6 +65,7 @@ dependencies:
   wakelock_plus: ^1.2.8
   zulip_plugin:
     path: ./packages/zulip_plugin
+  unorm_dart: ^0.3.1+1
   # Keep list sorted when adding dependencies; it helps prevent merge conflicts.
 
 dependency_overrides:


Test patch:
diff --git a/test/model/autocomplete_test.dart b/test/model/autocomplete_test.dart
index 20f8adb6be..a560512553 100644
--- a/test/model/autocomplete_test.dart
+++ b/test/model/autocomplete_test.dart
@@ -27,9 +27,6 @@ import 'autocomplete_checks.dart';
 typedef MarkedTextParse = ({int? expectedSyntaxStart, TextEditingValue value});
 
 final zulipLocalizations = GlobalLocalizations.zulipLocalizations;
-final zulipLocalizationsArabic =
-  lookupZulipLocalizations(ZulipLocalizations.supportedLocales
-    .firstWhere((locale) => locale.languageCode == 'ar'));
 
 void main() {
   ({int? expectedSyntaxStart, TextEditingValue value}) parseMarkedText(String markedText) {
@@ -922,26 +919,49 @@ void main() {
       });
     }
 
-    final localizedTestCases = [
-      ('ال',        channelNarrow, [WildcardMentionOption.all, WildcardMentionOption.topic]),
-      ('الجميع',    topicNarrow,   [WildcardMentionOption.all]),
-      ('الموضوع',   channelNarrow, [WildcardMentionOption.topic]),
-      ('ق',         topicNarrow,   [WildcardMentionOption.channel]),
-      ('دفق',       channelNarrow, [WildcardMentionOption.stream]),
-      ('الكل',      dmNarrow,      [WildcardMentionOption.everyone]),
+    WildcardTester wildcardTesterForLocale(bool Function(Locale) localePredicate) {
+      final locale = ZulipLocalizations.supportedLocales.firstWhere(localePredicate);
+      final localizations = lookupZulipLocalizations(locale);
 
-      ('top',       channelNarrow, [WildcardMentionOption.topic]),
-      ('channel',   topicNarrow,   [WildcardMentionOption.channel]),
-      ('every',     dmNarrow,      [WildcardMentionOption.everyone]),
-    ];
+      return (String query, Narrow narrow, List<WildcardMentionOption> expected) {
+        test('locale "$locale" -> query "$query" in ${narrow.runtimeType} -> $expected', () {
+          check(getWildcardOptionsFor(query, narrow: narrow,
+            localizations: localizations)).deepEquals(expected);
+        });
+      };
+    }
 
-    for (final (String localizedQuery, Narrow narrow, List<WildcardMentionOption> wildcardOptions) in localizedTestCases) {
-      test('different locale -> query "$localizedQuery" in ${narrow.runtimeType} -> $wildcardOptions', () async {
-        check(getWildcardOptionsFor(localizedQuery, narrow: narrow,
-          localizations: zulipLocalizationsArabic)).deepEquals(wildcardOptions);
-      });
+    for (final option in WildcardMentionOption.values) {
+      // These are hard-coded, and they happened to be lowercase and without
+      // diacritics when written.
+      // Throw if that changes, to not accidentally break fuzzy matching.
+      check(option.canonicalString).equals(
+        AutocompleteQuery.lowercaseAndStripDiacritics(option.canonicalString));
     }
 
+    final testArabic = wildcardTesterForLocale((locale) => locale.languageCode == 'ar');
+    testArabic('ال',        channelNarrow, [WildcardMentionOption.all, WildcardMentionOption.topic]);
+    testArabic('الجميع',    topicNarrow,   [WildcardMentionOption.all]);
+    testArabic('الموضوع',   channelNarrow, [WildcardMentionOption.topic]);
+    testArabic('ق',         topicNarrow,   [WildcardMentionOption.channel]);
+    testArabic('دفق',       channelNarrow, [WildcardMentionOption.stream]);
+    testArabic('الكل',      dmNarrow,      [WildcardMentionOption.everyone]);
+    testArabic('top',       channelNarrow, [WildcardMentionOption.topic]);
+    testArabic('channel',   topicNarrow,   [WildcardMentionOption.channel]);
+    testArabic('every',     dmNarrow,      [WildcardMentionOption.everyone]);
+
+    final testEnglish = wildcardTesterForLocale((locale) => locale.languageCode == 'en');
+    testEnglish('topic',     topicNarrow,   [WildcardMentionOption.topic]);
+    testEnglish('Topic',     topicNarrow,   [WildcardMentionOption.topic]);
+
+    final testGerman = wildcardTesterForLocale((locale) => locale.languageCode == 'de');
+    testGerman('Thema',     topicNarrow,   [WildcardMentionOption.topic]);
+    testGerman('thema',     topicNarrow,   [WildcardMentionOption.topic]);
+
+    final testPolish = wildcardTesterForLocale((locale) => locale.languageCode == 'pl');
+    testPolish('wątek',     topicNarrow,   [WildcardMentionOption.topic]);
+    testPolish('watek',     topicNarrow,   [WildcardMentionOption.topic]);
+
     test('no wildcards for a silent mention', () {
       check(getWildcardOptionsFor('', isSilent: true, narrow: channelNarrow))
         .isEmpty();
@@ -1061,6 +1081,14 @@ void main() {
       check(rankOf(query, a)!).equals(rankOf(query, b)!);
     }
 
+    void checkAllSameRank(String query, Iterable<Object> candidates) {
+      // (i.e. throw here if it's not a match)
+      final firstCandidateRank = rankOf(query, candidates.first)!;
+
+      final ranks = candidates.skip(1).map((candidate) => rankOf(query, candidate));
+      check(ranks).every((it) => it.equals(firstCandidateRank));
+    }
+
     test('wildcards, then users', () {
       checkSameRank('', WildcardMentionOption.all, WildcardMentionOption.topic);
       checkPrecedes('', WildcardMentionOption.topic, eg.user());
@@ -1073,13 +1101,28 @@ void main() {
       checkPrecedes(user.fullName, WildcardMentionOption.channel, user);
     });
 
-    test('user name matched case-insensitively', () {
-      final user1 = eg.user(fullName: 'Chris Bobbe');
-      final user2 = eg.user(fullName: 'chris bobbe');
+    test('user name match is case- and diacritics-insensitive', () {
+      final users = [
+        eg.user(fullName: 'Édith Piaf'),
+        eg.user(fullName: 'édith piaf'),
+        eg.user(fullName: 'Edith Piaf'),
+        eg.user(fullName: 'edith piaf'),
+      ];
+
+      checkAllSameRank('Édith Piaf', users); // exact
+      checkAllSameRank('Edith Piaf', users); // exact
+      checkAllSameRank('édith piaf', users); // exact
+      checkAllSameRank('edith piaf', users); // exact
+
+      checkAllSameRank('Édith Pi',   users); // total-prefix
+      checkAllSameRank('Edith Pi',   users); // total-prefix
+      checkAllSameRank('édith pi',   users); // total-prefix
+      checkAllSameRank('edith pi',   users); // total-prefix
 
-      checkSameRank('chris bobbe', user1, user2); // exact
-      checkSameRank('chris bo',    user1, user2); // total-prefix
-      checkSameRank('chr bo',      user1, user2); // word-prefixes
+      checkAllSameRank('Éd Pi',      users); // word-prefixes
+      checkAllSameRank('Ed Pi',      users); // word-prefixes
+      checkAllSameRank('éd pi',      users); // word-prefixes
+      checkAllSameRank('ed pi',      users); // word-prefixes
     });
 
     test('user name match: exact over total-prefix', () {
@@ -1096,13 +1139,16 @@ void main() {
       checkPrecedes('so m', user1, user2);
     });
 
-    test('group name matched case-insensitively', () {
-      final userGroup1 = eg.userGroup(name: 'Mobile Team');
-      final userGroup2 = eg.userGroup(name: 'mobile team');
+    test('group name is case- and diacritics-insensitive', () {
+      final userGroups = [
+        eg.userGroup(name: 'Mobile Team'),
+        eg.userGroup(name: 'mobile team'),
+        eg.userGroup(name: 'möbile team'),
+      ];
 
-      checkSameRank('mobile team', userGroup1, userGroup2); // exact
-      checkSameRank('mobile te',   userGroup1, userGroup2); // total-prefix
-      checkSameRank('mob te',      userGroup1, userGroup2); // word-prefixes
+      checkAllSameRank('mobile team', userGroups); // exact
+      checkAllSameRank('mobile te',   userGroups); // total-prefix
+      checkAllSameRank('mob te',      userGroups); // word-prefixes
     });
 
     test('group name match: exact over total-prefix', () {
@@ -1119,16 +1165,19 @@ void main() {
       checkPrecedes('so m', userGroup1, userGroup2);
     });
 
-    test('email matched case-insensitively', () {
+    test('email match is case- and diacritics-insensitive', () {
       // "z" name to prevent accidental name match with example data
-      final user1 = eg.user(fullName: 'z', deliveryEmail: 'email@example.com');
-      final user2 = eg.user(fullName: 'z', deliveryEmail: 'EmAiL@ExAmPlE.com');
-
-      checkSameRank('email@example.com', user1, user2);
-      checkSameRank('email@e',           user1, user2);
-      checkSameRank('email@',            user1, user2);
-      checkSameRank('email',             user1, user2);
-      checkSameRank('ema',               user1, user2);
+      final users = [
+        eg.user(fullName: 'z', deliveryEmail: 'email@example.com'),
+        eg.user(fullName: 'z', deliveryEmail: 'EmAiL@ExAmPlE.com'),
+        eg.user(fullName: 'z', deliveryEmail: 'ēmail@example.com'),
+      ];
+
+      checkAllSameRank('email@example.com', users);
+      checkAllSameRank('email@e',           users);
+      checkAllSameRank('email@',            users);
+      checkAllSameRank('email',             users);
+      checkAllSameRank('ema',               users);
     });
 
     test('email match is by prefix only', () {
@@ -1259,3 +1308,5 @@ void main() {
     });
   });
 }
+
+typedef WildcardTester = void Function(String query, Narrow narrow, List<WildcardMentionOption> expected);
diff --git a/test/model/emoji_test.dart b/test/model/emoji_test.dart
index 5a89722d6e..6ce03854b7 100644
--- a/test/model/emoji_test.dart
+++ b/test/model/emoji_test.dart
@@ -558,8 +558,10 @@ void main() {
       check(matchOfName('blue dia', 'large_blue_diamond')).wordAligned;
     });
 
-    test('query is lower-cased', () {
+    test('case-insensitive', () {
       check(matchOfName('Smi', 'smile')).prefix;
+      check(matchOfName('smi', 'SMILE')).prefix;
+      check(matchOfName('SmI', 'sMiLe')).prefix;
     });
 
     test('query matches aliases same way as primary name', () {
@@ -577,6 +579,8 @@ void main() {
       check(matchOfNames('blue_dia', ['x', 'large_blue_diamond'])).wordAligned;
 
       check(matchOfNames('Smi', ['x', 'smile'])).prefix;
+      check(matchOfNames('smi', ['x', 'SMILE'])).prefix;
+      check(matchOfNames('SmI', ['x', 'sMiLe'])).prefix;
     });
 
     test('best match among name and aliases prevails', () {
diff --git a/test/widgets/new_dm_sheet_test.dart b/test/widgets/new_dm_sheet_test.dart
index a11e862b48..aff2e8ba6e 100644
--- a/test/widgets/new_dm_sheet_test.dart
+++ b/test/widgets/new_dm_sheet_test.dart
@@ -123,6 +123,7 @@ void main() {
       eg.user(fullName: 'Alice Anderson'),
       eg.user(fullName: 'Bob Brown'),
       eg.user(fullName: 'Charlie Carter'),
+      eg.user(fullName: 'Édith Piaf'),
     ];
 
     testWidgets('shows full list initially', (tester) async {
@@ -130,7 +131,7 @@ void main() {
       check(findText(includePlaceholders: false, 'Alice Anderson')).findsOne();
       check(findText(includePlaceholders: false, 'Bob Brown')).findsOne();
       check(findText(includePlaceholders: false, 'Charlie Carter')).findsOne();
-      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(3);
+      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(testUsers.length);
       check(find.byIcon(ZulipIcons.check_circle_checked)).findsNothing();
     });
 
@@ -150,7 +151,7 @@ void main() {
         users: [...testUsers, deactivatedUser]);
       check(findText(includePlaceholders: false, 'Impostor Charlie')).findsNothing();
       check(findText(includePlaceholders: false, 'Charlie Carter')).findsOne();
-      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(3);
+      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(testUsers.length);
 
       // … and after a query that would match their name.
       await tester.enterText(find.byType(TextField), 'Charlie');
@@ -168,7 +169,7 @@ void main() {
       check(findText(includePlaceholders: false, 'Someone Muted')).findsNothing();
       check(findText(includePlaceholders: false, 'Muted user')).findsNothing();
       check(findText(includePlaceholders: false, 'Alice Anderson')).findsOne();
-      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(3);
+      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(testUsers.length);
 
       // … and after a query.  One which matches both the user's actual name and
       // the replacement text "Muted user", for good measure.
@@ -178,13 +179,14 @@ void main() {
       check(findText(includePlaceholders: false, 'Muted user')).findsNothing();
       check(findText(includePlaceholders: false, 'Alice Anderson')).findsOne();
       check(findText(includePlaceholders: false, 'Charlie Carter')).findsOne();
-      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(2);
+      check(findText(includePlaceholders: false, 'Édith Piaf')).findsOne();
+      check(find.byIcon(ZulipIcons.check_circle_unchecked)).findsExactly(3);
     });
 
     // TODO test sorting by recent-DMs
     // TODO test that scroll position resets on query change
 
-    testWidgets('search is case-insensitive', (tester) async {
+    testWidgets('search is case- and diacritics-insensitive', (tester) async {
       await setupSheet(tester, users: testUsers);
       await tester.enterText(find.byType(TextField), 'alice');
       await tester.pump();
@@ -193,6 +195,14 @@ void main() {
       await tester.enterText(find.byType(TextField), 'ALICE');
       await tester.pump();
       check(findText(includePlaceholders: false, 'Alice Anderson')).findsOne();
+
+      await tester.enterText(find.byType(TextField), 'alicé');
+      await tester.pump();
+      check(findText(includePlaceholders: false, 'Alice Anderson')).findsOne();
+
+      await tester.enterText(find.byType(TextField), 'edith');
+      await tester.pump();
+      check(findText(includePlaceholders: false, 'Édith Piaf')).findsOne();
     });
 
     testWidgets('partial name and last name search handling', (tester) async {