Repository: kellpossible/cargo-i18n (language: rust)

Issue:
i18n-embed: get message_id for desired locale, falling back directly to base fallback language
Hello!

Sorry for the verbose title.

My web application has N available languages. On runtime, I check for the user's preferred locale, let's say pt-BR. Then I want to get the message for this locale, falling back directly en-US, but with all other locale files loaded, to proper populate the email template with the user's language.

Is there a method with this behaviour now on `FluentLanguageLoader`?

This would make it possible to avoid bundling each language with "en-US", the way I do it now with raw Fluent.

If I understood correctly, `FluentLanguageLoader` works in a strict fallback manner, as seen [here](https://github.com/kellpossible/cargo-i18n/blob/59bf01ccbeeff053e36103aadec695d7e217a0d8/i18n-embed/tests/loader.rs#L39), it tries to get the message at the highest priority locale, if it can't find it, goes to next locale and so on until the base one.

A concrete example:

```rust
#[test]
    fn fallbacks_ru_to_base() {
        setup();
        let ru: LanguageIdentifier = "ru".parse().unwrap();
        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();

        let loader = FluentLanguageLoader::new("test", en_us.clone());
        loader
            .load_languages(&Localizations, &[&ru, &en_gb])
            .unwrap();
        pretty_assertions::assert_eq!("only GB (US Version)", loader.get_with(ru, "only-gb"));
        pretty_assertions::assert_eq!("только русский", loader.get_with(ru, "only-ru"));
    }
```

Would this be useful or should I go for another approach?


Discussion / hints:
> On runtime, I check for the user's preferred locale, let's say pt-BR

Is this localization occurring on the front-end or the back-end of your web application?

>  but with all other locale files loaded, to proper populate the email template with the user's language.

Sorry I don't understand this use-case completely. You want to have all available locales loaded, or just the ones which the user is requesting? Why do they all need to be loaded?

> This would make it possible to avoid bundling each language with "en-US",

Why are you bundling each language with "en-US" and why do you want to avoid it?

> A concrete example:

Would you be able to write more about the proposed `get_with()` method?
Do you think your use-case might be similar to this feature request? https://github.com/kellpossible/cargo-i18n/issues/37 


> Is this localization occurring on the front-end or the back-end of your web application?

This section of localization occurs at the back-end. The user's preferred locale is stored on the database, I retrieve it, get all messages-ids from the .ftl that match his preferred locale that I need, falling back to English if it doesn't exist (my use-case).

```rust
// retrieved from database
struct User {
	locale: String  // stored as "pt-BR"
}

// strum helps me connect everything dynamically
#[derive(AsStaticStr, EnumString)]
enum LocalesAvailable {
	#[strum(serialize = "pt-BR")]
	Portuguese
	#[strum(serialize = "en-US")]
	English
	// ... other languages
}

impl LocalesAvailable {
	// With this, I can dynamically fetch the message-id
	fn to_langid(&self) -> LanguageIdentifier {
		self.into().parse().unwrap()
	}
}


fn render_email_template(user: User, loader: &'static FluentLanguageLoader) {
	let user_locale = LocalesAvailable::from_str(&*user.locale); // LocalesAvailable::Portuguese
	
	// loader is statically embedded at the server, containing all N translations
	let email_subject = loader.get_with(user_locale.to_langid(), "email-confirmation-subject");

	// ..
}
```

Hopefully, this makes sense.

To be honest, this should not happen very often, since it is very important to keep all locales on par with the most complete one. The reasoning is that it doesn't make much sense to me if the user main language is Russian, to fallback for Spanish. His preferences are stored, there is no language negotiation anymore, so simply fallback to one specific lang.

But I feel that this may be very specific to me. I created a middleware that takes care of the initial lang negotiation, so this a more of a separate step much later in the process.

> Why do they all need to be loaded?

Simply because they are so tiny, aggregating to barely 200kb. With the language loader allowing me to get the message-ids of the locale of interest, and falling back to a specific one. Much more convenient and minimal overhead.

Today I need to bundle (es-AR, en-us), (pt-BR, en-us), (ru, en-us), to get this behaviour.

Sorry if in some bits I was redundant, English is not my main language, hope you understand the core idea. I'm not even sure if something like this would be useful for the crate.
The `get_with(locale, message_id)` method would skip the regular fallback negotiation, and either try to get the message in the specified locale, or fall back to the global fallback locale?

It does seem very similar to what is being requested in https://github.com/kellpossible/cargo-i18n/issues/37 (which would probably implement the same behaviour). I'm leaning more towards having a separate implementation of `FluentLanuageLoader`  (perhaps called `FluentMultiLanguageLoader`), for this purpose which has no concept of a single "current language", all languages are loaded, and which optionally calculates fallbacks at runtime with each `get` request, and by default is set up in a fast path to fall back to the globally specified fallback locale.

> Simply because they are so tiny, aggregating to barely 200kb. With the language loader allowing me to get the message-ids of the locale of interest, and falling back to a specific one. Much more convenient and minimal overhead.

Thanks I think I'm understanding. You want to avoid creating a separate bundle for each language to get the specific fallback behaviour you want because it uses more memory, and creating a bundle the fly for each request would be worse right?

> Sorry if in some bits I was redundant, English is not my main language, hope you understand the core idea. I'm not even sure if something like this would be useful for the crate.

That's okay, sorry for my questions too, and thanks for posting this issue, it's an important topic if this library is to be used in backend services I think. At the moment it is currently geared towards desktop applications and front-end where the language/fallbacks being requested do not change very often.

> The `get_with(locale, message_id)` method would skip the regular fallback negotiation, and either try to get the message in the specified locale or fall back to the global fallback locale?

Yes, exactly.

> It does seem very similar to what is being requested in #37 (which would probably implement the same behaviour). I'm leaning more towards having a separate implementation of `FluentLanuageLoader` (perhaps called `FluentMultiLanguageLoader`), for this purpose which has no concept of a single "current language", all languages are loaded, and which optionally calculates fallbacks at runtime with each `get` request, and by default is set up in a fast path to fall back to the globally specified fallback locale.

Following your thoughts on a different loader, `FluentMultiLanguageLoader` could have even two methods, one to get a locale, falling back directly to a global language, and another, to fallback to specified locales, or maybe that is too much?

```rust
/// fallback to global fallback language
fn get_with(&self, locale: LanguageIdentifier, message_id: &str) {}

// fallback to custom fallback languages
// could work if es-ES is ready but es-AR is still being translated
// if message_id does not exist in either languages, fallsback to global
fn get_with_custom_fallback(&self, locales: &[LanguageIdentifier], message_id: &str) {}
```

> Thanks I think I'm understanding. You want to avoid creating a separate bundle for each language to get the specific fallback behaviour you want because it uses more memory, and creating a bundle the fly for each request would be worse right?

That would be it. I think this is more of a convenience. `FluentMultiLanguageLoader` is acting as a map type to hold every existent translation, instead of bundling them individually, matching and retrieving based on some rules that a backend would require.

The use case here would be where memory is plenty since `cargo-i18` has alternatives for embedded systems with more limited hardware.
 
> That's okay, sorry for my questions too, and thanks for posting this issue, it's an important topic if this library is to be used in backend services I think. At the moment it is currently geared towards desktop applications and front-end where the language/fallbacks being requested do not change very often.

That is great to hear, are you interested in a PR? I could set up a draft and you can decide if this is going in the direction you want.

> Following your thoughts on a different loader, FluentMultiLanguageLoader could have even two methods, one to get a locale, falling back directly to a global language, and another, to fallback to specified locales, or maybe that is too much?

That sounds like a good idea!

> That is great to hear, are you interested in a PR? I could set up a draft and you can decide if this is going in the direction you want.

Sure, I would definitely appreciate that, and happy to review
I was working on something that for my use-case it worked perfectly, and decided to push the PR as draft: https://github.com/kellpossible/cargo-i18n/pull/62

There are four new methods for the MultiLoader, `get_with_locale` and `get_with_locale_and_args`, for retrieval of a single translated string, for a specific locale, falling back to the global fallback language.

And `get_with_custom_fallback` and `get_with_custom_fallback_and_args` for custom fallback behaviour, based on an array of `LanguageIdentifier`, but ultimately falling back to the local fallback language too.

I've added tests and a string to the FTL to make sure it behaves accordingly.

There is duplicated code since they load FTL files the same way, perhaps they could be moved to some free functions. 

Also, `current_language` doesn't make much sense in the MultiLoader context, since there is no concept of it, then I left it as unimplemented. What do you think?
@saskenuba Thanks for your contribution! I've been away for the past few days, but I'll make sure to do a proper reply soon and review your PR

Fix patch:
diff --git a/i18n-embed/src/fluent.rs b/i18n-embed/src/fluent.rs
index d376506..2e0747a 100644
--- a/i18n-embed/src/fluent.rs
+++ b/i18n-embed/src/fluent.rs
@@ -13,16 +13,9 @@ use fluent::{bundle::FluentBundle, FluentArgs, FluentMessage, FluentResource, Fl
 use fluent_syntax::ast::{self, Pattern};
 use intl_memoizer::concurrent::IntlLangMemoizer;
 use parking_lot::RwLock;
-use std::{borrow::Cow, collections::HashMap, fmt::Debug, sync::Arc};
+use std::{borrow::Cow, collections::HashMap, fmt::Debug, iter::FromIterator, sync::Arc};
 use unic_langid::LanguageIdentifier;
 
-lazy_static::lazy_static! {
-    static ref CURRENT_LANGUAGE: RwLock<LanguageIdentifier> = {
-        let language = LanguageIdentifier::default();
-        RwLock::new(language)
-    };
-}
-
 struct LanguageBundle {
     language: LanguageIdentifier,
     bundle: FluentBundle<Arc<FluentResource>, IntlLangMemoizer>,
@@ -59,6 +52,9 @@ impl Debug for LanguageBundle {
 struct LanguageConfig {
     current_language: LanguageIdentifier,
     language_bundles: Vec<LanguageBundle>,
+    /// This maps a `LanguageIdentifier` to the index inside the
+    /// `language_bundles` vector.
+    language_map: HashMap<LanguageIdentifier, usize>,
 }
 
 /// [LanguageLoader] implemenation for the `fluent` localization
@@ -85,6 +81,7 @@ impl FluentLanguageLoader {
         let config = LanguageConfig {
             current_language: fallback_language.clone(),
             language_bundles: Vec::new(),
+            language_map: HashMap::new(),
         };
 
         Self {
@@ -106,40 +103,114 @@ impl FluentLanguageLoader {
 
     /// Get a localized message referenced by the `message_id`.
     pub fn get(&self, message_id: &str) -> String {
-        self.get_args_concrete(message_id, HashMap::new())
+        self.get_args_fluent(message_id, None)
     }
 
     /// A non-generic version of [FluentLanguageLoader::get_args()].
-    pub fn get_args_concrete<'source>(
+    pub fn get_args_concrete<'args>(
         &self,
         message_id: &str,
-        args: HashMap<&'source str, FluentValue<'source>>,
+        args: HashMap<&'args str, FluentValue<'args>>,
     ) -> String {
-        let args_option = if args.is_empty() {
-            None
-        } else {
-            let mut fluent_args = FluentArgs::with_capacity(args.len());
+        self.get_args_fluent(message_id, hash_map_to_fluent_args(args).as_ref())
+    }
 
-            for (key, value) in args {
-                fluent_args.set(key, value);
-            }
+    /// A non-generic version of [FluentLanguageLoader::get_args()]
+    /// accepting [FluentArgs] instead of a [HashMap].
+    pub fn get_args_fluent<'args>(
+        &self,
+        message_id: &str,
+        args: Option<&'args FluentArgs<'args>>,
+    ) -> String {
+        let language_config = self.language_config.read();
+        self._get(
+            language_config.language_bundles.iter(),
+            &language_config.current_language,
+            message_id,
+            args,
+        )
+    }
 
-            Some(fluent_args)
-        };
+    /// Get a localized message referenced by the `message_id`, and
+    /// formatted with the specified `args`.
+    pub fn get_args<'a, S, V>(&self, id: &str, args: HashMap<S, V>) -> String
+    where
+        S: Into<Cow<'a, str>> + Clone,
+        V: Into<FluentValue<'a>> + Clone,
+    {
+        self.get_args_fluent(id, hash_map_to_fluent_args(args).as_ref())
+    }
 
-        self.get_args_fluent(message_id, args_option.as_ref())
+    /// Get a localized message referenced by the `message_id`.
+    pub fn get_lang(&self, lang: &[&LanguageIdentifier], message_id: &str) -> String {
+        self.get_lang_args_fluent(lang, message_id, None)
     }
 
-    /// A non-generic version of [FluentLanguageLoader::get_args()]
+    /// A non-generic version of [FluentLanguageLoader::get_lang_args()].
+    pub fn get_lang_args_concrete<'source>(
+        &self,
+        lang: &[&LanguageIdentifier],
+        message_id: &str,
+        args: HashMap<&'source str, FluentValue<'source>>,
+    ) -> String {
+        self.get_lang_args_fluent(lang, message_id, hash_map_to_fluent_args(args).as_ref())
+    }
+
+    /// A non-generic version of [FluentLanguageLoader::get_lang_args()]
     /// accepting [FluentArgs] instead of a [HashMap].
-    pub fn get_args_fluent<'args>(
+    pub fn get_lang_args_fluent<'args>(
         &self,
+        lang: &[&LanguageIdentifier],
         message_id: &str,
         args: Option<&'args FluentArgs<'args>>,
     ) -> String {
+        let current_language = if lang.is_empty() {
+            &self.fallback_language
+        } else {
+            lang[0]
+        };
+        let fallback_language = if lang.contains(&&self.fallback_language) {
+            None
+        } else {
+            Some(&self.fallback_language)
+        };
         let config_lock = self.language_config.read();
+        let language_bundles = lang
+            .iter()
+            .chain(fallback_language.as_ref().into_iter())
+            .filter_map(|id| {
+                config_lock
+                    .language_map
+                    .get(id)
+                    .map(|idx| &config_lock.language_bundles[*idx])
+            });
+        self._get(language_bundles, current_language, message_id, args)
+    }
+
+    /// Get a localized message for the given language identifiers, referenced
+    /// by the `message_id` and formatted with the specified `args`.
+    pub fn get_lang_args<'a, S, V>(
+        &self,
+        lang: &[&LanguageIdentifier],
+        id: &str,
+        args: HashMap<S, V>,
+    ) -> String
+    where
+        S: Into<Cow<'a, str>> + Clone,
+        V: Into<FluentValue<'a>> + Clone,
+    {
+        let fluent_args = hash_map_to_fluent_args(args);
+        self.get_lang_args_fluent(lang, id, fluent_args.as_ref())
+    }
 
-        config_lock.language_bundles.iter().filter_map(|language_bundle| {
+    fn _get<'a, 'args>(
+        &'a self,
+        language_bundles: impl Iterator<Item = &'a LanguageBundle>,
+        current_language: &LanguageIdentifier,
+        message_id: &str,
+        args: Option<&'args FluentArgs<'args>>,
+    ) -> String {
+        language_bundles.filter_map(|language_bundle| {
             language_bundle
                 .bundle
                 .get_message(message_id)
@@ -151,57 +222,24 @@ impl FluentLanguageLoader {
                         log::error!(
                             target:"i18n_embed::fluent",
                             "Failed to format a message for language \"{}\" and id \"{}\".\nErrors\n{:?}.",
-                            &config_lock.current_language, message_id, errors
+                            current_language, message_id, errors
                         )
                     }
-
                     value.into()
-                    })
+                })
             })
             .next()
             .unwrap_or_else(|| {
                 log::error!(
                     target:"i18n_embed::fluent",
                     "Unable to find localization for language \"{}\" and id \"{}\".",
-                    config_lock.current_language,
+                    current_language,
                     message_id
                 );
                 format!("No localization for id: \"{}\"", message_id)
             })
     }
 
-    /// Get a localized message referenced by the `message_id`, and
-    /// formatted with the specified `args`.
-    pub fn get_args<'a, S, V>(&self, id: &str, args: HashMap<S, V>) -> String
-    where
-        S: Into<Cow<'a, str>> + Clone,
-        V: Into<FluentValue<'a>> + Clone,
-    {
-        let mut keys: Vec<Cow<'a, str>> = Vec::new();
-
-        let mut map: HashMap<&str, FluentValue<'_>> = HashMap::with_capacity(args.len());
-
-        let mut values = Vec::new();
-
-        for (key, value) in args.into_iter() {
-            keys.push(key.into());
-            values.push(value.into());
-        }
-
-        for (i, key) in keys.iter().rev().enumerate() {
-            let value = values.pop().unwrap_or_else(|| {
-                panic!(
-                    "expected a value corresponding with key \"{}\" at position {}",
-                    key, i
-                )
-            });
-
-            map.insert(&*key, value);
-        }
-
-        self.get_args_concrete(id, map)
-    }
-
     /// Returns true if a message with the specified `message_id` is
     /// available in any of the languages currently loaded (including
     /// the fallback language).
@@ -371,10 +409,28 @@ impl LanguageLoader for FluentLanguageLoader {
         }
 
         let mut config_lock = self.language_config.write();
-        config_lock.language_bundles = language_bundles;
         config_lock.current_language = current_language.clone();
+        config_lock.language_bundles = language_bundles;
+        config_lock.language_map = config_lock
+            .language_bundles
+            .iter()
+            .enumerate()
+            .map(|(i, language_bundle)| (language_bundle.language.clone(), i))
+            .collect();
         drop(config_lock);
 
         Ok(())
     }
 }
+
+fn hash_map_to_fluent_args<'args, K, V>(map: HashMap<K, V>) -> Option<FluentArgs<'args>>
+where
+    K: Into<Cow<'args, str>>,
+    V: Into<FluentValue<'args>>,
+{
+    if map.is_empty() {
+        None
+    } else {
+        Some(FluentArgs::from_iter(map))
+    }
+}


Test patch:
diff --git a/i18n-embed/i18n/ftl/en-GB/test.ftl b/i18n-embed/i18n/ftl/en-GB/test.ftl
index 996517e..28543b9 100644
--- a/i18n-embed/i18n/ftl/en-GB/test.ftl
+++ b/i18n-embed/i18n/ftl/en-GB/test.ftl
@@ -1,3 +1,4 @@
 hello-world = Hello World Localisation!
 only-gb = only GB
-different-args = this message has {$different} {$args} in different languages
\ No newline at end of file
+only-gb-args = Hello {$userName}!
+different-args = this message has {$different} {$args} in different languages
diff --git a/i18n-embed/i18n/ftl/en-US/test.ftl b/i18n-embed/i18n/ftl/en-US/test.ftl
index 1288029..a1f145b 100644
--- a/i18n-embed/i18n/ftl/en-US/test.ftl
+++ b/i18n-embed/i18n/ftl/en-US/test.ftl
@@ -2,6 +2,7 @@ hello-world = Hello World Localization!
 only-us = only US
 only-ru = only RU
 only-gb = only GB (US Version)
+only-gb-args = Hello {$userName}! (US Version)
 different-args = this message has different {$arg}s in different languages
 isolation-chars = inject a { $thing } here
 multi-line =
@@ -19,4 +20,4 @@ multi-line-args =
 
     { $argTwo }
 
-    Finished!
\ No newline at end of file
+    Finished!
diff --git a/i18n-embed/tests/loader.rs b/i18n-embed/tests/loader.rs
index 4b90aea..396fb67 100644
--- a/i18n-embed/tests/loader.rs
+++ b/i18n-embed/tests/loader.rs
@@ -188,6 +188,95 @@ mod fluent {
             msg
         );
     }
+
+    #[test]
+    fn get_lang_default_fallback() {
+        setup();
+        let ru: LanguageIdentifier = "ru".parse().unwrap();
+        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();
+        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
+        let loader = FluentLanguageLoader::new("test", en_us);
+
+        loader
+            .load_languages(&Localizations, &[&ru, &en_gb])
+            .unwrap();
+
+        let msg = loader.get_lang(&[&ru], "only-ru");
+        assert_eq!("только русский", msg);
+
+        let msg = loader.get_lang(&[&ru], "only-gb");
+        assert_eq!("only GB (US Version)", msg);
+    }
+
+    #[test]
+    fn get_lang_args_default_fallback() {
+        setup();
+        let ru: LanguageIdentifier = "ru".parse().unwrap();
+        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();
+        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
+        let loader = FluentLanguageLoader::new("test", en_us);
+
+        loader
+            .load_languages(&Localizations, &[&ru, &en_gb])
+            .unwrap();
+
+        let args = maplit::hashmap! {
+            "argOne" => "1",
+            "argTwo" => "2",
+        };
+
+        let msg = loader.get_lang_args(&[&ru], "multi-line-args", args);
+        assert_eq!(
+            "Это многострочное сообщение с параметрами.\n\n\
+            \u{2068}1\u{2069}\n\n\
+            Это многострочное сообщение с параметрами.\n\n\
+            \u{2068}2\u{2069}\n\n\
+            Законченный!",
+            msg
+        );
+    }
+
+    #[test]
+    fn get_lang_custom_fallback() {
+        setup();
+        let ru: LanguageIdentifier = "ru".parse().unwrap();
+        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();
+        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
+        let loader = FluentLanguageLoader::new("test", en_us);
+
+        loader
+            .load_languages(&Localizations, &[&ru, &en_gb])
+            .unwrap();
+
+        let msg = loader.get_lang(&[&ru, &en_gb], "only-gb");
+        assert_eq!("only GB", msg);
+
+        let msg = loader.get_lang(&[&ru, &en_gb], "only-us");
+        assert_eq!("only US", msg);
+    }
+
+    #[test]
+    fn get_lang_args_custom_fallback() {
+        setup();
+        let ru: LanguageIdentifier = "ru".parse().unwrap();
+        let en_gb: LanguageIdentifier = "en-GB".parse().unwrap();
+        let en_us: LanguageIdentifier = "en-US".parse().unwrap();
+        let loader = FluentLanguageLoader::new("test", en_us);
+
+        loader
+            .load_languages(&Localizations, &[&ru, &en_gb])
+            .unwrap();
+
+        let args = maplit::hashmap! {
+            "userName" => "username",
+        };
+
+        let msg = loader.get_lang_args(&[&ru], "only-gb-args", args.clone());
+        assert_eq!("Hello \u{2068}username\u{2069}! (US Version)", msg);
+
+        let msg = loader.get_lang_args(&[&ru, &en_gb], "only-gb-args", args.clone());
+        assert_eq!("Hello \u{2068}username\u{2069}!", msg);
+    }
 }
 
 #[cfg(feature = "gettext-system")]