Repository: nvaccess/nvda

Issue:
L10N (possible release blocker): messages with plural forms and/or formatted string literals with translated messages do not translate correctly under some languages
Hi,

I think this is a release blocker (@LeonardDeR, any ideas about what I'm describing below):

### Background:
GNU Gettext defines ngettext function for specifying singular and plural forms of messages based on the value stored inside a variable. The signature is:

gettext.ngettext(singular, plural, value)

For example, in English:

gettext.ngettext("test", "tests", testValue)

Then:

* testValue == 1: test
* testValue == 2: tests

Whereas in languages such as Korean with one value for singular and plural forms:

* testValue == 1: test
* testValue == 2: test

So far, translation is working.

### Problems:
Three major problems:

1. Translated text, together with the value, is used to build formatted string literals for actual output. If spaces are inserted between variable value and translated string, it causes text to be shown incorrectly for languages that expect no spaces between value and text (e.g. Korean). For example, if we have f"{number} {message}" with number being 2 and message being "tests", respectivley, then it becomes "2 tests" in English and "2test" in Korean; if the formatted string literal is output as is, it becomes "2 test" in Korean which goes against spacing rules in Korean text.
2. Some plural text may contain string interpolations. This causes an issue with languages with no distinction between singular and plural forms (e.g. Korean) where the translator cannot translate singular and plural form together. The ideal solution is just translating plural form. An example is "1 test" (singular) versus "{tests} tests" (plural) in English, whereas in Korean both texts can be translated as "{tests} test".
3. Some languages define different plural rules as opposed to common ones such as singular versus plural (English) or single text (Korean). A good example is Russian where up to three text forms can be defined to describe singular and plural forms.

### Steps to reproduce:

1. Attempt to translate latest alpha build (ercently merged into beta branch). First, use scons pot to generate the catalog template, then apply it to a po file for languages other than English (try Korean).
2. Attempt to translate text (for example, "category" and "categories").

### Actual behavior:
Text cannot be translated into some languages (e.g. Korean).

### Expected behavior:
NVDA announces translated messages.

### System configuration
#### NVDA installed/portable/running from source:
Installed

#### NVDA version:
alpha-22762,1007f9e2

#### Windows version:
Windows 10 Version 21H1 (build 19043)

#### Name and version of other software in use when reproducing the issue:
Poedit 1.8.9

#### Other information about your system:
NVDA development workstation, also involved in translating messages into Korean.

### Other questions
#### Does the issue still occur after restarting your computer?
Yes

#### Have you tried any other versions of NVDA? If so, please report their behaviors.
Works correctly in 2020.4 as there is one-to-one mapping between messages and translations.

#### If add-ons are disabled, is your problem still occurring?
Yes (this also affects add-ons that call gettext.ngettext).

#### Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Not applicable

### Impact and possible mitigations:
This may become a release blocker - people who do not speak English may feel new messages were not translated when in fact they were translated.

One possible mitigation is reverting gettext.ngettext call when announcing message categories, or directing speakers of affected languages to apply specific rules about plural texts that might be contradictory to what they have known. Also, if gettext.ngettext is to be used going forward, to avoid spacing issues, I advise making actual output messages translatable.

Thanks.


Discussion:
Hi,

Active thread on this issue:

https://groups.io/g/nvda-translations/topic/plural_forms/82818213?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,82818213

@josephsl While it probably stems from my ignorance of translations 
workflow, I do not understand what has changed to make this blocking now, 
when it wasn't before. It seems like something that would have been an 
issue for a long time.

Hi,

What i have done for my languages is that I have  added the plural forms myself.

There is a comprehensive list of these.

Hi,

It wasn't an issue until now because translatable strings are sent to translators once alpha is merged into beta, which just happened. In the old days (2020.4 and earlier), there was a one-to-one correspondence between translatable strings and resulting translations simply because we didn't use gettext.ngettext (we simply used gettext.gettext, or the convenience method of using _()); for ambiguous strings, we use pgettext function which was pulled into built-in namespace because Python 3.7 and earlier didn't define gettext.pgettext (that's why I posted a PR (reverted later due to discussions surrounding Python 3.8 and lifffi issue) that allowed NVDA to pull in gettext.pgettext). Using gettext.ngettext, while allowing singular and plural forms to be defined for languages such as English and Spanish, does not work well for languages with a variety of grammatical rules about singular versus plural forms (as I described above, Korean only defines one way to handle singular and plural forms, whereas others define multiple forms to deal with different number ranges); it is up to translators to provide expressions that define plural forms, and thankfully GNU Gettext (the Python binding that we are using) comes with plural forms for common cases such as Korean.

The biggest issue though is how will translators handle plural forms in their languages and using their favorite tools. After a catalog is merged into a po file, the resulting po file will have a field named "msgid_plural" used to define plural forms. Then it is up to translators to define translations for specific plural forms, or in case of Korean, only a single text would suffice. At least this can be done easily in Notepad++ and tools that consume po files as text files; the problem may arise for folks using Poedit and similar tools.

Another reason why this is a release blocker is that, for Outlook message categories, two steps are involved when announcing number of categories (actually, four steps):

1. Outlook will report how many categories a message defines, and this is stored inside a variable.
2. gettext.ngettext will then select the appropriate text (category/categories) based on category count value and is then translated.
3. The translated text, together with category count, is then combined to form the complete output. This is where formatted string literals (f strings) are employed.
4. Actual output takes place.

The problem is that the formatted string literal contains a space between category count and the translated string. This works for languageswith clear spacing between words, not so for languages such as Korean where spaces are not allowed between numbers and words. Making matters worse is that the complete output text isn't translatable (because it is an f string), meaning that for languages with no spacing rules, the final output not only looks weird, but is also grammatically incorrect (for translators, their concern is how a given message written in the source language (English) can be best translated while obeying grammar rules of the target language; this is one of the reasons for using pgettext function under specific contexts, and I vividly remember a conversation almost a decade ago regarding this issue).

The reason for this being a release blocker is because of the impact this will have on the international reputation of NVDA. NVDA's biggest strength (besides cost) is internationalization. Consider a bug report from a user whose native language isn't English, saying that a new string isn't translated into their language when in fact translators say it is; imagine this being a case for messages presented as part of a feature the bug reporter would use on a daily basis. Contributors then can propose possible solutions, but translators must then wait until the fix makes it to beta branch, and the person who reported this bug must be told to wait until the next stable version of NVDA is released. To me, this reason alone makes this a release blocker.

To help investigate this, I'm testing possible solutions through my add-ons (at least three of my add-ons include messages that could be enhanced by defining plural forms). For now I am using ternary syntax (x if z else y) to provide singular and plural forms (at least for English and Korean), which isn't ideal considering that many languages have multiple plural forms. Another reason for testing with add-ons is to look for ways to make the original (English) messages more concise and easier to understand and translate (I spend considerable time just working on translatable messages).

Hope this helps. Thanks.
Hi,

Have you tried to add the plural form to your header?

List of these is here:

http://docs.translatehouse.org/projects/localization-guide/en/latest/l10n/pluralforms.html?id=l10n/pluralforms

See my previous Comment: I’ve added the plural forms. Guide me how I can test these.


Hi, for now you must locate where NVDA is installed and copy locale files manually (this is when portable copies are useful); be sure to create the .mo file. Thanks.

Hi, yep I have added plural forms (testing with Add-on Updater translation because it does use gettext.ngettext in test builds I have). Thanks.


I'm happy to come up with a pr that stops using ngettext 
Hi, thanks, although I think we might as well look for ways to make better use of gettext.ngettext to improve localization support in the future (it will benefit add-ons, too). Thanks.


Plural forms will hopefully allow my translations being on higher levels.

I’ve tested the plurals for my language, and they work!

> ### Problems:
> Three major problems:
> 
> 1. Translated text, together with the value, is used to build formatted string literals for actual output. If spaces are inserted between variable value and translated string, it causes text to be shown incorrectly for languages that expect no spaces between value and text (e.g. Korean). For example, if we have f"{number} {message}" with number being 2 and message being "tests", respectivley, then it becomes "2 tests" in English and "2test" in Korean; if the formatted string literal is output as is, it becomes "2 test" in Korean which goes against spacing rules in Korean text.

I don't think this applies to the current case, as the number of categories isn't reflected in the actual message.

> 2. Some plural text may contain string interpolations. This causes an issue with languages with no distinction between singular and plural forms (e.g. Korean) where the translator cannot translate singular and plural form together. The ideal solution is just translating plural form. An example is "1 test" (singular) versus "{tests} tests" (plural) in English, whereas in Korean both texts can be translated as "{tests} test".

I also don't see how this relates to the actual use of ngettext in the outlook appModule.

> 3. Some languages define different plural rules as opposed to common ones such as singular versus plural (English) or single text (Korean). A good example is Russian where up to three text forms can be defined to describe singular and plural forms.

This is rather a limitation of ngettext, if I understand it correctly.
Hello,

just to clarify, because, I'm the author of a topic on a mailing list linked above:
string "category(ies)", which was added in the last update, of a po catalog, allows me to translate it in a singular and plural form. but other strings, where plural forms are possible are (at least in poedit) translatable only in one form. so it is behaviour which has been seed until 2020.4. if we allow only some strings to be in a correct form, then I think, that change should be reverted due to problems, which would occur, like, why there are "category" and "categories" but not "table with 1 column" or "table with x columns"
than translators will need to explaine to our local communities, why there are different solutions for same problem.
in slovenian we have 4 plural versions in case of x columns:
1 column = 1 stolpec
2 columns = 2 stolpca
3 or 4 columns = 3 or 4 stolpci
5 columns = 5 stolpcev
so to summerise, until there is a complete solution I'm against of distinct between singular and plural, due to bringing more problems then resolving them.
if I missed something, I would step back gladly and would learn, how to make all appropriate strings in a correct forms, so there will be "1 column - 1 stolpec" and "3 columns - 3 stolpci"
I hope, that I'm clear, why I raised this issue again.
first it was discussed years ago, think around 2012.3. should look into message archive.
anywayy, thanks for your great work.
The impact of this isn't really clear to me. But I have added the 2021.1 label so that we look at this more closely before the release.
I've looked at this, it seems like the conversation has prematurely generalized. The only example of `ngettext` in NVDA is the following in `source/appModules/outlook.py`:
``` python
		# Translators: Part of a message reported when on a calendar appointment with one or more categories
		# in Microsoft Outlook.
		categoriesText = ngettext("category", "categories", categoriesCount)
		return f"{categoriesText} {categories}"
```

This is clearly incorrect by constructing the sentence. It prevents the translators from having control over the order of the placeholder, and as @josephsl points out also the spacing. 

@josephsl in future, please start with the specific example. 

To me it seems the solution is simple: 
``` python
		# Translators: Part of a message reported when on a calendar appointment with one or more categories
		# in Microsoft Outlook.
		categoriesText = ngettext("category {categories}", "categories {categories}", categoriesCount)
		return categoriesText
```

Resolving patch:
diff --git a/source/appModules/outlook.py b/source/appModules/outlook.py
--- a/source/appModules/outlook.py
+++ b/source/appModules/outlook.py
@@ -357,8 +357,8 @@ def _generateCategoriesText(appointment):
 
 		# Translators: Part of a message reported when on a calendar appointment with one or more categories
 		# in Microsoft Outlook.
-		categoriesText = ngettext("category", "categories", categoriesCount)
-		return f"{categoriesText} {categories}"
+		categoriesText = ngettext("category {categories}", "categories {categories}", categoriesCount)
+		return categoriesText.format(categories)
 
 	def isDuplicateIAccessibleEvent(self,obj):
 		return False


Test patch: