Repository: WeblateOrg/weblate

Issue:
plural clash after first edit in Weblate
The project had a lt.po translation with following plural form:

"Plural-Forms: nplurals=4; plural=(n%10==1 ? 0 : n%10==1 && n%100!=11 ? 1 : n"
"%10>=2 && (n%100<10 || n%100>=20) ? 2 : 3);\n"
"X-Generator: Lokalize 1.4\n"

Translations correctly had 4 plurals for each string.

Then the translator did several edits. None of them affected strings that have a plural form.

After the commit, Weblate forcefully changed plural form, without any change of strings with plural forms.

New plural form:
"Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && "
"(n%100<10 || n%100>=20) ? 1 : 2;\n"
"X-Generator: Weblate 2.3\n"

The result was a broken translation.

https://github.com/openSUSE/libstorage/blob/34a1ef71ad3750152f4c691b5866972c4b3dd0b7/po/lt.po

(Note that the translation was also broken by https://github.com/nijel/weblate/issues/875.)



Discussion:
For Leap please use this header:

"Plural-Forms: nplurals=4; plural=(n==1 ? 0 : n%10==1 && n%100!=11 ? 1 : n%"
"10>=2 && (n%100<10 || n%100>=20) ? 2 : 3);\n"

https://github.com/openSUSE/libstorage/pull/152

@stanislav-brabec can you post instructions to reproduce the problem?

Pick a language with n plurals pre-defined.
Upload a file with m plurals, where m ≠ n.
Check, what was uploaded:
header contains n plural form equation
translations contain m plurals

msgfmt -c is currently the only standard gettext tool that is capable to detect the problem.

You can eventually try to edit translation in the web interface. Maybe it will make things even worse.

thanks

it appears that the language definition is the only place where plural forms are stored, therefore the weblate config is authoritative in this sense.

So this issue can be solved either by adding per-translation plural config, or by filtering the plural forms at import. (no sense doing that at export, as it's impossible to edit the overflowing plural forms 
I'm not too keen on implementing configurable translation matrices, ISTM that this problem is rare enough for hard-coded solutions for particular languages

@stanislav-brabec, @nijel, what do you think?

alternately we could specify a translation matrix at upload

after a brief discussion with @stanislav-brabec, the following solution appears viable:
- at upload, we make sure that the uploaded file has the same plural-forms definition as the weblate instance
- if not, the upload fails
- unless you check "plural conversion" box on the upload screen, and provide a conversion matrix
  (i'm thinking a very simple "0:0 1:0 2:1 3:2" format for expressing Lithuanian expansion mentioned in #901)

And again: It would be best, if Webplate don't hardcode and don't change header at all – respect, accept, and use as user upload, like other editors – e.g. Lokalize.

I will change (re-arrange) plural rules in trunk as sone as Leap 42.1 translations will be branched from trunk. So using one rules for one openSUSE version and another rules for other version will be bad using hardcoded rules in Webplate.

It would be best, if Webplate will only check if number of plural forms is same in both: header and body of PO file

@embar-: Is there any reason, why a single language should have more plural forms in one repository? It would only complicate translators' work: e. g. always keep in mind, that I am now translating project x, that have a reversed plural form and I have to enter plurals in reverse order.

IMHO the best matching plural form for particular language should be defined globally, and all projects should migrate to it. (This is the way how it works just now, just import with different plural form does not work properly.)

No, only checking number of plurals is not sufficient. I seen languages that use several plural forms with the same number of plurals, but different order of plurals.

@matejcik: Please add a checkbox to the import: “Mark strings as fuzzy after conversion”, eventually checkbox for Weblate internal “Check needed” mark (first will exclude imported strings with plurals unless somebody will review them, second will use them, but will be displayed in “Check needed” section.

Because now openSUSE Lithuanian translations use plural header, that is not used by any other project (though at introducing it, I thought It is same as KDE). 

Most popular plural header for Lithuanian project has 3 plural forms, but this number is insufficient, because of numbers like 21, 31, 41, 51... E.g. plural strings without precize number like in English "There is an update" vs "There is some updates", will be the same displayed for 1, 21, 31, 41 (i.e. displayed incorrectly).

I would like to unify header with used in KDE team nowadays (4 plural forms, but different order). This form (compared to currently used for openSUSE) is superior, because 2nd and 3rd form is identical to most popular header, so accidental replacing header is not so harmful.

@embar-: Do I understand correctly that Lithuanian language uses a different form for 1, 2, 3 and 21.
Then it apparently cannot be covered by 3 forms plural, and files should be converted to 4-forms plural.

Theoretically (without applying to software localization), 3 plural forms is enough for Lithuania:
1) 1, 21, 31, 41, 51, 61 ...
2) 2...9, 22...29, 32...39, 42...49 ...
3) 10...20, 30, 40, 50, 60...

But practically, some software strings in "1)" case must be translated differently:
1.1) as the only ONE thing in mind
1.2) as more than one –  21, 31, 41, 51, 61 ...

P.S., in replaying to Stanislav comment 

> It would only complicate translators' work: e. g. always keep in mind, that I am now translating project x, that have a reversed plural form and I have to enter plurals in reverse order."

The is (still) only ONE active translator into Lithuanian language for openSUSE... 

> No, only checking number of plurals is not sufficient. I seen languages that use several plural forms with the same number of plurals, but different order of plurals.

Please, make exception for Lithuanian :)

Another suggested scenario:

For Lithuanian, accept two hardcoded rules:
a) Historical
"Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && "
"(n%100<10 || n%100>=20) ? 1 : 2;\n"
b) Advanced
"Plural-Forms: nplurals=4; plural=(n==1 ? 0 : n%10>=2 && (n%100<10 || n"
"%100>=20) ? 1 : n%10==0 || (n%100>10 && n%100<20) ? 2 : 3);\n"

If uploaded header is (as in openSUSE Leap and trunk)
"Plural-Forms: nplurals=4; plural=(n==1 ? 0 : n%10==1 && n%100!=11 ? 1 : n%"
"10>=2 && (n%100<10 || n%100>=20) ? 2 : 3);\n"
then please convert to "advanced" (by automagically rearranging translations in body of PO file, and replacing header)

otherwise mark plural strings as fuzzy and rewrite header to "historical" (if 3 forms in uploaded header) or to "advanced"  (if 4 forms in uploaded header).

@embar-: Well, now I understand the 3 or 4 plurals problem. Both are valid for Lithuanian. Most project can live with 3 forms, and only in some cases 4 forms are needed.

Current Weblate implementation heavily depends on a single plural form for each language. Changing this rule will require more changes.

So the simplest (for implementation) would be conversion of all po files on initial import, and then use 4-form plural for all projects.

It's always wise to check the whole plural form. If you consider it as safe, you can always select identity as mapping and uncheck both “Make fuzzy” and “Need review”. (And if it will be added to API, you can make scripts for it.)

However I agree, that there are disadvantages:
• Say 9 of 10 projects could live with 3-forms plural, and 4-forms plural would be rarely needed. Now Weblate will force 4 forms even if is needed for a single project.
• 99.9% of string will have the same form for 1 and 21. It will make editing less comfortable. (Well, I can imagine that filling of form for 1 will pre-fill also form for 21 with the same, but even it will not help for people using external editor.)

But also an advantage:
• It would be easier to translate, if all project will use the same form.
• If Weblate will support both forms at once, there it could happen that a particular project lived with 3 plural forms, but upstream added a string that needs 4 plural forms. What now? Nowaday there is no tool for plural remapping. => Weblate will need a button: “Migrate to another plural form.”

Note @matejcik: Reverse conversion (4 to 3) would require not only simple matrix, but a matrix with comparison, e. g.:
Map oldp0 to newp0
If (oldp1 == oldp3) then map oldp1 to newp1 else map oldp1 to newp1 and mark fuzzy
Map oldp2 to newp2
Discard oldp3

By the way, Czech language had a similar problem up to 1993. It had 3 plural forms:
one for 1, 21, 31,… 91, 101, 121,…
another for 2, 3, 4, 22, 23, 24,… 104, 122, 123,…
and yet another for 5, 6, 7, 8, 9, 10, 11,… 19, 25,…

As it was not used by people, Institute of the Czech Language changed it in 1993 to a simpler form.

So it was abandoned earlier than translators encountered the problem that both 1 and 21 need singular form, but in some cases, 1 needs a different sentence than 21.

ok, so
i'm going to go ahead and solve this _bug_ by disallowing upload with mismatched number of plurals.

and will consider the proper handling of Lithuaninan (and similar) as a separate feature, for which probably #901 is more appropriate ;) and maybe work on that at some other time


Resolving patch:
diff --git a/weblate/trans/models/translation.py b/weblate/trans/models/translation.py
--- a/weblate/trans/models/translation.py
+++ b/weblate/trans/models/translation.py
@@ -1200,6 +1200,13 @@ def merge_upload(self, request, fileobj, overwrite, author=None,
         if author is None:
             author = get_author_name(request.user)
 
+        # Check valid plural forms
+        if hasattr(store.store, 'parseheader'):
+            header = store.store.parseheader()
+            if 'Plural-Forms' in header and \
+                    self.language.get_plural_form() != header['Plural-Forms']:
+                raise Exception('Plural forms do not match the language.')
+
         # List translations we should process
         # Filter out those who don't want automatic update, but keep ourselves
         translations = Translation.objects.filter(


Test patch:
diff --git a/weblate/trans/tests/data/cs-badplurals.po b/weblate/trans/tests/data/cs-badplurals.po
new file mode 100644
--- /dev/null
+++ b/weblate/trans/tests/data/cs-badplurals.po
@@ -0,0 +1,42 @@
+# Czech translations for PACKAGE package.
+# Copyright (C) 2012 THE PACKAGE'S COPYRIGHT HOLDER
+# This file is distributed under the same license as the PACKAGE package.
+# Automatically generated, 2012.
+# Testing Weblate, 2015.
+#
+msgid ""
+msgstr ""
+"Project-Id-Version: Weblate Hello World 2012\n"
+"Report-Msgid-Bugs-To: <noreply@example.net>\n"
+"POT-Creation-Date: 2012-03-14 15:54+0100\n"
+"PO-Revision-Date: 2012-03-05 15:55+0100\n"
+"Last-Translator: Automatically generated\n"
+"Language-Team: Test Team <noreply@weblate.org>\n"
+"Language: cs\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Plural-Forms: nplurals=4; plural=(n==1) ? 0 : (n>=2 && n<=4) ? 1 : (n % 10 == 1) ? 3 : 2;\n"
+
+#: main.c:11
+#, c-format
+msgid "Hello, world!\n"
+msgstr "Ahoj světe!\n"
+
+#: main.c:12
+#, c-format
+msgid "Orangutan has %d banana.\n"
+msgid_plural "Orangutan has %d bananas.\n"
+msgstr[0] ""
+msgstr[1] ""
+msgstr[2] ""
+msgstr[3] ""
+
+#: main.c:13
+#, c-format
+msgid "Try Weblate at <http://demo.weblate.org/>!\n"
+msgstr ""
+
+#: main.c:14
+msgid "Thank you for using Weblate."
+msgstr ""
diff --git a/weblate/trans/tests/test_files.py b/weblate/trans/tests/test_files.py
--- a/weblate/trans/tests/test_files.py
+++ b/weblate/trans/tests/test_files.py
@@ -33,6 +33,7 @@
 TEST_CSV = get_test_file('cs.csv')
 TEST_PO_BOM = get_test_file('cs-bom.po')
 TEST_FUZZY_PO = get_test_file('cs-fuzzy.po')
+TEST_BADPLURALS = get_test_file('cs-badplurals.po')
 TEST_MO = get_test_file('cs.mo')
 TEST_ANDROID = get_test_file('strings-cs.xml')
 
@@ -52,11 +53,14 @@ def setUp(self):
         self.user.is_superuser = True
         self.user.save()
 
-    def do_import(self, **kwargs):
+    def do_import(self, test_file=None, follow=False, **kwargs):
         '''
         Helper to perform file import.
         '''
-        with open(self.test_file) as handle:
+        if test_file is None:
+            test_file = self.test_file
+
+        with open(test_file) as handle:
             params = {'file': handle}
             params.update(kwargs)
             return self.client.post(
@@ -64,7 +68,8 @@ def do_import(self, **kwargs):
                     'upload_translation',
                     kwargs=self.kw_translation
                 ),
-                params
+                params,
+                follow=follow
             )
 
 
@@ -209,6 +214,24 @@ def test_import_suggest(self):
         )
 
 
+class ImportErrorTest(ImportBaseTest):
+    '''
+    Testing import of broken files.
+    '''
+    def test_mismatched_plurals(self):
+        '''
+        Test importing a file with different number of plural forms.
+        In response to issue #900
+        '''
+        from django.contrib.messages import ERROR
+        response = self.do_import(test_file=TEST_BADPLURALS, follow=True)
+        self.assertRedirects(response, self.translation_url)
+        messages = list(response.context["messages"])
+        self.assertEquals(len(messages), 1)
+        self.assertEquals(messages[0].level, ERROR)
+        self.assertIn("Plural forms do not match", messages[0].message)
+
+
 class BOMImportTest(ImportTest):
     test_file = TEST_PO_BOM