Repository: translate/translate

Issue:
json2po output strings in wrong order
Running `json2po` outputs a valid PO files, but the strings are in the wrong order. I’m not sure exactly _which_ order it is, but is _neither_ the original order, the reverse order or alphabetically. Here’s an example JSON file:

``` json
{
    "foo": "foo",
    "bar": "bar",
    "baz": "baz"
}
```

Running `json2po` on this file results in the following (with a PO file header):

``` po
#: .baz
msgid "baz"
msgstr ""

#: .foo
msgid "foo"
msgstr ""

#: .bar
msgid "bar"
msgstr ""
```

Having the strings in a different order than in the original file makes it much harder to correctly translate the file, as one loses context.

I observe this bug both in version 1.13.0 and 1.14.0-rc1.



Discussion:
I think this might be quite hard to fix, reasons follow.

There is no real ordering in JSON, it's just a data structure.  From the Python side its just a dictionary, which has no order.  We rely on a JSON parser to help us here and to most programmers the ordering isn't important at all.  The parser might expose some way for us to get such ordering, things like [OrderedDict](https://docs.python.org/2/library/collections.html#collections.OrderedDict) do exist in Python.

From a translator's perspective the change in ordering is really problematic.  But writing our own parser is really not an option.  Note to programmers, use proper localisation formats, JSON isn't one of those.

@dwaynebailey decides to experiment a bit to confirm things.

Ouch, that was an embarrassingly easy fix.

Thanks @huftis for spotting this.  And thanks for a really nice simple example that replicates the issue, that saved me a lot of time.

Would appreciate if you could confirm the PR I'll post in a bit.

Getting the tests to pass though was not as easy.  This will only work in Python >= 2.7

Thank you! I can confirm that the fix works fine. (`po2json` has a similar issue, but I see that that’s already mentioned in the TODO list in the source code (`jsonl10n.py`).)


Resolving patch:
diff --git a/translate/storage/jsonl10n.py b/translate/storage/jsonl10n.py
--- a/translate/storage/jsonl10n.py
+++ b/translate/storage/jsonl10n.py
@@ -64,7 +64,6 @@
 
 - Handle ``\u`` and other escapes in Unicode
 - Manage data type storage and conversion. True --> "True" --> True
-- Sort the extracted data to the order of the JSON file
 
 """
 
@@ -159,11 +158,14 @@ def __init__(self, inputfile=None, filter=None, **kwargs):
             self.parse(inputfile)
 
     def serialize(self, out):
-        units = {}
+        if OrderedDict is not None:
+            units = OrderedDict()
+        else:
+            units = {}
         for unit in self.unit_iter():
             path = unit.getid().lstrip('.')
             units[path] = unit.target
-        out.write(json.dumps(units, sort_keys=True, separators=(',', ': '),
+        out.write(json.dumps(units, separators=(',', ': '),
                              indent=4, ensure_ascii=False).encode(self.encoding))
         out.write(b'\n')
 


Test patch:
diff --git a/translate/convert/test_po2json.py b/translate/convert/test_po2json.py
--- a/translate/convert/test_po2json.py
+++ b/translate/convert/test_po2json.py
@@ -1,4 +1,7 @@
 # -*- coding: utf-8 -*-
+import sys
+import pytest
+
 from translate.convert import po2json
 from translate.misc import wStringIO
 from translate.storage import po
@@ -25,6 +28,36 @@ def test_basic(self):
         expected_json = '''{
     "text": "Du texte simple"
 }
+'''
+        json_out = self.po2json(input_po, json_template)
+        assert json_out == expected_json
+
+    @pytest.mark.skipif(sys.version_info < (2, 7),
+                        reason="json.loads() can't order in Python 2.6")
+    def test_ordering_serialize(self):
+        json_template = '''{
+    "foo": "foo",
+    "bar": "bar",
+    "baz": "baz"
+}'''
+        input_po =  '''
+#: .foo
+msgid "foo"
+msgstr "oof"
+
+#: .bar
+msgid "bar"
+msgstr "rab"
+
+#: .baz
+msgid "baz"
+msgstr "zab"
+'''
+        expected_json = '''{
+    "foo": "oof",
+    "bar": "rab",
+    "baz": "zab"
+}
 '''
         json_out = self.po2json(input_po, json_template)
         assert json_out == expected_json