Repository: translate/translate

Issue:
WebExtension JSON is not actually JSON
The current parser for WebExtension JSON assumes that it js JSON based, however it's not as it expects preparsing to strip comments:

> Standards-compliant browsers are expected to ignore "//" comments. Disregarding comments, the manifest.json file format for browser extensions is expected to be fully JSON compliant.

See https://browserext.github.io/browserext/#manifest-json

Discussion:
@nijel: This is going to be tricky - there doesn't seem to be any Python parser which would allow round trip - all just strip comments and then use standard json library.@nschonni: Maybe https://pypi.org/project/json_tricks/@nijel: It doesn't support saving as well:

> Since comments aren't stored in the Python representation of the data, loading and then saving a json file will remove the comments (it also likely changes the indentation).@rugk: Adapted from https://github.com/WeblateOrg/weblate/issues/2043#issuecomment-393633466:

When importing:

* Convert comments to some special JSON field so it can be represented in JSON, e.g. "// comment 1 2 3" to "{idSpecialComment_comment_1_2_3}"
* Exclude them from translations and only use them behind-the-scenes.
* When exporting (i.e. converting back to JSON) just do the same vice versa.@rugk: Any news?@nijel: No news, if there would be any news it would be written here.@nijel: https://pypi.org/project/jsoncparser/ might be actually a viable solution (at least it claims it supports saving as well)@nijel: Looking at this again (after https://github.com/mozilla/moz-l10n/issues/47). I've realized that using jsonc is not the best approach either. It can also have `/* */` comments, so using jsonc parser for this would accept files which do not confirm to this *JSON and "//"-style comments* thingy. So let's just strip these comments on parsing and allow processing such files and discard them on saving. Something more fancy can always be implemented, but there is existing `description` syntax in the web extension translations to pass comments, so it really should not be used to pass information to translators.

Solution patch:
diff --git a/translate/storage/jsonl10n.py b/translate/storage/jsonl10n.py
index 74110493..1dca0dc2 100644
--- a/translate/storage/jsonl10n.py
+++ b/translate/storage/jsonl10n.py
@@ -68,8 +68,12 @@ Todo:
 
 """
 
+from __future__ import annotations
+
 import json
+import re
 import uuid
+from typing import BinaryIO, TextIO, cast
 
 from translate.lang.data import cldr_plural_categories, plural_tags
 from translate.misc.multistring import multistring
@@ -219,25 +223,36 @@ class JsonFile(base.DictStore):
             unit.set_unitid(prev)
             yield unit
 
-    def parse(self, input):
+    def preprocess_input(self, text: str) -> str:
+        return text
+
+    def parse(self, data: str | bytes | TextIO | BinaryIO):
         """Parse the given file or file source string."""
-        if hasattr(input, "name"):
-            self.filename = input.name
+        text: str | bytes
+        if hasattr(data, "name"):
+            self.filename = data.name
         elif not getattr(self, "filename", ""):
             self.filename = ""
-        if hasattr(input, "read"):
-            src = input.read()
-            input.close()
-            input = src
-        if isinstance(input, bytes):
+        if hasattr(data, "read"):
+            # Make type checking happy
+            data = cast("BinaryIO", data)
+
+            text = data.read()
+            data.close()
+        else:
+            text = data
+
+        if isinstance(text, bytes):
             # The JSON files should be UTF-8, but implementations
             # that parse JSON texts MAY ignore the presence of a byte order mark
             # rather than treating it as an error, see RFC7159
-            input, self.encoding = self.detect_encoding(input)
-            if input is None:
+            decoded, self.encoding = self.detect_encoding(text)
+            if decoded is None:
                 raise base.ParseError(ValueError("Failed to decode JSON string."))
+            text = decoded
+        text = self.preprocess_input(text)
         try:
-            self._file = json.loads(input)
+            self._file = json.loads(text)
         except ValueError as e:
             raise base.ParseError(e)
 
@@ -277,6 +292,11 @@ class WebExtensionJsonFile(JsonFile):
 
     UnitClass = WebExtensionJsonUnit
 
+    COMMENT_RE = re.compile(r"""^((?:[^"]|"(?:[^"\\]|\.)*")*?)//.*$""", re.MULTILINE)
+
+    def preprocess_input(self, text: str) -> str:
+        return self.COMMENT_RE.sub(r"\1", text)
+
     def _extract_units(
         self,
         data,


Test patch:
diff --git a/tests/translate/storage/test_jsonl10n.py b/tests/translate/storage/test_jsonl10n.py
index 1d3d271b..7aecd60e 100644
--- a/tests/translate/storage/test_jsonl10n.py
+++ b/tests/translate/storage/test_jsonl10n.py
@@ -843,6 +843,34 @@ class TestWebExtensionStore(test_monolingual.TestMonolingualStore):
 
         assert out.getvalue() == DATA
 
+    def test_comments(self):
+        DATA = """{
+    "test": {
+        // Comment
+        "message": "Message //"
+    },
+    "test//": { // Comment
+        "message": "Message // Other"
+    }
+}
+"""
+        store = self.StoreClass()
+        store.parse(DATA)
+        assert len(store.units) == 2
+        # Any comments will be stripped
+        assert (
+            bytes(store).decode()
+            == """{
+    "test": {
+        "message": "Message //"
+    },
+    "test//": {
+        "message": "Message // Other"
+    }
+}
+"""
+        )
+
 
 class TestI18NextStore(test_monolingual.TestMonolingualStore):
     StoreClass = jsonl10n.I18NextFile