Repository: i18next/i18next-parser
Pull request #413 (2021-09-10 12:22:16)

Escape non-printable Unicode characters
Revert d3858b6dd1399cf9bf2fa6c36de93f7bab693380

Changed #362 to escape only non-printable Unicode character.

### Why am I submitting this PR

Enhanced version of #362. Escaping only Unicode Separators `\p{Z}`, Control `\p{Cc}` and Format `\p{Cf}` characters ([doc](https://unicode.org/reports/tr18/#General_Category_Property)).

See also: [node.green](https://node.green/#ES2018-features--RegExp-Unicode-Property-Escapes), [caniuse](https://caniuse.com/mdn-javascript_builtins_regexp_property_escapes).

Tested also in proprietary project with Cyrillic translation files.
@kobrynsky, @damien-monni, @bodryi (menthioned from #362), feel free to test it in your projects too.

### Does it fix an existing ticket?

Yes #361

### Checklist

- [x] only relevant code is changed (make a diff before you submit the PR)
- [x] tests are included and pass (see [details here](https://github.com/i18next/i18next-parser/blob/master/docs/development.md#tests))
- [ ] documentation is changed or added
- [ ] I ran `yarn build` to compile and prettify the code


codecov-commenter: # [Codecov](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next) Report
> Merging [#413](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next) (8810f65) into [master](https://codecov.io/gh/i18next/i18next-parser/commit/618708a13a3b685091a2b5fb07f71bfddce051b1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next) (618708a) will **increase** coverage by `0.03%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/i18next/i18next-parser/pull/413/graphs/tree.svg?width=650&height=150&src=pr&token=CJ74Vps41L&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next)](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next)

```diff
@@            Coverage Diff             @@
##           master     #413      +/-   ##
==========================================
+ Coverage   93.09%   93.12%   +0.03%     
==========================================
  Files           9        9              
  Lines         608      611       +3     
==========================================
+ Hits          566      569       +3     
  Misses         42       42              
```


| [Impacted Files](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next) | Coverage Δ | |
|---|---|---|
| [src/transform.js](https://codecov.io/gh/i18next/i18next-parser/pull/413/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next#diff-c3JjL3RyYW5zZm9ybS5qcw==) | `81.21% <100.00%> (+0.34%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next). Last update [618708a...8810f65](https://codecov.io/gh/i18next/i18next-parser/pull/413?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=i18next).

karellm: Thanks for the PR. I have tested it quickly and it seems to work. I will deploy this as `4.5.0`

Patch:
diff --git a/src/transform.js b/src/transform.js
index 7436d14..6910302 100644
--- a/src/transform.js
+++ b/src/transform.js
@@ -294,6 +294,12 @@ export default class i18nTransform extends Transform {
       })
     } else {
       text = JSON.stringify(contents, null, this.options.indentation) + '\n'
+      // Convert non-printable Unicode characters to unicode escape sequence
+      // https://unicode.org/reports/tr18/#General_Category_Property
+      text = text.replace(/[\p{Z}\p{Cc}\p{Cf}]/gu, (chr) => {
+        const n = chr.charCodeAt(0)
+        return n < 128 ? chr : `\\u${`0000${n.toString(16)}`.substr(-4)}`
+      })
     }
 
     if (this.options.lineEnding === 'auto') {


Test patch:
diff --git a/test/parser.test.js b/test/parser.test.js
index af268fa..172cb56 100644
--- a/test/parser.test.js
+++ b/test/parser.test.js
@@ -990,6 +990,31 @@ describe('parser', () => {
       i18nextParser.end(fakeFile)
     })
 
+    it('writes non-breaking space output to json', (done) => {
+      let result
+      const expected = '{\n  "nbsp": "Oné\\u00a0Twó\\u3000Three Four"\n}\n'
+
+      const i18nextParser = new i18nTransform({
+        output: 'locales/$LOCALE/$NAMESPACE.json',
+      })
+      const fakeFile = new Vinyl({
+        contents: Buffer.from("t('nbsp', 'Oné\\u00A0Twó\\u3000Three Four')"),
+        path: 'file.js',
+      })
+
+      i18nextParser.on('data', (file) => {
+        if (file.relative.endsWith(path.normalize('en/translation.json'))) {
+          result = file.contents.toString('utf8')
+        }
+      })
+      i18nextParser.once('end', () => {
+        assert.equal(result, expected)
+        done()
+      })
+
+      i18nextParser.end(fakeFile)
+    })
+
     it('supports outputing to yml', (done) => {
       let result
       const i18nextParser = new i18nTransform({