Repository: i18next/i18next-parser
Pull request #222 (2020-06-25 11:25:36)

Extract translations from comments
Adds support for extracting translations from comments. Relevant for: #83 #215

#### Checklist

- [x] only relevant code is changed (make a diff before you submit the PR)
- [x] run tests `npm run test`
- [x] tests are included
- [x] documentation is changed or added

karellm: I've merge your other PR and it's not fast-forward anymore. Do you mind rebasing?
vincaslt: Yeah, no problem

Patch:
diff --git a/README.md b/README.md
index f37c0ea..1e33f33 100644
--- a/README.md
+++ b/README.md
@@ -19,6 +19,7 @@ Finally, if you want to make this process even less painful, I invite you to che
 - Creates one catalog file per locale and per namespace
 - Backs up the old keys your code doesn't use anymore in `namespace_old.json` catalog
 - Restores keys from the `_old` file if the one in the translation file is empty
+- Parses comments for static keys to support dynamic key translations.
 - Supports i18next features:
   - **Context**: keys of the form `key_context`
   - **Plural**: keys of the form `key_plural` and `key_0`, `key_1` as described [here](https://www.i18next.com/translation-function/plurals)
@@ -28,7 +29,6 @@ Finally, if you want to make this process even less painful, I invite you to che
 
 `1.x` has been in beta for a good while. You can follow the pre-releases [here](https://github.com/i18next/i18next-parser/releases). It is a deep rewrite of this package that solves many issues, the main one being that it was slowly becoming unmaintainable. The [migration](docs/migration.md) from `0.x` contains all the breaking changes. Everything that follows is related to `1.x`. You can still find the old `0.x` documentation on its dedicated [branch](https://github.com/i18next/i18next-parser/tree/0.x.x).
 
-
 ## Usage
 
 ### CLI
@@ -87,9 +87,8 @@ npm install --save-dev i18next-parser
 [Broccoli.js](https://github.com/broccolijs/broccoli) defines itself as a fast, reliable asset pipeline, supporting constant-time rebuilds and compact build definitions.
 
 ```javascript
-
 const Funnel = require('broccoli-funnel')
-const i18nextParser = require('i18next-parser').broccoli;
+const i18nextParser = require('i18next-parser').broccoli
 
 const appRoot = 'broccoli'
 
@@ -188,7 +187,7 @@ module.exports = {
   verbose: false,
   // Display info about the parsing including some stats
 
-  customValueTemplate: null,
+  customValueTemplate: null
   // If you wish to customize the value output the value as an object, you can set your own format.
   // ${defaultValue} is the default value you set in your translation function.
   // Any other custom property will be automatically extracted.
@@ -211,47 +210,8 @@ There are 4 lexers available: `HandlebarsLexer`, `HTMLLexer`, `JavascriptLexer`
 If you need to change the defaults, you can do it like so:
 
 #### Javascript
-The Javascript lexer uses [Acorn](https://github.com/acornjs/acorn) to walk through your code and extract references
-translation functions. If your code uses features not supported natively by Acorn, you can enable support through
-`injectors` and `plugins` configuration. Note that you must install these additional dependencies yourself through
-`yarn` or `npm`; they are not included in this package. This is an example configuration that adds all non-jsx plugins supported by acorn
-at the time of writing:
-```javascript
-const injectAcornStaticClassPropertyInitializer = require('acorn-static-class-property-initializer/inject');
-const injectAcornStage3 = require('acorn-stage3/inject');
-const injectAcornEs7 = require('acorn-es7');
-
-// ...
-  js: [{
-    lexer: 'JavascriptLexer'
-    functions: ['t'], // Array of functions to match
-
-    // acorn config (for more information on the acorn options, see here: https://github.com/acornjs/acorn#main-parser)
-    acorn: {
-      injectors: [
-          injectAcornStaticClassPropertyInitializer,
-          injectAcornStage3,
-          injectAcornEs7,
-        ],
-      plugins: {
-        // The presence of these plugin options is important -
-        // without them, the plugins will be available but not
-        // enabled.
-        staticClassPropertyInitializer: true,
-        stage3: true,
-        es7: true,
-      }
-    }
-  }],
-// ...
-```
 
-If you receive an error that looks like this:
-```bash
-TypeError: baseVisitor[type] is not a function
-# rest of stack trace...
-```
-The problem is likely that you are missing a plugin that supports a feature that your code uses.
+The Javascript lexer uses [Typescript compiler](https://github.com/microsoft/TypeScript/wiki/Using-the-Compiler-API) to walk through your code and extract translation functions.
 
 The default configuration is below:
 
@@ -262,27 +222,16 @@ The default configuration is below:
     lexer: 'JavascriptLexer'
     functions: ['t'], // Array of functions to match
 
-    // acorn config (for more information on the acorn options, see here: https://github.com/acornjs/acorn#main-parser)
-    acorn: {
-      sourceType: 'module',
-      ecmaVersion: 9, // forward compatibility
-      // Allows additional acorn plugins via the exported injector functions
-      injectors: [],
-      plugins: {},
-    }
   }],
 }
 ```
 
 #### Jsx
-The JSX lexer builds off of the Javascript lexer, and additionally requires the `acorn-jsx` plugin. To use it, add `acorn-jsx` to your dev dependencies:
-```bash
-npm install -D acorn-jsx
-# or
-yarn add -D acorn-jsx@4.1.1
-```
+
+The JSX lexer builds off of the Javascript lexer and extends it with support for JSX syntax.
 
 Default configuration:
+
 ```js
 {
   // JsxLexer default config (jsx)
@@ -290,31 +239,30 @@ Default configuration:
   jsx: [{
     lexer: 'JsxLexer',
     attr: 'i18nKey', // Attribute for the keys
-
-    // acorn config (for more information on the acorn options, see here: https://github.com/acornjs/acorn#main-parser)
-    acorn: {
-      sourceType: 'module',
-      ecmaVersion: 9, // forward compatibility
-      injectors: [],
-      plugins: {},
-    }
   }],
 }
 ```
+
 #### Ts(x)
+
 Typescript is supported via Javascript and Jsx lexers. If you are using Javascript syntax (e.g. with React), follow the steps in Jsx section, otherwise Javascript section.
 
 #### Handlebars
+
 ```js
 {
   // HandlebarsLexer default config (hbs, handlebars)
-  handlebars: [{
-    lexer: 'HandlebarsLexer',
-    functions: ['t'] // Array of functions to match
-  }]
+  handlebars: [
+    {
+      lexer: 'HandlebarsLexer',
+      functions: ['t'] // Array of functions to match
+    }
+  ]
 }
 ```
+
 #### Html
+
 ```js
 {
   // HtmlLexer default config (htm, html)
@@ -327,7 +275,9 @@ Typescript is supported via Javascript and Jsx lexers. If you are using Javascri
 ```
 
 #### Custom lexers
+
 You can provide function instead of string as a custom lexer.
+
 ```js
 const CustomJsLexer = require('./CustomJsLexer');
 
@@ -342,6 +292,30 @@ const CustomJsLexer = require('./CustomJsLexer');
 // ...
 ```
 
+### Caveats
+
+While i18next extracts translation keys in runtime, i18next-parser doesn't run the code, so it can't interpolate values in these expressions:
+
+```
+t(key)
+t('key' + id)
+t(`key${id}`)
+```
+
+As a workaround you should specify possible static values in comments anywhere in your file:
+
+```
+// t('key_1')
+// t('key_2')
+t(key)
+
+/*
+t('key1')
+t('key2')
+*/
+t('key' + id)
+```
+
 ## Events
 
 The transform emits a `reading` event for each file it parses:
@@ -356,8 +330,6 @@ The transform emits a `warning:variable` event if the file has a key that contai
 
 `.pipe( i18next().on('warning:variable', (path, key) => {}) )`
 
-
-
 ## Contribute
 
 Any contribution is welcome. Please [read the guidelines](docs/development.md) first.
@@ -372,7 +344,7 @@ If you use this package and like it, supporting me on [Patreon](https://www.patr
   </a>
 </p>
 
---------------
+---
 
 ## Gold Sponsors
 
diff --git a/dist/lexers/javascript-lexer.js b/dist/lexers/javascript-lexer.js
index 8e30ecc..4ece9f5 100644
--- a/dist/lexers/javascript-lexer.js
+++ b/dist/lexers/javascript-lexer.js
@@ -1,20 +1,42 @@
 'use strict';Object.defineProperty(exports, "__esModule", { value: true });var _typeof = typeof Symbol === "function" && typeof Symbol.iterator === "symbol" ? function (obj) {return typeof obj;} : function (obj) {return obj && typeof Symbol === "function" && obj.constructor === Symbol && obj !== Symbol.prototype ? "symbol" : typeof obj;};var _createClass = function () {function defineProperties(target, props) {for (var i = 0; i < props.length; i++) {var descriptor = props[i];descriptor.enumerable = descriptor.enumerable || false;descriptor.configurable = true;if ("value" in descriptor) descriptor.writable = true;Object.defineProperty(target, descriptor.key, descriptor);}}return function (Constructor, protoProps, staticProps) {if (protoProps) defineProperties(Constructor.prototype, protoProps);if (staticProps) defineProperties(Constructor, staticProps);return Constructor;};}();var _baseLexer = require('./base-lexer');var _baseLexer2 = _interopRequireDefault(_baseLexer);
-var _typescript = require('typescript');var ts = _interopRequireWildcard(_typescript);function _interopRequireWildcard(obj) {if (obj && obj.__esModule) {return obj;} else {var newObj = {};if (obj != null) {for (var key in obj) {if (Object.prototype.hasOwnProperty.call(obj, key)) newObj[key] = obj[key];}}newObj.default = obj;return newObj;}}function _interopRequireDefault(obj) {return obj && obj.__esModule ? obj : { default: obj };}function _classCallCheck(instance, Constructor) {if (!(instance instanceof Constructor)) {throw new TypeError("Cannot call a class as a function");}}function _possibleConstructorReturn(self, call) {if (!self) {throw new ReferenceError("this hasn't been initialised - super() hasn't been called");}return call && (typeof call === "object" || typeof call === "function") ? call : self;}function _inherits(subClass, superClass) {if (typeof superClass !== "function" && superClass !== null) {throw new TypeError("Super expression must either be null or a function, not " + typeof superClass);}subClass.prototype = Object.create(superClass && superClass.prototype, { constructor: { value: subClass, enumerable: false, writable: true, configurable: true } });if (superClass) Object.setPrototypeOf ? Object.setPrototypeOf(subClass, superClass) : subClass.__proto__ = superClass;}var
+var _typescript = require('typescript');var ts = _interopRequireWildcard(_typescript);function _interopRequireWildcard(obj) {if (obj && obj.__esModule) {return obj;} else {var newObj = {};if (obj != null) {for (var key in obj) {if (Object.prototype.hasOwnProperty.call(obj, key)) newObj[key] = obj[key];}}newObj.default = obj;return newObj;}}function _interopRequireDefault(obj) {return obj && obj.__esModule ? obj : { default: obj };}function _toConsumableArray(arr) {if (Array.isArray(arr)) {for (var i = 0, arr2 = Array(arr.length); i < arr.length; i++) {arr2[i] = arr[i];}return arr2;} else {return Array.from(arr);}}function _classCallCheck(instance, Constructor) {if (!(instance instanceof Constructor)) {throw new TypeError("Cannot call a class as a function");}}function _possibleConstructorReturn(self, call) {if (!self) {throw new ReferenceError("this hasn't been initialised - super() hasn't been called");}return call && (typeof call === "object" || typeof call === "function") ? call : self;}function _inherits(subClass, superClass) {if (typeof superClass !== "function" && superClass !== null) {throw new TypeError("Super expression must either be null or a function, not " + typeof superClass);}subClass.prototype = Object.create(superClass && superClass.prototype, { constructor: { value: subClass, enumerable: false, writable: true, configurable: true } });if (superClass) Object.setPrototypeOf ? Object.setPrototypeOf(subClass, superClass) : subClass.__proto__ = superClass;}var
 
 JavascriptLexer = function (_BaseLexer) {_inherits(JavascriptLexer, _BaseLexer);
   function JavascriptLexer() {var options = arguments.length > 0 && arguments[0] !== undefined ? arguments[0] : {};_classCallCheck(this, JavascriptLexer);var _this = _possibleConstructorReturn(this, (JavascriptLexer.__proto__ || Object.getPrototypeOf(JavascriptLexer)).call(this,
     options));
 
+    _this.callPattern = '(?<=^|\\s|\\.)' + _this.functionPattern() + '\\(.*\\)';
     _this.functions = options.functions || ['t'];
     _this.attr = options.attr || 'i18nKey';return _this;
   }_createClass(JavascriptLexer, [{ key: 'extract', value: function extract(
 
     content) {var _this2 = this;var filename = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : '__default.js';
+      var visitedComments = new Set();
       var keys = [];
 
       var parseTree = function parseTree(node) {
         var entry = void 0;
 
+        ts.forEachLeadingCommentRange(
+        content,
+        node.getFullStart(),
+        function (pos, end, kind) {
+          var commentId = pos + '_' + end;
+          if (
+          (kind === ts.SyntaxKind.MultiLineCommentTrivia ||
+          kind === ts.SyntaxKind.SingleLineCommentTrivia) &&
+          !visitedComments.has(commentId))
+          {
+            visitedComments.add(commentId);
+            var text = content.slice(pos, end);
+            var commentKeys = _this2.commentExtractor.call(_this2, text);
+            if (commentKeys) {
+              keys.push.apply(keys, _toConsumableArray(commentKeys));
+            }
+          }
+        });
+
+
         if (node.kind === ts.SyntaxKind.CallExpression) {
           entry = _this2.expressionExtractor.call(_this2, node);
         }
@@ -125,6 +147,24 @@ JavascriptLexer = function (_BaseLexer) {_inherits(JavascriptLexer, _BaseLexer);
       }
 
       return null;
+    } }, { key: 'commentExtractor', value: function commentExtractor(
+
+    commentText) {var _this3 = this;
+      var regexp = new RegExp(this.callPattern, 'g');
+      var expressions = commentText.match(regexp);
+
+      if (!expressions) {
+        return null;
+      }
+
+      var keys = [];
+      expressions.forEach(function (expression) {
+        var expressionKeys = _this3.extract(expression);
+        if (expressionKeys) {
+          keys.push.apply(keys, _toConsumableArray(expressionKeys));
+        }
+      });
+      return keys;
     } }, { key: 'concatenateString', value: function concatenateString(
 
     binaryExpression) {var string = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : '';
diff --git a/src/lexers/javascript-lexer.js b/src/lexers/javascript-lexer.js
index dc36606..f92d81b 100644
--- a/src/lexers/javascript-lexer.js
+++ b/src/lexers/javascript-lexer.js
@@ -5,16 +5,38 @@ export default class JavascriptLexer extends BaseLexer {
   constructor(options = {}) {
     super(options)
 
+    this.callPattern = '(?<=^|\\s|\\.)' + this.functionPattern() + '\\(.*\\)'
     this.functions = options.functions || ['t']
     this.attr = options.attr || 'i18nKey'
   }
 
   extract(content, filename = '__default.js') {
+    const visitedComments = new Set()
     const keys = []
 
     const parseTree = (node) => {
       let entry
 
+      ts.forEachLeadingCommentRange(
+        content,
+        node.getFullStart(),
+        (pos, end, kind) => {
+          const commentId = `${pos}_${end}`
+          if (
+            (kind === ts.SyntaxKind.MultiLineCommentTrivia ||
+              kind === ts.SyntaxKind.SingleLineCommentTrivia) &&
+            !visitedComments.has(commentId)
+          ) {
+            visitedComments.add(commentId)
+            const text = content.slice(pos, end)
+            const commentKeys = this.commentExtractor.call(this, text)
+            if (commentKeys) {
+              keys.push(...commentKeys)
+            }
+          }
+        }
+      )
+
       if (node.kind === ts.SyntaxKind.CallExpression) {
         entry = this.expressionExtractor.call(this, node)
       }
@@ -127,6 +149,24 @@ export default class JavascriptLexer extends BaseLexer {
     return null
   }
 
+  commentExtractor(commentText) {
+    const regexp = new RegExp(this.callPattern, 'g')
+    const expressions = commentText.match(regexp)
+
+    if (!expressions) {
+      return null
+    }
+
+    const keys = []
+    expressions.forEach((expression) => {
+      const expressionKeys = this.extract(expression)
+      if (expressionKeys) {
+        keys.push(...expressionKeys)
+      }
+    })
+    return keys
+  }
+
   concatenateString(binaryExpression, string = '') {
     if (binaryExpression.operatorToken.kind !== ts.SyntaxKind.PlusToken) {
       return


Test patch:
diff --git a/test/lexers/javascript-lexer.test.js b/test/lexers/javascript-lexer.test.js
index f2bd878..6766bdd 100644
--- a/test/lexers/javascript-lexer.test.js
+++ b/test/lexers/javascript-lexer.test.js
@@ -71,6 +71,43 @@ describe('JavascriptLexer', () => {
     done()
   })
 
+  it('extracts keys from single line comments', (done) => {
+    const Lexer = new JavascriptLexer()
+    const content = `
+    // i18n.t('commentKey1')
+    i18n.t('commentKey' + i)
+    // i18n.t('commentKey2')
+    i18n.t(\`commentKey\${i}\`)
+    // Irrelevant comment
+    // i18n.t('commentKey3')
+    `
+    assert.deepEqual(Lexer.extract(content), [
+      { key: 'commentKey1' },
+      { key: 'commentKey2' },
+      { key: 'commentKey3' },
+    ])
+    done()
+  })
+
+  it('extracts keys from multiline comments', (done) => {
+    const Lexer = new JavascriptLexer()
+    const content = `
+    /*
+      i18n.t('commentKey1')
+      i18n.t('commentKey2')
+    */
+    i18n.t(\`commentKey\${i}\`)
+    // Irrelevant comment
+    /* i18n.t('commentKey3') */
+    `
+    assert.deepEqual(Lexer.extract(content), [
+      { key: 'commentKey1' },
+      { key: 'commentKey2' },
+      { key: 'commentKey3' },
+    ])
+    done()
+  })
+
   it("does not parse text with `doesn't` or isolated `t` in it", (done) => {
     const Lexer = new JavascriptLexer()
     const js =