Repository: lingui/js-lingui
Pull request #1065 (2021-05-18 09:40:09)

feat: configurable and extendable extractors with Lingui config
Now with this pull request users can create his own extractors:

```
type ExtractorType = {
  match(filename: string): boolean
  extract(filename: string, targetDir: string, options?: any): void
}
```

### Little breaking change
Before we had two extractors `[babel, typescript]`, the typescript step it's useless since babel support typescript practically out of the box, since this compiling it's done twice first by babel and then by typescript was causing a lot of failures on Typescript projects with bad origin's and strange errors about NODE_ENV.

Related issues: 
https://github.com/lingui/js-lingui/issues/1058
https://github.com/lingui/js-lingui/discussions/1047

Now if your package it's pure Typescript and doesn't contain Babel, you could just use typescript extractor instead of using the babel one.

```js
  extractors: [
    require.resolve("@lingui/cli/api/extractors/typescript")
  ],
```

- [x] Document this feature

vercel[bot]: This pull request is being automatically deployed with Vercel ([learn more](https://vercel.link/github-learn-more)).  
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: [https://vercel.com/lingui-js/js-lingui/8evJGbHAwdS1JHmdBvNZcYnjktRN](https://vercel.com/lingui-js/js-lingui/8evJGbHAwdS1JHmdBvNZcYnjktRN)  
✅ Preview: [https://js-lingui-git-feat-extractors-lingui-js.vercel.app](https://js-lingui-git-feat-extractors-lingui-js.vercel.app)


semoal: @nathanforce
github-actions[bot]: ## size-limit report 📦 
| Path                                                               | Size         |
| ------------------------------------------------------------------ | ------------ |
| ./packages/core/build/cjs/core.production.min.js                   | 2.78 KB (0%) |
| ./packages/detect-locale/build/cjs/detect-locale.production.min.js | 798 B (0%)   |
| ./packages/react/build/cjs/react.production.min.js                 | 4.85 KB (0%) |
codecov[bot]: # [Codecov](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui) Report
> Merging [#1065](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui) (4a9b4c9) into [main](https://codecov.io/gh/lingui/js-lingui/commit/78ad09fc88e8ec07daefb455457e778471977f2f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui) (78ad09f) will **increase** coverage by `0.14%`.
> The diff coverage is `80.00%`.

[![Impacted file tree graph](https://codecov.io/gh/lingui/js-lingui/pull/1065/graphs/tree.svg?width=650&height=150&src=pr&token=CuJ1VD5BY5&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui)](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui)

```diff
@@            Coverage Diff             @@
##             main    #1065      +/-   ##
==========================================
+ Coverage   83.06%   83.20%   +0.14%     
==========================================
  Files          53       53              
  Lines        1624     1620       -4     
  Branches      443      441       -2     
==========================================
- Hits         1349     1348       -1     
- Misses        158      159       +1     
+ Partials      117      113       -4     
```


| [Impacted Files](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui) | Coverage Δ | |
|---|---|---|
| [packages/cli/src/api/catalog.ts](https://codecov.io/gh/lingui/js-lingui/pull/1065/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui#diff-cGFja2FnZXMvY2xpL3NyYy9hcGkvY2F0YWxvZy50cw==) | `81.56% <ø> (ø)` | |
| [packages/conf/src/index.ts](https://codecov.io/gh/lingui/js-lingui/pull/1065/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui#diff-cGFja2FnZXMvY29uZi9zcmMvaW5kZXgudHM=) | `80.76% <ø> (ø)` | |
| [packages/cli/src/api/extract.ts](https://codecov.io/gh/lingui/js-lingui/pull/1065/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui#diff-cGFja2FnZXMvY2xpL3NyYy9hcGkvZXh0cmFjdC50cw==) | `88.46% <50.00%> (+4.59%)` | :arrow_up: |
| [packages/cli/src/api/extractors/index.ts](https://codecov.io/gh/lingui/js-lingui/pull/1065/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui#diff-cGFja2FnZXMvY2xpL3NyYy9hcGkvZXh0cmFjdG9ycy9pbmRleC50cw==) | `69.23% <100.00%> (+10.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui). Last update [ff7a235...4a9b4c9](https://codecov.io/gh/lingui/js-lingui/pull/1065?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lingui).


Patch:
diff --git a/docs/ref/conf.rst b/docs/ref/conf.rst
index 9403d981..0120aebf 100644
--- a/docs/ref/conf.rst
+++ b/docs/ref/conf.rst
@@ -29,6 +29,7 @@ Default config:
     "fallbackLocales": {},
     "format": "po",
     "locales": [],
+    "extractors": ["babel"],
     "orderBy": "messageId",
     "pseudoLocale": "",
     "rootDir": ".",
@@ -506,3 +507,26 @@ providing custom translation.
 The difference between :conf:`fallbackLocales` and :conf:`sourceLocale` is that
 :conf:`fallbackLocales` is used in translation, while :conf:`sourceLocale` is
 used for the message ID.
+
+extractors
+------------
+
+Default: ``[babel]``
+
+Extractors it's the way to customize which extractor you want for your codebase, a long time ago Babel wasn't ready yet to work with Typescript,
+so we added two extractors as default ``[babel, typescript]``, but right now Babel already works good with Typescript so isn't a requirement anymore to compile two times the same code.
+
+Anyway, if you want to use the typescript extractor in conjuntion with babel you can do:
+
+.. code-block:: js
+
+   {
+      "extractors": [
+         require.resolve("@lingui/cli/api/extractors/babel"),
+         require.resolve("@lingui/cli/api/extractors/typescript"),
+      ]
+   }
+
+Of course you can build your own extractor, take a look to babel and typescript extractors to see how you should do it, but basically exports two methods:
+ - match: regex to a filename extension, should return true|false
+ - extract: is the responsible of transforming the code and using @lingui/babel-plugin-extract-messages
diff --git a/packages/cli/src/api/catalog.ts b/packages/cli/src/api/catalog.ts
index 7bd24973..f6b98f5d 100644
--- a/packages/cli/src/api/catalog.ts
+++ b/packages/cli/src/api/catalog.ts
@@ -155,6 +155,8 @@ export class Catalog {
         extract(filename, tmpDir, {
           verbose: options.verbose,
           babelOptions: this.config.extractBabelOptions,
+          // @ts-ignore
+          extractors: options.extractors,
           projectType: options.projectType,
         })
       )
diff --git a/packages/cli/src/api/extract.ts b/packages/cli/src/api/extract.ts
index 38012608..63d02935 100644
--- a/packages/cli/src/api/extract.ts
+++ b/packages/cli/src/api/extract.ts
@@ -6,13 +6,12 @@ import * as R from "ramda"
 
 import { prettyOrigin } from "./utils"
 
-import * as extractors from "./extractors"
-import { ExtractorType } from "./extractors"
-
+import cliExtractor, { ExtractorType } from "./extractors"
 
 type ExtractOptions = {
   ignore?: Array<string>
   verbose?: boolean
+  extractors?: ExtractorType[]
   projectType?: string
   babelOptions?: Object
 }
@@ -44,7 +43,7 @@ export function extract(
   targetPath: string,
   options: ExtractOptions = {}
 ) {
-  const { ignore = [], verbose = false } = options
+  const { ignore = [] } = options
   const ignorePattern = ignore.length ? new RegExp(ignore.join("|"), "i") : null
 
   srcPaths.forEach((srcFilename) => {
@@ -64,17 +63,7 @@ export function extract(
       return
     }
 
-    R.values(extractors).some((ext: ExtractorType) => {
-      if (!ext.match || !ext.match(srcFilename)) return false
-
-      let spinner
-      if (verbose) spinner = ora().start(srcFilename)
-
-      ext.extract(srcFilename, targetPath, options)
-      if (verbose && spinner) spinner.succeed()
-
-      return true
-    })
+    cliExtractor(srcFilename, targetPath, options)
   })
 }
 
diff --git a/packages/cli/src/api/extractors/index.ts b/packages/cli/src/api/extractors/index.ts
index 76c03d34..cb5d23b0 100644
--- a/packages/cli/src/api/extractors/index.ts
+++ b/packages/cli/src/api/extractors/index.ts
@@ -1,10 +1,7 @@
 import ora from "ora"
 import babel from "./babel"
-import typescript from "./typescript"
-import * as R from "ramda"
-
-const extractors = { babel, typescript }
 
+const DEFAULT_EXTRACTORS: ExtractorType[] = [babel]
 
 export type BabelOptions = {
   plugins?: Array<string>
@@ -14,6 +11,7 @@ export type BabelOptions = {
 export type ExtractOptions = {
   verbose?: boolean
   projectType?: string
+  extractors?: ExtractorType[]
   babelOptions?: BabelOptions
 }
 
@@ -27,7 +25,9 @@ export default function extract(
   targetPath: string,
   options: ExtractOptions
 ): boolean {
-  return R.values(extractors).some((ext: ExtractorType) => {
+  const extractorsToExtract = options.extractors ?? DEFAULT_EXTRACTORS
+
+  return extractorsToExtract.some((ext) => {
     if (!ext.match(filename)) return false
 
     let spinner
@@ -48,5 +48,3 @@ export default function extract(
     return true
   })
 }
-
-export { babel, typescript }
diff --git a/packages/cli/src/lingui-extract.ts b/packages/cli/src/lingui-extract.ts
index 8bc186a9..b88eb9ec 100644
--- a/packages/cli/src/lingui-extract.ts
+++ b/packages/cli/src/lingui-extract.ts
@@ -42,6 +42,7 @@ export default function command(
     catalog.make({
       ...options,
       orderBy: config.orderBy,
+      extractors: config.extractors,
       projectType: detect(),
     })
 
diff --git a/packages/conf/index.d.ts b/packages/conf/index.d.ts
index 408c1520..fe47871e 100644
--- a/packages/conf/index.d.ts
+++ b/packages/conf/index.d.ts
@@ -23,6 +23,11 @@ export type DefaultLocaleObject = {
 
 export declare type FallbackLocales = LocaleObject | DefaultLocaleObject
 
+declare type ExtractorType = {
+    match(filename: string): boolean;
+    extract(filename: string, targetDir: string, options?: any): void;
+}
+
 export declare type LinguiConfig = {
     catalogs: CatalogConfig[];
     compileNamespace: "es" | "cjs" | "ts" | string;
@@ -30,6 +35,7 @@ export declare type LinguiConfig = {
     compilerBabelOptions: GeneratorOptions;
     fallbackLocales: FallbackLocales;
     format: CatalogFormat;
+    extractors?: ExtractorType[];
     prevFormat: CatalogFormat;
     formatOptions: CatalogFormatOptions;
     localeDir: string;
diff --git a/packages/conf/src/index.ts b/packages/conf/src/index.ts
index 688295ea..f57ff115 100644
--- a/packages/conf/src/index.ts
+++ b/packages/conf/src/index.ts
@@ -32,12 +32,18 @@ export type FallbackLocales = LocaleObject | DefaultLocaleObject | false
 
 type ModuleSource = [string, string?]
 
+type ExtractorType = {
+  match(filename: string): boolean
+  extract(filename: string, targetDir: string, options?: any): void
+}
+
 export type LinguiConfig = {
   catalogs: CatalogConfig[]
   compileNamespace: "es" | "ts" | "cjs" | string
   extractBabelOptions: Record<string, unknown>
   compilerBabelOptions: GeneratorOptions
   fallbackLocales?: FallbackLocales
+  extractors?: ExtractorType[]
   format: CatalogFormat
   formatOptions: CatalogFormatOptions
   locales: string[]


Test patch:
diff --git a/packages/cli/src/api/extract.test.ts b/packages/cli/src/api/extract.test.ts
index f1442c74..ccc9b3dc 100644
--- a/packages/cli/src/api/extract.test.ts
+++ b/packages/cli/src/api/extract.test.ts
@@ -53,6 +53,9 @@ describe("extract", function () {
     })
   })
 
+  afterEach(() => {
+    jest.clearAllMocks()
+  })
   afterAll(() => {
     mockFs.restore()
   })
@@ -60,6 +63,10 @@ describe("extract", function () {
   it("should traverse directory and call extractors", function () {
     extract(["src"], "locale", {
       ignore: ["forbidden"],
+      extractors: [
+        babel,
+        typescript
+      ],
       babelOptions: {},
     })
 
@@ -92,8 +99,8 @@ describe("extract", function () {
     )
 
     const extractArgs = [
-      "locale", 
-      { babelOptions: {}, ignore: ["forbidden"]}
+      "locale",
+      { extractors: [babel, typescript], babelOptions: {}, ignore: ["forbidden"]}
     ]
     expect(babel.extract).toHaveBeenCalledWith(
       path.join("src", "components", "Babel.js"),
@@ -145,6 +152,95 @@ describe("extract", function () {
       ...extractArgs
     )
   })
+
+  it("by default the traverse directory only uses babel", function () {
+    extract(["src"], "locale", {
+      ignore: ["forbidden"],
+      babelOptions: {},
+    })
+
+    expect(typescript.match).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Typescript.ts")
+    )
+    expect(babel.match).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.js")
+    )
+    expect(babel.match).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.jsx")
+    )
+    expect(babel.match).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es6")
+    )
+    expect(babel.match).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es")
+    )
+    expect(babel.match).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.mjs")
+    )
+		
+    expect(babel.match).toHaveBeenCalledWith(
+			path.join("src", "index.html")
+    )
+
+    // This file is ignored
+    expect(babel.extract).not.toHaveBeenCalledWith(
+      path.join("src", "index.html")
+    )
+
+    const extractArgs = [
+      "locale",
+      { babelOptions: {}, ignore: ["forbidden"]}
+    ]
+    expect(babel.extract).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.js"),
+      ...extractArgs
+    )
+    expect(babel.extract).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.jsx"),
+      ...extractArgs
+    )
+    expect(babel.extract).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es6"),
+      ...extractArgs
+    )
+    expect(babel.extract).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es"),
+      ...extractArgs
+    )
+    expect(babel.extract).toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.mjs"),
+      ...extractArgs
+    )
+    expect(babel.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Typescript.ts"),
+      ...extractArgs
+    )
+
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.js"),
+      ...extractArgs
+    )
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.jsx"),
+      ...extractArgs
+    )
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es6"),
+      ...extractArgs
+    )
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.es"),
+      ...extractArgs
+    )
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Babel.mjs"),
+      ...extractArgs
+    )
+    expect(typescript.extract).not.toHaveBeenCalledWith(
+      path.join("src", "components", "Typescript.ts"),
+      ...extractArgs
+    )
+  })
 })
 
 describe("collect", function () {