Repository: shikijs/shiki
Pull request #912 (2025-01-30 07:42:24)

feat(rehype): Support `fallbackLanguage` on lazy mode
<!-- DO NOT IGNORE THE TEMPLATE!

Thank you for contributing!

Before submitting the PR, please make sure you do the following:

- Read the [Contributing Guide](https://github.com/shikijs/shiki/blob/main/CONTRIBUTING.md).
- Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
- Provide a description in this PR that addresses **what** the PR is solving, or reference the issue that it solves (e.g. `fixes #123`).
- Ideally, include relevant tests that fail without this PR but pass with it.

-->

### Description

Support `fallbackLanguage` when `lazy` is set to true, you can consider this as a successor of #821.

Since `highlighter.loadLanguage` throws error on languages that's not included in the bundle, we can use it to detect whether fallback is needed.

### Linked Issues

N/A

### Additional context

Unit test added


netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-next* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 93a5cf0b386356998b43a6529e9b1af30d3b164a |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-next/deploys/679b2d63a18e42000832db1d |
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-matsu* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 93a5cf0b386356998b43a6529e9b1af30d3b164a |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-matsu/deploys/679b2d63bece100008fc2fed |
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/shikijs/shiki/pull/912?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) Report
Attention: Patch coverage is `87.50000%` with `3 lines` in your changes missing coverage. Please review.
> Project coverage is 94.23%. Comparing base [(`c55865f`)](https://app.codecov.io/gh/shikijs/shiki/commit/c55865fb140983dc611e5a8534878e8562808bdb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) to head [(`93a5cf0`)](https://app.codecov.io/gh/shikijs/shiki/commit/93a5cf0b386356998b43a6529e9b1af30d3b164a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).

| [Files with missing lines](https://app.codecov.io/gh/shikijs/shiki/pull/912?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) | Patch % | Lines |
|---|---|---|
| [packages/rehype/src/core.ts](https://app.codecov.io/gh/shikijs/shiki/pull/912?src=pr&el=tree&filepath=packages%2Frehype%2Fsrc%2Fcore.ts&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs#diff-cGFja2FnZXMvcmVoeXBlL3NyYy9jb3JlLnRz) | 87.50% | [3 Missing :warning: ](https://app.codecov.io/gh/shikijs/shiki/pull/912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #912      +/-   ##
==========================================
- Coverage   94.27%   94.23%   -0.04%     
==========================================
  Files          91       91              
  Lines        7471     7481      +10     
  Branches     1535     1539       +4     
==========================================
+ Hits         7043     7050       +7     
- Misses        419      422       +3     
  Partials        9        9              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/shikijs/shiki/pull/912?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).


Patch:
diff --git a/packages/rehype/src/core.ts b/packages/rehype/src/core.ts
index d8d49ee7..c9e2304f 100644
--- a/packages/rehype/src/core.ts
+++ b/packages/rehype/src/core.ts
@@ -98,30 +98,29 @@ function rehypeShikiFromHighlighter(
       if (node.tagName === 'pre') {
         handler = PreHandler
       }
-
-      if (node.tagName === 'code' && inline) {
+      else if (node.tagName === 'code' && inline) {
         handler = InlineCodeHandlers[inline]
       }
-
-      if (!handler)
+      else {
         return
+      }
 
-      const res = handler(tree, node)
-      if (!res)
+      const parsed = handler(tree, node)
+      if (!parsed)
         return
 
       let lang: string | undefined
       let lazyLoad = false
 
-      if (!res.lang) {
+      if (!parsed.lang) {
         lang = defaultLanguage
       }
-      else if (highlighter.getLoadedLanguages().includes(res.lang) || isSpecialLang(res.lang)) {
-        lang = res.lang
+      else if (highlighter.getLoadedLanguages().includes(parsed.lang) || isSpecialLang(parsed.lang)) {
+        lang = parsed.lang
       }
       else if (lazy) {
         lazyLoad = true
-        lang = res.lang
+        lang = parsed.lang
       }
       else if (fallbackLanguage) {
         lang = fallbackLanguage
@@ -130,14 +129,14 @@ function rehypeShikiFromHighlighter(
       if (!lang)
         return
 
-      const processNode = (): void => {
-        const meta = res.meta ? parseMetaString?.(res.meta, node, tree) : undefined
+      const meta = parsed.meta ? parseMetaString?.(parsed.meta, node, tree) : undefined
 
-        const fragment = highlight(lang, res.code, res.meta, meta ?? {})
+      const processNode = (targetLang: string): void => {
+        const fragment = highlight(targetLang, parsed.code, parsed.meta, meta ?? {})
         if (!fragment)
           return
 
-        if (res.type === 'inline') {
+        if (parsed.type === 'inline') {
           const head = fragment.children[0]
           if (head.type === 'element' && head.tagName === 'pre') {
             head.tagName = 'span'
@@ -148,10 +147,20 @@ function rehypeShikiFromHighlighter(
       }
 
       if (lazyLoad) {
-        queue.push(highlighter.loadLanguage(lang).then(() => processNode()))
+        try {
+          // passed language is checked in sync, promise `.catch()` wouldn't work
+          queue.push(highlighter.loadLanguage(lang).then(() => processNode(lang)))
+        }
+        catch (error) {
+          if (fallbackLanguage)
+            return processNode(fallbackLanguage)
+          else if (onError)
+            onError(error)
+          else throw error
+        }
       }
       else {
-        processNode()
+        processNode(lang)
       }
 
       // don't visit processed nodes
diff --git a/packages/rehype/src/types.ts b/packages/rehype/src/types.ts
index cd34223c..92c6e4f8 100644
--- a/packages/rehype/src/types.ts
+++ b/packages/rehype/src/types.ts
@@ -26,9 +26,7 @@ export interface RehypeShikiExtraOptions {
   defaultLanguage?: string
 
   /**
-   * The fallback language to use when specified language is not loaded
-   *
-   * Ignored if `lazy` is enabled
+   * The fallback language to use when specified language is not loaded, or not included in the bundle
    */
   fallbackLanguage?: string
 


Test patch:
diff --git a/packages/rehype/test/core.test.ts b/packages/rehype/test/core.test.ts
index 0878764c..f19485a5 100644
--- a/packages/rehype/test/core.test.ts
+++ b/packages/rehype/test/core.test.ts
@@ -99,3 +99,27 @@ it('run with rehype-raw', async () => {
 
   await expect(file.toString()).toMatchFileSnapshot('./fixtures/a.core.out.html')
 })
+
+it('run with lazy + fallback language', async () => {
+  const highlighter = await createHighlighter({
+    themes: [
+      'vitesse-light',
+    ],
+    langs: [],
+  })
+
+  const file = await unified()
+    .use(remarkParse)
+    .use(remarkRehype)
+    .use(rehypeShikiFromHighlighter, highlighter, {
+      lazy: true,
+      theme: 'vitesse-light',
+      defaultLanguage: 'text',
+      fallbackLanguage: 'text',
+      langs: [],
+    })
+    .use(rehypeStringify)
+    .process(await fs.readFile(new URL('./fixtures/d.md', import.meta.url)))
+
+  await expect(file.toString()).toMatchFileSnapshot('./fixtures/d.out.html')
+})
diff --git a/packages/rehype/test/fixtures/d.md b/packages/rehype/test/fixtures/d.md
new file mode 100644
index 00000000..ff507daa
--- /dev/null
+++ b/packages/rehype/test/fixtures/d.md
@@ -0,0 +1,3 @@
+```invalid
+Should use fallback language (plaintext)
+```
\ No newline at end of file
diff --git a/packages/rehype/test/fixtures/d.out.html b/packages/rehype/test/fixtures/d.out.html
new file mode 100644
index 00000000..6aa0cf2c
--- /dev/null
+++ b/packages/rehype/test/fixtures/d.out.html
@@ -0,0 +1,1 @@
+<pre class="shiki vitesse-light" style="background-color:#ffffff;color:#393a34" tabindex="0"><code><span class="line"><span>Should use fallback language (plaintext)</span></span></code></pre>
\ No newline at end of file