Repository: shikijs/shiki (ts)

Issue:
Using transformerRenderWhitespace together with the structure "inline" option throws an error
### Validations

- [x] Follow our [Code of Conduct](https://github.com/shikijs/shiki/blob/main/CODE_OF_CONDUCT.md)
- [x] Read the [Contributing Guide](https://github.com/shikijs/shiki/blob/main/CONTRIBUTING.md).
- [x] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

### Describe the bug

Hi,

When using `transformerRenderWhitespace` together with the `structure: "inline"` setting I receive this error:

```javascript
TypeError: Cannot read properties of undefined (reading 'forEach')
    at Object.root (http://localhost:3000/node_modules/.vite/deps/@shikijs_transformers.js?v=82b8e6ae:538:21)
    at tokensToHast (http://localhost:3000/node_modules/.vite/deps/chunk-EBWDM6ZB.js?v=82b8e6ae:1315:92)
    at codeToHast (http://localhost:3000/node_modules/.vite/deps/chunk-EBWDM6ZB.js?v=82b8e6ae:1188:10)
    at codeToHtml (http://localhost:3000/node_modules/.vite/deps/chunk-EBWDM6ZB.js?v=82b8e6ae:1400:27)
    at Object.codeToHtml (http://localhost:3000/node_modules/.vite/deps/chunk-EBWDM6ZB.js?v=82b8e6ae:1842:37)
    at Proxy.highlightCode (http://localhost:3000/lib/markdownHelpers.ts:89:22)
    at Proxy._sfc_render (http://localhost:3000/components/CodeBlock.vue:37:23)
    at renderComponentRoot (http://localhost:3000/node_modules/.vite/deps/chunk-IWYC5R2O.js?v=82b8e6ae:8581:17)
    at ReactiveEffect.componentUpdateFn [as fn] (http://localhost:3000/node_modules/.vite/deps/chunk-IWYC5R2O.js?v=82b8e6ae:7403:46)
    at ReactiveEffect.run (http://localhost:3000/node_modules/.vite/deps/chunk-IWYC5R2O.js?v=82b8e6ae:481:19)
```

Here is my test setup:

```typescript
const code = `
  const foo = 'bar'
  console.log(foo)
  `;

appEl.innerHTML = await codeToHtml(code, {
  lang: 'js',
  theme: 'nord',
  structure: "inline", // try removing this
  transformers: [
    transformerRenderWhitespace(),
  ],
});
```

Packages:
```json
{
  "name": "shiki-bug2",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build",
    "preview": "vite preview"
  },
  "devDependencies": {
    "typescript": "~5.7.2",
    "vite": "^6.2.0"
  },
  "dependencies": {
    "@shikijs/transformers": "^3.1.0",
    "shiki": "^3.1.0"
  }
}
```

Repro [over here](https://stackblitz.com/~/github.com/marpe/shiki-structure-whitespace-bug)

### Reproduction

https://stackblitz.com/~/github.com/marpe/shiki-structure-whitespace-bug

### Contributes

- [ ] I am willing to submit a PR to fix this issue
- [ ] I am willing to submit a PR with failing tests

Pull request description:
fix(transformers): whitespace rendering transformer for inline structure
<!-- DO NOT IGNORE THE TEMPLATE!

Thank you for contributing!

Before submitting the PR, please make sure you do the following:

- Read the [Contributing Guide](https://github.com/shikijs/shiki/blob/main/CONTRIBUTING.md).
- Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
- Provide a description in this PR that addresses **what** the PR is solving, or reference the issue that it solves (e.g. `fixes #123`).
- Ideally, include relevant tests that fail without this PR but pass with it.

-->

### Description

<!-- Please insert your description here and provide especially info about the "what" this PR is solving -->

Fixes issue where calling `await codeToHtml('...', { structure: 'inline', transformers: [transformerRenderWhitespace()] })` would throw an error (see #954).

### Linked Issues

Fixes #954.

### Additional context

<!-- e.g. is there anything you'd like reviewers to focus on? -->


Patch:
diff --git a/docs/.vitepress/components/HomeDemo.vue b/docs/.vitepress/components/HomeDemo.vue
index c78fdc19..94f47acd 100644
--- a/docs/.vitepress/components/HomeDemo.vue
+++ b/docs/.vitepress/components/HomeDemo.vue
@@ -1,5 +1,5 @@
 <template>
-  <div class="container max-w-250 mx-a my-10 min-h-250 vp-doc p4" of-hidden>
+  <div class="container max-w-250 mx-a my-10 vp-doc p4" of-hidden>
     <ShikiMiniPlayground />
   </div>
 </template>
diff --git a/packages/colorized-brackets/package.json b/packages/colorized-brackets/package.json
index 7fd9a361..f43ef49f 100644
--- a/packages/colorized-brackets/package.json
+++ b/packages/colorized-brackets/package.json
@@ -2,7 +2,7 @@
   "name": "@shikijs/colorized-brackets",
   "type": "module",
   "version": "3.2.2",
-  "description": "Collective of common transformers transformers for Shiki",
+  "description": "VSCode-style colorized brackets transformer for Shiki",
   "author": "Michael Moore <mscottmoore@pm.me>",
   "license": "MIT",
   "homepage": "https://github.com/shikijs/shiki#readme",
diff --git a/packages/colorized-brackets/src/colorizeBracketTokens.ts b/packages/colorized-brackets/src/colorizeBracketTokens.ts
index 25b69f55..6d9f88fd 100644
--- a/packages/colorized-brackets/src/colorizeBracketTokens.ts
+++ b/packages/colorized-brackets/src/colorizeBracketTokens.ts
@@ -107,7 +107,8 @@ function assignColorToToken(
       = shikiOptions
     const styles: Record<string, string> = typeof token.htmlStyle === 'string'
       ? {}
-      : token.htmlStyle || {}
+      // clone to make sure we're not mutating a shared style object
+      : structuredClone(token.htmlStyle ?? {})
 
     for (const [colorName, theme] of Object.entries(shikiOptions.themes)) {
       const themeName = typeof theme === 'string' ? theme : theme?.name
@@ -143,7 +144,7 @@ function getColor(
   const colors
     = themeName == null
       ? DEFAULT_BRACKETS_COLORS
-      : themes[themeName] ?? builtInThemes[themeName] ?? DEFAULT_BRACKETS_COLORS
+      : getThemeColors(themeName, themes) ?? getThemeColors(themeName, builtInThemes) ?? DEFAULT_BRACKETS_COLORS
 
   const isUnexpected = level === -1
   if (isUnexpected) {
@@ -153,3 +154,16 @@ function getColor(
     return colors[level % (colors.length - 1)]
   }
 }
+
+function getThemeColors(themeName: string, themes: Record<string, string[]>): string[] | null {
+  if (themes[themeName])
+    return themes[themeName]
+
+  // check if the start of the name matches any themes
+  // this improves compatibility with "Expressive Code", which appends unique IDs to the end of themeNames
+  const startsWithName = Object.keys(themes).sort().reverse().find(key => themeName.startsWith(key))
+  if (startsWithName)
+    return themes[startsWithName]
+
+  return null
+}
diff --git a/packages/colorized-brackets/src/index.ts b/packages/colorized-brackets/src/index.ts
index 467c7300..3cf584ef 100644
--- a/packages/colorized-brackets/src/index.ts
+++ b/packages/colorized-brackets/src/index.ts
@@ -28,7 +28,7 @@ const jinjaLikeBracketPairs: BracketPair[] = [
  *
  * @param options
  * @param options.themes - custom themes; all Shiki built-in themes are supported without additional configuration
- * @param options.bracketPairs - bracket definitions; be default [], {}, (), and <> (TS-only)
+ * @param options.bracketPairs - bracket definitions; by default [], {}, (), and <> (TS-only)
  * @param options.langs - language-specific overrides for themes and bracketPairs
  * @returns Shiki transformer
  */
diff --git a/packages/core/src/highlight/code-to-hast.ts b/packages/core/src/highlight/code-to-hast.ts
index 68b3767d..5ace2996 100644
--- a/packages/core/src/highlight/code-to-hast.ts
+++ b/packages/core/src/highlight/code-to-hast.ts
@@ -47,6 +47,7 @@ export function codeToHast(
 
   const {
     mergeWhitespaces = true,
+    mergeSameStyleTokens = false,
   } = options
 
   if (mergeWhitespaces === true)
@@ -54,6 +55,10 @@ export function codeToHast(
   else if (mergeWhitespaces === 'never')
     tokens = splitWhitespaceTokens(tokens)
 
+  if (mergeSameStyleTokens) {
+    tokens = mergeAdjacentStyledTokens(tokens)
+  }
+
   const contextSource = {
     ...transformerContext,
     get source() {
@@ -306,3 +311,35 @@ function splitWhitespaceTokens(tokens: ThemedToken[][]): ThemedToken[][] {
     })
   })
 }
+
+function mergeAdjacentStyledTokens(tokens: ThemedToken[][]): ThemedToken[][] {
+  return tokens.map((line) => {
+    const newLine: ThemedToken[] = []
+    for (const token of line) {
+      if (newLine.length === 0) {
+        newLine.push({ ...token })
+        continue
+      }
+
+      const prevToken = newLine[newLine.length - 1]
+      const prevStyle = prevToken.htmlStyle || stringifyTokenStyle(getTokenStyleObject(prevToken))
+      const currentStyle = token.htmlStyle || stringifyTokenStyle(getTokenStyleObject(token))
+      const isPrevDecorated = prevToken.fontStyle && (
+        (prevToken.fontStyle & FontStyle.Underline)
+        || (prevToken.fontStyle & FontStyle.Strikethrough)
+      )
+      const isDecorated = token.fontStyle && (
+        (token.fontStyle & FontStyle.Underline)
+        || (token.fontStyle & FontStyle.Strikethrough)
+      )
+
+      if (!isPrevDecorated && !isDecorated && prevStyle === currentStyle) {
+        prevToken.content += token.content
+      }
+      else {
+        newLine.push({ ...token })
+      }
+    }
+    return newLine
+  })
+}
diff --git a/packages/transformers/src/transformers/render-whitespace.ts b/packages/transformers/src/transformers/render-whitespace.ts
index a9404a29..7a21eef7 100644
--- a/packages/transformers/src/transformers/render-whitespace.ts
+++ b/packages/transformers/src/transformers/render-whitespace.ts
@@ -43,9 +43,9 @@ export function transformerRenderWhitespace(
     // We use `root` hook here to ensure it runs after all other transformers
     root(root) {
       const pre = root.children[0] as Element
-      const code = pre.children[0] as Element
+      const code = pre.tagName === 'pre' ? pre.children[0] as Element : { children: [root] }
       code.children.forEach((line) => {
-        if (line.type !== 'element')
+        if (line.type !== 'element' && line.type !== 'root')
           return
         const elements = line.children.filter(token => token.type === 'element')
         const last = elements.length - 1
diff --git a/packages/types/src/options.ts b/packages/types/src/options.ts
index ef989b85..588329b8 100644
--- a/packages/types/src/options.ts
+++ b/packages/types/src/options.ts
@@ -142,6 +142,14 @@ export interface CodeToHastOptionsCommon<Languages extends string = string>
    */
   mergeWhitespaces?: boolean | 'never'
 
+  /**
+   * Merge consecutive tokens with the same style to reduce the number of DOM nodes.
+   * This can improve rendering performance but may affect the structure of the output.
+   *
+   * @default false
+   */
+  mergeSameStyleTokens?: boolean
+
   /**
    * The structure of the generated HAST and HTML.
    *


Test patch:
diff --git a/packages/colorized-brackets/test/dual-themes.test.ts b/packages/colorized-brackets/test/dual-themes.test.ts
index 4d635b5a..aba8f717 100644
--- a/packages/colorized-brackets/test/dual-themes.test.ts
+++ b/packages/colorized-brackets/test/dual-themes.test.ts
@@ -16,7 +16,7 @@ describe('dual themes', async () => {
   })
 
   it('light and dark', () => {
-    const htmlStr = highlighter.codeToHtml('{}', {
+    const htmlStr = highlighter.codeToHtml('[({})](', {
       lang,
       themes: { light: 'light-plus', dark: 'dark-plus' },
       transformers: [
@@ -28,7 +28,14 @@ describe('dual themes', async () => {
         }),
       ],
     })
-    expect(htmlStr).toContain('<span style="color:Y;--shiki-dark:y">{</span>')
+
+    expect(htmlStr).toContain('<span style="color:Y;--shiki-dark:y">[</span>')
+    expect(htmlStr).toContain('<span style="color:P;--shiki-dark:p">(</span>')
+    expect(htmlStr).toContain('<span style="color:B;--shiki-dark:b">{</span>')
+    expect(htmlStr).toContain('<span style="color:B;--shiki-dark:b">}</span>')
+    expect(htmlStr).toContain('<span style="color:P;--shiki-dark:p">)</span>')
+    expect(htmlStr).toContain('<span style="color:Y;--shiki-dark:y">]</span>')
+    expect(htmlStr).toContain('<span style="color:R;--shiki-dark:r">(</span>')
   })
 
   it('custom prefix', () => {
diff --git a/packages/core/test/hast.test.ts b/packages/core/test/hast.test.ts
index ba0bf4d9..2a802526 100644
--- a/packages/core/test/hast.test.ts
+++ b/packages/core/test/hast.test.ts
@@ -94,3 +94,44 @@ it('render whitespace', async () => {
       <span class="line"><span style="color:#59873A">		tab</span><span style="color:#999999">()</span></span></code></pre>"
     `)
 })
+
+describe('merge same style', () => {
+  it('merges adjacent tokens with same style', async () => {
+    using shiki = await createHighlighter({
+      themes: ['min-light'],
+      langs: ['yaml'],
+    })
+
+    const code = 'name: CI'
+    const html = await shiki.codeToHtml(code, {
+      lang: 'yaml',
+      theme: 'min-light',
+      mergeSameStyleTokens: true,
+    })
+
+    expect(html).toMatchInlineSnapshot(`"<pre class="shiki min-light" style="background-color:#ffffff;color:#24292eff" tabindex="0"><code><span class="line"><span style="color:#D32F2F">name:</span><span style="color:#22863A"> CI</span></span></code></pre>"`)
+  })
+
+  it('does not merge tokens with decorations', async () => {
+    using shiki = await createHighlighter({
+      themes: ['min-light'],
+      langs: ['yaml'],
+    })
+
+    const code = 'name: CI'
+    const html = await shiki.codeToHtml(code, {
+      lang: 'yaml',
+      theme: 'min-light',
+      mergeSameStyleTokens: true,
+      decorations: [
+        {
+          start: { line: 0, character: 0 },
+          end: { line: 0, character: 4 },
+          properties: { class: 'highlighted-word' },
+        },
+      ],
+    })
+
+    expect(html).toMatchInlineSnapshot(`"<pre class="shiki min-light" style="background-color:#ffffff;color:#24292eff" tabindex="0"><code><span class="line"><span style="color:#D32F2F" class="highlighted-word">name</span><span style="color:#D32F2F">:</span><span style="color:#22863A"> CI</span></span></code></pre>"`)
+  })
+})
diff --git a/packages/transformers/test/whitespace-inline.test.ts b/packages/transformers/test/whitespace-inline.test.ts
new file mode 100644
index 00000000..5da08b95
--- /dev/null
+++ b/packages/transformers/test/whitespace-inline.test.ts
@@ -0,0 +1,57 @@
+import { createHighlighter } from 'shiki'
+import { expect, it } from 'vitest'
+import { transformerRenderWhitespace } from '../src/transformers/render-whitespace'
+
+it('transformerRenderWhitespace', async () => {
+  using shiki = await createHighlighter({
+    themes: ['vitesse-dark', 'vitesse-light', 'nord'],
+    langs: ['typescript'],
+  })
+
+  const transformer = transformerRenderWhitespace()
+
+  const code = `
+    const a = Math.random() > 0.5 ? 1 : \`foo\`
+  `.trim()
+
+  const result = shiki.codeToHtml(code, {
+    lang: 'typescript',
+    themes: {
+      dark: 'vitesse-dark',
+      light: 'vitesse-light',
+      nord: 'nord',
+    },
+    defaultColor: false,
+    transformers: [transformer],
+    structure: 'inline',
+  })
+
+  expect(result.replace(/<span/g, '\n<span'))
+    .toMatchInlineSnapshot(`
+      "
+      <span style="--shiki-dark:#CB7676;--shiki-light:#AB5959;--shiki-nord:#81A1C1">const</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#BD976A;--shiki-light:#B07D48;--shiki-nord:#D8DEE9">a</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#666666;--shiki-light:#999999;--shiki-nord:#81A1C1">=</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#BD976A;--shiki-light:#B07D48;--shiki-nord:#D8DEE9">Math</span>
+      <span style="--shiki-dark:#666666;--shiki-light:#999999;--shiki-nord:#ECEFF4">.</span>
+      <span style="--shiki-dark:#80A665;--shiki-light:#59873A;--shiki-nord:#88C0D0">random</span>
+      <span style="--shiki-dark:#666666;--shiki-light:#999999;--shiki-nord:#D8DEE9FF">()</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#666666;--shiki-light:#999999;--shiki-nord:#81A1C1">></span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#4C9A91;--shiki-light:#2F798A;--shiki-nord:#B48EAD">0.5</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#CB7676;--shiki-light:#AB5959;--shiki-nord:#81A1C1">?</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#4C9A91;--shiki-light:#2F798A;--shiki-nord:#B48EAD">1</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#CB7676;--shiki-light:#AB5959;--shiki-nord:#81A1C1">:</span>
+      <span class="space"> </span>
+      <span style="--shiki-dark:#C98A7D77;--shiki-light:#B5695977;--shiki-nord:#ECEFF4">\`</span>
+      <span style="--shiki-dark:#C98A7D;--shiki-light:#B56959;--shiki-nord:#A3BE8C">foo</span>
+      <span style="--shiki-dark:#C98A7D77;--shiki-light:#B5695977;--shiki-nord:#ECEFF4">\`</span>"
+    `)
+})