Repository: shikijs/shiki
Pull request #896 (2025-01-18 15:36:59)

fix(rehype): improve `lazy` performance
<!-- DO NOT IGNORE THE TEMPLATE!

Thank you for contributing!

Before submitting the PR, please make sure you do the following:

- Read the [Contributing Guide](https://github.com/shikijs/shiki/blob/main/CONTRIBUTING.md).
- Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
- Provide a description in this PR that addresses **what** the PR is solving, or reference the issue that it solves (e.g. `fixes #123`).
- Ideally, include relevant tests that fail without this PR but pass with it.

-->

### Description

This PR improves the performance of `lazy` by:

- Instead of always add tasks to a queue, it highlights nodes in sync if the language is loaded.
- Only return promise (async) if unloaded language is found.
- Cleaned some code, and removed one unnecessary function.

### Linked Issues

N/A

### Additional context

<!-- e.g. is there anything you'd like reviewers to focus on? -->


netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-matsu* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b644dfde88cc47ff5c17c4176caff042e92b5afd |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-matsu/deploys/678bca9e9bc62c000731052b |
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-next* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b644dfde88cc47ff5c17c4176caff042e92b5afd |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-next/deploys/678bca9e323e040008bf582b |

Patch:
diff --git a/docs/guide/best-performance.md b/docs/guide/best-performance.md
index 74c545f0..9e9652b9 100644
--- a/docs/guide/best-performance.md
+++ b/docs/guide/best-performance.md
@@ -4,7 +4,7 @@ outline: deep
 
 # Best Performance Practices
 
-This guide will help you to improve the performance your Shiki usage.
+This guide will help you to improve the performance of your Shiki usage.
 
 ## Cache the Highlighter Instance
 
@@ -23,7 +23,7 @@ export async function highlightCode(code: string, lang: string) {
 }
 ```
 
-When you no longer need a highlighter instance, you can call `dispose()` method to release the resources. (It can't not be GC-ed automatically, you need to do it explicitly)
+When you no longer need a highlighter instance, you can call the `dispose()` method to release the resources. (It can't be GC-ed automatically, you need to do it explicitly.)
 
 ```ts
 highlighter.dispose()
@@ -31,11 +31,11 @@ highlighter.dispose()
 
 ## Fine-Grained Bundle
 
-The pre-built bundles are for easy usage, and mostly intended for Node.js environment where you don't worry too much about the bundle size. If you are building a web application, or in a resource-constrained environment, it's always better to use the fine-grained bundles to reduce the bundle size and memory usage.
+The pre-built bundles are for easy usage, and mostly intended for a Node.js environment where you aren't worried about the bundle size. If you are building a web application or in a resource-constrained environment, it's always better to use the fine-grained bundles to reduce the bundle size and memory usage.
 
 **Avoid importing `shiki`, `shiki/bundle/full`, `shiki/bundle/web` directly**.
 
-Instead, import the fine-grained modules like `shiki/core`, `shiki/engine/javascript`, `@shikijs/langs/typescript`, `@shikijs/themes/dark-plus`, etc.
+Instead, import fine-grained modules like `shiki/core`, `shiki/engine/javascript`, `@shikijs/langs/typescript`, `@shikijs/themes/dark-plus`, etc.
 
 ```ts
 import { createHighlighterCore } from 'shiki/core'
@@ -56,13 +56,13 @@ const highlighter = await createHighlighterCore({
 })
 ```
 
-To compose the fine-grained bundles easily, we also provide the [`shiki-codegen`](/packages/codegen) tool to generate the fine-grained bundles for you.
+To compose the fine-grained bundles easily, we also provide the [`shiki-codegen`](/packages/codegen) tool to generate fine-grained bundles for you.
 
 Learn more about [Fine-Grained Bundles](/guide/bundles#fine-grained-bundle).
 
 ## Use Shorthands
 
-`createHighlighter` and `createHighlighterCore` loads all the themes and languages **upfront** to ensure the subsequent highlight operations are synchronous. This can add a significant overhead to the startup time specially when you have a lot of themes and languages. Shorthands abstract the theme and language loading process and maintains an internal highlighter instance underneath, and only load the necessary themes and languages when needed. When your highlight process can be asynchronous, you can use the shorthands to reduce the startup time.
+`createHighlighter` and `createHighlighterCore` load all the themes and languages **upfront** to ensure subsequent highlight operations are synchronous. This can add significant overhead to startup time, especially when you have a lot of themes and languages. Shorthands abstract the theme and language loading process and maintain an internal highlighter instance underneath, only loading the necessary themes and languages when needed. When your highlighting process can be asynchronous, you can use shorthands to reduce startup time.
 
 ```ts
 import { codeToHtml } from 'shiki'
@@ -78,15 +78,15 @@ You can also create your own shorthands with fine-grained bundles. Check out the
 
 ## JavaScript Engine and Pre-compiled Languages
 
-Shiki provides [two engines](/guide/regex-engines) for executing Regular Expressions: [`JavaScript`](/guide/regex-engines#javascript-regexp-engine) and [`Oniguruma`](/guide/regex-engines#oniguruma-engine). The Oniguruma engine is a WebAssembly-based that compiled from C++ code, and `JavaScript` is a pure JavaScript engine that translates Oniguruma-flavored regex to JavaScript regex.
+Shiki provides [two engines](/guide/regex-engines) for executing regular expressions: [`JavaScript`](/guide/regex-engines#javascript-regexp-engine) and [`Oniguruma`](/guide/regex-engines#oniguruma-engine). The Oniguruma engine is WebAssembly-based and compiled from C code, and `JavaScript` is a pure JavaScript engine that translates Oniguruma-flavored regexes to JavaScript regexes.
 
-If you are bundling Shiki for the web, using the JavaScript engine would be smaller in bundle size and faster in startup time. Meanwhile, the [precompiled languages](/guide/regex-engines#pre-compiled-languages) can also reduce the bundle size and startup time, if your target browsers support the latest RegExp features.
+If you are bundling Shiki for the web, using the JavaScript engine results in a smaller bundle size and faster startup time. The [precompiled languages](/guide/regex-engines#pre-compiled-languages) can further reduce bundle size and startup time, if your target browsers support the latest RegExp features.
 
 Check the [RegExp Engines](/guide/regex-engines) guide for more details.
 
 ## Use Workers
 
-Shiki hightlights the code using Regular Expressions, which can be CPU-intensive. You can offload the highlighting work to a Web Worker/Node Worker to avoid blocking the main thread.
+Shiki hightlights code using regular expressions, which can be CPU-intensive. You can offload the highlighting work to a Web Worker/Node Worker to avoid blocking the main thread.
 
 ::: info
 
diff --git a/docs/guide/regex-engines.md b/docs/guide/regex-engines.md
index 2fd17b00..188e4684 100644
--- a/docs/guide/regex-engines.md
+++ b/docs/guide/regex-engines.md
@@ -85,13 +85,13 @@ const jsEngine = createJavaScriptRegexEngine({
 
 ### Pre-compiled Languages
 
-Instead of compiling the regular expressions at on-the-fly, we also provided pre-compiled languages for the JavaScript engine to further reduce the startup time.
+Instead of compiling regular expressions on-the-fly, we also provide pre-compiled languages for the JavaScript engine to further reduce startup time.
 
 ::: info
-Pre-compiled languages requires RegExp uncoide sets support (`v` flag), which is targeting **ES2024** or Node.js 20+, and may not work in older environments. [Can I use](https://caniuse.com/mdn-javascript_builtins_regexp_unicode).
+Pre-compiled languages require support for RegExp UnicodeSets (the `v` flag), which requires **ES2024** or Node.js 20+, and may not work in older environments. [Can I use](https://caniuse.com/mdn-javascript_builtins_regexp_unicodesets).
 :::
 
-You can install them with `@shikijs/langs-precompiled`, and change your `@shikijs/langs` imports to `@shikijs/langs-precompiled`:
+You can install them with `@shikijs/langs-precompiled`, and then change your `@shikijs/langs` imports to `@shikijs/langs-precompiled`:
 
 ```ts
 import { createHighlighterCore } from 'shiki/core'
@@ -113,6 +113,6 @@ const highlighter = await createHighlighterCore({
 })
 ```
 
-If you are not using the custom grammars that requires transpilation, you can use the `createJavaScriptRawEngine` to skip the transpilation step further reducing bundle size.
+If you are not using custom grammars that require transpilation, you can use `createJavaScriptRawEngine` to skip the transpilation step, further reducing bundle size.
 
-If you are using [`shiki-codegen`](/packages/codegen), you can generate the pre-compiled languages with the `--precompiled` and `--engine=javascript-raw` flags.
+If you are using [`shiki-codegen`](/packages/codegen), you can generate pre-compiled languages with the `--precompiled` and `--engine=javascript-raw` flags.
diff --git a/docs/references/engine-js-compat.md b/docs/references/engine-js-compat.md
index 85c19bcb..af895568 100644
--- a/docs/references/engine-js-compat.md
+++ b/docs/references/engine-js-compat.md
@@ -2,9 +2,9 @@
 
 Compatibility reference of all built-in grammars with the [JavaScript RegExp engine](/guide/regex-engines#javascript-regexp-engine).
 
-> Generated on Tuesday, January 14, 2025
+> Generated on Monday, January 20, 2025
 >
-> Version `1.27.0`
+> Version `1.27.2`
 >
 > Runtime: Node.js v22.11.0
 
@@ -262,5 +262,5 @@ Languages that throw with the JavaScript RegExp engine, either because they cont
 | sass       | ✅ OK           |                67 |               2 |      |
 | purescript | ❌ Error        |                72 |               1 |      |
 | csharp     | ❌ Error        |               310 |               3 |  137 |
+| razor      | ❌ Error        |               959 |               3 |      |
 | swift      | ❌ Error        |               326 |               3 |      |
-| razor      | ❌ Error        |               957 |               5 |      |
diff --git a/packages/engine-javascript/src/engine-compile.ts b/packages/engine-javascript/src/engine-compile.ts
index 5f46c9cd..4346e237 100644
--- a/packages/engine-javascript/src/engine-compile.ts
+++ b/packages/engine-javascript/src/engine-compile.ts
@@ -41,6 +41,9 @@ export function defaultJavaScriptRegexConstructor(pattern: string, options?: Oni
         // Oniguruma uses depth limit `20`; lowered here to keep regexes shorter and maybe
         // sometimes faster, but can be increased if issues reported due to low limit
         recursionLimit: 5,
+        // Oniguruma option for `^`->`\A`, `$`->`\Z`; improves search performance without any
+        // change in meaning since TM grammars search line by line
+        singleline: true,
       },
       ...options,
     },
diff --git a/packages/rehype/src/core.ts b/packages/rehype/src/core.ts
index 3a838a5d..d8d49ee7 100644
--- a/packages/rehype/src/core.ts
+++ b/packages/rehype/src/core.ts
@@ -86,24 +86,7 @@ function rehypeShikiFromHighlighter(
 
   return (tree) => {
     // use this queue if lazy is enabled
-    const languageQueue: string[] = []
-    const queue: (() => void)[] = []
-
-    function getLanguage(lang: string | undefined): string | undefined {
-      if (!lang)
-        return defaultLanguage
-
-      if (highlighter.getLoadedLanguages().includes(lang) || isSpecialLang(lang))
-        return lang
-
-      if (lazy) {
-        languageQueue.push(lang)
-        return lang
-      }
-
-      if (fallbackLanguage)
-        return fallbackLanguage
-    }
+    const queue: Promise<void>[] = []
 
     visit(tree, 'element', (node, index, parent) => {
       let handler: RehypeShikiHandler | undefined
@@ -127,7 +110,23 @@ function rehypeShikiFromHighlighter(
       if (!res)
         return
 
-      const lang = getLanguage(res.lang)
+      let lang: string | undefined
+      let lazyLoad = false
+
+      if (!res.lang) {
+        lang = defaultLanguage
+      }
+      else if (highlighter.getLoadedLanguages().includes(res.lang) || isSpecialLang(res.lang)) {
+        lang = res.lang
+      }
+      else if (lazy) {
+        lazyLoad = true
+        lang = res.lang
+      }
+      else if (fallbackLanguage) {
+        lang = fallbackLanguage
+      }
+
       if (!lang)
         return
 
@@ -145,24 +144,26 @@ function rehypeShikiFromHighlighter(
           }
         }
 
-        parent.children.splice(index, 1, ...fragment.children)
+        parent.children[index] = fragment as any
       }
 
-      if (lazy)
-        queue.push(processNode)
-      else
+      if (lazyLoad) {
+        queue.push(highlighter.loadLanguage(lang).then(() => processNode()))
+      }
+      else {
         processNode()
+      }
 
       // don't visit processed nodes
       return 'skip'
     })
 
-    if (lazy) {
-      return highlighter
-        .loadLanguage(...languageQueue)
-        .then(() => {
-          queue.forEach(fn => fn())
-        })
+    if (queue.length > 0) {
+      async function run(): Promise<void> {
+        await Promise.all(queue)
+      }
+
+      return run()
     }
   }
 }
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index d1088c9e..290e53e4 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -148,8 +148,8 @@ catalogs:
       specifier: ^1.1.4
       version: 1.1.4
     oniguruma-to-es:
-      specifier: ^2.0.0
-      version: 2.0.0
+      specifier: ^2.2.0
+      version: 2.2.0
     picocolors:
       specifier: ^1.1.1
       version: 1.1.1
@@ -578,7 +578,7 @@ importers:
         version: 10.0.1
       oniguruma-to-es:
         specifier: 'catalog:'
-        version: 2.0.0
+        version: 2.2.0
 
   packages/engine-oniguruma:
     dependencies:
@@ -610,7 +610,7 @@ importers:
         version: link:../types
       oniguruma-to-es:
         specifier: 'catalog:'
-        version: 2.0.0
+        version: 2.2.0
     devDependencies:
       tm-grammars:
         specifier: 'catalog:'
@@ -4116,8 +4116,8 @@ packages:
     resolution: {integrity: sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ==}
     engines: {node: '>=18'}
 
-  oniguruma-to-es@2.0.0:
-    resolution: {integrity: sha512-pE7+9jQgomy10aK6BJKRNHj1Nth0YLOzb3iRuhlz4gRzNSBSd7hga6U8BE6o0SoSuSkqv+PPtt511Msd1Hkl0w==}
+  oniguruma-to-es@2.2.0:
+    resolution: {integrity: sha512-EEsso27ri0sf+t4uRFEj5C5gvXQj0d0w1Y2qq06b+hDLBnvzO1rWTwEW4C7ytan6nhg4WPwE26eLoiPhHUbvKg==}
 
   optionator@0.9.4:
     resolution: {integrity: sha512-6IpQ7mKUxRcZNLIObR0hz7lxsapSSIYNZJwXPGeF0mTVqGKFIXj1DQcMoT22S3ROcLyY/rz0PWaWZ9ayWmad9g==}
@@ -9142,7 +9142,7 @@ snapshots:
     dependencies:
       mimic-function: 5.0.1
 
-  oniguruma-to-es@2.0.0:
+  oniguruma-to-es@2.2.0:
     dependencies:
       emoji-regex-xs: 1.0.0
       regex: 5.1.1
diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml
index f592debf..65829003 100644
--- a/pnpm-workspace.yaml
+++ b/pnpm-workspace.yaml
@@ -56,7 +56,7 @@ catalog:
   monaco-editor-core: ^0.52.2
   ofetch: ^1.4.1
   ohash: ^1.1.4
-  oniguruma-to-es: ^2.0.0
+  oniguruma-to-es: ^2.2.0
   picocolors: ^1.1.1
   pinia: ^2.3.0
   pnpm: ^9.15.4


Test patch:
diff --git a/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap b/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap
index 91bea6db..aa363561 100644
--- a/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap
+++ b/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap
@@ -13,14 +13,14 @@ exports[`precompile 1`] = `
     "syntax",
     "sublime-syntax",
   ],
-  firstLineMatch: /(?<=^|\\n(?!$))%YAML( ?1[^\\n]\\p{Nd}+)?/dgv,
+  firstLineMatch: /^%YAML( ?1[^\\n]\\p{Nd}+)?/dgv,
   name: "yaml",
   patterns: [
     { include: "#comment" },
     { include: "#property" },
     { include: "#directive" },
-    { match: /(?<=^|\\n(?!$))---/dgv, name: "entity.other.document.begin.yaml" },
-    { match: /(?<=^|\\n(?!$))\\.{3}/dgv, name: "entity.other.document.end.yaml" },
+    { match: /^---/dgv, name: "entity.other.document.begin.yaml" },
+    { match: /^\\.{3}/dgv, name: "entity.other.document.end.yaml" },
     { include: "#node" },
   ],
   repository: {
@@ -44,7 +44,7 @@ exports[`precompile 1`] = `
           beginCaptures: {
             "1": { name: "punctuation.definition.key-value.begin.yaml" },
           },
-          end: /(?=\\?)|(?<=^|\\n(?!$)) *(:)|(:)/dgv,
+          end: /(?=\\?)|^ *(:)|(:)/dgv,
           endCaptures: {
             "1": { name: "punctuation.separator.key-value.mapping.yaml" },
             "2": { name: "invalid.illegal.expected-newline.yaml" },
@@ -54,8 +54,8 @@ exports[`precompile 1`] = `
         },
         {
           begin:
-            /(?=(?:[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-]\\P{space})([^\\p{space}\\:]|:\\P{space}|\\p{space}+(?![\\#\\p{space}]))*\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
-          end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
+            /(?=(?:[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-]\\P{space})([^\\p{space}\\:]|:\\P{space}|\\p{space}+(?![\\#\\p{space}]))*\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
+          end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
           patterns: [
             { include: "#flow-scalar-plain-out-implicit-type" },
             {
@@ -63,13 +63,13 @@ exports[`precompile 1`] = `
                 /[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-]\\P{space}/dgv,
               beginCaptures: { "0": { name: "entity.name.tag.yaml" } },
               contentName: "entity.name.tag.yaml",
-              end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
+              end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
               name: "string.unquoted.plain.out.yaml",
             },
           ],
         },
         {
-          match: /:(?=\\p{space}|(?=$|\\n))/dgv,
+          match: /:(?=\\p{space}|(?=\\n?$))/dgv,
           name: "punctuation.separator.key-value.mapping.yaml",
         },
       ],
@@ -91,15 +91,13 @@ exports[`precompile 1`] = `
           ],
         },
       },
-      end: /*@__PURE__*/ new EmulatedRegExp(
-        "(?<=^|\\\\n(?!$))(?=\\\\P{space})|(?!^)",
-        "dgv",
-        { strategy: "search_start_clip" },
-      ),
+      end: /*@__PURE__*/ new EmulatedRegExp("^(?=\\\\P{space})|(?!^)", "dgv", {
+        strategy: "search_start_clip",
+      }),
       patterns: [
         {
-          begin: /(?<=^|\\n(?!$))([ ]+)(?! )/dgv,
-          end: /(?<=^|\\n(?!$))(?!\\1|\\p{space}*(?=$|\\n))()/dgv,
+          begin: /^([ ]+)(?! )/dgv,
+          end: /^(?!\\1|\\p{space}*(?=\\n?$))()/dgv,
           name: "string.unquoted.block.yaml",
         },
       ],
@@ -110,7 +108,7 @@ exports[`precompile 1`] = `
     },
     comment: {
       begin:
-        /(?:((?<=^|\\n(?!$))[ \\t]*)|[ \\t]+)(?=#[[\\P{space}&&\\P{cntrl}&&\\P{Cn}&&\\P{Cs}]\\p{Zs}]*(?=$|\\n))/dgv,
+        /(?:(^[ \\t]*)|[ \\t]+)(?=#[[\\P{space}&&\\P{cntrl}&&\\P{Cn}&&\\P{Cs}]\\p{Zs}]*(?=\\n?$))/dgv,
       beginCaptures: {
         "1": { name: "punctuation.whitespace.comment.leading.yaml" },
       },
@@ -129,11 +127,11 @@ exports[`precompile 1`] = `
       ],
     },
     directive: {
-      begin: /(?<=^|\\n(?!$))%/dgv,
+      begin: /^%/dgv,
       beginCaptures: {
         "0": { name: "punctuation.definition.directive.begin.yaml" },
       },
-      end: /(?=(?=$|\\n)|[ \\t]+((?=$|\\n)|#))/dgv,
+      end: /(?=(?=\\n?$)|[ \\t]+((?=\\n?$)|#))/dgv,
       name: "meta.directive.yaml",
       patterns: [
         {
@@ -213,7 +211,7 @@ exports[`precompile 1`] = `
             { include: "#flow-pair" },
             { include: "#flow-node" },
             {
-              begin: /:(?=\\p{space}|(?=$|\\n)|[\\[\\]\\{\\}\\,])/dgv,
+              begin: /:(?=\\p{space}|(?=\\n?$)|[\\[\\]\\{\\}\\,])/dgv,
               beginCaptures: {
                 "0": { name: "punctuation.separator.key-value.mapping.yaml" },
               },
@@ -224,8 +222,8 @@ exports[`precompile 1`] = `
         },
         {
           begin:
-            /(?=(?:[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-][^\\p{space}\\[\\]\\{\\}\\,])([^\\p{space}\\:\\[\\]\\{\\}\\,]|:[^\\p{space}\\[\\]\\{\\}\\,]|\\p{space}+(?![\\#\\p{space}]))*\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
-          end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
+            /(?=(?:[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-][^\\p{space}\\[\\]\\{\\}\\,])([^\\p{space}\\:\\[\\]\\{\\}\\,]|:[^\\p{space}\\[\\]\\{\\}\\,]|\\p{space}+(?![\\#\\p{space}]))*\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
+          end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
           name: "meta.flow-pair.key.yaml",
           patterns: [
             { include: "#flow-scalar-plain-in-implicit-type" },
@@ -234,14 +232,14 @@ exports[`precompile 1`] = `
                 /[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-][^\\p{space}\\[\\]\\{\\}\\,]/dgv,
               beginCaptures: { "0": { name: "entity.name.tag.yaml" } },
               contentName: "entity.name.tag.yaml",
-              end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
+              end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
               name: "string.unquoted.plain.in.yaml",
             },
           ],
         },
         { include: "#flow-node" },
         {
-          begin: /:(?=\\p{space}|(?=$|\\n)|[\\[\\]\\{\\}\\,])/dgv,
+          begin: /:(?=\\p{space}|(?=\\n?$)|[\\[\\]\\{\\}\\,])/dgv,
           captures: {
             "0": { name: "punctuation.separator.key-value.mapping.yaml" },
           },
@@ -284,7 +282,7 @@ exports[`precompile 1`] = `
         {
           begin:
             /[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-][^\\p{space}\\[\\]\\{\\}\\,]/dgv,
-          end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
+          end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
           name: "string.unquoted.plain.in.yaml",
         },
       ],
@@ -302,7 +300,7 @@ exports[`precompile 1`] = `
             "7": { name: "constant.language.merge.yaml" },
           },
           match:
-            /(?:(null|Null|NULL|~)|(y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF)|((?:[\\-\\+]?0b[0-1_]+|[\\-\\+]?0[0-7_]+|[\\-\\+]?(?:0|[1-9][0-9_]*)|[\\-\\+]?0x[0-9a-fA-F_]+|[\\-\\+]?[1-9][0-9_]*(?::[0-5]?[0-9])+))|((?:[\\-\\+]?(?:[0-9][0-9_]*)?\\.[0-9\\.]*(?:[eE][\\-\\+][0-9]+)?|[\\-\\+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[\\-\\+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN)))|((?:\\p{Nd}{4}-\\p{Nd}{2}-\\p{Nd}{2}|\\p{Nd}{4}-\\p{Nd}{1,2}-\\p{Nd}{1,2}(?:[Tt]|[ \\t]+)\\p{Nd}{1,2}:\\p{Nd}{2}:\\p{Nd}{2}(?:\\.\\p{Nd}*)?(?:[ \\t]*Z|[\\-\\+]\\p{Nd}{1,2}(?::\\p{Nd}{1,2})?)?))|(=)|(<<))(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
+            /(?:(null|Null|NULL|~)|(y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF)|((?:[\\-\\+]?0b[0-1_]+|[\\-\\+]?0[0-7_]+|[\\-\\+]?(?:0|[1-9][0-9_]*)|[\\-\\+]?0x[0-9a-fA-F_]+|[\\-\\+]?[1-9][0-9_]*(?::[0-5]?[0-9])+))|((?:[\\-\\+]?(?:[0-9][0-9_]*)?\\.[0-9\\.]*(?:[eE][\\-\\+][0-9]+)?|[\\-\\+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[\\-\\+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN)))|((?:\\p{Nd}{4}-\\p{Nd}{2}-\\p{Nd}{2}|\\p{Nd}{4}-\\p{Nd}{1,2}-\\p{Nd}{1,2}(?:[Tt]|[ \\t]+)\\p{Nd}{1,2}:\\p{Nd}{2}:\\p{Nd}{2}(?:\\.\\p{Nd}*)?(?:[ \\t]*Z|[\\-\\+]\\p{Nd}{1,2}(?::\\p{Nd}{1,2})?)?))|(=)|(<<))(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$))|\\p{space}*:[\\[\\]\\{\\}\\,]|\\p{space}*[\\[\\]\\{\\}\\,])/dgv,
         },
       ],
     },
@@ -312,7 +310,7 @@ exports[`precompile 1`] = `
         {
           begin:
             /[^\\p{space}\\-\\?\\:\\,\\[\\]\\{\\}\\#\\&\\*\\!\\|\\>'"\\%\\@\\\`]|[\\?\\:\\-]\\P{space}/dgv,
-          end: /(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
+          end: /(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
           name: "string.unquoted.plain.out.yaml",
         },
       ],
@@ -330,7 +328,7 @@ exports[`precompile 1`] = `
             "7": { name: "constant.language.merge.yaml" },
           },
           match:
-            /(?:(null|Null|NULL|~)|(y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF)|((?:[\\-\\+]?0b[0-1_]+|[\\-\\+]?0[0-7_]+|[\\-\\+]?(?:0|[1-9][0-9_]*)|[\\-\\+]?0x[0-9a-fA-F_]+|[\\-\\+]?[1-9][0-9_]*(?::[0-5]?[0-9])+))|((?:[\\-\\+]?(?:[0-9][0-9_]*)?\\.[0-9\\.]*(?:[eE][\\-\\+][0-9]+)?|[\\-\\+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[\\-\\+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN)))|((?:\\p{Nd}{4}-\\p{Nd}{2}-\\p{Nd}{2}|\\p{Nd}{4}-\\p{Nd}{1,2}-\\p{Nd}{1,2}(?:[Tt]|[ \\t]+)\\p{Nd}{1,2}:\\p{Nd}{2}:\\p{Nd}{2}(?:\\.\\p{Nd}*)?(?:[ \\t]*Z|[\\-\\+]\\p{Nd}{1,2}(?::\\p{Nd}{1,2})?)?))|(=)|(<<))(?=\\p{space}*(?=$|\\n)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=$|\\n)))/dgv,
+            /(?:(null|Null|NULL|~)|(y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF)|((?:[\\-\\+]?0b[0-1_]+|[\\-\\+]?0[0-7_]+|[\\-\\+]?(?:0|[1-9][0-9_]*)|[\\-\\+]?0x[0-9a-fA-F_]+|[\\-\\+]?[1-9][0-9_]*(?::[0-5]?[0-9])+))|((?:[\\-\\+]?(?:[0-9][0-9_]*)?\\.[0-9\\.]*(?:[eE][\\-\\+][0-9]+)?|[\\-\\+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*|[\\-\\+]?\\.(?:inf|Inf|INF)|\\.(?:nan|NaN|NAN)))|((?:\\p{Nd}{4}-\\p{Nd}{2}-\\p{Nd}{2}|\\p{Nd}{4}-\\p{Nd}{1,2}-\\p{Nd}{1,2}(?:[Tt]|[ \\t]+)\\p{Nd}{1,2}:\\p{Nd}{2}:\\p{Nd}{2}(?:\\.\\p{Nd}*)?(?:[ \\t]*Z|[\\-\\+]\\p{Nd}{1,2}(?::\\p{Nd}{1,2})?)?))|(=)|(<<))(?=\\p{space}*(?=\\n?$)|\\p{space}+#|\\p{space}*:(\\p{space}|(?=\\n?$)))/dgv,
         },
       ],
     },
@@ -395,7 +393,7 @@ exports[`precompile 1`] = `
         },
         {
           match:
-            /(?:!<(?:%[0-9A-Fa-f]{2}|[0-9A-Za-z\\-\\#\\;\\/\\?\\:\\@\\&\\=\\+\\$\\,_\\.\\!\\~\\*'\\(\\)\\[\\]])+>|!(?:[0-9A-Za-z\\-]*!)?(?:%[0-9A-Fa-f]{2}|[0-9A-Za-z\\-\\#\\;\\/\\?\\:\\@\\&\\=\\+\\$_\\.\\~\\*'\\(\\)])+|!)(?= |\\t|(?=$|\\n))/dgvy,
+            /(?:!<(?:%[0-9A-Fa-f]{2}|[0-9A-Za-z\\-\\#\\;\\/\\?\\:\\@\\&\\=\\+\\$\\,_\\.\\!\\~\\*'\\(\\)\\[\\]])+>|!(?:[0-9A-Za-z\\-]*!)?(?:%[0-9A-Fa-f]{2}|[0-9A-Za-z\\-\\#\\;\\/\\?\\:\\@\\&\\=\\+\\$_\\.\\~\\*'\\(\\)])+|!)(?= |\\t|(?=\\n?$))/dgvy,
           name: "storage.type.tag-handle.yaml",
         },
         { match: /\\P{space}+/dgv, name: "invalid.illegal.tag-handle.yaml" },