Repository: shikijs/shiki
Pull request #847 (2024-11-24 10:24:35)

feat: improve js engine support (and run tests) in pre-ES2024 envs
Bumps `oniguruma-to-es` to patch [v0.6.1](https://github.com/slevithan/oniguruma-to-es/releases) to improve JS engine support when using `target: 'ES2018'`. This includes cases when `target: 'auto'` is used in pre-ES2024 environments like Node.js 18. I've also updated the JS engine tests to run in such environments.

netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-next* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 8c598029869de30928f18cf6c3e5155766038f4c |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-next/deploys/6742fee5a2c3e3000803b4aa |
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *shiki-matsu* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 8c598029869de30928f18cf6c3e5155766038f4c |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/shiki-matsu/deploys/6742fee517e7a600082a59fe |
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/shikijs/shiki/pull/847?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 95.09%. Comparing base [(`fc87512`)](https://app.codecov.io/gh/shikijs/shiki/commit/fc87512fc825fbdebfc1721e6bc52fe07dd80e1e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs) to head [(`8c59802`)](https://app.codecov.io/gh/shikijs/shiki/commit/8c598029869de30928f18cf6c3e5155766038f4c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #847   +/-   ##
=======================================
  Coverage   95.09%   95.09%           
=======================================
  Files          83       83           
  Lines        6927     6927           
  Branches     1458     1458           
=======================================
  Hits         6587     6587           
  Misses        331      331           
  Partials        9        9           
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/shikijs/shiki/pull/847?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=shikijs).

----
🚨 Try these New Features:

- [Flaky Tests Detection](https://docs.codecov.com/docs/test-result-ingestion-beta) - Detect and resolve failed and flaky tests
- [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis) - Avoid shipping oversized bundles

Patch:
diff --git a/docs/guide/regex-engines.md b/docs/guide/regex-engines.md
index d6617c41..d1ff0345 100644
--- a/docs/guide/regex-engines.md
+++ b/docs/guide/regex-engines.md
@@ -79,7 +79,7 @@ The JavaScript engine is best when running in the browser and in cases when you
 
 For the best result, [Oniguruma-To-ES](https://github.com/slevithan/oniguruma-to-es) uses the [RegExp `v` flag](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicodeSets), which is available in Node.js v20+ and ES2024 ([Browser compatibility](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicodeSets#browser_compatibility)).
 
-For older environments, it can use the `u` flag but somewhat fewer grammars are supported.
+For older environments, it can use the `u` flag but this results in a few less grammars being supported.
 
 By default, the runtime target is automatically detected. You can override this behavior by setting the `target` option:
 
diff --git a/docs/references/engine-js-compat.md b/docs/references/engine-js-compat.md
index e1c90c05..8eccfc18 100644
--- a/docs/references/engine-js-compat.md
+++ b/docs/references/engine-js-compat.md
@@ -2,7 +2,7 @@
 
 Compatibility reference of all built-in grammars with the [JavaScript RegExp engine](/guide/regex-engines#javascript-regexp-engine-experimental).
 
-> Generated on Friday, November 22, 2024
+> Generated on Sunday, November 24, 2024
 >
 > Version `1.23.1`
 >
@@ -20,7 +20,7 @@ Compatibility reference of all built-in grammars with the [JavaScript RegExp eng
 ## Supported Languages
 
 Languages that work with the JavaScript RegExp engine, and will produce the same result as the WASM engine (with the [sample snippets in the registry](https://github.com/shikijs/textmate-grammars-themes/tree/main/samples)).
-In some edge cases, it's not guaranteed that the the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.
+In some edge cases, it's not guaranteed that the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.
 
 | Language           | Highlight Match | Patterns Parsable | Patterns Failed | Diff |
 | ------------------ | :-------------- | ----------------: | --------------: | ---: |
@@ -210,7 +210,7 @@ In some edge cases, it's not guaranteed that the the highlighting will be 100% t
 | wenyan             | ✅ OK           |                18 |               - |      |
 | wgsl               | ✅ OK           |                44 |               - |      |
 | wikitext           | ✅ OK           |               104 |               - |      |
-| wolfram            | ✅ OK           |               501 |               - |      |
+| wolfram            | ✅ OK           |               501 |               - |   10 |
 | xml                | ✅ OK           |               169 |               - |      |
 | xsl                | ✅ OK           |               171 |               - |      |
 | yaml               | ✅ OK           |                48 |               - |      |
@@ -254,7 +254,7 @@ Languages that throw with the JavaScript RegExp engine, either because they cont
 | rst        | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=rst) |              1855 |               2 |   62 |
 | hack       | ❌ Error                                                                 |               947 |               1 |  114 |
 | purescript | ❌ Error                                                                 |                72 |               1 |   42 |
-| cpp        | ❌ Error                                                                 |               510 |               2 |   27 |
+| cpp        | ❌ Error                                                                 |               510 |               2 |    8 |
 | csharp     | ❌ Error                                                                 |               306 |               3 |  204 |
 | markdown   | ❌ Error                                                                 |               115 |               3 |  857 |
 | swift      | ❌ Error                                                                 |               326 |               3 |   40 |
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 83c797e7..47241ab6 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -139,8 +139,8 @@ catalogs:
       specifier: ^1.4.1
       version: 1.4.1
     oniguruma-to-es:
-      specifier: 0.6.0
-      version: 0.6.0
+      specifier: 0.6.1
+      version: 0.6.1
     picocolors:
       specifier: ^1.1.1
       version: 1.1.1
@@ -553,7 +553,7 @@ importers:
         version: 9.3.0
       oniguruma-to-es:
         specifier: 'catalog:'
-        version: 0.6.0
+        version: 0.6.1
 
   packages/engine-oniguruma:
     dependencies:
@@ -4250,8 +4250,8 @@ packages:
     resolution: {integrity: sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ==}
     engines: {node: '>=18'}
 
-  oniguruma-to-es@0.6.0:
-    resolution: {integrity: sha512-YaRxSzO6b7hnCzpq3IiyImHPnTNnhqFvhRdhg68gFMAaTEiFm0Wa8vUKSWqVz89a691o14KBPVJTcybjdRA6aQ==}
+  oniguruma-to-es@0.6.1:
+    resolution: {integrity: sha512-YQmbeCaNmSrSwuMuTq1oYCvb+jkISsruMu2XQTJAgLvHQkFKEMqJI33NVtq8EGPuRvFa0CHcH32l2CpTS9Fs/Q==}
 
   optionator@0.9.4:
     resolution: {integrity: sha512-6IpQ7mKUxRcZNLIObR0hz7lxsapSSIYNZJwXPGeF0mTVqGKFIXj1DQcMoT22S3ROcLyY/rz0PWaWZ9ayWmad9g==}
@@ -9502,7 +9502,7 @@ snapshots:
     dependencies:
       mimic-function: 5.0.1
 
-  oniguruma-to-es@0.6.0:
+  oniguruma-to-es@0.6.1:
     dependencies:
       emoji-regex-xs: 1.0.0
       regex: 5.0.2
diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml
index 73a7a488..7a01016b 100644
--- a/pnpm-workspace.yaml
+++ b/pnpm-workspace.yaml
@@ -53,7 +53,7 @@ catalog:
   minimist: ^1.2.8
   monaco-editor-core: ^0.52.0
   ofetch: ^1.4.1
-  oniguruma-to-es: 0.6.0
+  oniguruma-to-es: 0.6.1
   picocolors: ^1.1.1
   pinia: ^2.2.6
   pnpm: ^9.13.2
diff --git a/scripts/report-engine-js-compat.ts b/scripts/report-engine-js-compat.ts
index fd919e99..5f2dba33 100644
--- a/scripts/report-engine-js-compat.ts
+++ b/scripts/report-engine-js-compat.ts
@@ -223,7 +223,7 @@ async function run() {
     '## Supported Languages',
     '',
     'Languages that work with the JavaScript RegExp engine, and will produce the same result as the WASM engine (with the [sample snippets in the registry](https://github.com/shikijs/textmate-grammars-themes/tree/main/samples)).',
-    'In some edge cases, it\'s not guaranteed that the the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.',
+    'In some edge cases, it\'s not guaranteed that the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.',
     '',
     createTable(reportOk),
     '',


Test patch:
diff --git a/packages/engine-javascript/test/compare.test.ts b/packages/engine-javascript/test/compare.test.ts
index dbae8080..b63f671e 100644
--- a/packages/engine-javascript/test/compare.test.ts
+++ b/packages/engine-javascript/test/compare.test.ts
@@ -142,9 +142,7 @@ const cases: Cases[] = [
   },
 ]
 
-describe.skipIf(
-  +process.versions.node.split('.')[0] < 20,
-)('cases', async () => {
+describe('cases', async () => {
   await loadWasm(import('@shikijs/core/wasm-inlined'))
 
   const resolved = await Promise.all(cases.map(async (c) => {
@@ -163,7 +161,6 @@ describe.skipIf(
       const engineWasm = createWasmOnigLibWrapper()
       const engineJs = createJavaScriptRegexEngine({
         forgiving: true,
-        target: 'ES2024',
       })
 
       const shiki1 = await createHighlighterCore({
diff --git a/packages/engine-javascript/test/general.test.ts b/packages/engine-javascript/test/general.test.ts
index 4d52e22f..7fbe236c 100644
--- a/packages/engine-javascript/test/general.test.ts
+++ b/packages/engine-javascript/test/general.test.ts
@@ -2,9 +2,7 @@ import { describe, expect, it } from 'vitest'
 import { createHighlighter } from '../../shiki/src/index'
 import { createJavaScriptRegexEngine } from '../src'
 
-describe.skipIf(
-  +process.versions.node.split('.')[0] < 20,
-)('should', () => {
+describe('should', () => {
   it('works', async () => {
     const shiki = await createHighlighter({
       themes: ['vitesse-light'],
diff --git a/packages/shiki/test/core-sync.test.ts b/packages/shiki/test/core-sync.test.ts
index d16cb776..6afca662 100644
--- a/packages/shiki/test/core-sync.test.ts
+++ b/packages/shiki/test/core-sync.test.ts
@@ -4,10 +4,7 @@ import { createHighlighterCoreSync } from '../src/core'
 import js from '../src/langs/javascript.mjs'
 import nord from '../src/themes/nord.mjs'
 
-describe.skipIf(
-  // JavaScript engine requires Node v20+
-  +process.versions.node.split('.')[0] < 20,
-)('should', () => {
+describe('should', () => {
   const engine = createJavaScriptRegexEngine()
 
   it('works', () => {