Repository: QwikDev/qwik
Pull request #7417: fix: under-prefetching — bundle graph missing dynamic dependencies
Author: login: maiieul
Created: 2025-03-13T11:31:54Z | Merged: 2025-03-25T19:18:46Z

<!--
The Qwik Team and Community appreciate all PRs. Thank you for your effort! Not all PRs can be merged, but those that meet the following criteria will be prioritized:

a) Core fixes, and

b) Framework functionality achievable only by the core.

If this PR can be done as a 3rd-Party Community Add-On, we encourage that for quicker adoption.

If you believe your functionality is valuable to the entire Qwik Community, discuss it in the Qwik Discord channels for potential inclusion in the core.

First of all, make sure your PR title is descriptive and matches our commit title guidelines.

Also make sure your PR follows all the guidelines in the [CONTRIBUTING.md](./CONTRIBUTING.md) document.

-->

# What is it?

<!-- pick one and remove the others -->

- Bug

# Description

Repro: https://github.com/maiieul/qwik-under-prefetching-repro -> TLDR: `open.value && <Comp />` has always led to under-prefetching. This could be worked around (as was the case on the qwik.dev docs) by using `qwikVite`'s `entryStrategy.manual` (https://qwik.dev/docs/guides/bundle/).

I managed to get a pretty good result by checking for each bundle if their dynamic dependencies are entries, reducing the under-prefetching a lot. Then I noticed that removing `manualChunks` logic on top of that completely gets rid of it with the qwik-city ServiceWorkerRegister. The qwik PrefetchServiceWorker also works but is a bit buggy.  

Results https://github.com/QwikDev/qwik/pull/7417: 
- On qwik UI, with the qwik PrefetchServiceWorker there is a bit of over-prefetching (but that’s due to specific PrefetchServiceWorker bugs): ~6s TTI. With the qwik-city ServiceWorkerRegister it is ~2s TTI so ~3x faster. Without the change, it is ~1.5s, but with under-prefetching. 
- There is zero under-prefetching.
- Since I removed manualChunks, there are less chunks outputted overall, which means the interactions feel snappier. For example the qwik.dev docs search now takes ~50ms to serve all the chunks from SW instead of ~300ms.    

Remaining issues/questions:
- The qwik `PrefetchServiceWorker`  doesn't prioritize the chunks  on user interaction. This means if all the chunks need 5s to be prefetched, the app won’t be interactive before that point, and also that the user must re-interact to actually get a result. 
- Both service workers can't prioritize chunks need for user interaction before DOMContentLoaded. For the qwik-city service worker for example, it can start to prefetch chunks before DOMContentLoaded, but cannot re-prioritize. This would be a good optimization to have for 3G networks.
- There is now a bit more of over-prefetching due to the static imports being wrong. For example copy-icon component on qwik UI statically imports the toggle-group.js, which dynamically imports all of the toogle-group chunks. So instead of over-prefetching only the toggle-group.js, we now over-prefetch all the chunks that come with it :/. I’m hopeful we can find a way to fix this. Probably a setting in rollup. 
- It is unclear to me whether `clearTransitiveDeps` and the direct `<dynamic>` check do anything useful. They don’t seem to change anything on qwik.dev and qwik-ui.com I’m concerned these might lead to under-prefetching in some untested situations.

Bonus: this also fixes the docs search 🥳 

Malus: I disabled qwik insights because it was logging lots of `error` on the network tab 🤔


Fixes #4149

# list of PRs that created problems with prefetching:
- https://github.com/QwikDev/qwik/pull/4673 -> change visible tasks to tasks change the symbols hashes, which broke the `entryStrategy.manual` for the docs search.
- https://github.com/QwikDev/qwik/pull/6468 -> removal of `document.dispatchEvent(new CustomEvent("qprefetch",{detail:${JSON.stringify(data)}}))` broke the qwik-city SW. Brought back in https://github.com/QwikDev/qwik/pull/7286.
- https://github.com/QwikDev/qwik/pull/6670 -> introduced severe over-prefetching by letting rollup handle entries. Also introduced returning segments (hooks) in `manualChunks`, therefore preventing rollup to bundle those chunks with others, which affects the bundle-graph, creating even more over-prefetching but also under-prefetching.
https://github.com/QwikDev/qwik/pull/6853 -> Fixed over-prefetching by only allowing tasks dynamic dependencies to be prefetched, because prefetching all kinds of dynamic imports can for example prefetch all dynamic imports from external dependencies (e.g. all of the shiki syntax definitions 😬 ), but also introduced more under-prefetching because some non tasks chunks also need to be prefetched.

<!--
* Include a summary of the motivation and context for this PR
* Is it related to any opened issues? (please add them here)
-->

# Checklist

<!--
* delete the items that are not relevant, so it's easy to tell if the PR is ready to be merged
* add items that are relevant and need to be done before merging
-->

- [x] My code follows the [developer guidelines of this project](https://github.com/QwikDev/qwik/blob/main/CONTRIBUTING.md)
- [x] I performed a self-review of my own code
- [x] I added a changeset with `pnpm change`
- [ ] I made corresponding changes to the Qwik docs
- [ ] I added new tests to cover the fix / functionality


Linked issue #4149: [🐞] Qwik fails to prefetch leading to input delay
### Which component is affected?

Qwik Runtime

### Describe the bug

I have been running into this issue since around v0.18.0 and have spent maybe 20+ hours trying to work out what has been going on.

Qwik fails to preload the Javascript which leads to an interaction delay. This only occurs on the first load. It works perfectly fine after the service worker has cached the file. I have provided a reproduction which shows the issue.

https://97710ab2.preload-3fv.pages.dev/

### Reproduction

https://github.com/EamonHeffernan/qwik-prefetch-issue

### Steps to reproduce

pnpm install
pnpm build
pnpm run deploy

Once deployed, click the green square. This will load instantly as it should. Wait for any preloading to occur, press the red button. There will be a delay as javascript is fetched from the server to display the text.

### System Info

```shell
System:
    OS: Linux 5.10 Ubuntu 22.04.1 LTS 22.04.1 LTS (Jammy Jellyfish)
    CPU: (16) x64 AMD Ryzen 7 3700X 8-Core Processor
    Memory: 10.02 GB / 12.42 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 19.8.1 - /usr/bin/node
    Yarn: 1.22.19 - /usr/bin/yarn
    npm: 9.5.1 - /usr/bin/npm
  npmPackages:
    @builder.io/qwik: ^1.1.0 => 1.1.0 
    @builder.io/qwik-city: ^1.1.0 => 1.1.0 
    undici: 5.22.0 => 5.22.0 
    vite: 4.3.5 => 4.3.5
```


### Additional Information

_No response_
Linked issue comments: user: login: astraldev
created_at: 2023-05-15T09:28:49Z
body: Same issue here


There's a slight delay when handling on click events on buttons. 
author_association: NONE
user: login: EamonHeffernan
created_at: 2023-06-05T06:58:43Z
body: > Same issue here
> 
> There's a slight delay when handling on click events on buttons.

I often have very noticeable delays of up to a second.
author_association: CONTRIBUTOR
user: login: NexonTreeHouse
created_at: 2023-06-08T04:57:19Z
body: Again have this issue, any updates?
author_association: NONE
user: login: EamonHeffernan
created_at: 2023-06-30T10:21:29Z
body: This still occurs in the latest version of Qwik, v1.2. I've updated the demo to the latest version https://3741894d.preload-3fv.pages.dev/
author_association: CONTRIBUTOR
user: login: hamatoyogi
created_at: 2023-10-30T07:51:54Z
body: I'm not seeing this delay... Is it still reproducing for you?
Video shows no delay
https://github.com/BuilderIO/qwik/assets/6682981/725cbfe7-e8d6-4ea4-8a21-d1850e298747
author_association: CONTRIBUTOR
user: login: mhevery
created_at: 2023-10-31T23:07:39Z
body: There will always be corner cases like this. This is why it is handled with Qwik-Insights https://qwik.builder.io/docs/labs/insights/ 

Not sure there is anything to do with this other than to suggest you use Qwik Insights?
author_association: CONTRIBUTOR
user: login: intellix
created_at: 2024-03-24T18:10:39Z
body: In this thread there was talk about waterfalling caused by transitive dependencies not being prefetched (w00t called it a missed opportunity for optimisation): https://discord.com/channels/842438759945601056/1145076169469870120/1145076169469870120

We're using insights but we're still seeing waterfalling, even though Insights has plenty of data. I've a feeling that it's only prefetching a few symbols but not everything (even after waiting).

We're also seeing that hovering over links doesn't trigger bundles to download, but we are seeing an API call to fetch builder.io CMS data on hovering. 

Right now we're just prefetching everything with this strategy and it's noticeably better for us (3 seconds to 500ms to change page):

```ts
prefetchStrategy: {
  implementation: {
    linkInsert: 'html-append',
    linkRel: 'prefetch',
    workerFetchInsert: 'no-link-support',
  },
  symbolsToPrefetch: ({ manifest }) => {
    const buildBase = extractBase(opts);
    return Object.keys(manifest.bundles).map((bundleFileName) => ({ url: `${buildBase}/${bundleFileName}`, imports: [] }));
  },
},
```

I'll need to get a reproduction obviously, but just wanted to say that I think it's still fishy and not quite working as intended.

We're basically testing by: 
- Application -> Clear site data 
- Right click refresh and "Empty cache and hard reload" (I'm assuming we don't need to use clear site data first, but...)
- Wait a couple of seconds for Chrome to stop downloading things
- Click a link in the header -> we see resources are being downloaded. The thing is, if I go to Cache Storage, I can seeeeee the files referenced, but they say 0 bytes? Not sure what's going on. "Disable cache" is not ticked
author_association: CONTRIBUTOR
user: login: JaxCavalera
created_at: 2024-06-10T00:05:46Z
body: I can confirm this is still an issue in the latest version v1.5.5

I did a prod bundle build and ran preview to test how it goes vs dev (which I knew was going to load the kitchen sink for hot loading purposes), it's better, but still not where it should be. If simulating Australian mobile network speeds I see a 2+ second delay between the first click event and the corresponding action taking place when browser cache has been cleared.

My temp solution has been to add a nasty pure CSS spinner onto the Navigation menu. Till this is fixed it gives me a way to 
 let users know something is being worked on by the server (a.k.a a waterfall of code is being downloaded .. because it was never prefetched when the page first loaded).

Hopefully a fix for this is on the way 🙏 It looks like it was working at one point in the past.
author_association: NONE
user: login: ziimakc
created_at: 2024-08-22T20:26:33Z
body: That's the reason I stopped using qwik, if you set `Slow 4g` in chrome and visit https://qwikui.com/docs/styled/checkbox/ for the first time, opening a menu will take ~3 seconds, interacting with any button or checkbox ~1 second, this is just unusable and will ruin first user impression of any website build with qwik.
author_association: NONE

Diff:
diff --git a/.changeset/approach-bed-gift.md b/.changeset/approach-bed-gift.md
new file mode 100644
index 00000000000..eed78464df8
--- /dev/null
+++ b/.changeset/approach-bed-gift.md
@@ -0,0 +1,5 @@
+---
+'@builder.io/qwik': patch
+---
+
+🐞 🩹 The qwik-city ServiceWorkerRegister and qwik PrefetchServiceWorker now prefetch all their qrls to prevent under-prefetching
diff --git a/packages/docs/src/root.tsx b/packages/docs/src/root.tsx
index 7e3274e1710..1935489e3fa 100644
--- a/packages/docs/src/root.tsx
+++ b/packages/docs/src/root.tsx
@@ -55,6 +55,7 @@ export default component$(() => {
         <meta charset="utf-8" />
         <script dangerouslySetInnerHTML={uwu} />
         <RouterHead />
+
         <ServiceWorkerRegister />
 
         <script dangerouslySetInnerHTML={`(${collectSymbols})()`} />
@@ -68,7 +69,6 @@ export default component$(() => {
       >
         <RouterOutlet />
         <RealMetricsOptimization builderApiKey={BUILDER_PUBLIC_API_KEY} />
-        {/* Core Web Vitals experiment until November 8: Do not bring back any SW until then! Reach out to @maiieul first if you believe you have a good reason to change this. */}
       </body>
     </QwikCityProvider>
   );
diff --git a/packages/docs/src/routes/api/qwik-optimizer/api.json b/packages/docs/src/routes/api/qwik-optimizer/api.json
index e12f43ae489..04faea738c8 100644
--- a/packages/docs/src/routes/api/qwik-optimizer/api.json
+++ b/packages/docs/src/routes/api/qwik-optimizer/api.json
@@ -406,7 +406,7 @@
         }
       ],
       "kind": "Interface",
-      "content": "```typescript\nexport interface QwikBundle \n```\n\n\n<table><thead><tr><th>\n\nProperty\n\n\n</th><th>\n\nModifiers\n\n\n</th><th>\n\nType\n\n\n</th><th>\n\nDescription\n\n\n</th></tr></thead>\n<tbody><tr><td>\n\n[dynamicImports?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[imports?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[isTask?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nboolean\n\n\n</td><td>\n\n_(Optional)_ Not precise, but an indication of whether this import may be a task\n\n\n</td></tr>\n<tr><td>\n\n[origins?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[size](#)\n\n\n</td><td>\n\n\n</td><td>\n\nnumber\n\n\n</td><td>\n\n\n</td></tr>\n<tr><td>\n\n[symbols?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n</tbody></table>",
+      "content": "```typescript\nexport interface QwikBundle \n```\n\n\n<table><thead><tr><th>\n\nProperty\n\n\n</th><th>\n\nModifiers\n\n\n</th><th>\n\nType\n\n\n</th><th>\n\nDescription\n\n\n</th></tr></thead>\n<tbody><tr><td>\n\n[dynamicImports?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[hasSymbols?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nboolean\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[imports?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[origins?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n<tr><td>\n\n[size](#)\n\n\n</td><td>\n\n\n</td><td>\n\nnumber\n\n\n</td><td>\n\n\n</td></tr>\n<tr><td>\n\n[symbols?](#)\n\n\n</td><td>\n\n\n</td><td>\n\nstring\\[\\]\n\n\n</td><td>\n\n_(Optional)_\n\n\n</td></tr>\n</tbody></table>",
       "editUrl": "https://github.com/QwikDev/qwik/tree/main/packages/qwik/src/optimizer/src/types.ts",
       "mdFile": "qwik.qwikbundle.md"
     },
diff --git a/packages/docs/src/routes/api/qwik-optimizer/index.md b/packages/docs/src/routes/api/qwik-optimizer/index.md
index d2fe9605443..1664f10545f 100644
--- a/packages/docs/src/routes/api/qwik-optimizer/index.md
+++ b/packages/docs/src/routes/api/qwik-optimizer/index.md
@@ -1320,13 +1320,13 @@ _(Optional)_
 </td></tr>
 <tr><td>
 
-[imports?](#)
+[hasSymbols?](#)
 
 </td><td>
 
 </td><td>
 
-string[]
+boolean
 
 </td><td>
 
@@ -1335,17 +1335,17 @@ _(Optional)_
 </td></tr>
 <tr><td>
 
-[isTask?](#)
+[imports?](#)
 
 </td><td>
 
 </td><td>
 
-boolean
+string[]
 
 </td><td>
 
-_(Optional)_ Not precise, but an indication of whether this import may be a task
+_(Optional)_
 
 </td></tr>
 <tr><td>
diff --git a/packages/qwik-city/src/buildtime/runtime-generation/generate-service-worker.ts b/packages/qwik-city/src/buildtime/runtime-generation/generate-service-worker.ts
index 3285d39f27c..e4b5eb1364c 100644
--- a/packages/qwik-city/src/buildtime/runtime-generation/generate-service-worker.ts
+++ b/packages/qwik-city/src/buildtime/runtime-generation/generate-service-worker.ts
@@ -58,15 +58,27 @@ export function generateAppBundles(appBundles: AppBundle[], manifest: QwikManife
     const symbolHashesInBundle: string[] = [];
 
     const manifestBundle = manifest.bundles[appBundleName];
-    const importedBundleNames = Array.isArray(manifestBundle.imports) ? manifestBundle.imports : [];
+    const staticDepsNames = Array.isArray(manifestBundle.imports) ? manifestBundle.imports : [];
 
-    const depsSet = new Set(importedBundleNames);
+    const dynamicDepsNames = [];
+    for (const dynamicDepName of manifestBundle.dynamicImports || []) {
+      const dynamicDep = manifest.bundles[dynamicDepName];
+      if (!manifest.bundles[dynamicDepName]) {
+        continue;
+      }
+      if (dynamicDep.hasSymbols) {
+        dynamicDepsNames.push(dynamicDepName);
+      }
+    }
+    const depsNames = [...staticDepsNames, ...dynamicDepsNames];
+    const depsNamesSet = new Set(depsNames);
 
-    for (const importedBundleName of importedBundleNames) {
-      clearTransitiveDeps(depsSet, new Set(), importedBundleName);
+    for (const depName of depsNames) {
+      clearTransitiveDeps(depsNamesSet, new Set(), depName);
     }
+
     // set the imports based on the sorted index number
-    appBundle[1] = Array.from(depsSet).map((dep) => sortedBundles.indexOf(dep));
+    appBundle[1] = Array.from(depsNamesSet).map((dep) => sortedBundles.indexOf(dep));
 
     if (manifestBundle.symbols) {
       for (const manifestBundleSymbolName of manifestBundle.symbols) {
diff --git a/packages/qwik/src/optimizer/src/api.md b/packages/qwik/src/optimizer/src/api.md
index 5dc12594d81..0d3f1adc115 100644
--- a/packages/qwik/src/optimizer/src/api.md
+++ b/packages/qwik/src/optimizer/src/api.md
@@ -177,8 +177,9 @@ export interface QwikBundle {
     // (undocumented)
     dynamicImports?: string[];
     // (undocumented)
+    hasSymbols?: boolean;
+    // (undocumented)
     imports?: string[];
-    isTask?: boolean;
     // (undocumented)
     origins?: string[];
     // (undocumented)
diff --git a/packages/qwik/src/optimizer/src/manifest.ts b/packages/qwik/src/optimizer/src/manifest.ts
index 286397b1776..54a2a7b5375 100644
--- a/packages/qwik/src/optimizer/src/manifest.ts
+++ b/packages/qwik/src/optimizer/src/manifest.ts
@@ -279,10 +279,10 @@ export function generateManifestFromBundles(
 
     const bundle: QwikBundle = {
       size: outputBundle.code.length,
+      hasSymbols: false,
     };
 
     let hasSymbols = false;
-    let hasHW = false;
     for (const symbol of outputBundle.exports) {
       if (qrlNames.has(symbol)) {
         // When not minifying we see both the entry and the segment file
@@ -292,12 +292,9 @@ export function generateManifestFromBundles(
           manifest.mapping[symbol] = bundleFileName;
         }
       }
-      if (symbol === '_hW') {
-        hasHW = true;
-      }
     }
-    if (hasSymbols && hasHW) {
-      bundle.isTask = true;
+    if (hasSymbols) {
+      bundle.hasSymbols = true;
     }
 
     const bundleImports = outputBundle.imports
diff --git a/packages/qwik/src/optimizer/src/plugins/plugin.ts b/packages/qwik/src/optimizer/src/plugins/plugin.ts
index e2910f49266..f676c6e900b 100644
--- a/packages/qwik/src/optimizer/src/plugins/plugin.ts
+++ b/packages/qwik/src/optimizer/src/plugins/plugin.ts
@@ -879,9 +879,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
   function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
     if ((opts.entryStrategy as SmartEntryStrategy).manual) {
       const module = getModuleInfo(id)!;
@@ -895,18 +892,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
         }
       }
     }
-
-    if (id.includes('node_modules')) {
-      return null;
-    }
-
-    // Patch to prevent over-prefetching, we must clearly separate .tsx/.jsx chunks so that rollup doesn't mix random imports into non-entry files such as hooks.
-    // Maybe a better solution would be to mark those files as entires earlier in the chain so that we can remove this check and the one above altogether.
-    // We check .(tsx|jsx) after node_modules in case some node_modules end with .jsx or .tsx.
-    if (/\.(tsx|jsx)$/.test(id)) {
-      return id;
-    }
-
     return null;
   }
 
diff --git a/packages/qwik/src/optimizer/src/plugins/vite.ts b/packages/qwik/src/optimizer/src/plugins/vite.ts
index 465f6a7af47..b30f1023a99 100644
--- a/packages/qwik/src/optimizer/src/plugins/vite.ts
+++ b/packages/qwik/src/optimizer/src/plugins/vite.ts
@@ -1179,7 +1179,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if (dep.hasSymbols) {
         if (!didAdd) {
           deps.add('<dynamic>');
           didAdd = true;
diff --git a/packages/qwik/src/optimizer/src/types.ts b/packages/qwik/src/optimizer/src/types.ts
index 1b1febb7620..2efa40aa325 100644
--- a/packages/qwik/src/optimizer/src/types.ts
+++ b/packages/qwik/src/optimizer/src/types.ts
@@ -270,8 +270,7 @@ export interface QwikSymbol {
 /** @public */
 export interface QwikBundle {
   size: number;
-  /** Not precise, but an indication of whether this import may be a task */
-  isTask?: boolean;
+  hasSymbols?: boolean;
   symbols?: string[];
   imports?: string[];
   dynamicImports?: string[];


Reviews:
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T16:56:52Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T20:15:26Z
user: login: wmertens
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T05:21:57Z
user: login: wmertens
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T05:46:20Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T07:14:31Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T08:09:22Z
user: login: shairez
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T13:23:58Z
user: login: shairez
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T13:26:39Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T13:37:12Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T18:26:15Z
user: login: maiieul
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T00:20:57Z
user: login: wmertens
body: needs a little cleanup but I think we can merge this even though #7453 will follow soon
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-25T09:59:51Z
user: login: wmertens
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-25T19:18:33Z

Review comments:
diff_hunk: @@ -1175,7 +1175,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if ((dep.isTask || dep.isEntry) && !dep.isPlan) {
path: packages/qwik/src/optimizer/src/plugins/vite.ts
user: login: maiieul
body: @wmertens just FYI, the `!dep.isPlan` check doesn't change anything in terms of actual prefetching on either qwik-ui webiste or qwik docs website. It is unnecessary imo but I'm fine with keeping it ^^
created_at: 2025-03-16T16:56:52Z
author_association: MEMBER
diff_hunk: @@ -871,32 +871,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
-  function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
-    const module = getModuleInfo(id)!;
-    const segment = module.meta.segment;
-
-    // We need to specifically return segment.entry for qwik-insights
-    if (segment) {
-      return segment.entry;
-    }
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: maiieul
body: Removing this alongside handling dynamic deps completely fixes the under-prefetching waterfalls. I might be missing something, but I don't see how returning `segment.entry` is any helpful for qwik insights @wmertens.
created_at: 2025-03-16T20:15:26Z
author_association: MEMBER
diff_hunk: @@ -1175,7 +1175,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if ((dep.isTask || dep.isEntry) && !dep.isPlan) {
path: packages/qwik/src/optimizer/src/plugins/vite.ts
user: login: wmertens
body: Hmm how is that possible? If we're fetching the full dynamic graph, doesn't the plan then cause the entire application to fetch?
created_at: 2025-03-17T05:21:57Z
author_association: MEMBER
diff_hunk: @@ -871,32 +871,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
-  function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
-    const module = getModuleInfo(id)!;
-    const segment = module.meta.segment;
-
-    // We need to specifically return segment.entry for qwik-insights
-    if (segment) {
-      return segment.entry;
-    }
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: wmertens
body: The insights are given to the optimizer and that then puts the group names into entry. So `segment.entry` contains the combined information from insights and the optimizer entry strategy
created_at: 2025-03-17T05:46:20Z
author_association: MEMBER
diff_hunk: @@ -871,32 +871,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
-  function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
-    const module = getModuleInfo(id)!;
-    const segment = module.meta.segment;
-
-    // We need to specifically return segment.entry for qwik-insights
-    if (segment) {
-      return segment.entry;
-    }
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: maiieul
body: Okay so that's what I thought. The hashes seem to remain deterministic so shouldn't be a problem.

Below is my repro before and after adding a small <Visible1> component to my repro:
<img width="1189" alt="image" src="https://github.com/user-attachments/assets/940173c2-6f66-4805-942f-0296713bfe0a" />


But my understanding is that insights actually uses the symbols s_[hash] to gather its data, like it's done in [bundle optimization](https://qwik.dev/docs/guides/bundle/) in the docs. So we don't need a deterministic q-[hash] for it to improve based on past data. So even if the q- hashes weren't deterministic, we wouldn't need to return `segment.entry` in manualChunks.
created_at: 2025-03-17T07:14:31Z
author_association: MEMBER
diff_hunk: @@ -1175,7 +1175,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if ((dep.isTask || dep.isEntry) && !dep.isPlan) {
path: packages/qwik/src/optimizer/src/plugins/vite.ts
user: login: maiieul
body: I think the qwik SW never imports the qwik-city plan, and the qwik-city SW uses it for the LinkBundles maybe? I'm not sure about that part. 
created_at: 2025-03-17T08:09:22Z
author_association: MEMBER
diff_hunk: @@ -55,10 +55,11 @@ export default component$(() => {
         <meta charset="utf-8" />
         <script dangerouslySetInnerHTML={uwu} />
         <RouterHead />
+
         <ServiceWorkerRegister />
 
         <script dangerouslySetInnerHTML={`(${collectSymbols})()`} />
-        <Insights publicApiKey={import.meta.env.PUBLIC_QWIK_INSIGHTS_KEY} />
+        {/* <Insights publicApiKey={import.meta.env.PUBLIC_QWIK_INSIGHTS_KEY} /> */}
path: packages/docs/src/root.tsx
user: login: shairez
body: is this on purpose? 
created_at: 2025-03-19T13:23:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -871,32 +871,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
-  function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
-    const module = getModuleInfo(id)!;
-    const segment = module.meta.segment;
-
-    // We need to specifically return segment.entry for qwik-insights
-    if (segment) {
-      return segment.entry;
-    }
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: shairez
body: @wmertens so is it OK to remove this?
@maiieul I saw you discussed this with @mhevery , what was the conclusion?
created_at: 2025-03-19T13:26:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -55,10 +55,11 @@ export default component$(() => {
         <meta charset="utf-8" />
         <script dangerouslySetInnerHTML={uwu} />
         <RouterHead />
+
         <ServiceWorkerRegister />
 
         <script dangerouslySetInnerHTML={`(${collectSymbols})()`} />
-        <Insights publicApiKey={import.meta.env.PUBLIC_QWIK_INSIGHTS_KEY} />
+        {/* <Insights publicApiKey={import.meta.env.PUBLIC_QWIK_INSIGHTS_KEY} /> */}
path: packages/docs/src/root.tsx
user: login: maiieul
body: Yes, I left a comment somewhere which I probably edited.. to explain it was logging lots of `error` in the network logs, even though everything still worked normally 🤷‍♂️ . I think we can investigate later.
created_at: 2025-03-19T13:37:12Z
author_association: MEMBER
diff_hunk: @@ -871,32 +871,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
-  // optimization opportunity: group small segments that don't import anything into smallish chunks
-  // order by discovery time, so that related segments are more likely to group together
-  function manualChunks(id: string, { getModuleInfo }: Rollup.ManualChunkMeta) {
-    const module = getModuleInfo(id)!;
-    const segment = module.meta.segment;
-
-    // We need to specifically return segment.entry for qwik-insights
-    if (segment) {
-      return segment.entry;
-    }
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: maiieul
body: what we discussed with Misko was a different API 
```
qwikVite({
  entryStrategy: {
    type: "smart",
    manual: ...
  }
})
```
which also uses `manualChunks` under the hood (but in a different place in the code) to let the developer pass a list of symbols collected at runtime to create one for all those symbols, reducing waterfalls. See [bundle optimization](https://qwik.dev/docs/guides/bundle/).
created_at: 2025-03-19T18:26:15Z
author_association: MEMBER
diff_hunk: @@ -879,9 +879,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
     }
   }
 
-  // This groups all QRL segments into their respective entry points
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: maiieul
body: Removed those comments as they seem outdated. Insights and entryStrategy manual will do their job. And if not used, Rollup does a pretty good job at merging small files into bigger ones already.
created_at: 2025-03-25T00:17:53Z
author_association: MEMBER
diff_hunk: @@ -895,18 +892,6 @@ export const manifest = ${JSON.stringify(manifest)};\n`;
         }
       }
     }
-
-    if (id.includes('node_modules')) {
-      return null;
-    }
-
-    // Patch to prevent over-prefetching, we must clearly separate .tsx/.jsx chunks so that rollup doesn't mix random imports into non-entry files such as hooks.
-    // Maybe a better solution would be to mark those files as entires earlier in the chain so that we can remove this check and the one above altogether.
-    // We check .(tsx|jsx) after node_modules in case some node_modules end with .jsx or .tsx.
-    if (/\.(tsx|jsx)$/.test(id)) {
path: packages/qwik/src/optimizer/src/plugins/plugin.ts
user: login: maiieul
body: removed this workaround as does not seem to be needed with insights. If insights is not used then rollup does a good job at bundling.
created_at: 2025-03-25T00:18:56Z
author_association: MEMBER
diff_hunk: @@ -1179,7 +1179,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if (dep.hasSymbols) {
path: packages/qwik/src/optimizer/src/plugins/vite.ts
user: login: maiieul
body: Simpler implementation. The result is the same as checking if `dep.isEntry`. No need to check for `dep.isPlan` as @qwik-city-plan chunk doesn't contain symbols.
created_at: 2025-03-25T00:20:52Z
author_association: MEMBER
path: packages/qwik/src/optimizer/src/qwik-binding-map.ts
user: login: wmertens
body: this should be undone
created_at: 2025-03-25T08:50:16Z
author_association: MEMBER
path: packages/docs/src/routes/api/qwik/api.json
user: login: wmertens
body: run `pnpm api.update` to undo
created_at: 2025-03-25T08:54:01Z
author_association: MEMBER
diff_hunk: @@ -1179,7 +1179,8 @@ export function convertManifestToBundleGraph(manifest: QwikManifest): QwikBundle
         // external dependency
         continue;
       }
-      if (dep.isTask) {
+      // prevent importing all segments chunks
+      if (dep.hasSymbols) {
path: packages/qwik/src/optimizer/src/plugins/vite.ts
user: login: wmertens
body: Ok, in #7453 I just removed all of it and only disallow more than 10 dynamic deps because we're probably not going to guess right. That works fine too it seems.
created_at: 2025-03-25T09:58:23Z
author_association: MEMBER

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-03-13T11:32:02Z
body: ###  🦋  Changeset detected

Latest commit: 81b3c3b160c87e5feaacd59e4ecf45aa06f8a2cf

**The changes in this PR will be included in the next version bump.**

<details><summary>This PR includes changesets to release 4 packages</summary>

  | Name                  | Type  |
| --------------------- | ----- |
| @builder.io/qwik      | Patch |
| eslint-plugin-qwik    | Patch |
| @builder.io/qwik-city | Patch |
| create-qwik           | Patch |

</details>

Not sure what this means? [Click here  to learn what changesets are](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add another changeset to this PR](https://github.com/maiieul/qwik/new/fix-under-prefetching?filename=.changeset/curly-suns-join.md&value=---%0A%22qwik-docs%22%3A%20patch%0A%22%40builder.io%2Fqwik-city%22%3A%20patch%0A%22%40builder.io%2Fqwik%22%3A%20patch%0A---%0A%0Afix%3A%20under-prefetching%20%E2%80%94%20bundle%20graph%20missing%20dynamic%20dependencies%0A)


author_association: NONE
user: login: pkg-pr-new[bot]
created_at: 2025-03-13T11:48:50Z
body: 
[Open in Stackblitz](https://pkg.pr.new/template/69a1c371-91f4-4b56-85eb-e60efbbdf825)


```
npm i https://pkg.pr.new/@builder.io/qwik@7417
```


```
npm i https://pkg.pr.new/@builder.io/qwik-city@7417
```


```
npm i https://pkg.pr.new/eslint-plugin-qwik@7417
```


```
npm i https://pkg.pr.new/create-qwik@7417
```


_commit: <a href="https://github.com/QwikDev/qwik/runs/39366847629"><code>81b3c3b</code></a>_

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-13T11:49:10Z
body: <!-- refined-cf-pages-action:deployment-summary:qwik -->

###### built with [Refined Cloudflare Pages Action](https://github.com/AdrianGonz97/refined-cf-pages-action)
## ⚡ Cloudflare Pages Deployment
| Name | Status | Preview | Last Commit |
| :--- | :----- | :------ | :---------- |
| **qwik-docs** | ✅ Ready ([View Log](https://github.com/QwikDev/qwik/actions/runs/14059259459)) | [Visit Preview](https://maiieul-fix-under-prefetchin.qwik-8nx.pages.dev) | 81b3c3b160c87e5feaacd59e4ecf45aa06f8a2cf |

author_association: CONTRIBUTOR
user: login: wmertens
created_at: 2025-03-14T20:50:20Z
body: @maiieul I think you can add the isPlan and we can merge this already as a bit of an improvement?
author_association: MEMBER
user: login: maiieul
created_at: 2025-03-14T21:37:20Z
body: Yup, I think it's actually quite a big improvement. Without `isEntry` we were missing a lot of direct level 0 dynamic imports. This also happens to fix the docs search slowness btw 🥹
author_association: MEMBER
user: login: maiieul
created_at: 2025-03-25T11:45:37Z
body: > Ok, in https://github.com/QwikDev/qwik/pull/7453 I just removed all of it and only disallow more than 10 dynamic deps because we're probably not going to guess right. That works fine too it seems.

We need the `hasSymbols` guard, otherwise we will prefetch external dynamic deps again (e.g. shiki). 10 sounds like a reasonable safeguard, but I wonder if it's needed at all. The dynamic deps graph seems to stop on its own after a certain number of nodes. There is still a bit of over-prefetching on qwik-ui but I'm quite sure it's because some static deps that have dynamic deps aren't being tree-shaken somehow.
author_association: MEMBER