Repository: vercel/next.js
Pull request: fix: Add Chrome-ligthouse to htmlLimitedBots

We have observed that metadata is not found in PageSpeed runs:

<img width="580" alt="Screenshot 2025-06-18 at 15 03 24" src="https://github.com/user-attachments/assets/812fcacc-41b0-4de3-8b14-eec3eea34592" />

Although the title is detected. Let's consider taking the slower path for PageSpeed runs and block on metadata.


For light house there're two cases:
* Using local chrome browser lighthouse, it's based on the environment we cannot control
* Some people are using [PageSpeed Insights](https://pagespeed.web.dev/) to test their web page.

The UA of [pagesspeed.web.dev](https://pagespeed.web.dev/) are: 
- `UA Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse`
- `UA Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse`

We're going to add `Chrome-Lighthouse` in html limited bots for now to ensure it can crawl metadata correctly. We can remove this in the future once it's resolved

Changed files:
filename: packages/next/src/shared/lib/router/utils/html-bots.ts
patch: @@ -1,4 +1,4 @@
 // This regex contains the bots that we need to do a blocking render for and can't safely stream the response
 // due to how they parse the DOM. For example, they might explicitly check for metadata in the `head` tag, so we can't stream metadata tags after the `head` was sent.
 export const HTML_LIMITED_BOT_UA_RE =
-  /Mediapartners-Google|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti/i
+  /Mediapartners-Google|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti/i

filename: test/e2e/app-dir/metadata-streaming/metadata-streaming.test.ts
patch: @@ -182,5 +182,20 @@ describe('app-dir - metadata-streaming', () => {
       expect($('title').length).toBe(1)
       expect($('head title').text()).toBe('partial static page')
     })
+
+    it('should still render blocking metadata for Google speed insights bot (special case)', async () => {
+      const $ = await next.render$(
+        '/static/partial',
+        {},
+        {
+          headers: {
+            'user-agent':
+              'UA Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse',
+          },
+        }
+      )
+      expect($('title').length).toBe(1)
+      expect($('head title').text()).toBe('partial static page')
+    })
   })
 })

filename: test/production/app-dir/metadata-streaming-config/metadata-streaming-config.test.ts
patch: @@ -11,7 +11,7 @@ describe('app-dir - metadata-streaming-config', () => {
     )
 
     expect(requiredServerFiles.config.htmlLimitedBots).toMatchInlineSnapshot(
-      `"Mediapartners-Google|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti"`
+      `"Mediapartners-Google|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti"`
     )
 
     const prerenderManifest = JSON.parse(
@@ -38,7 +38,7 @@ describe('app-dir - metadata-streaming-config', () => {
        "/ppr": {
          "key": "user-agent",
          "type": "header",
-         "value": "Mediapartners-Google|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti",
+         "value": "Mediapartners-Google|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti",
        },
      }
     `)