Repository: vercel/next.js
Pull request: [Segment Cache] Detect third-party redirect + static export w/ HEAD request

The PR superseds #85903.

#81247 was introduced to:

- Some CDN/server configuration that performs extra redirections (e.g., Netlify's `_redirects`, Cloudflare Pages w/ `_redirects`/Page Rules, nginx conf on the origin server), we want Next.js can still be able to fetch the RSC payload after redirections.
- But there are also cases where the target is not our Next.js app, for which we should not fetch the RSC payload. Thus, a build id check is implemented to ensure the target is our Next.js app. This is done via a range request to check the first 64 bytes of the HTML response, looking for the injected build id.

However, as mentioned in #85903, some popular static hosting providers (like Cloudflare Pages and `Render.com`) don't support range requests, and in the worst case, the entire HTML will be returned (which is wasteful).

#85903 was an attempt to combat the waste, but it is not ideal:

- The client might not save much bandwidth: with a somewhat good Internet connection, the extra HTML can still be returned in one chunk, and in the worst case, the entire HTML is still returned.
- This doesn't ease the server loads

This PR changes the approach:

- We now use the `HEAD` request to detect third-party redirections, which should ease the server loads
- We add a check for `1xx`/`4xx`/`5xx` response (can be a WAF returning 403 or a backend server returning 502), where we still bail out
- We no longer enforce the "redirection target belongs to our Next.js app" check. We might be storming prefetch requests toward non-Next.js apps. In the future, we might add an option to disable preflight prefetch to combat that.


Changed files:
filename: packages/next/src/client/components/segment-cache/cache.ts
patch: @@ -87,10 +87,6 @@ import {
 import { STATIC_STALETIME_MS } from '../router-reducer/reducers/navigate-reducer'
 import { pingVisibleLinks } from '../links'
 import { PAGE_SEGMENT_KEY } from '../../../shared/lib/segment'
-import {
-  DOC_PREFETCH_RANGE_HEADER_VALUE,
-  doesExportedHtmlMatchBuildId,
-} from '../../../shared/lib/segment-cache/output-export-prefetch-encoding'
 import { FetchStrategy } from './types'
 import { createPromiseWithResolvers } from '../../../shared/lib/promise-with-resolvers'
 
@@ -1337,30 +1333,40 @@ export async function fetchRouteOnCacheMiss(
       // location, but we shouldn't assume or expect that they also redirect all
       // the segment files, too.
       //
-      // To check whether the page is redirected, we perform a range request of
-      // the first N bytes of the HTML document. The canonical URL is determined
-      // from the response.
+      // To check whether the page is redirected, previously we perform a range
+      // request of 64 bytes of the HTML document to check if the target page
+      // is part of this app (by checking if build id matches). Only if the target
+      // page is part of this app do we determine the final canonical URL.
+      //
+      // However, as mentioned in https://github.com/vercel/next.js/pull/85903,
+      // some popular static hosting providers (like Cloudflare Pages or Render.com)
+      // do not support range requests, in the worst case, the entire HTML instead
+      // of 64 bytes could be returned, which is wasteful.
       //
-      // Then we can use the canonical URL to request the route tree.
+      // So instead, we drops the check for build id here, and simply perform
+      // a HEAD request to rejects 1xx/4xx/5xx responses, and then determine the
+      // final URL after redirects.
       //
       // NOTE: We could embed the route tree into the HTML document, to avoid
       // a second request. We're not doing that currently because it would make
       // the HTML document larger and affect normal page loads.
-      const htmlResponse = await fetch(url, {
-        headers: {
-          Range: DOC_PREFETCH_RANGE_HEADER_VALUE,
-        },
+      const headResponse = await fetch(url, {
+        method: 'HEAD',
       })
-      const partialHtml = await htmlResponse.text()
-      if (!doesExportedHtmlMatchBuildId(partialHtml, getAppBuildId())) {
-        // The target page is not part of this app, or it belongs to a
-        // different build.
+      if (headResponse.status < 200 || headResponse.status >= 400) {
+        // The target page responded w/o a successful status code
+        // Could be a WAF serving a 403, or a 5xx from a backend
+        //
+        // Note that we can't use headResponse.ok here, because
+        // Response#ok returns `false` with 3xx responses.
         rejectRouteCacheEntry(entry, Date.now() + 10 * 1000)
         return null
       }
-      urlAfterRedirects = htmlResponse.redirected
-        ? new URL(htmlResponse.url)
+
+      urlAfterRedirects = headResponse.redirected
+        ? new URL(headResponse.url)
         : url
+
       response = await fetchPrefetchResponse(
         addSegmentPathToUrlInOutputExportMode(urlAfterRedirects, segmentPath),
         headers

filename: packages/next/src/shared/lib/segment-cache/output-export-prefetch-encoding.ts
patch: @@ -16,9 +16,6 @@
 const DOCTYPE_PREFIX = '<!DOCTYPE html>' // 15 bytes
 const MAX_BUILD_ID_LENGTH = 24
 
-// Request the first 64 bytes. The Range header is inclusive of the end value.
-export const DOC_PREFETCH_RANGE_HEADER_VALUE = 'bytes=0-63'
-
 function escapeBuildId(buildId: string) {
   // If the build id is longer than the given limit, it's OK for our purposes
   // to only match the beginning.
@@ -49,13 +46,3 @@ export function insertBuildIdComment(originalHtml: string, buildId: string) {
     DOCTYPE_PREFIX + '<!--' + escapeBuildId(buildId) + '-->'
   )
 }
-
-export function doesExportedHtmlMatchBuildId(
-  partialHtmlDocument: string,
-  buildId: string
-) {
-  // Check whether the document starts with the expected buildId.
-  return partialHtmlDocument.startsWith(
-    DOCTYPE_PREFIX + '<!--' + escapeBuildId(buildId) + '-->'
-  )
-}