Repository: nodejs/undici (JavaScript)

Issue:
Response headers `vary: Accept-Encoding` is stopping undici from caching the http request
## Bug Description

I was trying to try out undici cache with the following code snippets:

```ts
import undici, {
  interceptors,
  Agent,
  setGlobalDispatcher
} from 'undici';

setGlobalDispatcher(agent.compose(
  interceptors.cache({
    store: new undici.cacheStores.MemoryCacheStore()
  })
));

(async () => {
  console.time('fetch');
  await undici.request('https://easylist.to/easylist/easylist.txt');
  console.timeEnd('fetch');

  console.time('fetch 2');
  await undici.request('https://easylist.to/easylist/easylist.txt');
  console.timeEnd('fetch 2');
})();
```

However, I noticed that the request is never cached.

## Reproducible By

See code snippet above.

## Expected Behavior

The request should be cached and the `fetch 2` should happen "instantly".

## Environment

Node.js v22.11.0 on Darwin

### Additional context

I noticed that `https://easylist.to/easylist/easylist.txt` is returning the `Vary: Accept-Encoding` response header regardless of whether `Accept-Encoding` is included in the request header. This causes undici to panic because undici doesn't expect `Accept-Encoding` to be included in the `cacheKey`:

https://github.com/nodejs/undici/blob/2c3644247a53132267de2cf09cd3dbe3600c21d3/lib/handler/cache-handler.js#L133-L137

cc @flakey5 


Discussion: Many CDNs will always return the `Vary: Accept-Encoding` header whether the HTTP client is sending an `Accept-Encoding` request header or not. However `undici.request` doesn't include `Accept-Encoding` and gzip & br decompression.

Same for GitHub RAW. GitHub RAW always returns `Vary: Authorization,Accept-Encoding,Origin` header. However, the `Authorization` and `Origin` headers might not always exist. The cache should still work w/o those exists in the request headers
Thanks for reporting! Would you like to send a Pull Request to address this issue? Remember to add unit tests.
> Thanks for reporting! Would you like to send a Pull Request to address this issue? Remember to add unit tests.

I am not sure of the proper way to fix this:

We could simply ignore any missing request headers (those undici are not sending) when parsing the `Vary` response header, but I am afraid that later requests with those request headers presented will result in cache poisoning.

Or we could use a placeholder for those missing request headers, but the placeholder couldn't be a symbol because it can't be properly serialized for storing.
What does the standard say? cc @flakey5 
> If (after any normalization that might take place) a header field [specified in the vary header] is absent from a request, it can only match another request if it is also absent there.

[section 4.1-4](https://www.rfc-editor.org/rfc/rfc9111.html#section-4.1-4)

So I think we can ignore it if it's present in the vary header but isn't actually present in the request. The fix would be removing this else block:
https://github.com/nodejs/undici/blob/3eeeeb79b9b890cce39607016fd9e91d235c7134/lib/util/cache.js#L273-L275

I don't think cache poisoning would be a problem, but it is noteworthy that this change would mean that these two requests considered the same:

```http
GET / HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: asd
Vary: a

asd123
```

```http
GET / HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: asd

asd123
```

>  I am afraid that later requests with those request headers presented will result in cache poisoning.

It shouldn't, cache stores get the result of the parsed vary header which maps the header's name to the header's value. The store should be using it to find the request to use. If we ignore a header, it won't be in this mapping meaning it shouldn't be taken into consideration when deciding which response to use.

Regardless, that is definitely something that needs be covered in a unit test if we go down this road.
> but the placeholder couldn't be a symbol because it can't be properly serialized for storing.

Wdyt of using `null`?
> So I think we can ignore it if it's present in the vary header but isn't actually present in the request. The fix would be removing this else block:

> I don't think cache poisoning would be a problem, but it is noteworthy that this change would mean that these two requests considered the same:

> It shouldn't, cache stores get the result of the parsed vary header which maps the header's name to the header's value. The store should be using it to find the request to use. If we ignore a header, it won't be in this mapping meaning it shouldn't be taken into consideration when deciding which response to use.

I am still not sure if this is the path we should take. I am not sure how to properly implement this now.

> > but the placeholder couldn't be a symbol because it can't be properly serialized for storing.
> 
> Wdyt of using `null`?

This should work as long as someone doesn't ignore the typescript definitions and passes a `Record<string, string | null>`.
> I am still not sure if this is the path we should take

@flakey5 What's your suggestions on this?
Sorry for lateish response - looking a bit deeper into this and I think was wrong here:

> So I think we can ignore it if it's present in the vary header but isn't actually present in the request.

We can still continue parsing the vary header if we get to one that's in `vary` but isn't in the request ofc, however, cache stores still need to use it when deciding if a request is cached or not.

My original suggested change wouldn't allow for this and would mean that these two request & responses are seen as the same:

```http
GET / HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: asd
Vary: a

asd123
```

```http
GET / HTTP/1.1
Host: localhost
a: something

HTTP/1.1 200 OK
Date: asd

asd123
```

This goes against the spec. So we would need to do something similar to what @SukkaW mentioned here

> Or we could use a placeholder for those missing request headers

I'm not really sure on the best way to do this however. The best thing I can think of right now is what I mentioned here https://github.com/nodejs/undici/issues/3959#issuecomment-2552247987 but I don't particularly like it



Fix (patch):
diff --git a/lib/cache/memory-cache-store.js b/lib/cache/memory-cache-store.js
index dd5ac00b5a3..6fa7356b366 100644
--- a/lib/cache/memory-cache-store.js
+++ b/lib/cache/memory-cache-store.js
@@ -79,7 +79,13 @@ class MemoryCacheStore {
     const entry = this.#entries.get(topLevelKey)?.find((entry) => (
       entry.deleteAt > now &&
       entry.method === key.method &&
-      (entry.vary == null || Object.keys(entry.vary).every(headerName => entry.vary[headerName] === key.headers?.[headerName]))
+      (entry.vary == null || Object.keys(entry.vary).every(headerName => {
+        if (entry.vary[headerName] === null) {
+          return key.headers[headerName] === undefined
+        }
+
+        return entry.vary[headerName] === key.headers[headerName]
+      }))
     ))
 
     return entry == null
diff --git a/lib/cache/sqlite-cache-store.js b/lib/cache/sqlite-cache-store.js
index 682aac31326..e027cff84ea 100644
--- a/lib/cache/sqlite-cache-store.js
+++ b/lib/cache/sqlite-cache-store.js
@@ -411,10 +411,6 @@ module.exports = class SqliteCacheStore {
       let matches = true
 
       if (value.vary) {
-        if (!headers) {
-          return undefined
-        }
-
         const vary = JSON.parse(value.vary)
 
         for (const header in vary) {
@@ -440,18 +436,21 @@ module.exports = class SqliteCacheStore {
  * @returns {boolean}
  */
 function headerValueEquals (lhs, rhs) {
+  if (lhs == null && rhs == null) {
+    return true
+  }
+
+  if ((lhs == null && rhs != null) ||
+      (lhs != null && rhs == null)) {
+    return false
+  }
+
   if (Array.isArray(lhs) && Array.isArray(rhs)) {
     if (lhs.length !== rhs.length) {
       return false
     }
 
-    for (let i = 0; i < lhs.length; i++) {
-      if (rhs.includes(lhs[i])) {
-        return false
-      }
-    }
-
-    return true
+    return lhs.every((x, i) => x === rhs[i])
   }
 
   return lhs === rhs
diff --git a/lib/util/cache.js b/lib/util/cache.js
index 35c53512b2a..41d1c1b7656 100644
--- a/lib/util/cache.js
+++ b/lib/util/cache.js
@@ -26,10 +26,14 @@ function makeCacheKey (opts) {
       if (typeof key !== 'string' || typeof val !== 'string') {
         throw new Error('opts.headers is not a valid header map')
       }
-      headers[key] = val
+      headers[key.toLowerCase()] = val
     }
   } else if (typeof opts.headers === 'object') {
-    headers = opts.headers
+    headers = {}
+
+    for (const key of Object.keys(opts.headers)) {
+      headers[key.toLowerCase()] = opts.headers[key]
+    }
   } else {
     throw new Error('opts.headers is not an object')
   }
@@ -260,19 +264,16 @@ function parseVaryHeader (varyHeader, headers) {
     return headers
   }
 
-  const output = /** @type {Record<string, string | string[]>} */ ({})
+  const output = /** @type {Record<string, string | string[] | null>} */ ({})
 
   const varyingHeaders = typeof varyHeader === 'string'
     ? varyHeader.split(',')
     : varyHeader
+
   for (const header of varyingHeaders) {
     const trimmedHeader = header.trim().toLowerCase()
 
-    if (headers[trimmedHeader]) {
-      output[trimmedHeader] = headers[trimmedHeader]
-    } else {
-      return undefined
-    }
+    output[trimmedHeader] = headers[trimmedHeader] ?? null
   }
 
   return output
diff --git a/types/cache-interceptor.d.ts b/types/cache-interceptor.d.ts
index 1713ca7e747..e53be60a611 100644
--- a/types/cache-interceptor.d.ts
+++ b/types/cache-interceptor.d.ts
@@ -70,7 +70,7 @@ declare namespace CacheHandler {
     statusCode: number
     statusMessage: string
     headers: Record<string, string | string[]>
-    vary?: Record<string, string | string[]>
+    vary?: Record<string, string | string[] | null>
     etag?: string
     cacheControlDirectives?: CacheControlDirectives
     cachedAt: number
@@ -88,7 +88,7 @@ declare namespace CacheHandler {
     statusCode: number
     statusMessage: string
     headers: Record<string, string | string[]>
-    vary?: Record<string, string | string[]>
+    vary?: Record<string, string | string[] | null>
     etag?: string
     body?: Readable | Iterable<Buffer> | AsyncIterable<Buffer> | Buffer | Iterable<string> | AsyncIterable<string> | string
     cacheControlDirectives: CacheControlDirectives,