Repository: electric-sql/electric
Pull request #2593: feat: Do not cache empty live responses
Author: login: msfstef
Created: 2025-04-14T12:24:47Z | Merged: 2025-04-15T15:26:17Z

Fixes https://github.com/electric-sql/electric/issues/2589

~We introduced the concept of a cursor in https://github.com/electric-sql/electric/pull/1826 in order to avoid infinite loops of clients running into cached live responses by artificially "moving the cache forward" via the time based, coordinated cache buster.~

~However we should not need it, as our cache buster is already the `offset` parameter. We were running into this issue because we were caching live responses that do not move the cache forward, i.e. empty live responses, so clients would continuously hit the same cache over and over again.~

~Even with the "cursor" fix we still run into this issue but in a different way - empty live responses create a "chain" rather than a loop of cache hits, that can be arbitrarily long as we allow these cached live responses to be revalidated as well. This means someone who is at the tip of the log might end up following a huge chain of responses with no changes in them, and each of those requests made might trigger a separate revalidation request to the origin.~

~Request collapsing on cache misses works regardless of the cache policy you set on a response (since the CDN does not yet know what the caching policy on the response will be). This is [a canonical example](https://developers.cloudflare.com/cache/concepts/revalidation/#example-2) from Cloudflare.~

~Request collapsing on stale cache hits, as discussed [in the other Cloudflare example](https://developers.cloudflare.com/cache/concepts/revalidation/#example-1) will send back only a single revalidation out of the collapsed requests.~

~Therefore to avoid these infinite loops, we can simply _not cache_ live responses with no changes in them, which retains the request collapsing behaviour without creating any infinite loops.~

~For live responses that _do_ contain changes, we set the usual 5 second lifetime + 5 second stale lifetime, so that clients that are slightly behind can catch up using the cache, but the cursor is not needed as these cached live responses move the `offset` forward.~

~If we go ahead with this change we do need to keep the cursor header present as we require it in our official client, although I've ripped out any logic for it since it won't actually be used and it is better to not change it between requests to ensure cache consistency - we can discuss a path to deprecation~


### UPDATE

We just use an etag that is always different for live responses that contain no changes to ensure they never get revalidated.


Diff:
diff --git a/.changeset/cool-plums-speak.md b/.changeset/cool-plums-speak.md
new file mode 100644
index 0000000000..d394169ab2
--- /dev/null
+++ b/.changeset/cool-plums-speak.md
@@ -0,0 +1,5 @@
+---
+"@core/sync-service": patch
+---
+
+Always add invalid etag for empty live requests
diff --git a/packages/sync-service/lib/electric/shapes/api.ex b/packages/sync-service/lib/electric/shapes/api.ex
index 9bb94a5744..e47e57d7d9 100644
--- a/packages/sync-service/lib/electric/shapes/api.ex
+++ b/packages/sync-service/lib/electric/shapes/api.ex
@@ -551,9 +551,10 @@ defmodule Electric.Shapes.Api do
     %{response: response, global_last_seen_lsn: global_last_seen_lsn} =
       update_attrs(request, %{ot_is_empty_response: true})
 
-    %{
+    %Response{
       response
       | status: 200,
+        no_changes: true,
         body: encode_log(request, [up_to_date_ctl(global_last_seen_lsn)])
     }
   end
diff --git a/packages/sync-service/lib/electric/shapes/api/response.ex b/packages/sync-service/lib/electric/shapes/api/response.ex
index 27d4037c5b..878c359aac 100644
--- a/packages/sync-service/lib/electric/shapes/api/response.ex
+++ b/packages/sync-service/lib/electric/shapes/api/response.ex
@@ -28,6 +28,7 @@ defmodule Electric.Shapes.Api.Response do
     api: %Api{},
     chunked: false,
     up_to_date: false,
+    no_changes: false,
     params: %Api.Params{},
     status: 200,
     trace_attrs: %{},
@@ -44,6 +45,7 @@ defmodule Electric.Shapes.Api.Response do
           chunked: boolean(),
           params: Api.Params.t(),
           up_to_date: boolean(),
+          no_changes: boolean(),
           status: pos_integer(),
           trace_attrs: %{optional(atom()) => term()},
           body: Enum.t()
@@ -194,38 +196,33 @@ defmodule Electric.Shapes.Api.Response do
     |> put_cache_header("cache-control", "no-cache", api)
   end
 
-  defp put_cache_headers(conn, %__MODULE__{api: api} = response) do
-    case response do
-      # If the offset is -1, set a 1 week max-age, 1 hour s-maxage (shared cache)
-      # and 1 month stale-while-revalidate We want private caches to cache the
-      # initial offset for a long time but for shared caches to frequently
-      # revalidate so they're serving a fairly fresh copy of the initials shape
-      # log.
-      %{params: %{offset: @before_all_offset}} ->
-        conn
-        |> put_cache_header(
-          "cache-control",
-          "public, max-age=604800, s-maxage=3600, stale-while-revalidate=2629746",
-          api
-        )
+  # If the offset is -1, set a 1 week max-age, 1 hour s-maxage (shared cache)
+  # and 1 month stale-while-revalidate We want private caches to cache the
+  # initial offset for a long time but for shared caches to frequently
+  # revalidate so they're serving a fairly fresh copy of the initials shape
+  # log.
+  defp put_cache_headers(conn, %__MODULE__{params: %{offset: @before_all_offset}, api: api}) do
+    conn
+    |> put_cache_header(
+      "cache-control",
+      "public, max-age=604800, s-maxage=3600, stale-while-revalidate=2629746",
+      api
+    )
+  end
 
-      # For live requests we want short cache lifetimes and to update the live cursor
-      %{params: %{live: true}, api: api} ->
-        conn
-        |> put_cache_header(
-          "cache-control",
-          "public, max-age=5, stale-while-revalidate=5",
-          api
-        )
+  # For live requests we want short cache lifetimes and to update the live cursor
+  defp put_cache_headers(conn, %__MODULE__{params: %{live: true}, api: api}) do
+    conn
+    |> put_cache_header("cache-control", "public, max-age=5, stale-while-revalidate=5", api)
+  end
 
-      %{params: %{live: false}, api: api} ->
-        conn
-        |> put_cache_header(
-          "cache-control",
-          "public, max-age=#{api.max_age}, stale-while-revalidate=#{api.stale_age}",
-          api
-        )
-    end
+  defp put_cache_headers(conn, %__MODULE__{params: %{live: false}, api: api}) do
+    conn
+    |> put_cache_header(
+      "cache-control",
+      "public, max-age=#{api.max_age}, stale-while-revalidate=#{api.stale_age}",
+      api
+    )
   end
 
   defp put_cache_header(conn, header, value, %{send_cache_headers?: true}) do
@@ -236,27 +233,25 @@ defmodule Electric.Shapes.Api.Response do
     conn
   end
 
-  defp put_cursor_headers(conn, %__MODULE__{} = response) do
-    case response do
-      # For live requests we want short cache lifetimes and to update the live cursor
-      %{params: %{live: true}, api: api} ->
-        conn
-        |> Plug.Conn.put_resp_header(
-          @electric_cursor_header,
-          api.long_poll_timeout
-          |> Utils.get_next_interval_timestamp(conn.query_params["cursor"])
-          |> Integer.to_string()
-        )
-
-      _response ->
-        conn
-    end
-  end
-
-  defp put_etag_headers(conn, %{handle: nil}) do
+  # For live requests we want short cache lifetimes and to update the live cursor
+  defp put_cursor_headers(
+         %{query_params: query_params} = conn,
+         %__MODULE__{params: %{live: true}, api: api} = _response
+       ) do
     conn
+    |> Plug.Conn.put_resp_header(
+      "electric-cursor",
+      api.long_poll_timeout
+      |> Utils.get_next_interval_timestamp(query_params["cursor"])
+      |> Integer.to_string()
+    )
   end
 
+  defp put_cursor_headers(conn, _), do: conn
+
+  # Responses that don't correspond to a shape should not be revalidateable
+  defp put_etag_headers(conn, %__MODULE__{handle: nil}), do: conn
+
   defp put_etag_headers(conn, %__MODULE__{} = response) do
     # etag values should be in double quotes: https://www.rfc-editor.org/rfc/rfc7232#section-2.3
     Plug.Conn.put_resp_header(conn, "etag", etag(response))
@@ -313,14 +308,26 @@ defmodule Electric.Shapes.Api.Response do
     Plug.Conn.assign(conn, :streaming_bytes_sent, bytes_sent)
   end
 
-  def etag(%__MODULE__{handle: handle, offset: offset, params: params} = _response, opts \\ []) do
-    etag = "#{handle}:#{params.offset}:#{offset}"
+  def etag(response, opts \\ [])
+
+  # When response contains no changes, in order to uniquely identify it and avoid
+  # infinite revalidations we add the current monotonic time to the etag.
+  # Any attempt by the CDN to revalidate an empty live response will fail and require
+  # actually holding and collapsing live requests.
+  def etag(%__MODULE__{handle: handle, offset: offset, params: params, no_changes: true}, opts) do
+    "#{handle}:#{params.offset}:#{offset}:#{System.monotonic_time()}"
+    |> format_etag(opts)
+  end
+
+  def etag(%__MODULE__{handle: handle, offset: offset, params: params}, opts) do
+    "#{handle}:#{params.offset}:#{offset}"
+    |> format_etag(opts)
+  end
 
-    if Keyword.get(opts, :quote, true) do
-      ~s|"#{etag}"|
-    else
-      etag
-    end
+  defp format_etag(etag, opts) do
+    if Keyword.get(opts, :quote, true),
+      do: ~s|"#{etag}"|,
+      else: etag
   end
 
   def electric_headers, do: @electric_headers
diff --git a/packages/sync-service/test/electric/plug/serve_shape_plug_test.exs b/packages/sync-service/test/electric/plug/serve_shape_plug_test.exs
index 8f3dc97c79..b39a03e683 100644
--- a/packages/sync-service/test/electric/plug/serve_shape_plug_test.exs
+++ b/packages/sync-service/test/electric/plug/serve_shape_plug_test.exs
@@ -668,6 +668,9 @@ defmodule Electric.Plug.ServeShapePlugTest do
                "public, max-age=5, stale-while-revalidate=5"
              ]
 
+      expected_etag_part = "\"#{@test_shape_handle}:#{@test_offset}:#{@test_offset}:"
+      assert [^expected_etag_part <> _rest] = get_resp_header(conn, "etag")
+
       assert get_resp_header(conn, "electric-up-to-date") == [""]
     end
 
diff --git a/packages/typescript-client/test/cache.test.ts b/packages/typescript-client/test/cache.test.ts
index 49a73e23ef..a0e2f8e6e6 100644
--- a/packages/typescript-client/test/cache.test.ts
+++ b/packages/typescript-client/test/cache.test.ts
@@ -4,6 +4,8 @@ import { exec } from 'child_process'
 import { setTimeout as sleep } from 'node:timers/promises'
 import { testWithIssuesTable } from './support/test-context'
 import { CHUNK_LAST_OFFSET_HEADER, SHAPE_HANDLE_HEADER } from '../src/constants'
+import { ShapeStream } from '../src'
+import { isUpToDateMessage } from '../src/helpers'
 
 // FIXME: pull from environment?
 const maxAge = 1 // seconds
@@ -105,6 +107,79 @@ describe(`HTTP Proxy Cache`, { timeout: 30000 }, () => {
     expect(getCacheStatus(cachedRes)).toBe(CacheStatus.HIT)
   })
 
+  it(`should collapse requests in live mode`, async ({
+    insertIssues,
+    proxyCacheBaseUrl,
+    issuesTableUrl,
+    aborter,
+  }) => {
+    const numClients = 10
+    const eventTarget = new EventTarget()
+
+    let reqStats = {
+      reqs: 0,
+      cacheHits: 0,
+    }
+
+    const resetReqStats = () => {
+      reqStats = {
+        reqs: 0,
+        cacheHits: 0,
+      }
+    }
+
+    const fetchClient = async (...args: Parameters<typeof fetch>) => {
+      const resp = await fetch(...args)
+      reqStats.reqs++
+      if (getCacheStatus(resp) === CacheStatus.HIT) {
+        reqStats.cacheHits++
+      }
+
+      return resp
+    }
+
+    const waitForClients = () =>
+      new Promise<void>((res) => {
+        let ctr = 0
+        const listener = () => {
+          if (++ctr === numClients) {
+            eventTarget.removeEventListener(`up-to-date`, listener)
+            res()
+          }
+        }
+        eventTarget.addEventListener(`up-to-date`, listener)
+      })
+
+    for (let i = 0; i < numClients; i++) {
+      const stream = new ShapeStream({
+        url: `${proxyCacheBaseUrl}/v1/shape`,
+        signal: aborter.signal,
+        fetchClient,
+        params: {
+          table: issuesTableUrl,
+          foo: `cache-test`,
+        },
+      })
+
+      stream.subscribe((messages) => {
+        if (isUpToDateMessage(messages[messages.length - 1])) {
+          eventTarget.dispatchEvent(new Event(`up-to-date`))
+        }
+      })
+    }
+
+    // wait for clients to catch up
+    await waitForClients()
+
+    // add some data, should collapse requests and respond to
+    // all of them but one with cache hits
+    resetReqStats()
+    await insertIssues({ title: `foo` })
+    await waitForClients()
+    expect(reqStats.reqs).toBe(numClients)
+    expect(reqStats.cacheHits).toBe(numClients - 1)
+  })
+
   it(`should get cached response on second request`, async ({
     proxyCacheBaseUrl,
     issuesTableUrl,


Reviews:
user: login: KyleAMathews
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T16:46:28Z
user: login: msfstef
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T16:47:38Z
user: login: msfstef
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T16:48:43Z
user: login: KyleAMathews
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T17:01:40Z
user: login: icehaunter
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:17:02Z

Review comments:
diff_hunk: @@ -554,9 +554,10 @@ defmodule Electric.Shapes.Api do
     %{response: response, global_last_seen_lsn: global_last_seen_lsn} =
       update_attrs(request, %{ot_is_empty_response: true})
 
-    %{
+    %Response{
       response
       | status: 200,
+        no_changes: true,
path: packages/sync-service/lib/electric/shapes/api.ex
user: login: KyleAMathews
body: what is this?
created_at: 2025-04-14T16:45:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -298,13 +293,21 @@ defmodule Electric.Shapes.Api.Response do
     Plug.Conn.assign(conn, :streaming_bytes_sent, bytes_sent)
   end
 
-  def etag(%__MODULE__{handle: handle, offset: offset, params: params} = _response, opts \\ []) do
-    etag = "#{handle}:#{params.offset}:#{offset}"
+  def etag(response, opts \\ [])
+
+  def etag(%__MODULE__{handle: handle, offset: offset, params: params, no_changes: true}, opts) do
+    "#{handle}:#{params.offset}:#{offset}:#{System.monotonic_time()}"
path: packages/sync-service/lib/electric/shapes/api/response.ex
user: login: KyleAMathews
body: add a comment
created_at: 2025-04-14T16:45:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -105,6 +107,79 @@ describe(`HTTP Proxy Cache`, { timeout: 30000 }, () => {
     expect(getCacheStatus(cachedRes)).toBe(CacheStatus.HIT)
   })
 
+  it.only(`should collapse requests in live mode`, async ({
path: packages/typescript-client/test/cache.test.ts
user: login: KyleAMathews
body: ```suggestion
  it(`should collapse requests in live mode`, async ({
```

Is this test still useful?
created_at: 2025-04-14T16:46:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -105,6 +107,79 @@ describe(`HTTP Proxy Cache`, { timeout: 30000 }, () => {
     expect(getCacheStatus(cachedRes)).toBe(CacheStatus.HIT)
   })
 
+  it.only(`should collapse requests in live mode`, async ({
path: packages/typescript-client/test/cache.test.ts
user: login: msfstef
body: I think so, good to have as a safety check like the other cache ones - I mostly added it cause I realised we didn't have that anywhere
created_at: 2025-04-14T16:47:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -554,9 +554,10 @@ defmodule Electric.Shapes.Api do
     %{response: response, global_last_seen_lsn: global_last_seen_lsn} =
       update_attrs(request, %{ot_is_empty_response: true})
 
-    %{
+    %Response{
       response
       | status: 200,
+        no_changes: true,
path: packages/sync-service/lib/electric/shapes/api.ex
user: login: msfstef
body: It sets the response as having "no changes" - the alternative would be to compare the response offset with the request offset when calculating the etag instead
created_at: 2025-04-14T16:48:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -554,9 +554,10 @@ defmodule Electric.Shapes.Api do
     %{response: response, global_last_seen_lsn: global_last_seen_lsn} =
       update_attrs(request, %{ot_is_empty_response: true})
 
-    %{
+    %Response{
       response
       | status: 200,
+        no_changes: true,
path: packages/sync-service/lib/electric/shapes/api.ex
user: login: KyleAMathews
body: Ah ok, so it's used down below for calculating the etag
created_at: 2025-04-14T17:01:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: thruflo
created_at: 2025-04-14T12:28:09Z
body: Very interesting! Have you verified the request collapsing behaviour with other reverse proxies / CDNs?
author_association: CONTRIBUTOR
user: login: netlify[bot]
created_at: 2025-04-14T12:39:55Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *electric-next* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 20ff8a93e78524f36cb34b8f94f7d49f65bf0076 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/electric-next/deploys/67fd2812e677170008b4b6e2 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-2593--electric-next.netlify.app](https://deploy-preview-2593--electric-next.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTI1OTMtLWVsZWN0cmljLW5leHQubmV0bGlmeS5hcHAifQ.Lj8UN9-SQBaDKNnnsjBpoJOZsr-QEaM8EBIMB602pAI)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [electric-next Preview](https://deploy-preview-2593--electric-next.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify site configuration](https://app.netlify.com/sites/electric-next/configuration/notifications#deploy-webhooks)._
author_association: NONE
user: login: msfstef
created_at: 2025-04-14T12:40:40Z
body: @thruflo I've added a request collapsing test that we didn't have before to our typescript client that uses Nginx's `proxy_cache_lock` collapsing behaviour - it asserts that if you run 10 clients at tip of stream and insert a row, you get 10 requests with 9 cache hits.

I haven't run this on Cloudflare but I definitely will, although I'd like to get @KyleAMathews 's insight on how he tested this before to ensure we get the same behaviour. Ideally we get some more caching behaviour tests in our suite as well.
author_association: CONTRIBUTOR
user: login: KyleAMathews
created_at: 2025-04-14T12:43:05Z
body: The behavior for Fastly/Cloudflare that I've seen is that once a request gets back `no-cache` then all subsequent requests to that URL won't get cache busted.

All CDNs/proxies behave a bit differently — which is why there's multiple layers of protection here. Writing actual tests that we can verify against all of them would be excellent. I just did a lot of manual testing w/ custom SST setups deploying to Fastly/Cloudflare.
author_association: CONTRIBUTOR
user: login: KyleAMathews
created_at: 2025-04-14T12:48:19Z
body: For more context, we didn't see any of these issues until I started testing on CDNs.
author_association: CONTRIBUTOR
user: login: msfstef
created_at: 2025-04-14T13:11:05Z
body: @KyleAMathews you're right and indeed Fastly collapses requests but then "uncollapses" them if the response is not cacheable:

![image](https://github.com/user-attachments/assets/a6c17b0e-cec0-4ef8-b148-fba70bf9e5ba)
from [their docs](https://www.fastly.com/documentation/guides/concepts/edge-state/cache/request-collapsing/)

Therefore if we _need_ to make the response cacheable we can never make sure we don't run into a cycle :/ so I think I can reinstate the cursor, which makes it safer to make this change as well, but still make sure we never _revalidate_ requests with no changes, which we can do by not including etags on them.

We could also skip revalidation for all live requests.
author_association: CONTRIBUTOR