Repository: pubpub/platform
Pull request #1220: Reapply "Add valkey cache handler to next app (#1210)" (#1215)
Author: login: kalilsn
Created: 2025-04-30T21:16:40Z | Merged: 2025-05-06T13:17:29Z

## Issue(s) Resolved
Resolves #1131
## High-level Explanation of PR
This PR changes core to use valkey as its cache, as opposed to the filesystem. Theoretically it should be automatically instrumented by sentry since we're using a supported redis client.

I haven't yet set a max size, because I'm really not sure what it should be. IMO we shouldn't really worry about that yet, since the large values are a problem for the node server which can't gracefully run out of memory (which we should fix), whereas valkey will simply evict keys. Once we have some data about how the cache is performing, we can tune that configuration more.

## Test Plan
- Start the app without valkey running. Ensure that it still serves requests.
- Start the app with valkey running. In a separate terminal, run `redis-cli monitor`. Send a few requests and verify that you  see valkey being used. Stop valkey (`pnpm -w dev:cache:stop`). Confirm the app still serves requests.
- not yet working without a manual restart of the app: After either of the above scenarios, when valkey is stopped, restart valkey without restarting the app. New requests should use the valkey cache.


Linked issue #1131: Autocache using redis
## Motivation
<!-- Why are we building this? -->
So we don't rely on file system based storage that slows down or crashes the app for large queries.

## Requirements
<!-- What does this accomplish? -->
- A [ValKey](https://github.com/valkey-io/valkey) service is added to our dev, self-host, and production environments.
- Next is configured to store data cached with `unstable_cache` in ValKey.
- A maximum entry size is set to prevent extremely large queries from being cached.

## Acceptance Criteria
<!-- What technical work must be completed for this to be accepted? -->
- Include the output of `mask tf plan` in the PR, and run `mask tf apply` only after the PR is approved.
- Justify the maximum entry size with some napkin math.

## Extra work
<!-- Not required, but could be added in if there's time -->
- Instrument ValKey to report cache hits/misses _somewhere_. Neither [Honeycomb](https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/), nor [Sentry](https://github.com/getsentry/sentry-python/issues/3543) support ValKey as a data source. So this might be tricky.

<!-- ## Mockups (if available) -->

<!-- ## Use this if you want to build a collapsible box
<details>
  <summary>Detail Box</summary>
- Thing one
- Thing two
</details>
-->

Linked issue comments: user: login: tefkah
created_at: 2025-04-05T11:05:57Z
body: See example here: https://github.com/vercel/next.js/blob/canary/examples/cache-handler-redis/README.md

One requirement id add is to allow us to set some maximum cache size, both for individual entries and for the total size of the cache stored. If the total size of the to-be-cached data > MAX_ENTRY_SIZE, it should not be stored. i think redis itself can probably handle the evicting of cached data if a certain limit is reached.

One suggestion id make is that we use [ValKey](https://github.com/valkey-io/valkey) rather than straight redis. I have a bit more faith in it continuing to exist in its current form than redis, and switching the backend later would be a hassle. but on the other hand redis is much more standardized ofc. 


author_association: MEMBER
user: login: tefkah
created_at: 2025-04-05T11:17:44Z
body: Also, and this might be too much: it would be nice if we could trace the cache hit rate somewhere, likely honeycomb. Don't really know the status of our honeycomb setup, but while we do have a (kind of jank admittedly) way of logging out whether there was a hit or not, it would be nice to also keep track of how many times a certain cache was used before it was busted
author_association: MEMBER
user: login: 3mcd
created_at: 2025-04-07T13:14:35Z
body: I updated the ticket to include ValKey as our cache db, but I want some feedback on whether or not the lack of instrumentation is a deal breaker for ValKey, since Redis seems supported by both Sentry and Honeycomb out of the box
author_association: COLLABORATOR
user: login: kalilsn
created_at: 2025-04-07T19:31:31Z
body: I don't think the automatic instrumentation support should guide our decision here. We can manually capture the cache metrics we're interested in as [open telemetry span attributes](https://opentelemetry.io/docs/languages/js/instrumentation/#attributes) in the debug logging code in the `memoize` function. If we want to capture metrics from ValKey itself, the [AWS cloudwatch metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html) should be plenty for now. Eventually the valkey collector should let us collect those with opentelemetry too: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/37005.

That said, I'd love to hear why not use redis - I know they changed their license but I don't really understand the implications.

Also, given the questions about honeycomb/sentry here, I want to point out that we are using sentry pretty heavily right now and honeycomb not at all. So maybe we don't need honeycomb? [Sentry can ingest our opentelemetry spans and metrics](https://docs.sentry.io/platforms/javascript/guides/node/opentelemetry/) so we wouldn't even have to change much.
author_association: CONTRIBUTOR
user: login: 3mcd
created_at: 2025-04-07T19:59:57Z
body: That all makes sense to me!

We should do a cross comparison to make sure we understand Honeycomb's intended role in our observability stack. I think Honeycomb was selected because it can take advantage of high-dimensional telemetry, e.g. able to construct views that consider many different dimensions (not just time-series). But I only partly know what I'm talking about here so we should regroup on this soon, maybe our next symposium meeting.

We'll stick with ValKey for this ticket. If the final implementation doesn't include metrics capture we can create a follow up ticket.
author_association: COLLABORATOR

Diff:
diff --git a/.env.docker-compose.dev b/.env.docker-compose.dev
index 09a6d46a4c..a9fc15eac3 100644
--- a/.env.docker-compose.dev
+++ b/.env.docker-compose.dev
@@ -12,4 +12,4 @@ POSTGRES_USER=postgres
 POSTGRES_PASSWORD=postgres
 POSTGRES_DB=postgres
 
-VALKEY_URL='redis://cache:6379'
\ No newline at end of file
+VALKEY_HOST='cache'
\ No newline at end of file
diff --git a/.env.docker-compose.test b/.env.docker-compose.test
index b2c1a5796b..0e8455c368 100644
--- a/.env.docker-compose.test
+++ b/.env.docker-compose.test
@@ -38,4 +38,4 @@ MAILGUN_SMTP_USERNAME=omitted
 OTEL_SERVICE_NAME=core.core
 PUBPUB_URL=http://localhost:3000
 API_KEY=xxx
-VALKEY_URL='redis://cache:6379'
\ No newline at end of file
+VALKEY_HOST='cache'
\ No newline at end of file
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index bf096df68b..9709483221 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -132,7 +132,7 @@ jobs:
 
             - name: Print container logs
               if: ${{failure() || cancelled()}}
-              run: docker compose -f docker-compose.test.yml --profile integration logs
+              run: docker compose -f docker-compose.test.yml --profile integration logs -t
               env:
                   INTEGRATION_TESTS_IMAGE: ${{steps.label.outputs.core_label}}
                   JOBS_IMAGE: ${{steps.label.outputs.jobs_label}}
diff --git a/Dockerfile b/Dockerfile
index f23946bb53..ffa93e6465 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -15,7 +15,7 @@ ARG PNPM_VERSION=9.10.0
 
 ################################################################################
 # Use node image for base image for all stages.
-FROM node:${NODE_VERSION}-alpine${ALPINE_VERSION} as base
+FROM node:${NODE_VERSION}-alpine${ALPINE_VERSION} AS base
 
 # these are necessary to be able to use them inside of `base`
 ARG BASE_IMAGE
@@ -39,22 +39,22 @@ RUN --mount=type=cache,target=/root/.npm \
   npm install -g pnpm@${PNPM_VERSION}
 
 
-FROM base as fetch-deps
+FROM base AS fetch-deps
 
 
 # Copy pnpm-lock.yaml so that we can use pnpm to install dependencies
 COPY pnpm-lock.yaml ./
 
-# Could possibly be sped up using `turbo prune` 
+# Could possibly be sped up using `turbo prune`
 # https://turbo.build/repo/docs/guides/tools/docker
 RUN pnpm fetch
 
 # Install dependencies we only need to run pnpm install
-RUN apk add g++ make py3-pip 
+RUN apk add g++ make py3-pip
 
 ################################################################################
 # Create a stage for building the application.
-FROM fetch-deps as monorepo
+FROM fetch-deps AS monorepo
 
 # Copy over the rest of the files
 ADD . ./
@@ -76,19 +76,19 @@ RUN test -n "$PACKAGE" || (echo "PACKAGE  not set, required for this target" &&
 ENV DOCKERBUILD=1
 
 ARG CI
-ENV CI $CI
+ENV CI=$CI
 
 RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN,env=SENTRY_AUTH_TOKEN \
-  pnpm --filter $PACKAGE build 
+  pnpm --filter $PACKAGE build
 
-FROM withpackage as prepare-jobs
+FROM withpackage AS prepare-jobs
 
 ARG PACKAGE
 
 RUN pnpm --filter $PACKAGE --prod deploy /tmp/app
 
 
-FROM base as jobs
+FROM base AS jobs
 
 WORKDIR /usr/src/app
 
@@ -96,19 +96,19 @@ COPY --from=prepare-jobs --chown=node:node /tmp/app .
 
 USER node
 
-CMD pnpm start
+CMD ["pnpm", "start"]
 
 ################################################################################
 # Create a new stage to run the application with minimal runtime dependencies
 # where the necessary files are copied from the build stage.
 # this is separated by package to make it slightly more clear what happens
 # and because you cannot conditionally copy from a different folder
-# based on the argument 
-FROM base as prod-setup
+# based on the argument
+FROM base AS prod-setup
 ARG PORT
 
 # Use production node environment by default.
-ENV NODE_ENV production
+ENV NODE_ENV=production
 
 # Run the application as a non-root user.
 USER node
@@ -117,10 +117,10 @@ USER node
 EXPOSE $PORT
 
 # Use production node environment by default.
-ENV NODE_ENV production
+ENV NODE_ENV=production
 
 # otherwise it will use the strange default docker hostname
-ENV HOSTNAME "0.0.0.0"
+ENV HOSTNAME="0.0.0.0"
 
 ### Core
 
@@ -132,4 +132,4 @@ COPY --from=withpackage --chown=node:node /usr/src/app/core/public ./core/public
 # needed to set the database url correctly based on PGHOST variables
 COPY --from=withpackage --chown=node:node /usr/src/app/core/.env.docker ./core/.env
 
-CMD node core/server.js
+CMD ["node", "core/server.js"]
diff --git a/core/.env.development b/core/.env.development
index fc87fb9514..c30d247cc9 100644
--- a/core/.env.development
+++ b/core/.env.development
@@ -25,4 +25,4 @@ DATACITE_PASSWORD=""
 DATACITE_API_URL="https://api.test.datacite.org"
 
 GCLOUD_KEY_FILE='xxx'
-VALKEY_URL='redis://cache:6379'
\ No newline at end of file
+VALKEY_HOST='localhost'
\ No newline at end of file
diff --git a/core/.env.test b/core/.env.test
index 47fdfe7d93..f90d463c1b 100644
--- a/core/.env.test
+++ b/core/.env.test
@@ -19,4 +19,4 @@ HONEYCOMB_API_KEY="xxx"
 # KYSELY_DEBUG="true"
 
 GCLOUD_KEY_FILE='xxx'
-VALKEY_URL='redis://cache:6379'
\ No newline at end of file
+VALKEY_HOST='cache'
\ No newline at end of file
diff --git a/core/app/api/health/route.ts b/core/app/api/health/route.ts
index 1be19c1a65..ac0c7a2e31 100644
--- a/core/app/api/health/route.ts
+++ b/core/app/api/health/route.ts
@@ -2,25 +2,25 @@ import type { NextRequest } from "next/server";
 
 import { NextResponse } from "next/server";
 
+import { logger } from "logger";
+
 import { db } from "~/kysely/database";
+import { getRedisClient } from "~/lib/redis";
 import { handleErrors } from "~/lib/server";
 
 export async function GET(req: NextRequest) {
 	return await handleErrors(async () => {
-		const errors: string[] = [];
 		try {
-			const dbQuery = await db
+			const dbQuery = db
 				.selectFrom("communities")
-				.selectAll()
+				.select("id")
+				.limit(1)
 				.executeTakeFirstOrThrow();
+			const cacheQuery = (await getRedisClient()).ping();
+			await Promise.all([dbQuery, cacheQuery]);
 		} catch (err) {
-			if (err instanceof Error) {
-				errors.push(err.message);
-			}
-		}
-
-		if (errors.length > 0) {
-			return NextResponse.json({ errors }, { status: 500 });
+			logger.error({ msg: "error in health check", err });
+			return NextResponse.json({}, { status: 500 });
 		}
 
 		return NextResponse.json({});
diff --git a/core/cache-handler.mjs b/core/cache-handler.mjs
index ea52f41d35..5e08069d9d 100644
--- a/core/cache-handler.mjs
+++ b/core/cache-handler.mjs
@@ -1,11 +1,258 @@
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
+
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import { CacheHandler } from "@neshca/cache-handler";
+import { isImplicitTag } from "@neshca/cache-handler/helpers";
+import Redis from "ioredis";
+
+var REVALIDATED_TAGS_KEY = "__revalidated_tags__";
+
 /**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
+ * Creates a redis handler based on fortedigital's redis-strings handler, but using the ioredis
+ * client
  *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
+ * @param {{ client: Redis}} props
  */
+function createRedisHandler({
+	client,
+	keyPrefix = "",
+	sharedTagsKey = "__sharedTags__",
+	sharedTagsTtlKey = "__sharedTagsTtl__",
+	revalidateTagQuerySize = 1e4,
+}) {
+	function assertClientIsReady() {
+		if (!client.status === "ready") {
+			// Throwing here ensures that we immediately fall back to uncached behavior, rather than
+			// waiting for the command timeout
+			throw new Error("Redis client is not ready yet or connection is lost.");
+		}
+	}
+	async function revalidateTags(tag) {
+		const tagsMap = /* @__PURE__ */ new Map();
+		let cursor = 0;
+		let remoteTagsPortion;
+		do {
+			[cursor, remoteTagsPortion] = await client.hscan(
+				keyPrefix + sharedTagsKey,
+				cursor,
+				"COUNT",
+				revalidateTagQuerySize
+			);
+
+			for (let i = 0; i < remoteTagsPortion.length; i += 2) {
+				const field = remoteTagsPortion[i];
+				const value = remoteTagsPortion[i + 1];
+				tagsMap.set(field, JSON.parse(value));
+			}
+		} while (cursor !== "0");
+		const keysToDelete = [];
+		const tagsToDelete = [];
+		for (const [key, tags] of tagsMap) {
+			if (tags.includes(tag)) {
+				keysToDelete.push(keyPrefix + key);
+				tagsToDelete.push(key);
+			}
+		}
+		if (keysToDelete.length === 0) {
+			return;
+		}
+		await client.unlink(keysToDelete);
+		const updateTagsOperation = client.hdel(keyPrefix + sharedTagsKey, tagsToDelete);
+		const updateTtlOperation = client.hdel(keyPrefix + sharedTagsTtlKey, tagsToDelete);
+		await Promise.all([updateTtlOperation, updateTagsOperation]);
+	}
+	async function revalidateSharedKeys() {
+		const ttlMap = /* @__PURE__ */ new Map();
+		let cursor = 0;
+		let remoteTagsPortion;
+		do {
+			[cursor, remoteTagsPortion] = await client.hscan(
+				keyPrefix + sharedTagsKey,
+				cursor,
+				"COUNT",
+				revalidateTagQuerySize
+			);
+
+			for (let i = 0; i < remoteTagsPortion.length; i += 2) {
+				const field = remoteTagsPortion[i];
+				const value = remoteTagsPortion[i + 1];
+				ttlMap.set(field, Number(value));
+			}
+		} while (cursor !== "0");
+		const tagsAndTtlToDelete = [];
+		const keysToDelete = [];
+		for (const [key, ttlInSeconds] of ttlMap) {
+			if (/* @__PURE__ */ new Date().getTime() > ttlInSeconds * 1e3) {
+				tagsAndTtlToDelete.push(key);
+				keysToDelete.push(keyPrefix + key);
+			}
+		}
+		if (tagsAndTtlToDelete.length === 0) {
+			return;
+		}
+		await client.unlink(keysToDelete);
+		const updateTtlOperation = client.hdel(keyPrefix + sharedTagsTtlKey, tagsAndTtlToDelete);
+		const updateTagsOperation = client.hdel(keyPrefix + sharedTagsKey, tagsAndTtlToDelete);
+		await Promise.all([updateTagsOperation, updateTtlOperation]);
+	}
+	const revalidatedTagsKey = keyPrefix + REVALIDATED_TAGS_KEY;
+	return {
+		name: "pubpub-redis-strings",
+		async get(key, { implicitTags }) {
+			try {
+				assertClientIsReady();
+				const result = await client.get(keyPrefix + key);
+				if (!result) {
+					return null;
+				}
+				const cacheValue = JSON.parse(result);
+				if (!cacheValue) {
+					return null;
+				}
+				const sharedTagKeyExists = await client.hexists(keyPrefix + sharedTagsKey, key);
+				if (!sharedTagKeyExists) {
+					await client.unlink(keyPrefix + key);
+					return null;
+				}
+				const combinedTags = /* @__PURE__ */ new Set([...cacheValue.tags, ...implicitTags]);
+				if (combinedTags.size === 0) {
+					return cacheValue;
+				}
+				const revalidationTimes = await client.hmget(
+					revalidatedTagsKey,
+					Array.from(combinedTags)
+				);
+				for (const timeString of revalidationTimes) {
+					if (timeString && Number.parseInt(timeString, 10) > cacheValue.lastModified) {
+						await client.unlink(keyPrefix + key);
+						return null;
+					}
+				}
+				return cacheValue;
+			} catch (err) {
+				console.err("Cache get err", err);
+				throw err;
+			}
+		},
+		async set(key, cacheHandlerValue) {
+			try {
+				assertClientIsReady();
+				const lifespan = cacheHandlerValue.lifespan;
+				const setTagsOperation =
+					cacheHandlerValue.tags.length > 0
+						? client.hset(
+								keyPrefix + sharedTagsKey,
+								key,
+								JSON.stringify(cacheHandlerValue.tags)
+							)
+						: void 0;
+				const setSharedTtlOperation = lifespan
+					? client.hset(keyPrefix + sharedTagsTtlKey, key, lifespan.expireAt)
+					: void 0;
+				await Promise.all([setTagsOperation, setSharedTtlOperation]);
+				if (typeof lifespan?.expireAt === "number") {
+					await client.set(
+						keyPrefix + key,
+						JSON.stringify(cacheHandlerValue),
+						"EXAT",
+						lifespan.expireAt
+					);
+				} else {
+					await client.set(keyPrefix + key, JSON.stringify(cacheHandlerValue));
+				}
+			} catch (err) {
+				console.error("Cache set error", err);
+				throw err;
+			}
+		},
+		async revalidateTag(tag) {
+			try {
+				assertClientIsReady();
+				if (isImplicitTag(tag)) {
+					await client.hset(revalidatedTagsKey, tag, Date.now());
+				}
+				await Promise.all([revalidateTags(tag), revalidateSharedKeys()]);
+			} catch (err) {
+				console.error("Cache revalidate error", err);
+				throw err;
+			}
+		},
+		async delete(key) {
+			await client.unlink(keyPrefix + key);
+			await Promise.all([
+				client.hdel(keyPrefix + sharedTagsKey, key),
+				client.hdel(keyPrefix + sharedTagsTtlKey, key),
+			]);
+		},
+	};
+}
+
+async function getCacheHandlerPromise() {
+	let redisClient = null;
+	if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+		try {
+			redisClient = new Redis({
+				host: process.env.VALKEY_HOST,
+				lazyConnect: true,
+				commandTimeout: 1000,
+				retryStrategy: (times) => {
+					if (times >= 15) {
+						return;
+					}
+					console.log("Retrying redis connection attempt:", times);
+					return Math.pow(2, times) + Math.random() * 1000;
+				},
+			});
+
+			await redisClient.connect();
+			console.log("Successfully connected to redis for caching");
+			redisClient.on("error", (err) => {
+				console.error("Cache error:", err);
+			});
+		} catch (err) {
+			console.error("Failed to create Redis client:", err);
+		}
+	}
+	if (redisClient?.status !== "ready") {
+		console.error("Failed to initialize caching layer.");
+	}
+
+	const redisCacheHandler = createBufferStringHandler(
+		createRedisHandler({
+			client: redisClient,
+			keyPrefix: "nextjs:",
+		})
+	);
+
+	global.cacheHandlerConfigPromise = null;
+
+	global.cacheHandlerConfig = {
+		handlers: [redisCacheHandler],
+	};
+
+	return global.cacheHandlerConfig;
+}
+
+// Usual onCreation from @neshca/cache-handler
+CacheHandler.onCreation(() => {
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures only one instance get created
+	if (global.cacheHandlerConfig) {
+		return global.cacheHandlerConfig;
+	}
+
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures new instances are not created in a race condition
+	if (global.cacheHandlerConfigPromise) {
+		return global.cacheHandlerConfigPromise;
+	}
+
+	// Main promise initializing the handler
+	global.cacheHandlerConfigPromise = getCacheHandlerPromise();
 
-import FileSystemCache from "next/dist/server/lib/incremental-cache/file-system-cache.js";
+	return global.cacheHandlerConfigPromise;
+});
 
-export default FileSystemCache;
+export default new Next15CacheHandler();
diff --git a/core/instrumentation.node.mts b/core/instrumentation.node.mts
index 56f33a1524..3efb12f679 100644
--- a/core/instrumentation.node.mts
+++ b/core/instrumentation.node.mts
@@ -19,6 +19,11 @@ if (env.NODE_ENV === "production") {
 
 		// Setting this option to true will print useful information to the console while you're setting up Sentry.
 		debug: false,
+		integrations: [
+			Sentry.redisIntegration({
+				cachePrefixes: ["nextjs:"],
+			}),
+		],
 	});
 	logger.info("✅ Successfully instrumented Sentry");
 }
diff --git a/core/lib/env/env.ts b/core/lib/env/env.ts
index 89c15bb90e..7e707e5f5b 100644
--- a/core/lib/env/env.ts
+++ b/core/lib/env/env.ts
@@ -25,7 +25,7 @@ export const env = createEnv({
 		 * Whether or not to verbosely log `memoize` cache hits and misses
 		 */
 		CACHE_LOG: z.string().optional(),
-		VALKEY_URL: z.string().url(),
+		VALKEY_HOST: z.string(),
 		DATABASE_URL: z.string().url(),
 		ENV_NAME: z.string().optional(),
 		FLAGS: flagsSchema,
diff --git a/core/lib/redis.ts b/core/lib/redis.ts
new file mode 100644
index 0000000000..c7b99bfada
--- /dev/null
+++ b/core/lib/redis.ts
@@ -0,0 +1,39 @@
+import { captureException } from "@sentry/nextjs";
+import Redis from "ioredis";
+
+import { logger } from "logger";
+
+import { env } from "./env/env";
+
+let redisClient: Redis;
+
+export const getRedisClient = async () => {
+	if (!redisClient) {
+		logger.info({ msg: "Creating redis client" });
+		redisClient = new Redis({
+			host: env.VALKEY_HOST,
+			lazyConnect: true,
+			commandTimeout: 1000,
+			retryStrategy: (times) => {
+				if (times >= 15) {
+					return;
+				}
+				return Math.pow(2, times) + Math.random() * 1000;
+			},
+		});
+	}
+
+	if (redisClient.status !== "ready") {
+		logger.info({ msg: "Connecting redis client" });
+		try {
+			await redisClient.connect();
+			logger.info({ msg: "Redis client connected" });
+		} catch (err) {
+			logger.error("Failed to connect Redis client", err);
+			captureException(err);
+			redisClient.disconnect();
+		}
+	}
+
+	return redisClient;
+};
diff --git a/core/package.json b/core/package.json
index 2382dfa78a..abd06d27c2 100644
--- a/core/package.json
+++ b/core/package.json
@@ -62,11 +62,13 @@
 		"@dnd-kit/sortable": "^8.0.0",
 		"@dnd-kit/utilities": "^3.2.2",
 		"@faker-js/faker": "^9.0.0",
+		"@fortedigital/nextjs-cache-handler": "^1.2.0",
 		"@googleapis/drive": "^8.16.0",
 		"@handlewithcare/react-prosemirror": "catalog:",
 		"@honeycombio/opentelemetry-node": "catalog:",
 		"@hookform/resolvers": "catalog:",
 		"@icons-pack/react-simple-icons": "^10.2.0",
+		"@neshca/cache-handler": "^1.9.0",
 		"@nimpl/getters": "^2.0.0",
 		"@node-rs/argon2": "^1.8.3",
 		"@opentelemetry/auto-instrumentations-node": "catalog:",
@@ -99,6 +101,7 @@
 		"hast": "^1.0.0",
 		"hastscript": "^9.0.0",
 		"import-in-the-middle": "^1.13.1",
+		"ioredis": "^5.6.1",
 		"jsonpath-plus": "^10.2.0",
 		"jsonwebtoken": "^9.0.0",
 		"katex": "catalog:",
diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml
index 8d9843816f..ed6bc530b6 100644
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -45,6 +45,8 @@ services:
             service: cache
         networks:
             - app-network
+        ports:
+            - 6379:6379
 
     minio-init:
         env_file:
diff --git a/infrastructure/terraform/modules/container-generic/main.tf b/infrastructure/terraform/modules/container-generic/main.tf
index fc253030d2..0bfed825e9 100644
--- a/infrastructure/terraform/modules/container-generic/main.tf
+++ b/infrastructure/terraform/modules/container-generic/main.tf
@@ -49,6 +49,11 @@ module "ecs_service" {
         containerPort = var.listener.to_port
       }] : []
 
+      restartPolicy = {
+        enabled              = true
+        restartAttemptPeriod = 60
+      }
+
       # use concat() to add a computible variable
       environment = concat(
         var.configuration.environment,
@@ -158,13 +163,13 @@ resource "aws_lb_target_group" "this" {
   # when the container does not provide a more meaningful
   # one.
   health_check {
-    path                = "/legacy_healthcheck"
-    interval            = "30"
+    path                = coalesce(var.health_check_path, "/legacy_healthcheck")
+    interval            = "5"
     protocol            = "HTTP"
     matcher             = "200"
-    timeout             = "5"
+    timeout             = "2"
     unhealthy_threshold = "3"
-    healthy_threshold   = "5"
+    healthy_threshold   = "3"
   }
 }
 
diff --git a/infrastructure/terraform/modules/container-generic/variables.tf b/infrastructure/terraform/modules/container-generic/variables.tf
index 92de7ba07a..5f2ed802be 100644
--- a/infrastructure/terraform/modules/container-generic/variables.tf
+++ b/infrastructure/terraform/modules/container-generic/variables.tf
@@ -99,3 +99,9 @@ variable "command" {
   default = []
   type    = list(string)
 }
+
+variable "health_check_path" {
+  description = "A path to an endpoint on the container suitable for use as a health check"
+  type        = string
+  default     = null
+}
diff --git a/infrastructure/terraform/modules/core-services/outputs.tf b/infrastructure/terraform/modules/core-services/outputs.tf
index 7e57f8801e..f3a61f31d5 100644
--- a/infrastructure/terraform/modules/core-services/outputs.tf
+++ b/infrastructure/terraform/modules/core-services/outputs.tf
@@ -35,6 +35,6 @@ output "rds_connection_components" {
   }
 }
 
-output "valkey_url" {
+output "valkey_host" {
   value = aws_elasticache_replication_group.core_valkey.primary_endpoint_address
 }
diff --git a/infrastructure/terraform/modules/deployment/main.tf b/infrastructure/terraform/modules/deployment/main.tf
index 10a9dd6ab9..597fb9a0d0 100644
--- a/infrastructure/terraform/modules/deployment/main.tf
+++ b/infrastructure/terraform/modules/deployment/main.tf
@@ -52,8 +52,9 @@ module "service_core" {
   service_name = "core"
   cluster_info = module.cluster.cluster_info
 
-  repository_url = var.ecr_repository_urls.core
-  nginx_image    = "${var.ecr_repository_urls.nginx}:latest"
+  repository_url    = var.ecr_repository_urls.core
+  nginx_image       = "${var.ecr_repository_urls.nginx}:latest"
+  health_check_path = "/api/health"
 
   listener = {
     service_name  = "core"
@@ -100,7 +101,7 @@ module "service_core" {
       { name = "SUPABASE_PUBLIC_KEY", value = var.NEXT_PUBLIC_SUPABASE_PUBLIC_KEY },
       { name = "HOSTNAME", value = var.HOSTNAME },
       { name = "DATACITE_API_URL", value = var.DATACITE_API_URL },
-      { name = "VALKEY_URL", value = "redis://${module.core_dependency_services.valkey_url}" }
+      { name = "VALKEY_HOST", value = module.core_dependency_services.valkey_host }
     ]
 
     secrets = [
@@ -167,7 +168,7 @@ module "service_bastion" {
       { name = "SUPABASE_URL", value = var.NEXT_PUBLIC_SUPABASE_URL },
       { name = "HOSTNAME", value = var.HOSTNAME },
       { name = "PAGER", value = "less -S" },
-      { name = "VALKEY_URL", value = "redis://${module.core_dependency_services.valkey_url}" }
+      { name = "VALKEY_HOST", value = module.core_dependency_services.valkey_host }
     ]
 
     secrets = [
diff --git a/package.json b/package.json
index 3f8466deee..7edaf9ab9c 100644
--- a/package.json
+++ b/package.json
@@ -27,6 +27,8 @@
 		"dev:inbucket:stop": "docker compose -f docker-compose.dev.yml down inbucket",
 		"dev:minio:start": "docker compose -f docker-compose.dev.yml up minio -d && docker compose -f docker-compose.dev.yml run minio-init",
 		"dev:minio:stop": "docker compose -f docker-compose.dev.yml down minio",
+		"dev:cache:start": "docker compose -f docker-compose.dev.yml up cache -d",
+		"dev:cache:stop": "docker compose -f docker-compose.dev.yml down cache",
 		"dev:setup": "pnpm install && docker compose -f docker-compose.dev.yml up -d && pnpm p:dev && pnpm --filter core migrate-dev && pnpm --filter core reset",
 		"dev:teardown": "docker compose -f docker-compose.dev.yml down -v",
 		"integration:setup": "docker compose -f docker-compose.test.yml --profile integration up -d",
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 8f71c72c1e..b26ac04dd9 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -54,9 +54,6 @@ catalogs:
     date-fns:
       specifier: ^4.1.0
       version: 4.1.0
-    eslint:
-      specifier: ^9.9.0
-      version: 9.10.0
     katex:
       specifier: ^0.16.18
       version: 0.16.21
@@ -69,9 +66,6 @@ catalogs:
     postcss:
       specifier: ^8.4.27
       version: 8.4.49
-    prettier:
-      specifier: ^3.4.2
-      version: 3.4.2
     react-hook-form:
       specifier: ^7.54.2
       version: 7.54.2
@@ -272,6 +266,9 @@ importers:
       '@faker-js/faker':
         specifier: ^9.0.0
         version: 9.0.0
+      '@fortedigital/nextjs-cache-handler':
+        specifier: ^1.2.0
+        version: 1.2.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(redis@4.7.0)
       '@googleapis/drive':
         specifier: ^8.16.0
         version: 8.16.0
@@ -287,6 +284,9 @@ importers:
       '@icons-pack/react-simple-icons':
         specifier: ^10.2.0
         version: 10.2.0(react@19.0.0)
+      '@neshca/cache-handler':
+        specifier: ^1.9.0
+        version: 1.9.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(redis@4.7.0)
       '@nimpl/getters':
         specifier: ^2.0.0
         version: 2.0.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(react-dom@19.0.0(react@19.0.0))(react@19.0.0)
@@ -383,6 +383,9 @@ importers:
       import-in-the-middle:
         specifier: ^1.13.1
         version: 1.13.1
+      ioredis:
+        specifier: ^5.6.1
+        version: 5.6.1
       jsonpath-plus:
         specifier: ^10.2.0
         version: 10.2.0
@@ -3105,6 +3108,12 @@ packages:
   '@formatjs/intl-localematcher@0.6.0':
     resolution: {integrity: sha512-4rB4g+3hESy1bHSBG3tDFaMY2CH67iT7yne1e+0CLTsGLDcmoEWWpJjjpWVaYgYfYuohIRuo0E+N536gd2ZHZA==}
 
+  '@fortedigital/nextjs-cache-handler@1.2.0':
+    resolution: {integrity: sha512-dHu7+D6yVHI5ii1/DgNSZM9wVPk8uKAB0zrRoNNbZq6hggpRRwAExV4J6bSGOd26RN6ZnfYaGLBmdb0gLpeBQg==}
+    peerDependencies:
+      next: '>=13.5.1'
+      redis: '>=4.6'
+
   '@googleapis/drive@8.16.0':
     resolution: {integrity: sha512-Xi2mMrUTQ+gsfyouRGd0pfnL+jjg4n4sjKsJruM1y4DknuRfdSBTk5E//WrL0YJ/CqpcBgyd7L8DvaPRtxZD3Q==}
     engines: {node: '>=12.0.0'}
@@ -3286,6 +3295,9 @@ packages:
     cpu: [x64]
     os: [win32]
 
+  '@ioredis/commands@1.2.0':
+    resolution: {integrity: sha512-Sx1pU8EM64o2BrqNpEO1CNLtKQwyhuXuqyfH7oGKCk+1a33d2r5saW8zNwm3j6BTExtjrv2BxTgzzkMwts6vGg==}
+
   '@isaacs/cliui@8.0.2':
     resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==}
     engines: {node: '>=12'}
@@ -3563,6 +3575,12 @@ packages:
   '@napi-rs/wasm-runtime@0.2.4':
     resolution: {integrity: sha512-9zESzOO5aDByvhIAsOy9TbpZ0Ur2AJbUI7UT73kcUTS2mxAMHOBaa1st/jAymNoCtvrit99kkzT1FZuXVcgfIQ==}
 
+  '@neshca/cache-handler@1.9.0':
+    resolution: {integrity: sha512-dh0x4pdjDKvPRfZF5DZb8TtOUkbBfeTodOUdQsHDuv0oiuqQ3p7GLx38f6bPn8Sa4he8HsWo+rM4S20ZRqr7pA==}
+    peerDependencies:
+      next: '>= 13.5.1 < 15'
+      redis: '>= 4.6'
+
   '@next/env@15.0.4':
     resolution: {integrity: sha512-WNRvtgnRVDD4oM8gbUcRc27IAhaL4eXQ/2ovGbgLnPGUvdyDr8UdXP4Q/IBDdAdojnD2eScryIDirv0YUCjUVw==}
 
@@ -5650,6 +5668,35 @@ packages:
       react: '>=17'
       react-dom: '>=17'
 
+  '@redis/bloom@1.2.0':
+    resolution: {integrity: sha512-HG2DFjYKbpNmVXsa0keLHp/3leGJz1mjh09f2RLGGLQZzSHpkmZWuwJbAvo3QcRY8p80m5+ZdXZdYOSBLlp7Cg==}
+    peerDependencies:
+      '@redis/client': ^1.0.0
+
+  '@redis/client@1.6.0':
+    resolution: {integrity: sha512-aR0uffYI700OEEH4gYnitAnv3vzVGXCFvYfdpu/CJKvk4pHfLPEy/JSZyrpQ+15WhXe1yJRXLtfQ84s4mEXnPg==}
+    engines: {node: '>=14'}
+
+  '@redis/graph@1.1.1':
+    resolution: {integrity: sha512-FEMTcTHZozZciLRl6GiiIB4zGm5z5F3F6a6FZCyrfxdKOhFlGkiAqlexWMBzCi4DcRoyiOsuLfW+cjlGWyExOw==}
+    peerDependencies:
+      '@redis/client': ^1.0.0
+
+  '@redis/json@1.0.7':
+    resolution: {integrity: sha512-6UyXfjVaTBTJtKNG4/9Z8PSpKE6XgSyEb8iwaqDcy+uKrd/DGYHTWkUdnQDyzm727V7p21WUMhsqz5oy65kPcQ==}
+    peerDependencies:
+      '@redis/client': ^1.0.0
+
+  '@redis/search@1.2.0':
+    resolution: {integrity: sha512-tYoDBbtqOVigEDMAcTGsRlMycIIjwMCgD8eR2t0NANeQmgK/lvxNAvYyb6bZDD4frHRhIHkJu2TBRvB0ERkOmw==}
+    peerDependencies:
+      '@redis/client': ^1.0.0
+
+  '@redis/time-series@1.1.0':
+    resolution: {integrity: sha512-c1Q99M5ljsIuc4YdaCwfUEXsofakb9c8+Zse2qxTadu8TalLXuAESzLvFAvNVbkmSlvlzIQOLpBCmWI9wTOt+g==}
+    peerDependencies:
+      '@redis/client': ^1.0.0
+
   '@remirror/core-constants@3.0.0':
     resolution: {integrity: sha512-42aWfPrimMfDKDi4YegyS7x+/0tlzaqwPQCULLanv3DMIlu96KTJR0fM5isWX2UViOqlGnX6YFgqWepcX+XMNg==}
 
@@ -5846,6 +5893,11 @@ packages:
     cpu: [x64]
     os: [linux]
 
+  '@rollup/rollup-linux-x64-gnu@4.40.1':
+    resolution: {integrity: sha512-XiK5z70PEFEFqcNj3/zRSz/qX4bp4QIraTy9QjwJAb/Z8GM7kVUsD0Uk8maIPeTyPCP03ChdI+VVmJriKYbRHQ==}
+    cpu: [x64]
+    os: [linux]
+
   '@rollup/rollup-linux-x64-musl@4.21.2':
     resolution: {integrity: sha512-7twFizNXudESmC9oneLGIUmoHiiLppz/Xs5uJQ4ShvE6234K0VB1/aJYU3f/4g7PhssLGKBVCC37uRkkOi8wjg==}
     cpu: [x64]
@@ -7931,6 +7983,10 @@ packages:
     resolution: {integrity: sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==}
     engines: {node: '>=6'}
 
+  cluster-key-slot@1.1.2:
+    resolution: {integrity: sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==}
+    engines: {node: '>=0.10.0'}
+
   cmdk@1.0.4:
     resolution: {integrity: sha512-AnsjfHyHpQ/EFeAnG216WY7A5LiYCoZzCSygiLvfXC3H3LFGCprErteUcszaVluGOhuOTbJS3jWHrSDYPBBygg==}
     peerDependencies:
@@ -8440,6 +8496,10 @@ packages:
     resolution: {integrity: sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==}
     engines: {node: '>=0.4.0'}
 
+  denque@2.1.0:
+    resolution: {integrity: sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==}
+    engines: {node: '>=0.10'}
+
   dequal@2.0.3:
     resolution: {integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==}
     engines: {node: '>=6'}
@@ -9073,6 +9133,10 @@ packages:
     resolution: {integrity: sha512-Jh/AIwwgaxan+7ZUUmRLCjtchyDiqh4KjBJ5tW3plBZb5iL/BPcso8A5DlzeD9qlw0duCamnNdpFjxwaT0KyKg==}
     engines: {node: '>=14'}
 
+  generic-pool@3.9.0:
+    resolution: {integrity: sha512-hymDOu5B53XvN4QT9dBmZxPX4CWhBPPLguTZ9MMFeFa/Kg0xWVfylOVNlJji/E7yTZWFd/q9GO5TxDLq156D7g==}
+    engines: {node: '>= 4'}
+
   gensync@1.0.0-beta.2:
     resolution: {integrity: sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg==}
     engines: {node: '>=6.9.0'}
@@ -9515,6 +9579,10 @@ packages:
     resolution: {integrity: sha512-6xwYfHbajpoF0xLW+iwLkhwgvLoZDfjYfoFNu8ftMoXINzwuymNLd9u/KmwtdT2GbR+/Cz66otEGEVVUHX9QLQ==}
     engines: {node: '>=10.13.0'}
 
+  ioredis@5.6.1:
+    resolution: {integrity: sha512-UxC0Yv1Y4WRJiGQxQkP0hfdL0/5/6YvdfOOClRgJ0qppSarkhneSa6UvkMkms0AkdGimSH3Ikqm+6mkMmX7vGA==}
+    engines: {node: '>=12.22.0'}
+
   ip-address@9.0.5:
     resolution: {integrity: sha512-zHtQzGojZXTwZTHQqra+ETKd4Sn3vgi7uBmlPoXVWZqYvuKmtI0l/VZTjqGmJY9x88GGOaZ9+G9ES8hC4T4X8g==}
     engines: {node: '>= 12'}
@@ -9924,7 +9992,6 @@ packages:
     resolution: {integrity: sha512-t0etAxTUk1w5MYdNOkZBZ8rvYYN5iL+2dHCCx/DpkFm/bW28M6y5nUS83D4XdZiHy35Fpaw6LBb+F88fHZnVCw==}
     engines: {node: '>=8.17.0'}
     hasBin: true
-    bundledDependencies: []
 
   jsonfile@6.1.0:
     resolution: {integrity: sha512-5dgndWOriYSm5cnYaJNhalLNDKOqFwyDB/rr1E9ZsGciGvKPs8R2xYGCacuf3z6K1YKDz182fd+fY3cn3pMqXQ==}
@@ -10209,6 +10276,9 @@ packages:
   lodash.includes@4.3.0:
     resolution: {integrity: sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==}
 
+  lodash.isarguments@3.1.0:
+    resolution: {integrity: sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==}
+
   lodash.isboolean@3.0.3:
     resolution: {integrity: sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg==}
 
@@ -10276,8 +10346,8 @@ packages:
   lru-cache@10.4.3:
     resolution: {integrity: sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==}
 
-  lru-cache@11.0.1:
-    resolution: {integrity: sha512-CgeuL5uom6j/ZVrg7G/+1IXqRY8JXX4Hghfy5YE0EhoYQWvndP1kufu58cmZLNIDKnRhZrXfdS9urVWx98AipQ==}
+  lru-cache@11.1.0:
+    resolution: {integrity: sha512-QIXZUBJUx+2zHUdQujWejBkcD9+cs94tLn0+YL8UrCh+D5sCXZ4c7LaEH48pNwRY3MLDgqUFyhlCyjJPf1WP0A==}
     engines: {node: 20 || >=22}
 
   lru-cache@5.1.1:
@@ -11825,6 +11895,17 @@ packages:
     resolution: {integrity: sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg==}
     engines: {node: '>=8'}
 
+  redis-errors@1.2.0:
+    resolution: {integrity: sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==}
+    engines: {node: '>=4'}
+
+  redis-parser@3.0.0:
+    resolution: {integrity: sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==}
+    engines: {node: '>=4'}
+
+  redis@4.7.0:
+    resolution: {integrity: sha512-zvmkHEAdGMn+hMRXuMBtu4Vo5P6rHQjLoHftu+lBqq8ZTA3RCVC/WzD790bkKKiNFp7d5/9PcSD19fJyyRvOdQ==}
+
   reflect.getprototypeof@1.0.10:
     resolution: {integrity: sha512-00o4I+DVrefhv+nX0ulyi3biSHCPDe+yLv5o/p6d/UVlirijB8E16FtfwSAi4g3tcqrQ4lRAqQSoFEZJehYEcw==}
     engines: {node: '>= 0.4'}
@@ -12324,6 +12405,9 @@ packages:
     resolution: {integrity: sha512-KJP1OCML99+8fhOHxwwzyWrlUuVX5GQ0ZpJTd1DFXhdkrvg1szxfHhawXUZ3g9TkXORQd4/WG68jMlQZ2p8wlg==}
     engines: {node: '>=6'}
 
+  standard-as-callback@2.1.0:
+    resolution: {integrity: sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==}
+
   static-eval@2.0.2:
     resolution: {integrity: sha512-N/D219Hcr2bPjLxPiV+TQE++Tsmrady7TqAJugLy7Xk1EumfDWS/f5dtBbkRCGE7wKKXuYockQoj8Rm2/pVKyg==}
 
@@ -13350,6 +13434,9 @@ packages:
   yallist@3.1.1:
     resolution: {integrity: sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==}
 
+  yallist@4.0.0:
+    resolution: {integrity: sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==}
+
   yaml@1.10.2:
     resolution: {integrity: sha512-r3vXyErRCYJ7wg28yvBY5VSoAF8ZvlcW9/BwUzEtUsjvX/DKs24dIkuwjtuprwJJHsbyUbLApepYTR1BN4uHrg==}
     engines: {node: '>= 6'}
@@ -15487,6 +15574,16 @@ snapshots:
     dependencies:
       tslib: 2.8.1
 
+  '@fortedigital/nextjs-cache-handler@1.2.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(redis@4.7.0)':
+    dependencies:
+      '@neshca/cache-handler': 1.9.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(redis@4.7.0)
+      cluster-key-slot: 1.1.2
+      lru-cache: 11.1.0
+      next: 15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)
+      redis: 4.7.0
+    optionalDependencies:
+      '@rollup/rollup-linux-x64-gnu': 4.40.1
+
   '@googleapis/drive@8.16.0':
     dependencies:
       googleapis-common: 7.2.0
@@ -15675,6 +15772,8 @@ snapshots:
   '@img/sharp-win32-x64@0.33.5':
     optional: true
 
+  '@ioredis/commands@1.2.0': {}
+
   '@isaacs/cliui@8.0.2':
     dependencies:
       string-width: 5.1.2
@@ -16076,6 +16175,13 @@ snapshots:
       '@tybys/wasm-util': 0.9.0
     optional: true
 
+  '@neshca/cache-handler@1.9.0(next@15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0))(redis@4.7.0)':
+    dependencies:
+      cluster-key-slot: 1.1.2
+      lru-cache: 10.4.3
+      next: 15.2.3(@babel/core@7.25.2)(@opentelemetry/api@1.9.0)(@playwright/test@1.51.0)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)
+      redis: 4.7.0
+
   '@next/env@15.0.4': {}
 
   '@next/env@15.2.3': {}
@@ -18552,6 +18658,32 @@ snapshots:
       - '@types/react'
       - immer
 
+  '@redis/bloom@1.2.0(@redis/client@1.6.0)':
+    dependencies:
+      '@redis/client': 1.6.0
+
+  '@redis/client@1.6.0':
+    dependencies:
+      cluster-key-slot: 1.1.2
+      generic-pool: 3.9.0
+      yallist: 4.0.0
+
+  '@redis/graph@1.1.1(@redis/client@1.6.0)':
+    dependencies:
+      '@redis/client': 1.6.0
+
+  '@redis/json@1.0.7(@redis/client@1.6.0)':
+    dependencies:
+      '@redis/client': 1.6.0
+
+  '@redis/search@1.2.0(@redis/client@1.6.0)':
+    dependencies:
+      '@redis/client': 1.6.0
+
+  '@redis/time-series@1.1.0(@redis/client@1.6.0)':
+    dependencies:
+      '@redis/client': 1.6.0
+
   '@remirror/core-constants@3.0.0': {}
 
   '@remirror/core-helpers@4.0.0':
@@ -18720,6 +18852,9 @@ snapshots:
   '@rollup/rollup-linux-x64-gnu@4.35.0':
     optional: true
 
+  '@rollup/rollup-linux-x64-gnu@4.40.1':
+    optional: true
+
   '@rollup/rollup-linux-x64-musl@4.21.2':
     optional: true
 
@@ -21319,6 +21454,8 @@ snapshots:
 
   clsx@2.1.1: {}
 
+  cluster-key-slot@1.1.2: {}
+
   cmdk@1.0.4(@types/react-dom@19.0.3(@types/react@19.0.6))(@types/react@19.0.6)(react-dom@19.0.0(react@19.0.0))(react@19.0.0):
     dependencies:
       '@radix-ui/react-dialog': 1.1.4(@types/react-dom@19.0.3(@types/react@19.0.6))(@types/react@19.0.6)(react-dom@19.0.0(react@19.0.0))(react@19.0.0)
@@ -21834,6 +21971,8 @@ snapshots:
 
   delayed-stream@1.0.0: {}
 
+  denque@2.1.0: {}
+
   dequal@2.0.3: {}
 
   detect-indent@6.1.0: {}
@@ -22709,6 +22848,8 @@ snapshots:
       - encoding
       - supports-color
 
+  generic-pool@3.9.0: {}
+
   gensync@1.0.0-beta.2: {}
 
   get-caller-file@2.0.5: {}
@@ -23383,6 +23524,20 @@ snapshots:
 
   interpret@3.1.1: {}
 
+  ioredis@5.6.1:
+    dependencies:
+      '@ioredis/commands': 1.2.0
+      cluster-key-slot: 1.1.2
+      debug: 4.4.0
+      denque: 2.1.0
+      lodash.defaults: 4.2.0
+      lodash.isarguments: 3.1.0
+      redis-errors: 1.2.0
+      redis-parser: 3.0.0
+      standard-as-callback: 2.1.0
+    transitivePeerDependencies:
+      - supports-color
+
   ip-address@9.0.5:
     dependencies:
       jsbn: 1.1.0
@@ -23641,7 +23796,7 @@ snapshots:
 
   jest-worker@27.5.1:
     dependencies:
-      '@types/node': 20.17.12
+      '@types/node': 22.14.0
       merge-stream: 2.0.0
       supports-color: 8.1.1
 
@@ -24018,6 +24173,8 @@ snapshots:
 
   lodash.includes@4.3.0: {}
 
+  lodash.isarguments@3.1.0: {}
+
   lodash.isboolean@3.0.3: {}
 
   lodash.isequalwith@4.4.0: {}
@@ -24075,7 +24232,7 @@ snapshots:
 
   lru-cache@10.4.3: {}
 
-  lru-cache@11.0.1: {}
+  lru-cache@11.1.0: {}
 
   lru-cache@5.1.1:
     dependencies:
@@ -25342,7 +25499,7 @@ snapshots:
 
   path-scurry@2.0.0:
     dependencies:
-      lru-cache: 11.0.1
+      lru-cache: 11.1.0
       minipass: 7.1.2
 
   path-to-regexp@3.3.0: {}
@@ -26163,6 +26320,21 @@ snapshots:
       indent-string: 4.0.0
       strip-indent: 3.0.0
 
+  redis-errors@1.2.0: {}
+
+  redis-parser@3.0.0:
+    dependencies:
+      redis-errors: 1.2.0
+
+  redis@4.7.0:
+    dependencies:
+      '@redis/bloom': 1.2.0(@redis/client@1.6.0)
+      '@redis/client': 1.6.0
+      '@redis/graph': 1.1.1(@redis/client@1.6.0)
+      '@redis/json': 1.0.7(@redis/client@1.6.0)
+      '@redis/search': 1.2.0(@redis/client@1.6.0)
+      '@redis/time-series': 1.1.0(@redis/client@1.6.0)
+
   reflect.getprototypeof@1.0.10:
     dependencies:
       call-bind: 1.0.8
@@ -26904,6 +27076,8 @@ snapshots:
     dependencies:
       type-fest: 0.7.1
 
+  standard-as-callback@2.1.0: {}
+
   static-eval@2.0.2:
     dependencies:
       escodegen: 1.14.3
@@ -28093,6 +28267,8 @@ snapshots:
 
   yallist@3.1.1: {}
 
+  yallist@4.0.0: {}
+
   yaml@1.10.2: {}
 
   yaml@2.3.1: {}
diff --git a/self-host/.env.example b/self-host/.env.example
index 508222d393..3654f92d11 100644
--- a/self-host/.env.example
+++ b/self-host/.env.example
@@ -42,4 +42,4 @@ HONEYCOMB_API_KEY="xxx"
 GCLOUD_KEY_FILE='xxx'
 
 SELF_HOST="true"
-VALKEY_URL='redis://cache:6379'
\ No newline at end of file
+VALKEY_HOST='cache'
\ No newline at end of file


Reviews:
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T21:39:44Z
user: login: tefkah
body: if it would be possible to use a fallback i think that would be ideal to decrease possible downtime/manual fiddling! ideal behavior to me in case it can't reach valkey
- fallback to file system caching for now (or no caching, that's also fine)
- try to reconnect in the meantime
- if connection established, start using valkey cache
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-01T11:41:47Z
user: login: tefkah
body: whoops, i meant to request changes! if the behavior described above turns out not to be possible, i think at least creating a sentry issue would be nice (maybe it already does that if you throw)
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-05-01T11:42:59Z
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T14:21:57Z
user: login: tefkah
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T16:45:42Z
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T23:10:55Z
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T23:18:20Z
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T23:19:33Z
user: login: kalilsn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T04:04:55Z
user: login: 3mcd
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-06T10:54:09Z

Review comments:
diff_hunk: @@ -1,11 +1,96 @@
-/**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
- *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
- */
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
 
-import FileSystemCache from "next/dist/server/lib/incremental-cache/file-system-cache.js";
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import createRedisHandler from "@fortedigital/nextjs-cache-handler/redis-strings";
+import { CacheHandler } from "@neshca/cache-handler";
+import { createClient } from "redis";
 
-export default FileSystemCache;
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+// Usual onCreation from @neshca/cache-handler
+CacheHandler.onCreation(() => {
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures only one instance get created
+	if (global.cacheHandlerConfig) {
+		return global.cacheHandlerConfig;
+	}
+
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures new instances are not created in a race condition
+	if (global.cacheHandlerConfigPromise) {
+		return global.cacheHandlerConfigPromise;
+	}
+
+	// Main promise initializing the handler
+	global.cacheHandlerConfigPromise = (async () => {
+		console.info("Getting cache handler");
+		/** @type {import("redis").RedisClientType | null} */
+		let redisClient = null;
+		if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+			try {
+				redisClient = createClient({
+					url: process.env.VALKEY_URL,
+					pingInterval: 10000,
+				});
+				redisClient.on("error", (e) => {
+					console.warn("Redis error", e);
+					console.warn("Disabling caching");
+					global.cacheHandlerConfig = { handlers: [dummyHandler] };
+					global.cacheHandlerConfigPromise = null;
+					throw e;
path: core/cache-handler.mjs
user: login: kalilsn
body: If we don't throw here, the client just keeps trying to reconnect if there's no connectivity at startup and requests that involve the cache never get served. Throwing here unfortunately means that it won't reconnect on error, so we'll need to restart containers if they lose connectivity to the redis cluster. I'm not sure how to make it do both!
created_at: 2025-04-30T21:39:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,96 @@
-/**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
- *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
- */
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
 
-import FileSystemCache from "next/dist/server/lib/incremental-cache/file-system-cache.js";
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import createRedisHandler from "@fortedigital/nextjs-cache-handler/redis-strings";
+import { CacheHandler } from "@neshca/cache-handler";
+import { createClient } from "redis";
 
-export default FileSystemCache;
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+// Usual onCreation from @neshca/cache-handler
+CacheHandler.onCreation(() => {
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures only one instance get created
+	if (global.cacheHandlerConfig) {
+		return global.cacheHandlerConfig;
+	}
+
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures new instances are not created in a race condition
+	if (global.cacheHandlerConfigPromise) {
+		return global.cacheHandlerConfigPromise;
+	}
+
+	// Main promise initializing the handler
+	global.cacheHandlerConfigPromise = (async () => {
+		console.info("Getting cache handler");
+		/** @type {import("redis").RedisClientType | null} */
+		let redisClient = null;
+		if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+			try {
+				redisClient = createClient({
+					url: process.env.VALKEY_URL,
+					pingInterval: 10000,
+				});
+				redisClient.on("error", (e) => {
+					console.warn("Redis error", e);
+					console.warn("Disabling caching");
+					global.cacheHandlerConfig = { handlers: [dummyHandler] };
+					global.cacheHandlerConfigPromise = null;
+					throw e;
path: core/cache-handler.mjs
user: login: tefkah
body: i think it is possible with something like using `@fortedigital/nextjs-cache-handler/composite` to set a fallback and a strategy for when to use that fallback (ie when we can't connect to redis)

here's the type signature

```ts
type CreateCompositeHandlerOptions = {
    /**
     * A collection of handlers to manage the cache.
     */
    handlers: Handler[];
    /**
     * Strategy to determine which handler to use for the set operation.
     * Defaults to the first handler if not provided.
     * @param data - The data to be saved in one of the handlers. See {@link CacheHandlerValue}.
     * @returns The index of the handler for the set operation.
     */
    setStrategy: (data: CacheHandlerValue) => number;
};
/**
 * Creates a composite Handler for managing cache operations using multiple handlers.
 *
 * This function initializes a composite Handler that coordinates cache operations
 * across multiple handlers, providing methods for getting, setting, and revalidating cache values.
 *
 * @param options - The configuration options for the composite handler. See {@link CreateCompositeHandlerOptions}.
 *
 * @returns An object representing the composite cache handler, with methods for cache operations.
 *
 * @remarks
 * - The `get` method retrieves a value from the first handler that returns a non-null result, in the order defined by the handlers array.
 * - The `set` method uses the `setStrategy` to determine which handler should store the data. If no strategy is provided, the first handler is used.
 * - The `revalidateTag` and `delete` methods apply the operation across all handlers in parallel.
 */
declare function createHandler({ handlers, setStrategy: strategy, }: CreateCompositeHandlerOptions): Handler;
```

not sure how it works exactly but the `neshca` docs also mention the ability to set fallbacks, although im not sure how you can control their usage

https://caching-tools.github.io/next-shared-cache/api-reference/on-creation#example

I think ideally we should be able to fallback to the filesystem cache when redis is not available, and maybe (not sure if possible) create a Sentry error if so
created_at: 2025-05-01T11:39:07Z
author_association: MEMBER
diff_hunk: @@ -1,11 +1,96 @@
-/**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
- *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
- */
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
 
-import FileSystemCache from "next/dist/server/lib/incremental-cache/file-system-cache.js";
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import createRedisHandler from "@fortedigital/nextjs-cache-handler/redis-strings";
+import { CacheHandler } from "@neshca/cache-handler";
+import { createClient } from "redis";
 
-export default FileSystemCache;
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+// Usual onCreation from @neshca/cache-handler
+CacheHandler.onCreation(() => {
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures only one instance get created
+	if (global.cacheHandlerConfig) {
+		return global.cacheHandlerConfig;
+	}
+
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures new instances are not created in a race condition
+	if (global.cacheHandlerConfigPromise) {
+		return global.cacheHandlerConfigPromise;
+	}
+
+	// Main promise initializing the handler
+	global.cacheHandlerConfigPromise = (async () => {
+		console.info("Getting cache handler");
+		/** @type {import("redis").RedisClientType | null} */
+		let redisClient = null;
+		if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+			try {
+				redisClient = createClient({
+					url: process.env.VALKEY_URL,
+					pingInterval: 10000,
+				});
+				redisClient.on("error", (e) => {
+					console.warn("Redis error", e);
+					console.warn("Disabling caching");
+					global.cacheHandlerConfig = { handlers: [dummyHandler] };
+					global.cacheHandlerConfigPromise = null;
+					throw e;
path: core/cache-handler.mjs
user: login: kalilsn
body: >I think ideally we should be able to fallback to the filesystem cache when redis is not available, and maybe (not sure if possible) create a Sentry error if so

I don't actually think falling back to the filesystem cache is what we want in prod. If we use the filesystem cache, then each container has its own separate cache, and there's no guarantee it will be invalidated when it needs to, so we'd end up serving different data from different containers. I also have a suspicion that some of our crashes are being caused by the filesystem filling up, which then prevents the cache from working. So instead I'm falling back to not caching at all, which seems perfectly reasonable when redis is unavailable. I totally agree we need a sentry alert though, I'll add one explicitly.

As for the composite handler, I don't think it actually offers us anything we need. If we had certain requests that we wanted to skip the cache, or be sent to a different cache, I think that would be useful. But the "strategy" is just a function which returns the index of one of your two handlers so I'm not sure what improvement it would offer here! The real cause of the problems here is just that the redis client isn't very well behaved, so I will try using ioredis instead and see if I can get your ideal behavior working. I also think the sentry alert is probably enough because I don't really expect the app to automatically recover from redis going down at this point - just making sure it doesn't completely die seems good enough.
created_at: 2025-05-01T14:21:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,96 @@
-/**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
- *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
- */
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
 
-import FileSystemCache from "next/dist/server/lib/incremental-cache/file-system-cache.js";
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import createRedisHandler from "@fortedigital/nextjs-cache-handler/redis-strings";
+import { CacheHandler } from "@neshca/cache-handler";
+import { createClient } from "redis";
 
-export default FileSystemCache;
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+// Usual onCreation from @neshca/cache-handler
+CacheHandler.onCreation(() => {
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures only one instance get created
+	if (global.cacheHandlerConfig) {
+		return global.cacheHandlerConfig;
+	}
+
+	// Important - It's recommended to use global scope to ensure only one Redis connection is made
+	// This ensures new instances are not created in a race condition
+	if (global.cacheHandlerConfigPromise) {
+		return global.cacheHandlerConfigPromise;
+	}
+
+	// Main promise initializing the handler
+	global.cacheHandlerConfigPromise = (async () => {
+		console.info("Getting cache handler");
+		/** @type {import("redis").RedisClientType | null} */
+		let redisClient = null;
+		if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+			try {
+				redisClient = createClient({
+					url: process.env.VALKEY_URL,
+					pingInterval: 10000,
+				});
+				redisClient.on("error", (e) => {
+					console.warn("Redis error", e);
+					console.warn("Disabling caching");
+					global.cacheHandlerConfig = { handlers: [dummyHandler] };
+					global.cacheHandlerConfigPromise = null;
+					throw e;
path: core/cache-handler.mjs
user: login: tefkah
body: yeah since we are maybe relying on redis a bunch more in the future it probably makes sense just to alert us! 

and good point about not falling back to filesystem cache, i didn't really consider the cache divergence issue!
created_at: 2025-05-01T16:45:42Z
author_association: MEMBER
diff_hunk: @@ -1,11 +1,236 @@
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
+
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import { CacheHandler } from "@neshca/cache-handler";
+import { isImplicitTag } from "@neshca/cache-handler/helpers";
+import Redis from "ioredis";
+
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+var REVALIDATED_TAGS_KEY = "__revalidated_tags__";
+
 /**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
+ * Creates a redis handler based on fortedigital's redis-strings handler, but using the ioredis
+ * client
  *
- * We are reexporting `next.js`'s default cache handler to get around
- * the fixed 2mb limit for cached fetches.
- * We run into this limit somewhat quickly if we fetch more than 100 pubs.
+ * @param {{ client: Redis}} props
  */
+function createRedisHandler({
+	client,
+	keyPrefix = "",
+	sharedTagsKey = "__sharedTags__",
+	sharedTagsTtlKey = "__sharedTagsTtl__",
+	revalidateTagQuerySize = 1e4,
+}) {
+	async function revalidateTags(tag) {
+		const tagsMap = /* @__PURE__ */ new Map();
+		let cursor = 0;
+		do {
+			const remoteTagsPortion = await client.hscan(
+				keyPrefix + sharedTagsKey,
+				cursor,
+				"COUNT",
+				revalidateTagQuerySize
+			);
+			for (const { field, value } of remoteTagsPortion.tuples) {
+				tagsMap.set(field, JSON.parse(value));
+			}
+			cursor = remoteTagsPortion.cursor;
+		} while (cursor !== 0);
+		const keysToDelete = [];
+		const tagsToDelete = [];
+		for (const [key, tags] of tagsMap) {
+			if (tags.includes(tag)) {
+				keysToDelete.push(keyPrefix + key);
+				tagsToDelete.push(key);
+			}
+		}
+		if (keysToDelete.length === 0) {
+			return;
+		}
+		await client.unlink(keysToDelete);
+		const updateTagsOperation = client.hdel(keyPrefix + sharedTagsKey, tagsToDelete);
+		const updateTtlOperation = client.hdel(keyPrefix + sharedTagsTtlKey, tagsToDelete);
+		await Promise.all([updateTtlOperation, updateTagsOperation]);
+	}
+	async function revalidateSharedKeys() {
+		const ttlMap = /* @__PURE__ */ new Map();
+		let cursor = 0;
+		do {
+			const remoteTagsPortion = await client.hscan(
+				keyPrefix + sharedTagsKey,
+				cursor,
+				"COUNT",
+				revalidateTagQuerySize
+			);
+			for (const { field, value } of remoteTagsPortion.tuples) {
+				ttlMap.set(field, Number(value));
+			}
+			cursor = remoteTagsPortion.cursor;
+		} while (cursor !== 0);
+		const tagsAndTtlToDelete = [];
+		const keysToDelete = [];
+		for (const [key, ttlInSeconds] of ttlMap) {
+			if (/* @__PURE__ */ new Date().getTime() > ttlInSeconds * 1e3) {
+				tagsAndTtlToDelete.push(key);
+				keysToDelete.push(keyPrefix + key);
+			}
+		}
+		if (tagsAndTtlToDelete.length === 0) {
+			return;
+		}
+		await client.unlink(keysToDelete);
+		const updateTtlOperation = client.hdel(keyPrefix + sharedTagsTtlKey, tagsAndTtlToDelete);
+		const updateTagsOperation = client.hdel(keyPrefix + sharedTagsKey, tagsAndTtlToDelete);
+		await Promise.all([updateTagsOperation, updateTtlOperation]);
+	}
+	const revalidatedTagsKey = keyPrefix + REVALIDATED_TAGS_KEY;
+	return {
+		name: "pubpub-redis-strings",
+		async get(key, { implicitTags }) {
+			const result = await client.get(keyPrefix + key);
+			if (!result) {
+				return null;
+			}
+			const cacheValue = JSON.parse(result);
+			if (!cacheValue) {
+				return null;
+			}
+			const sharedTagKeyExists = await client.hexists(keyPrefix + sharedTagsKey, key);
+			if (!sharedTagKeyExists) {
+				await client.unlink(keyPrefix + key);
+				return null;
+			}
+			const combinedTags = /* @__PURE__ */ new Set([...cacheValue.tags, ...implicitTags]);
+			if (combinedTags.size === 0) {
+				return cacheValue;
+			}
+			const revalidationTimes = await client.hmget(
+				revalidatedTagsKey,
+				Array.from(combinedTags)
+			);
+			for (const timeString of revalidationTimes) {
+				if (timeString && Number.parseInt(timeString, 10) > cacheValue.lastModified) {
+					await client.unlink(keyPrefix + key);
+					return null;
+				}
+			}
+			return cacheValue;
+		},
+		async set(key, cacheHandlerValue) {
+			const lifespan = cacheHandlerValue.lifespan;
+			const setTagsOperation =
+				cacheHandlerValue.tags.length > 0
+					? client.hset(
+							keyPrefix + sharedTagsKey,
+							key,
+							JSON.stringify(cacheHandlerValue.tags)
+						)
+					: void 0;
+			const setSharedTtlOperation = lifespan
+				? client.hset(keyPrefix + sharedTagsTtlKey, key, lifespan.expireAt)
+				: void 0;
+			await Promise.all([setTagsOperation, setSharedTtlOperation]);
+			if (typeof lifespan?.expireAt === "number") {
+				await client.set(
+					keyPrefix + key,
+					JSON.stringify(cacheHandlerValue),
+					"EXAT",
+					lifespan.expireAt
+				);
+			} else {
+				await client.set(keyPrefix + key, JSON.stringify(cacheHandlerValue));
+			}
+		},
+		async revalidateTag(tag) {
+			if (isImplicitTag(tag)) {
+				await client.hset(revalidatedTagsKey, tag, Date.now());
+			}
+			await Promise.all([revalidateTags(tag), revalidateSharedKeys()]);
+		},
+		async delete(key) {
+			await client.unlink(keyPrefix + key);
+			await Promise.all([
+				client.hdel(keyPrefix + sharedTagsKey, key),
+				client.hdel(keyPrefix + sharedTagsTtlKey, key),
+			]);
+		},
+	};
+}
+
+async function getCacheHandlerPromise() {
+	let redisClient = null;
+	if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
+		try {
+			redisClient = new Redis({
+				host: process.env.VALKEY_HOST,
+				lazyConnect: true,
+				commandTimeout: 1000,
+				retryStrategy: (times) => {
+					console.log("Retrying redis connection attempt:", times);
+					return (2 ^ times) + Math.random() * 1000;
+				},
+			});
+
+			await redisClient.connect();
path: core/cache-handler.mjs
user: login: kalilsn
body: I'd love to reuse the singleton client from `lib/redis` but I can't seem to import it in this file (lib/redis.ts gets inlined/chunked somewhere, and the build process probably doesn't look at imports in this file). So we have to repeat some of the connection logic + params, and we have two clients per instance, but at least it's not more!
created_at: 2025-05-05T23:10:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,236 @@
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
+
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import { CacheHandler } from "@neshca/cache-handler";
+import { isImplicitTag } from "@neshca/cache-handler/helpers";
+import Redis from "ioredis";
+
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+var REVALIDATED_TAGS_KEY = "__revalidated_tags__";
+
 /**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
+ * Creates a redis handler based on fortedigital's redis-strings handler, but using the ioredis
+ * client
path: core/cache-handler.mjs
user: login: kalilsn
body: the `node-redis` client does not handle reconnecting well at all and it seems like many people have reported this issue for many years, with no fix. so I switched to ioredis, which both handles reconnection properly and has much clearer documentation. unfortunately `@nescha/cache-handler` and the `@fortedigital` handlers are tightly coupled to `node-redis`, so I've inlined a compatible version of the `redis-strings` handler here. the main changes were dropping the `client.isReady` check (which doesn't do anything), removing the per command timeouts (because that's set on the client), and replacing the method calls with lowercase versions to match the ioredis api. see https://github.com/pubpub/platform/pull/1220/commits/43461952190f20c6074827a8fb21738a398f26f0 for the exact diff from the fortedigital handler
created_at: 2025-05-05T23:18:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -132,4 +132,4 @@ COPY --from=withpackage --chown=node:node /usr/src/app/core/public ./core/public
 # needed to set the database url correctly based on PGHOST variables
 COPY --from=withpackage --chown=node:node /usr/src/app/core/.env.docker ./core/.env
 
-CMD node core/server.js
+CMD ["node", "core/server.js"]
path: Dockerfile
user: login: kalilsn
body: Apparently using non-json arguments here can prevent the server from responding to signals sent to the container
created_at: 2025-05-05T23:19:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,236 @@
+// Based on https://github.com/fortedigital/nextjs-cache-handler#full-example
+
+import { PHASE_PRODUCTION_BUILD } from "next/constants.js";
+import createBufferStringHandler from "@fortedigital/nextjs-cache-handler/buffer-string-decorator";
+import { Next15CacheHandler } from "@fortedigital/nextjs-cache-handler/next-15-cache-handler";
+import { CacheHandler } from "@neshca/cache-handler";
+import { isImplicitTag } from "@neshca/cache-handler/helpers";
+import Redis from "ioredis";
+
+// A cache that always misses - intended to let us disable caching when redis is unavailable. Should
+// be replaced if there's a better way to do that.
+const dummyHandler = {
+	name: "no-cache",
+	get: () => undefined,
+	set: () => undefined,
+	revalidateTag: () => undefined,
+};
+
+var REVALIDATED_TAGS_KEY = "__revalidated_tags__";
+
 /**
- * Solution taken from here: https://github.com/vercel/next.js/discussions/48324#discussioncomment-10542097
+ * Creates a redis handler based on fortedigital's redis-strings handler, but using the ioredis
+ * client
path: core/cache-handler.mjs
user: login: kalilsn
body: i spoke too soon. there were more differences (https://github.com/pubpub/platform/pull/1220/commits/84543b3ba57cb458dd22347171567f1639ceb102) and i also realized there was a use to the ready check (https://github.com/pubpub/platform/pull/1220/commits/d6035482614c1de88d2131123f580c07a61f69d8)
created_at: 2025-05-06T04:04:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: kalilsn
created_at: 2025-05-06T04:13:49Z
body: This works much better after switching to ioredis. The client will attempt to reconnect automatically, and it will simply fall back to uncached behavior when the cache is unreachable. Unfortunately I couldn't get `Sentry.captureException` working inside the `cache-handler.mjs` file so there's no explicit alert. But the health check now checks redis for connectivity and sentry auto-instrumentation should work there.

<details>
<summary>Terraform plan for env var change, health check, and restart policy</summary>

```
Terraform will perform the following actions:

  # module.deployment.module.service_core.aws_lb_target_group.this[0] will be updated in-place
  ~ resource "aws_lb_target_group" "this" {
        id                                 = "arn:aws:elasticloadbalancing:us-east-1:246372085946:targetgroup/stevie-94a0/1a912e6be01e0401"
        name                               = "stevie-94a0"
        tags                               = {}
        # (17 unchanged attributes hidden)

      ~ health_check {
          ~ healthy_threshold   = 5 -> 3
          ~ interval            = 30 -> 5
          ~ path                = "/legacy_healthcheck" -> "/api/health"
            # (6 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

  # module.deployment.module.service_bastion.module.ecs_service.data.aws_ecs_task_definition.this[0] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_ecs_task_definition" "this" {
      + arn                      = (known after apply)
      + arn_without_revision     = (known after apply)
      + container_definitions    = (known after apply)
      + cpu                      = (known after apply)
      + enable_fault_injection   = (known after apply)
      + ephemeral_storage        = (known after apply)
      + execution_role_arn       = (known after apply)
      + family                   = (known after apply)
      + id                       = (known after apply)
      + inference_accelerator    = (known after apply)
      + ipc_mode                 = (known after apply)
      + memory                   = (known after apply)
      + network_mode             = (known after apply)
      + pid_mode                 = (known after apply)
      + placement_constraints    = (known after apply)
      + proxy_configuration      = (known after apply)
      + requires_compatibilities = (known after apply)
      + revision                 = (known after apply)
      + runtime_platform         = (known after apply)
      + status                   = (known after apply)
      + task_definition          = "stevie-bastion"
      + task_role_arn            = (known after apply)
      + volume                   = (known after apply)
    }

  # module.deployment.module.service_bastion.module.ecs_service.aws_ecs_task_definition.this[0] must be replaced
+/- resource "aws_ecs_task_definition" "this" {
      ~ arn                      = "arn:aws:ecs:us-east-1:246372085946:task-definition/stevie-bastion:392" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:246372085946:task-definition/stevie-bastion" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [
              ~ {
                  ~ environment            = [
                        # (7 unchanged elements hidden)
                        {
                            name  = "SUPABASE_URL"
                            value = "https://dsleqjuvzuoycpeotdws.supabase.co"
                        },
                      ~ {
                          ~ name  = "VALKEY_URL" -> "VALKEY_HOST"
                          ~ value = "redis://stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com" -> "stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com"
                        },
                    ]
                    name                   = "bastion"
                    # (17 unchanged attributes hidden)
                },
            ] # forces replacement
        )
      ~ enable_fault_injection   = false -> (known after apply)
      ~ id                       = "stevie-bastion" -> (known after apply)
      ~ revision                 = 392 -> (known after apply)
        tags                     = {
            "Environment"         = "stevie-production"
            "LogicalName"         = "bastion"
            "Project"             = "Pubpub-v7"
            "Shortname"           = "590b"
            "ShortnameAnnotation" = "Shortname is calculated as first four characters of the sha1sum of the Logical Name."
        }
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.deployment.module.service_core.module.ecs_service.data.aws_ecs_task_definition.this[0] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_ecs_task_definition" "this" {
      + arn                      = (known after apply)
      + arn_without_revision     = (known after apply)
      + container_definitions    = (known after apply)
      + cpu                      = (known after apply)
      + enable_fault_injection   = (known after apply)
      + ephemeral_storage        = (known after apply)
      + execution_role_arn       = (known after apply)
      + family                   = (known after apply)
      + id                       = (known after apply)
      + inference_accelerator    = (known after apply)
      + ipc_mode                 = (known after apply)
      + memory                   = (known after apply)
      + network_mode             = (known after apply)
      + pid_mode                 = (known after apply)
      + placement_constraints    = (known after apply)
      + proxy_configuration      = (known after apply)
      + requires_compatibilities = (known after apply)
      + revision                 = (known after apply)
      + runtime_platform         = (known after apply)
      + status                   = (known after apply)
      + task_definition          = "stevie-core"
      + task_role_arn            = (known after apply)
      + volume                   = (known after apply)
    }

  # module.deployment.module.service_core.module.ecs_service.aws_ecs_task_definition.this[0] must be replaced
+/- resource "aws_ecs_task_definition" "this" {
      ~ arn                      = "arn:aws:ecs:us-east-1:246372085946:task-definition/stevie-core:393" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:246372085946:task-definition/stevie-core" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [
              ~ {
                  ~ environment            = [
                        # (18 unchanged elements hidden)
                        {
                            name  = "SUPABASE_URL"
                            value = "https://dsleqjuvzuoycpeotdws.supabase.co"
                        },
                      ~ {
                          ~ name  = "VALKEY_URL" -> "VALKEY_HOST"
                          ~ value = "redis://stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com" -> "stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com"
                        },
                    ]
                    name                   = "core"
                    # (17 unchanged attributes hidden)
                },
              ~ {
                  ~ environment            = [
                        # (18 unchanged elements hidden)
                        {
                            name  = "SUPABASE_URL"
                            value = "https://dsleqjuvzuoycpeotdws.supabase.co"
                        },
                      ~ {
                          ~ name  = "VALKEY_URL" -> "VALKEY_HOST"
                          ~ value = "redis://stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com" -> "stevie-core-valkey-production.we8a07.ng.0001.use1.cache.amazonaws.com"
                        },
                    ]
                    name                   = "migrations"
                    # (17 unchanged attributes hidden)
                },
                {
                    environment            = [
                        {
                            name  = "NGINX_LISTEN_PORT"
                            value = "8080"
                        },
                        {
                            name  = "NGINX_PREFIX"
                            value = "/"
                        },
                        {
                            name  = "NGINX_UPSTREAM_HOST"
                            value = "127.0.0.1"
                        },
                        {
                            name  = "NGINX_UPSTREAM_PORT"
                            value = "3000"
                        },
                        {
                            name  = "OTEL_SERVICE_NAME"
                            value = "core.nginx"
                        },
                    ]
                    essential              = true
                    image                  = "246372085946.dkr.ecr.us-east-1.amazonaws.com/nginx:latest"
                    interactive            = false
                    linuxParameters        = {
                        initProcessEnabled = true
                    }
                    logConfiguration       = {
                        logDriver = "awslogs"
                        options   = {
                            awslogs-group         = "stevie-ecs-production-container-logs"
                            awslogs-region        = "us-east-1"
                            awslogs-stream-prefix = "ecs"
                        }
                    }
                    mountPoints            = []
                    name                   = "nginx"
                    portMappings           = [
                        {
                            containerPort = 8080
                            hostPort      = 8080
                            name          = "core-nginx"
                            protocol      = "tcp"
                        },
                    ]
                    privileged             = false
                    pseudoTerminal         = false
                    readonlyRootFilesystem = false
                    startTimeout           = 30
                    stopTimeout            = 120
                    systemControls         = []
                    user                   = "0"
                    volumesFrom            = []
                },
            ] # forces replacement
        )
      ~ enable_fault_injection   = false -> (known after apply)
      ~ id                       = "stevie-core" -> (known after apply)
      ~ revision                 = 393 -> (known after apply)
        tags                     = {
            "Environment"         = "stevie-production"
            "LogicalName"         = "core"
            "Project"             = "Pubpub-v7"
            "Shortname"           = "94a0"
            "ShortnameAnnotation" = "Shortname is calculated as first four characters of the sha1sum of the Logical Name."
        }
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 2 to add, 1 to change, 2 to destroy.
```

</details>
author_association: CONTRIBUTOR
user: login: kalilsn
created_at: 2025-05-06T13:17:11Z
body: Changes applied after i lowered the health check timeout because of this error:
```
╷
│ Error: modifying ELBv2 Target Group (arn:aws:elasticloadbalancing:us-east-1:246372085946:targetgroup/stevie-94a0/1a912e6be01e0401): operation error Elastic Load Balancing v2: ModifyTargetGroup, https response error StatusCode: 400, RequestID: fdeecfb8-6bf6-4d01-998c-446a1c2f88b0, api error ValidationError: Health check interval must be greater than the timeout.
│
│   with module.deployment.module.service_core.aws_lb_target_group.this[0],
│   on ../../modules/container-generic/main.tf line 152, in resource "aws_lb_target_group" "this":
│  152: resource "aws_lb_target_group" "this" {
│
╵
```
author_association: CONTRIBUTOR