Repository: outline/outline
Pull request #9196: perf: Add cache for document structure
Author: login: tommoor
Created: 2025-05-13T12:50:02Z | Merged: 2025-05-18T22:45:00Z

There's an opportunity to push the implementation deeper and introduce a sequelize model cache, which we should consider. For now I'm interested in testing this in a very specific spot with large payload size that can easily be reverted if any unforeseen issues arise 

closes #8880

Linked issue #8880: Add cache for document structure
We can add a short redis cache for storing the collection structure.

There are only a couple of very defined points where this is updated and it can get quite large, making it a good candidate for caching in large workspaces.
Linked issue comments: user: login: linear[bot]
created_at: 2025-04-04T13:37:34Z
body: <p><a href="https://linear.app/outline/issue/OLN-1016/add-cache-for-document-structure">OLN-1016 Add cache for document structure</a></p>
author_association: NONE
user: login: Tess99854
created_at: 2025-04-27T21:21:14Z
body: Hey @tommoor I would like to work on this, is it possible to provide more information about it? 
author_association: CONTRIBUTOR

Diff:
diff --git a/server/models/Collection.ts b/server/models/Collection.ts
index 11cf8c3a3ac0..f7f52647beb5 100644
--- a/server/models/Collection.ts
+++ b/server/models/Collection.ts
@@ -14,7 +14,8 @@ import {
   EmptyResultError,
   type CreateOptions,
   type UpdateOptions,
-  ScopeOptions,
+  type ScopeOptions,
+  type SaveOptions,
 } from "sequelize";
 import {
   Sequelize,
@@ -39,6 +40,7 @@ import {
   BeforeCreate,
   BeforeUpdate,
   DefaultScope,
+  AfterSave,
 } from "sequelize-typescript";
 import isUUID from "validator/lib/isUUID";
 import type { CollectionSort, ProsemirrorData } from "@shared/types";
@@ -48,6 +50,7 @@ import { sortNavigationNodes } from "@shared/utils/collections";
 import slugify from "@shared/utils/slugify";
 import { CollectionValidation } from "@shared/validations";
 import { ValidationError } from "@server/errors";
+import { CacheHelper } from "@server/utils/CacheHelper";
 import removeIndexCollision from "@server/utils/removeIndexCollision";
 import { generateUrlId } from "@server/utils/url";
 import { ValidateIndex } from "@server/validation";
@@ -334,6 +337,34 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(
+    model: Collection,
+    options: SaveOptions<Collection>
+  ) {
+    if (model.changed("documentStructure")) {
+      const setData = () =>
+        CacheHelper.setData(
+          CacheHelper.getCollectionDocumentsKey(model.id),
+          model.documentStructure,
+          60
+        );
+
+      if (options.transaction) {
+        return (options.transaction.parent || options.transaction).afterCommit(
+          setData
+        );
+      }
+
+      await setData();
+    }
   }
 
   @BeforeDestroy
diff --git a/server/routes/api/collections/collections.ts b/server/routes/api/collections/collections.ts
index 962eab139865..a74afc3bf01d 100644
--- a/server/routes/api/collections/collections.ts
+++ b/server/routes/api/collections/collections.ts
@@ -38,6 +38,7 @@ import {
   presentFileOperation,
 } from "@server/presenters";
 import { APIContext } from "@server/types";
+import { CacheHelper } from "@server/utils/CacheHelper";
 import { RateLimiterStrategy } from "@server/utils/RateLimiter";
 import { collectionIndexing } from "@server/utils/indexing";
 import pagination from "../middlewares/pagination";
@@ -143,13 +144,25 @@ router.post(
     const { user } = ctx.state.auth;
     const collection = await Collection.findByPk(id, {
       userId: user.id,
-      includeDocumentStructure: true,
     });
 
     authorize(user, "readDocument", collection);
 
+    const documentStructure = await CacheHelper.getDataOrSet(
+      CacheHelper.getCollectionDocumentsKey(collection.id),
+      async () =>
+        (
+          await Collection.findByPk(collection.id, {
+            attributes: ["documentStructure"],
+            includeDocumentStructure: true,
+            rejectOnEmpty: true,
+          })
+        ).documentStructure,
+      60
+    );
+
     ctx.body = {
-      data: collection.documentStructure || [],
+      data: documentStructure || [],
     };
   }
 );
diff --git a/server/storage/__mocks__/redis.ts b/server/storage/__mocks__/redis.ts
new file mode 100644
index 000000000000..c0a349d4920d
--- /dev/null
+++ b/server/storage/__mocks__/redis.ts
@@ -0,0 +1,47 @@
+import { EventEmitter } from "events";
+
+// Create a mock Redis client with all needed methods mocked
+class RedisMock extends EventEmitter {
+  constructor() {
+    super();
+  }
+
+  get = jest.fn().mockResolvedValue(null);
+  set = jest.fn().mockResolvedValue("OK");
+  del = jest.fn().mockResolvedValue(1);
+  keys = jest.fn().mockResolvedValue([]);
+  ping = jest.fn().mockResolvedValue("PONG");
+  disconnect = jest.fn();
+  setMaxListeners = jest.fn();
+}
+
+// Mock the RedisAdapter class
+class RedisAdapter extends RedisMock {
+  constructor(_url: string | undefined, _options = {}) {
+    super();
+  }
+
+  private static client: RedisAdapter;
+  private static subscriber: RedisAdapter;
+
+  public static get defaultClient(): RedisAdapter {
+    return (
+      this.client ||
+      (this.client = new this(undefined, {
+        connectionNameSuffix: "client",
+      }))
+    );
+  }
+
+  public static get defaultSubscriber(): RedisAdapter {
+    return (
+      this.subscriber ||
+      (this.subscriber = new this(undefined, {
+        maxRetriesPerRequest: null,
+        connectionNameSuffix: "subscriber",
+      }))
+    );
+  }
+}
+
+export default RedisAdapter;
diff --git a/server/test/setup.ts b/server/test/setup.ts
index 07f0ca917668..8e4e38aad0d7 100644
--- a/server/test/setup.ts
+++ b/server/test/setup.ts
@@ -7,6 +7,11 @@ require("@server/storage/database");
 
 jest.mock("bull");
 
+// Enable mocks for Redis-related modules
+jest.mock("@server/storage/redis");
+jest.mock("@server/utils/MutexLock");
+jest.mock("@server/utils/CacheHelper");
+
 // This is needed for the relative manual mock to be picked up
 jest.mock("../queues");
 
@@ -34,7 +39,9 @@ jest.mock("@aws-sdk/s3-request-presigner", () => ({
   getSignedUrl: jest.fn(),
 }));
 
-afterAll(() => Redis.defaultClient.disconnect());
+afterAll(() => {
+  Redis.defaultClient.disconnect();
+});
 
 beforeEach(() => {
   env.URL = sharedEnv.URL = "https://app.outline.dev";
diff --git a/server/utils/CacheHelper.ts b/server/utils/CacheHelper.ts
index 9748ff16e116..bce71c404fef 100644
--- a/server/utils/CacheHelper.ts
+++ b/server/utils/CacheHelper.ts
@@ -125,4 +125,8 @@ export class CacheHelper {
   public static getUnfurlKey(teamId: string, url = "") {
     return `unfurl:${teamId}:${url}`;
   }
+
+  public static getCollectionDocumentsKey(collectionId: string) {
+    return `cd:${collectionId}`;
+  }
 }
diff --git a/server/utils/__mocks__/CacheHelper.ts b/server/utils/__mocks__/CacheHelper.ts
new file mode 100644
index 000000000000..178aac75b99f
--- /dev/null
+++ b/server/utils/__mocks__/CacheHelper.ts
@@ -0,0 +1,54 @@
+import { Day } from "@shared/utils/time";
+
+/**
+ * A Mock Helper class for server-side cache management
+ */
+export class CacheHelper {
+  // Default expiry time for cache data in seconds
+  private static defaultDataExpiry = Day.seconds;
+
+  /**
+   * Mocked method that resolves with the callback result
+   */
+  public static async getDataOrSet<T>(
+    key: string,
+    callback: () => Promise<T | undefined>,
+    _expiry: number,
+    _lockTimeout: number
+  ): Promise<T | undefined> {
+    return await callback();
+  }
+
+  /**
+   * Mocked method that resolves with undefined
+   */
+  public static async getData<T>(_key: string): Promise<T | undefined> {
+    return undefined;
+  }
+
+  /**
+   * Mocked method that resolves with void
+   */
+  public static async setData<T>(_key: string, _data: T, _expiry?: number) {
+    return;
+  }
+
+  /**
+   * Mocked method that resolves with void
+   */
+  public static async clearData(_prefix: string) {
+    return;
+  }
+
+  /**
+   * These are real methods that don't require mocking as they don't
+   * interact with Redis directly
+   */
+  public static getUnfurlKey(teamId: string, url = "") {
+    return `unfurl:${teamId}:${url}`;
+  }
+
+  public static getCollectionDocumentsKey(collectionId: string) {
+    return `cd:${collectionId}`;
+  }
+}
diff --git a/server/utils/__mocks__/MutexLock.ts b/server/utils/__mocks__/MutexLock.ts
new file mode 100644
index 000000000000..df106fd325f8
--- /dev/null
+++ b/server/utils/__mocks__/MutexLock.ts
@@ -0,0 +1,18 @@
+export class MutexLock {
+  // Default expiry time for acquiring lock in milliseconds
+  public static defaultLockTimeout = 4000;
+
+  /**
+   * Returns the mock redlock instance
+   */
+  public static get lock() {
+    return {
+      acquire: jest.fn().mockResolvedValue({
+        release: jest.fn().mockResolvedValue(true),
+        expiration: Date.now() + 10000,
+      }),
+    };
+  }
+
+  private static redlock: any;
+}
diff --git a/yarn.lock b/yarn.lock
index a86baa27d393..36e61a04ed39 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -4136,18 +4136,7 @@
     "@smithy/util-utf8" "^4.0.0"
     tslib "^2.6.2"
 
-"@smithy/credential-provider-imds@^4.0.2":
-  version "4.0.2"
-  resolved "https://registry.yarnpkg.com/@smithy/credential-provider-imds/-/credential-provider-imds-4.0.2.tgz#1ec34a04842fa69996b151a695b027f0486c69a8"
-  integrity sha512-32lVig6jCaWBHnY+OEQ6e6Vnt5vDHaLiydGrwYMW9tPqO688hPGTYRamYJ1EptxEC2rAwJrHWmPoKRBl4iTa8w==
-  dependencies:
-    "@smithy/node-config-provider" "^4.0.2"
-    "@smithy/property-provider" "^4.0.2"
-    "@smithy/types" "^4.2.0"
-    "@smithy/url-parser" "^4.0.2"
-    tslib "^2.6.2"
-
-"@smithy/credential-provider-imds@^4.0.4":
+"@smithy/credential-provider-imds@^4.0.2", "@smithy/credential-provider-imds@^4.0.4":
   version "4.0.4"
   resolved "https://registry.yarnpkg.com/@smithy/credential-provider-imds/-/credential-provider-imds-4.0.4.tgz#01315ab90c4cb3e017c1ee2c6e5f958aeaa7cf78"
   integrity sha512-jN6M6zaGVyB8FmNGG+xOPQB4N89M1x97MMdMnm1ESjljLS3Qju/IegQizKujaNcy2vXAvrz0en8bobe6E55FEA==


Reviews:
user: login: hmacr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T13:28:04Z
user: login: tommoor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:06:18Z
user: login: tommoor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-18T02:59:02Z
user: login: hmacr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-18T04:15:28Z

Review comments:
diff_hunk: @@ -332,6 +334,22 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(model: Collection) {
+    if (model.changed("documentStructure")) {
path: server/models/Collection.ts
user: login: hmacr
body: I think this should be in a `afterCommit` hook when a transaction is available, similar to [Event](https://github.com/outline/outline/blob/main/server/models/Event.ts#L81-L95) model.
created_at: 2025-05-16T13:25:36Z
author_association: COLLABORATOR
diff_hunk: @@ -332,6 +334,22 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(model: Collection) {
+    if (model.changed("documentStructure")) {
+      await CacheHelper.setData(
+        CacheHelper.getCollectionDocumentsKey(model.id),
+        model.documentStructure,
+        60
path: server/models/Collection.ts
user: login: hmacr
body: _nit:_ we can use [Minute.seconds](https://github.com/outline/outline/blob/main/shared/utils/time.ts#L10) for better readability. (likewise in `collections.documents` route) 🙈 
created_at: 2025-05-16T13:26:45Z
author_association: COLLABORATOR
diff_hunk: @@ -332,6 +334,22 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(model: Collection) {
+    if (model.changed("documentStructure")) {
path: server/models/Collection.ts
user: login: tommoor
body: Good call
created_at: 2025-05-16T14:06:17Z
author_association: MEMBER
diff_hunk: @@ -332,6 +334,22 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(model: Collection) {
+    if (model.changed("documentStructure")) {
+      await CacheHelper.setData(
+        CacheHelper.getCollectionDocumentsKey(model.id),
+        model.documentStructure,
+        60
path: server/models/Collection.ts
user: login: tommoor
body: eh, in this particular case I think `60` is fine and very clear. I'm not going to be dogmatic over use of that helper
created_at: 2025-05-18T02:59:02Z
author_association: MEMBER
diff_hunk: @@ -332,6 +334,22 @@ class Collection extends ParanoidModel<
     if (!model.content) {
       model.content = await DocumentHelper.toJSON(model);
     }
+    if (model.changed("documentStructure")) {
+      await CacheHelper.clearData(
+        CacheHelper.getCollectionDocumentsKey(model.id)
+      );
+    }
+  }
+
+  @AfterSave
+  static async cacheDocumentStructure(model: Collection) {
+    if (model.changed("documentStructure")) {
+      await CacheHelper.setData(
+        CacheHelper.getCollectionDocumentsKey(model.id),
+        model.documentStructure,
+        60
path: server/models/Collection.ts
user: login: hmacr
body: ok :)
created_at: 2025-05-18T04:15:28Z
author_association: COLLABORATOR

PR comments:
user: login: Mariemlahmar
created_at: 2025-05-13T17:12:43Z
body: Hi @tommoor,

I’m interested in contributing to the caching feature for the document structure (#8880). Could you please share more details about the current implementation
author_association: NONE
user: login: tommoor
created_at: 2025-05-13T18:04:52Z
body: This is literally the pr with that work?
author_association: MEMBER
user: login: Mariemlahmar
created_at: 2025-05-13T18:25:54Z
body: Thanks for clarifying, @tommoor! I’ll check out PR #9196 and see how I can help.
author_association: NONE