Repository: triggerdotdev/trigger.dev
Pull request #1773: Run Engine 2: More robust attempt failing/retrying (inc. OOM retrying)
Author: login: matt-aitken
Created: 2025-03-06T19:28:58Z | Merged: 2025-03-07T09:07:49Z

- The logic for failed attempt retrying has been extracted to a separate file to simplify things.
- Added tests for attempts failing in different conditions
- Added support for OOM retrying
- Set the `--max-old-space-size` on TaskRunProcess based on the machine size. So this now works for dev runs as well as prod runs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Added a new task configuration that streamlines package builds.
  - Introduced enhanced Node.js runtime options for dynamic memory management.
  - Added a function to deduplicate command line flags for improved command handling.

- **Bug Fixes**
  - Improved detection and handling of memory-related issues to increase overall system stability and reduce crashes.

- **Refactor**
  - Optimized retry and error-handling mechanisms for task runs.
  - Updated background processing to support customizable retry settings, enhancing recovery and performance.
  - Enhanced schema validation to enforce required machine configuration in task execution.
  - Updated test configurations to include additional test file paths for comprehensive coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


Diff:
diff --git a/.zed/tasks.json b/.zed/tasks.json
new file mode 100644
index 00000000000..8612e16bfb1
--- /dev/null
+++ b/.zed/tasks.json
@@ -0,0 +1,45 @@
+[
+  {
+    "label": "Build packages",
+    "command": "pnpm run build --filter \"@trigger.dev/*\" --filter trigger.dev",
+    //"args": [],
+    // Env overrides for the command, will be appended to the terminal's environment from the settings.
+    "env": { "foo": "bar" },
+    // Current working directory to spawn the command into, defaults to current project root.
+    //"cwd": "/path/to/working/directory",
+    // Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
+    "use_new_terminal": false,
+    // Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
+    "allow_concurrent_runs": false,
+    // What to do with the terminal pane and tab, after the command was started:
+    // * `always` — always show the task's pane, and focus the corresponding tab in it (default)
+    // * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
+    // * `never` — do not alter focus, but still add/reuse the task's tab in its pane
+    "reveal": "always",
+    // What to do with the terminal pane and tab, after the command has finished:
+    // * `never` — Do nothing when the command finishes (default)
+    // * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
+    // * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
+    "hide": "never",
+    // Which shell to use when running a task inside the terminal.
+    // May take 3 values:
+    // 1. (default) Use the system's default terminal configuration in /etc/passwd
+    //      "shell": "system"
+    // 2. A program:
+    //      "shell": {
+    //        "program": "sh"
+    //      }
+    // 3. A program with arguments:
+    //     "shell": {
+    //         "with_arguments": {
+    //           "program": "/bin/bash",
+    //           "args": ["--login"]
+    //         }
+    //     }
+    "shell": "system",
+    // Whether to show the task line in the output of the spawned task, defaults to `true`.
+    "show_summary": true,
+    // Whether to show the command line in the output of the spawned task, defaults to `true`.
+    "show_output": true
+  }
+]
diff --git a/apps/docker-provider/src/index.ts b/apps/docker-provider/src/index.ts
index 3ca5184c752..a0b0554fb23 100644
--- a/apps/docker-provider/src/index.ts
+++ b/apps/docker-provider/src/index.ts
@@ -8,7 +8,7 @@ import {
 } from "@trigger.dev/core/v3/apps";
 import { SimpleLogger } from "@trigger.dev/core/v3/apps";
 import { isExecaChildProcess } from "@trigger.dev/core/v3/apps";
-import { testDockerCheckpoint } from "@trigger.dev/core/v3/checkpoints";
+import { testDockerCheckpoint } from "@trigger.dev/core/v3/serverOnly";
 import { setTimeout } from "node:timers/promises";
 import { PostStartCauses, PreStopCauses } from "@trigger.dev/core/v3";
 
diff --git a/apps/webapp/app/v3/services/alerts/deliverAlert.server.ts b/apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
index 3c8d95cd5ee..c373b11f8a0 100644
--- a/apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
+++ b/apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
@@ -13,6 +13,7 @@ import {
   RunFailedWebhook,
   DeploymentFailedWebhook,
   DeploymentSuccessWebhook,
+  isOOMRunError,
 } from "@trigger.dev/core/v3";
 import assertNever from "assert-never";
 import { subtle } from "crypto";
@@ -39,7 +40,6 @@ import { generateFriendlyId } from "~/v3/friendlyIdentifiers";
 import { ProjectAlertChannelType, ProjectAlertType } from "@trigger.dev/database";
 import { alertsRateLimiter } from "~/v3/alertsRateLimiter.server";
 import { v3RunPath } from "~/utils/pathBuilder";
-import { isOOMError } from "../completeAttempt.server";
 import { ApiRetrieveRunPresenter } from "~/presenters/v3/ApiRetrieveRunPresenter.server";
 
 type FoundAlert = Prisma.Result<
@@ -375,7 +375,7 @@ export class DeliverAlertService extends BaseService {
                     idempotencyKey: alert.taskRun.idempotencyKey ?? undefined,
                     tags: alert.taskRun.runTags,
                     error,
-                    isOutOfMemoryError: isOOMError(error),
+                    isOutOfMemoryError: isOOMRunError(error),
                     machine: alert.taskRun.machinePreset ?? "Unknown",
                     dashboardUrl: `${env.APP_ORIGIN}${v3RunPath(
                       alert.project.organization,
diff --git a/apps/webapp/app/v3/services/completeAttempt.server.ts b/apps/webapp/app/v3/services/completeAttempt.server.ts
index 324750f1a84..73c8fbc88d3 100644
--- a/apps/webapp/app/v3/services/completeAttempt.server.ts
+++ b/apps/webapp/app/v3/services/completeAttempt.server.ts
@@ -11,6 +11,7 @@ import {
   TaskRunSuccessfulExecutionResult,
   flattenAttributes,
   isManualOutOfMemoryError,
+  isOOMRunError,
   sanitizeError,
   shouldRetryError,
   taskRunErrorEnhancer,
@@ -255,7 +256,7 @@ export class CompleteAttemptService extends BaseService {
 
     let retriableError = shouldRetryError(taskRunErrorEnhancer(completion.error));
     let isOOMRetry = false;
-    let isOOMAttempt = isOOMError(completion.error);
+    let isOOMAttempt = isOOMRunError(completion.error);
     let isOnMaxOOMMachine = false;
     let oomMachine: MachinePresetName | undefined;
 
@@ -738,45 +739,6 @@ async function findAttempt(prismaClient: PrismaClientOrTransaction, friendlyId:
   });
 }
 
-export function isOOMError(error: TaskRunError) {
-  if (error.type === "INTERNAL_ERROR") {
-    if (
-      error.code === "TASK_PROCESS_OOM_KILLED" ||
-      error.code === "TASK_PROCESS_MAYBE_OOM_KILLED"
-    ) {
-      return true;
-    }
-
-    // For the purposes of retrying on a larger machine, we're going to treat this is an OOM error.
-    // This is what they look like if we're executing using k8s. They then get corrected later, but it's too late.
-    // {"code": "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE", "type": "INTERNAL_ERROR", "message": "Process exited with code -1 after signal SIGKILL."}
-    if (
-      error.code === "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE" &&
-      error.message &&
-      error.message.includes("SIGKILL") &&
-      error.message.includes("-1")
-    ) {
-      return true;
-    }
-  }
-
-  if (error.type === "BUILT_IN_ERROR") {
-    // ffmpeg also does weird stuff
-    // { "name": "Error", "type": "BUILT_IN_ERROR", "message": "ffmpeg was killed with signal SIGKILL" }
-    if (error.message && error.message.includes("ffmpeg was killed with signal SIGKILL")) {
-      return true;
-    }
-  }
-
-  // Special `OutOfMemoryError` for doing a manual OOM kill.
-  // Useful if a native library does an OOM but doesn't actually crash the run and you want to manually
-  if (isManualOutOfMemoryError(error)) {
-    return true;
-  }
-
-  return false;
-}
-
 function exitRun(runId: string) {
   socketIo.coordinatorNamespace.emit("REQUEST_RUN_CANCELLATION", {
     version: "v1",
diff --git a/internal-packages/run-engine/src/engine/index.ts b/internal-packages/run-engine/src/engine/index.ts
index cddf3994d8b..fb14a2f9a73 100644
--- a/internal-packages/run-engine/src/engine/index.ts
+++ b/internal-packages/run-engine/src/engine/index.ts
@@ -71,6 +71,7 @@ import {
   isPendingExecuting,
 } from "./statuses";
 import { HeartbeatTimeouts, RunEngineOptions, TriggerParams } from "./types";
+import { retryOutcomeFromCompletion } from "./retrying";
 
 const workerCatalog = {
   finishWaitpoint: {
@@ -2867,228 +2868,193 @@ export class RunEngine {
 
         const failedAt = new Date();
 
-        if (
-          completion.error.type === "INTERNAL_ERROR" &&
-          completion.error.code === "TASK_RUN_CANCELLED"
-        ) {
-          // We need to cancel the task run instead of fail it
-          const result = await this.cancelRun({
-            runId,
-            completedAt: failedAt,
-            reason: completion.error.message,
-            finalizeRun: true,
-            tx: prisma,
-          });
-          return {
-            attemptStatus:
-              result.snapshot.executionStatus === "PENDING_CANCEL"
-                ? "RUN_PENDING_CANCEL"
-                : "RUN_FINISHED",
-            ...result,
-          };
-        }
-
-        const error = sanitizeError(completion.error);
-        const retriableError = shouldRetryError(taskRunErrorEnhancer(completion.error));
-
-        const permanentlyFailRun = async (run?: {
-          status: TaskRunStatus;
-          spanId: string;
-          createdAt: Date;
-          completedAt: Date | null;
-          taskEventStore: string;
-        }) => {
-          // Emit an event so we can complete any spans of stalled executions
-          if (forceRequeue && run) {
-            this.eventBus.emit("runAttemptFailed", {
-              time: failedAt,
-              run: {
-                id: runId,
-                status: run.status,
-                spanId: run.spanId,
-                error,
-                attemptNumber: latestSnapshot.attemptNumber ?? 0,
-                createdAt: run.createdAt,
-                completedAt: run.completedAt,
-                taskEventStore: run.taskEventStore,
-              },
-            });
-          }
-
-          return await this.#permanentlyFailRun({
-            runId,
-            snapshotId,
-            failedAt,
-            error,
-            workerId,
-            runnerId,
-          });
-        };
-
-        // Error is not retriable, fail the run
-        if (!retriableError) {
-          return await permanentlyFailRun();
-        }
-
-        // No retry config attached to completion, fail the run
-        if (completion.retry === undefined) {
-          return await permanentlyFailRun();
-        }
-
-        // Run attempts have reached the global maximum, fail the run
-        if (
-          latestSnapshot.attemptNumber !== null &&
-          latestSnapshot.attemptNumber >= MAX_TASK_RUN_ATTEMPTS
-        ) {
-          return await permanentlyFailRun();
-        }
+        const retryResult = await retryOutcomeFromCompletion(prisma, {
+          runId,
+          error: completion.error,
+          retryUsingQueue: forceRequeue ?? false,
+          retrySettings: completion.retry,
+          attemptNumber: latestSnapshot.attemptNumber,
+        });
 
-        const minimalRun = await prisma.taskRun.findFirst({
-          where: {
-            id: runId,
-          },
-          select: {
-            status: true,
-            spanId: true,
-            maxAttempts: true,
-            runtimeEnvironment: {
-              select: {
-                organizationId: true,
+        // Force requeue means it was crashed so the attempt span needs to be closed
+        if (forceRequeue) {
+          const minimalRun = await prisma.taskRun.findFirst({
+            where: {
+              id: runId,
+            },
+            select: {
+              status: true,
+              spanId: true,
+              maxAttempts: true,
+              runtimeEnvironment: {
+                select: {
+                  organizationId: true,
+                },
               },
+              taskEventStore: true,
+              createdAt: true,
+              completedAt: true,
             },
-            taskEventStore: true,
-            createdAt: true,
-            completedAt: true,
-          },
-        });
-
-        if (!minimalRun) {
-          throw new ServiceValidationError("Run not found", 404);
-        }
-
-        // Run doesn't have any max attempts set which is required for retrying, fail the run
-        if (!minimalRun.maxAttempts) {
-          return await permanentlyFailRun(minimalRun);
-        }
+          });
 
-        // Run has reached the maximum configured number of attempts, fail the run
-        if (
-          latestSnapshot.attemptNumber !== null &&
-          latestSnapshot.attemptNumber >= minimalRun.maxAttempts
-        ) {
-          return await permanentlyFailRun(minimalRun);
-        }
+          if (!minimalRun) {
+            throw new ServiceValidationError("Run not found", 404);
+          }
 
-        // This error didn't come from user code, so we need to emit an event to complete any spans
-        if (forceRequeue) {
           this.eventBus.emit("runAttemptFailed", {
             time: failedAt,
             run: {
               id: runId,
               status: minimalRun.status,
               spanId: minimalRun.spanId,
-              error,
+              error: completion.error,
               attemptNumber: latestSnapshot.attemptNumber ?? 0,
-              taskEventStore: minimalRun.taskEventStore,
               createdAt: minimalRun.createdAt,
               completedAt: minimalRun.completedAt,
+              taskEventStore: minimalRun.taskEventStore,
             },
           });
         }
 
-        const retryAt = new Date(completion.retry.timestamp);
+        switch (retryResult.outcome) {
+          case "cancel_run": {
+            const result = await this.cancelRun({
+              runId,
+              completedAt: failedAt,
+              reason: retryResult.reason,
+              finalizeRun: true,
+              tx: prisma,
+            });
+            return {
+              attemptStatus:
+                result.snapshot.executionStatus === "PENDING_CANCEL"
+                  ? "RUN_PENDING_CANCEL"
+                  : "RUN_FINISHED",
+              ...result,
+            };
+          }
+          case "fail_run": {
+            return await this.#permanentlyFailRun({
+              runId,
+              snapshotId,
+              failedAt,
+              error: retryResult.sanitizedError,
+              workerId,
+              runnerId,
+            });
+          }
+          case "retry": {
+            const retryAt = new Date(retryResult.settings.timestamp);
 
-        const run = await prisma.taskRun.update({
-          where: {
-            id: runId,
-          },
-          data: {
-            status: "RETRYING_AFTER_FAILURE",
-          },
-          include: {
-            runtimeEnvironment: {
+            const run = await prisma.taskRun.update({
+              where: {
+                id: runId,
+              },
+              data: {
+                status: "RETRYING_AFTER_FAILURE",
+                machinePreset: retryResult.machine,
+              },
               include: {
-                project: true,
-                organization: true,
-                orgMember: true,
+                runtimeEnvironment: {
+                  include: {
+                    project: true,
+                    organization: true,
+                    orgMember: true,
+                  },
+                },
               },
-            },
-          },
-        });
+            });
 
-        const nextAttemptNumber =
-          latestSnapshot.attemptNumber === null ? 1 : latestSnapshot.attemptNumber + 1;
+            const nextAttemptNumber =
+              latestSnapshot.attemptNumber === null ? 1 : latestSnapshot.attemptNumber + 1;
 
-        this.eventBus.emit("runRetryScheduled", {
-          time: failedAt,
-          run: {
-            id: run.id,
-            friendlyId: run.friendlyId,
-            attemptNumber: nextAttemptNumber,
-            queue: run.queue,
-            taskIdentifier: run.taskIdentifier,
-            traceContext: run.traceContext as Record<string, string | undefined>,
-            baseCostInCents: run.baseCostInCents,
-            spanId: run.spanId,
-          },
-          organization: {
-            id: run.runtimeEnvironment.organizationId,
-          },
-          environment: run.runtimeEnvironment,
-          retryAt,
-        });
+            if (retryResult.wasOOMError) {
+              this.eventBus.emit("runAttemptFailed", {
+                time: failedAt,
+                run: {
+                  id: runId,
+                  status: run.status,
+                  spanId: run.spanId,
+                  error: completion.error,
+                  attemptNumber: latestSnapshot.attemptNumber ?? 0,
+                  createdAt: run.createdAt,
+                  completedAt: run.completedAt,
+                  taskEventStore: run.taskEventStore,
+                },
+              });
+            }
 
-        //todo anything special for DEV? Ideally not.
+            this.eventBus.emit("runRetryScheduled", {
+              time: failedAt,
+              run: {
+                id: run.id,
+                friendlyId: run.friendlyId,
+                attemptNumber: nextAttemptNumber,
+                queue: run.queue,
+                taskIdentifier: run.taskIdentifier,
+                traceContext: run.traceContext as Record<string, string | undefined>,
+                baseCostInCents: run.baseCostInCents,
+                spanId: run.spanId,
+              },
+              organization: {
+                id: run.runtimeEnvironment.organizationId,
+              },
+              environment: run.runtimeEnvironment,
+              retryAt,
+            });
 
-        //if it's a long delay and we support checkpointing, put it back in the queue
-        if (
-          forceRequeue ||
-          (this.options.retryWarmStartThresholdMs !== undefined &&
-            completion.retry.delay >= this.options.retryWarmStartThresholdMs)
-        ) {
-          //we nack the message, requeuing it for later
-          const nackResult = await this.#tryNackAndRequeue({
-            run,
-            environment: run.runtimeEnvironment,
-            orgId: run.runtimeEnvironment.organizationId,
-            timestamp: retryAt.getTime(),
-            error: {
-              type: "INTERNAL_ERROR",
-              code: "TASK_RUN_DEQUEUED_MAX_RETRIES",
-              message: `We tried to dequeue the run the maximum number of times but it wouldn't start executing`,
-            },
-            tx: prisma,
-          });
+            //if it's a long delay and we support checkpointing, put it back in the queue
+            if (
+              forceRequeue ||
+              retryResult.method === "queue" ||
+              (this.options.retryWarmStartThresholdMs !== undefined &&
+                retryResult.settings.delay >= this.options.retryWarmStartThresholdMs)
+            ) {
+              //we nack the message, requeuing it for later
+              const nackResult = await this.#tryNackAndRequeue({
+                run,
+                environment: run.runtimeEnvironment,
+                orgId: run.runtimeEnvironment.organizationId,
+                timestamp: retryAt.getTime(),
+                error: {
+                  type: "INTERNAL_ERROR",
+                  code: "TASK_RUN_DEQUEUED_MAX_RETRIES",
+                  message: `We tried to dequeue the run the maximum number of times but it wouldn't start executing`,
+                },
+                tx: prisma,
+              });
+
+              if (!nackResult.wasRequeued) {
+                return {
+                  attemptStatus: "RUN_FINISHED",
+                  ...nackResult,
+                };
+              } else {
+                return { attemptStatus: "RETRY_QUEUED", ...nackResult };
+              }
+            }
+
+            //it will continue running because the retry delay is short
+            const newSnapshot = await this.#createExecutionSnapshot(prisma, {
+              run,
+              snapshot: {
+                executionStatus: "PENDING_EXECUTING",
+                description: "Attempt failed with a short delay, starting a new attempt",
+              },
+              environmentId: latestSnapshot.environmentId,
+              environmentType: latestSnapshot.environmentType,
+              workerId,
+              runnerId,
+            });
+            //the worker can fetch the latest snapshot and should create a new attempt
+            await this.#sendNotificationToWorker({ runId, snapshot: newSnapshot });
 
-          if (!nackResult.wasRequeued) {
             return {
-              attemptStatus: "RUN_FINISHED",
-              ...nackResult,
+              attemptStatus: "RETRY_IMMEDIATELY",
+              ...executionResultFromSnapshot(newSnapshot),
             };
-          } else {
-            return { attemptStatus: "RETRY_QUEUED", ...nackResult };
           }
         }
-
-        //it will continue running because the retry delay is short
-        const newSnapshot = await this.#createExecutionSnapshot(prisma, {
-          run,
-          snapshot: {
-            executionStatus: "PENDING_EXECUTING",
-            description: "Attempt failed with a short delay, starting a new attempt",
-          },
-          environmentId: latestSnapshot.environmentId,
-          environmentType: latestSnapshot.environmentType,
-          workerId,
-          runnerId,
-        });
-        //the worker can fetch the latest snapshot and should create a new attempt
-        await this.#sendNotificationToWorker({ runId, snapshot: newSnapshot });
-
-        return {
-          attemptStatus: "RETRY_IMMEDIATELY",
-          ...executionResultFromSnapshot(newSnapshot),
-        };
       });
     });
   }
diff --git a/internal-packages/run-engine/src/engine/retrying.ts b/internal-packages/run-engine/src/engine/retrying.ts
new file mode 100644
index 00000000000..974bcd11512
--- /dev/null
+++ b/internal-packages/run-engine/src/engine/retrying.ts
@@ -0,0 +1,175 @@
+import {
+  isOOMRunError,
+  RetryOptions,
+  shouldRetryError,
+  TaskRunError,
+  TaskRunExecutionRetry,
+  taskRunErrorEnhancer,
+  sanitizeError,
+  calculateNextRetryDelay,
+} from "@trigger.dev/core/v3";
+import { PrismaClientOrTransaction, TaskRunStatus } from "@trigger.dev/database";
+import { MAX_TASK_RUN_ATTEMPTS } from "./consts";
+import { ServiceValidationError } from ".";
+
+type Params = {
+  runId: string;
+  attemptNumber: number | null;
+  error: TaskRunError;
+  retryUsingQueue: boolean;
+  retrySettings: TaskRunExecutionRetry | undefined;
+};
+
+export type RetryOutcome =
+  | {
+      outcome: "cancel_run";
+      reason?: string;
+    }
+  | {
+      outcome: "fail_run";
+      sanitizedError: TaskRunError;
+      wasOOMError?: boolean;
+    }
+  | {
+      outcome: "retry";
+      method: "queue" | "immediate";
+      settings: TaskRunExecutionRetry;
+      machine?: string;
+      wasOOMError?: boolean;
+    };
+
+export async function retryOutcomeFromCompletion(
+  prisma: PrismaClientOrTransaction,
+  { runId, attemptNumber, error, retryUsingQueue, retrySettings }: Params
+): Promise<RetryOutcome> {
+  // Canceled
+  if (error.type === "INTERNAL_ERROR" && error.code === "TASK_RUN_CANCELLED") {
+    return { outcome: "cancel_run", reason: error.message };
+  }
+
+  const sanitizedError = sanitizeError(error);
+
+  // OOM error (retry on a larger machine or fail)
+  if (isOOMRunError(error)) {
+    const oomResult = await retryOOMOnMachine(prisma, runId);
+    if (!oomResult) {
+      return { outcome: "fail_run", sanitizedError, wasOOMError: true };
+    }
+
+    const delay = calculateNextRetryDelay(oomResult.retrySettings, attemptNumber ?? 1);
+
+    if (!delay) {
+      //no more retries left
+      return { outcome: "fail_run", sanitizedError, wasOOMError: true };
+    }
+
+    return {
+      outcome: "retry",
+      method: "queue",
+      machine: oomResult.machine,
+      settings: { timestamp: Date.now() + delay, delay },
+      wasOOMError: true,
+    };
+  }
+
+  // No retry settings
+  if (!retrySettings) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // Not a retriable error: fail
+  const retriableError = shouldRetryError(taskRunErrorEnhancer(error));
+  if (!retriableError) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // Exceeded global max attempts
+  if (attemptNumber !== null && attemptNumber > MAX_TASK_RUN_ATTEMPTS) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // Get the run settings
+  const run = await prisma.taskRun.findFirst({
+    where: {
+      id: runId,
+    },
+    select: {
+      maxAttempts: true,
+    },
+  });
+
+  if (!run) {
+    throw new ServiceValidationError("Run not found", 404);
+  }
+
+  // No max attempts set
+  if (!run.maxAttempts) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // No attempts left
+  if (attemptNumber !== null && attemptNumber >= run.maxAttempts) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  return {
+    outcome: "retry",
+    method: retryUsingQueue ? "queue" : "immediate",
+    settings: retrySettings,
+  };
+}
+
+async function retryOOMOnMachine(
+  prisma: PrismaClientOrTransaction,
+  runId: string
+): Promise<{ machine: string; retrySettings: RetryOptions } | undefined> {
+  try {
+    const run = await prisma.taskRun.findFirst({
+      where: {
+        id: runId,
+      },
+      select: {
+        machinePreset: true,
+        lockedBy: {
+          select: {
+            retryConfig: true,
+          },
+        },
+      },
+    });
+
+    if (!run || !run.lockedBy || !run.machinePreset) {
+      return;
+    }
+
+    const retryConfig = run.lockedBy?.retryConfig;
+    const parsedRetryConfig = RetryOptions.nullish().safeParse(retryConfig);
+
+    if (!parsedRetryConfig.success) {
+      return;
+    }
+
+    if (!parsedRetryConfig.data) {
+      return;
+    }
+
+    const retryMachine = parsedRetryConfig.data.outOfMemory?.machine;
+
+    if (!retryMachine) {
+      return;
+    }
+
+    if (run.machinePreset === retryMachine) {
+      return;
+    }
+
+    return { machine: retryMachine, retrySettings: parsedRetryConfig.data };
+  } catch (error) {
+    console.error("[FailedTaskRunRetryHelper] Failed to get execution retry", {
+      runId,
+      error,
+    });
+
+    return;
+  }
+}
diff --git a/internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts b/internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts
new file mode 100644
index 00000000000..232d090a735
--- /dev/null
+++ b/internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts
@@ -0,0 +1,830 @@
+import {
+  assertNonNullable,
+  containerTest,
+  setupAuthenticatedEnvironment,
+  setupBackgroundWorker,
+} from "@internal/testcontainers";
+import { trace } from "@opentelemetry/api";
+import { expect } from "vitest";
+import { EventBusEventArgs } from "../eventBus.js";
+import { RunEngine } from "../index.js";
+import { setTimeout } from "node:timers/promises";
+
+describe("RunEngine attempt failures", () => {
+  containerTest(
+    "Retry user error and succeed",
+    { timeout: 15_000 },
+    async ({ prisma, redisOptions }) => {
+      //create environment
+      const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+      const engine = new RunEngine({
+        prisma,
+        worker: {
+          redis: redisOptions,
+          workers: 1,
+          tasksPerWorker: 10,
+          pollIntervalMs: 100,
+        },
+        queue: {
+          redis: redisOptions,
+        },
+        runLock: {
+          redis: redisOptions,
+        },
+        machines: {
+          defaultMachine: "small-1x",
+          machines: {
+            "small-1x": {
+              name: "small-1x" as const,
+              cpu: 0.5,
+              memory: 0.5,
+              centsPerMs: 0.0001,
+            },
+          },
+          baseCostInCents: 0.0001,
+        },
+        tracer: trace.getTracer("test", "0.0.0"),
+      });
+
+      try {
+        const taskIdentifier = "test-task";
+
+        //create background worker
+        const backgroundWorker = await setupBackgroundWorker(
+          prisma,
+          authenticatedEnvironment,
+          taskIdentifier
+        );
+
+        //trigger the run
+        const run = await engine.trigger(
+          {
+            number: 1,
+            friendlyId: "run_1234",
+            environment: authenticatedEnvironment,
+            taskIdentifier,
+            payload: "{}",
+            payloadType: "application/json",
+            context: {},
+            traceContext: {},
+            traceId: "t12345",
+            spanId: "s12345",
+            masterQueue: "main",
+            queueName: "task/test-task",
+            isTest: false,
+            tags: [],
+          },
+          prisma
+        );
+
+        //dequeue the run
+        const dequeued = await engine.dequeueFromMasterQueue({
+          consumerId: "test_12345",
+          masterQueue: run.masterQueue,
+          maxRunCount: 10,
+        });
+
+        //create an attempt
+        const attemptResult = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: dequeued[0].snapshot.id,
+        });
+
+        //fail the attempt
+        const error = {
+          type: "BUILT_IN_ERROR" as const,
+          name: "UserError",
+          message: "This is a user error",
+          stackTrace: "Error: This is a user error\n    at <anonymous>:1:1",
+        };
+        const result = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult.snapshot.id,
+          completion: {
+            ok: false,
+            id: dequeued[0].run.id,
+            error,
+            retry: {
+              timestamp: Date.now(),
+              delay: 0,
+            },
+          },
+        });
+        expect(result.attemptStatus).toBe("RETRY_IMMEDIATELY");
+        expect(result.snapshot.executionStatus).toBe("PENDING_EXECUTING");
+        expect(result.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //state should be pending
+        const executionData3 = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData3);
+        expect(executionData3.snapshot.executionStatus).toBe("PENDING_EXECUTING");
+        //only when the new attempt is created, should the attempt be increased
+        expect(executionData3.run.attemptNumber).toBe(1);
+        expect(executionData3.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //create a second attempt
+        const attemptResult2 = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: executionData3.snapshot.id,
+        });
+        expect(attemptResult2.run.attemptNumber).toBe(2);
+
+        //now complete it successfully
+        const result2 = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult2.snapshot.id,
+          completion: {
+            ok: true,
+            id: dequeued[0].run.id,
+            output: `{"foo":"bar"}`,
+            outputType: "application/json",
+          },
+        });
+        expect(result2.snapshot.executionStatus).toBe("FINISHED");
+        expect(result2.run.attemptNumber).toBe(2);
+        expect(result2.run.status).toBe("COMPLETED_SUCCESSFULLY");
+
+        //waitpoint should have been completed, with the output
+        const runWaitpointAfter = await prisma.waitpoint.findMany({
+          where: {
+            completedByTaskRunId: run.id,
+          },
+        });
+        expect(runWaitpointAfter.length).toBe(1);
+        expect(runWaitpointAfter[0].type).toBe("RUN");
+        expect(runWaitpointAfter[0].output).toBe(`{"foo":"bar"}`);
+        expect(runWaitpointAfter[0].outputIsError).toBe(false);
+
+        //state should be completed
+        const executionData4 = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData4);
+        expect(executionData4.snapshot.executionStatus).toBe("FINISHED");
+        expect(executionData4.run.attemptNumber).toBe(2);
+        expect(executionData4.run.status).toBe("COMPLETED_SUCCESSFULLY");
+      } finally {
+        engine.quit();
+      }
+    }
+  );
+
+  containerTest("Fail (no more retries)", { timeout: 15_000 }, async ({ prisma, redisOptions }) => {
+    //create environment
+    const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+    const engine = new RunEngine({
+      prisma,
+      worker: {
+        redis: redisOptions,
+        workers: 1,
+        tasksPerWorker: 10,
+        pollIntervalMs: 100,
+      },
+      queue: {
+        redis: redisOptions,
+      },
+      runLock: {
+        redis: redisOptions,
+      },
+      machines: {
+        defaultMachine: "small-1x",
+        machines: {
+          "small-1x": {
+            name: "small-1x" as const,
+            cpu: 0.5,
+            memory: 0.5,
+            centsPerMs: 0.0001,
+          },
+        },
+        baseCostInCents: 0.0001,
+      },
+      tracer: trace.getTracer("test", "0.0.0"),
+    });
+
+    try {
+      const taskIdentifier = "test-task";
+
+      //create background worker
+      await setupBackgroundWorker(prisma, authenticatedEnvironment, taskIdentifier, undefined, {
+        maxAttempts: 1,
+      });
+
+      //trigger the run
+      const run = await engine.trigger(
+        {
+          number: 1,
+          friendlyId: "run_1234",
+          environment: authenticatedEnvironment,
+          taskIdentifier,
+          payload: "{}",
+          payloadType: "application/json",
+          context: {},
+          traceContext: {},
+          traceId: "t12345",
+          spanId: "s12345",
+          masterQueue: "main",
+          queueName: "task/test-task",
+          isTest: false,
+          tags: [],
+        },
+        prisma
+      );
+
+      //dequeue the run
+      const dequeued = await engine.dequeueFromMasterQueue({
+        consumerId: "test_12345",
+        masterQueue: run.masterQueue,
+        maxRunCount: 10,
+      });
+
+      //create an attempt
+      const attemptResult = await engine.startRunAttempt({
+        runId: dequeued[0].run.id,
+        snapshotId: dequeued[0].snapshot.id,
+      });
+
+      //fail the attempt
+      const error = {
+        type: "BUILT_IN_ERROR" as const,
+        name: "UserError",
+        message: "This is a user error",
+        stackTrace: "Error: This is a user error\n    at <anonymous>:1:1",
+      };
+      const result = await engine.completeRunAttempt({
+        runId: dequeued[0].run.id,
+        snapshotId: attemptResult.snapshot.id,
+        completion: {
+          ok: false,
+          id: dequeued[0].run.id,
+          error,
+          retry: {
+            timestamp: Date.now(),
+            delay: 0,
+          },
+        },
+      });
+      expect(result.attemptStatus).toBe("RUN_FINISHED");
+      expect(result.snapshot.executionStatus).toBe("FINISHED");
+      expect(result.run.status).toBe("COMPLETED_WITH_ERRORS");
+
+      //state should be pending
+      const executionData3 = await engine.getRunExecutionData({ runId: run.id });
+      assertNonNullable(executionData3);
+      expect(executionData3.snapshot.executionStatus).toBe("FINISHED");
+      //only when the new attempt is created, should the attempt be increased
+      expect(executionData3.run.attemptNumber).toBe(1);
+      expect(executionData3.run.status).toBe("COMPLETED_WITH_ERRORS");
+    } finally {
+      engine.quit();
+    }
+  });
+
+  containerTest(
+    "Fail (not a retriable error)",
+    { timeout: 15_000 },
+    async ({ prisma, redisOptions }) => {
+      //create environment
+      const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+      const engine = new RunEngine({
+        prisma,
+        worker: {
+          redis: redisOptions,
+          workers: 1,
+          tasksPerWorker: 10,
+          pollIntervalMs: 100,
+        },
+        queue: {
+          redis: redisOptions,
+        },
+        runLock: {
+          redis: redisOptions,
+        },
+        machines: {
+          defaultMachine: "small-1x",
+          machines: {
+            "small-1x": {
+              name: "small-1x" as const,
+              cpu: 0.5,
+              memory: 0.5,
+              centsPerMs: 0.0001,
+            },
+          },
+          baseCostInCents: 0.0001,
+        },
+        tracer: trace.getTracer("test", "0.0.0"),
+      });
+
+      try {
+        const taskIdentifier = "test-task";
+
+        //create background worker
+        await setupBackgroundWorker(prisma, authenticatedEnvironment, taskIdentifier, undefined, {
+          maxAttempts: 1,
+        });
+
+        //trigger the run
+        const run = await engine.trigger(
+          {
+            number: 1,
+            friendlyId: "run_1234",
+            environment: authenticatedEnvironment,
+            taskIdentifier,
+            payload: "{}",
+            payloadType: "application/json",
+            context: {},
+            traceContext: {},
+            traceId: "t12345",
+            spanId: "s12345",
+            masterQueue: "main",
+            queueName: "task/test-task",
+            isTest: false,
+            tags: [],
+          },
+          prisma
+        );
+
+        //dequeue the run
+        const dequeued = await engine.dequeueFromMasterQueue({
+          consumerId: "test_12345",
+          masterQueue: run.masterQueue,
+          maxRunCount: 10,
+        });
+
+        //create an attempt
+        const attemptResult = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: dequeued[0].snapshot.id,
+        });
+
+        //fail the attempt with an unretriable error
+        const error = {
+          type: "INTERNAL_ERROR" as const,
+          code: "DISK_SPACE_EXCEEDED" as const,
+        };
+        const result = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult.snapshot.id,
+          completion: {
+            ok: false,
+            id: dequeued[0].run.id,
+            error,
+            retry: {
+              timestamp: Date.now(),
+              delay: 0,
+            },
+          },
+        });
+        expect(result.attemptStatus).toBe("RUN_FINISHED");
+        expect(result.snapshot.executionStatus).toBe("FINISHED");
+        expect(result.run.status).toBe("CRASHED");
+
+        //state should be pending
+        const executionData3 = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData3);
+        expect(executionData3.snapshot.executionStatus).toBe("FINISHED");
+        //only when the new attempt is created, should the attempt be increased
+        expect(executionData3.run.attemptNumber).toBe(1);
+        expect(executionData3.run.status).toBe("CRASHED");
+      } finally {
+        engine.quit();
+      }
+    }
+  );
+
+  containerTest("OOM fail", { timeout: 15_000 }, async ({ prisma, redisOptions }) => {
+    //create environment
+    const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+    const engine = new RunEngine({
+      prisma,
+      worker: {
+        redis: redisOptions,
+        workers: 1,
+        tasksPerWorker: 10,
+        pollIntervalMs: 100,
+      },
+      queue: {
+        redis: redisOptions,
+      },
+      runLock: {
+        redis: redisOptions,
+      },
+      machines: {
+        defaultMachine: "small-1x",
+        machines: {
+          "small-1x": {
+            name: "small-1x" as const,
+            cpu: 0.5,
+            memory: 0.5,
+            centsPerMs: 0.0001,
+          },
+        },
+        baseCostInCents: 0.0001,
+      },
+      tracer: trace.getTracer("test", "0.0.0"),
+    });
+
+    try {
+      const taskIdentifier = "test-task";
+
+      //create background worker
+      await setupBackgroundWorker(prisma, authenticatedEnvironment, taskIdentifier);
+
+      //trigger the run
+      const run = await engine.trigger(
+        {
+          number: 1,
+          friendlyId: "run_1234",
+          environment: authenticatedEnvironment,
+          taskIdentifier,
+          payload: "{}",
+          payloadType: "application/json",
+          context: {},
+          traceContext: {},
+          traceId: "t12345",
+          spanId: "s12345",
+          masterQueue: "main",
+          queueName: "task/test-task",
+          isTest: false,
+          tags: [],
+        },
+        prisma
+      );
+
+      //dequeue the run
+      const dequeued = await engine.dequeueFromMasterQueue({
+        consumerId: "test_12345",
+        masterQueue: run.masterQueue,
+        maxRunCount: 10,
+      });
+
+      //create an attempt
+      const attemptResult = await engine.startRunAttempt({
+        runId: dequeued[0].run.id,
+        snapshotId: dequeued[0].snapshot.id,
+      });
+
+      //fail the attempt with an OOM error
+      const error = {
+        type: "INTERNAL_ERROR" as const,
+        code: "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE" as const,
+        message: "Process exited with code -1 after signal SIGKILL.",
+        stackTrace: "JavaScript heap out of memory",
+      };
+
+      const result = await engine.completeRunAttempt({
+        runId: dequeued[0].run.id,
+        snapshotId: attemptResult.snapshot.id,
+        completion: {
+          ok: false,
+          id: dequeued[0].run.id,
+          error,
+        },
+      });
+
+      // The run should be retried with a larger machine
+      expect(result.attemptStatus).toBe("RUN_FINISHED");
+      expect(result.snapshot.executionStatus).toBe("FINISHED");
+      expect(result.run.status).toBe("CRASHED");
+
+      //state should be pending
+      const executionData = await engine.getRunExecutionData({ runId: run.id });
+      assertNonNullable(executionData);
+      expect(executionData.snapshot.executionStatus).toBe("FINISHED");
+      expect(executionData.run.attemptNumber).toBe(1);
+      expect(executionData.run.status).toBe("CRASHED");
+    } finally {
+      engine.quit();
+    }
+  });
+
+  containerTest(
+    "OOM retry on larger machine",
+    { timeout: 15_000 },
+    async ({ prisma, redisOptions }) => {
+      //create environment
+      const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+      const engine = new RunEngine({
+        prisma,
+        worker: {
+          redis: redisOptions,
+          workers: 1,
+          tasksPerWorker: 10,
+          pollIntervalMs: 100,
+        },
+        queue: {
+          redis: redisOptions,
+        },
+        runLock: {
+          redis: redisOptions,
+        },
+        machines: {
+          defaultMachine: "small-1x",
+          machines: {
+            "small-1x": {
+              name: "small-1x" as const,
+              cpu: 0.5,
+              memory: 0.5,
+              centsPerMs: 0.0001,
+            },
+            "small-2x": {
+              name: "small-2x" as const,
+              cpu: 1,
+              memory: 1,
+              centsPerMs: 0.0002,
+            },
+          },
+          baseCostInCents: 0.0001,
+        },
+        tracer: trace.getTracer("test", "0.0.0"),
+      });
+
+      try {
+        const taskIdentifier = "test-task";
+
+        //create background worker
+        await setupBackgroundWorker(prisma, authenticatedEnvironment, taskIdentifier, undefined, {
+          outOfMemory: {
+            machine: "small-2x",
+          },
+        });
+
+        //trigger the run
+        const run = await engine.trigger(
+          {
+            number: 1,
+            friendlyId: "run_1234",
+            environment: authenticatedEnvironment,
+            taskIdentifier,
+            payload: "{}",
+            payloadType: "application/json",
+            context: {},
+            traceContext: {},
+            traceId: "t12345",
+            spanId: "s12345",
+            masterQueue: "main",
+            queueName: "task/test-task",
+            isTest: false,
+            tags: [],
+          },
+          prisma
+        );
+
+        //dequeue the run
+        const dequeued = await engine.dequeueFromMasterQueue({
+          consumerId: "test_12345",
+          masterQueue: run.masterQueue,
+          maxRunCount: 10,
+        });
+
+        //create an attempt
+        const attemptResult = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: dequeued[0].snapshot.id,
+        });
+
+        //fail the attempt with an OOM error
+        const error = {
+          type: "INTERNAL_ERROR" as const,
+          code: "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE" as const,
+          message: "Process exited with code -1 after signal SIGKILL.",
+          stackTrace: "JavaScript heap out of memory",
+        };
+
+        const result = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult.snapshot.id,
+          completion: {
+            ok: false,
+            id: dequeued[0].run.id,
+            error,
+          },
+        });
+
+        // The run should be retried with a larger machine
+        expect(result.attemptStatus).toBe("RETRY_QUEUED");
+        expect(result.snapshot.executionStatus).toBe("QUEUED");
+        expect(result.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //state should be pending
+        const executionData = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData);
+        expect(executionData.snapshot.executionStatus).toBe("QUEUED");
+        expect(executionData.run.attemptNumber).toBe(1);
+        expect(executionData.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //create a second attempt
+        const attemptResult2 = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: executionData.snapshot.id,
+        });
+        expect(attemptResult2.run.attemptNumber).toBe(2);
+
+        //now complete it successfully
+        const result2 = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult2.snapshot.id,
+          completion: {
+            ok: true,
+            id: dequeued[0].run.id,
+            output: `{"foo":"bar"}`,
+            outputType: "application/json",
+          },
+        });
+        expect(result2.snapshot.executionStatus).toBe("FINISHED");
+        expect(result2.run.attemptNumber).toBe(2);
+        expect(result2.run.status).toBe("COMPLETED_SUCCESSFULLY");
+
+        //waitpoint should have been completed, with the output
+        const runWaitpointAfter = await prisma.waitpoint.findMany({
+          where: {
+            completedByTaskRunId: run.id,
+          },
+        });
+        expect(runWaitpointAfter.length).toBe(1);
+        expect(runWaitpointAfter[0].type).toBe("RUN");
+        expect(runWaitpointAfter[0].output).toBe(`{"foo":"bar"}`);
+        expect(runWaitpointAfter[0].outputIsError).toBe(false);
+
+        //state should be completed
+        const executionData4 = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData4);
+        expect(executionData4.snapshot.executionStatus).toBe("FINISHED");
+        expect(executionData4.run.attemptNumber).toBe(2);
+        expect(executionData4.run.status).toBe("COMPLETED_SUCCESSFULLY");
+      } finally {
+        engine.quit();
+      }
+    }
+  );
+
+  containerTest(
+    "OOM fails after retrying on larger machine",
+    { timeout: 15_000 },
+    async ({ prisma, redisOptions }) => {
+      //create environment
+      const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
+
+      const engine = new RunEngine({
+        prisma,
+        worker: {
+          redis: redisOptions,
+          workers: 1,
+          tasksPerWorker: 10,
+          pollIntervalMs: 100,
+        },
+        queue: {
+          redis: redisOptions,
+        },
+        runLock: {
+          redis: redisOptions,
+        },
+        machines: {
+          defaultMachine: "small-1x",
+          machines: {
+            "small-1x": {
+              name: "small-1x" as const,
+              cpu: 0.5,
+              memory: 0.5,
+              centsPerMs: 0.0001,
+            },
+            "small-2x": {
+              name: "small-2x" as const,
+              cpu: 1,
+              memory: 1,
+              centsPerMs: 0.0002,
+            },
+          },
+          baseCostInCents: 0.0001,
+        },
+        tracer: trace.getTracer("test", "0.0.0"),
+      });
+
+      try {
+        const taskIdentifier = "test-task";
+
+        //create background worker
+        await setupBackgroundWorker(prisma, authenticatedEnvironment, taskIdentifier, undefined, {
+          maxTimeoutInMs: 10,
+          maxAttempts: 10,
+          outOfMemory: {
+            machine: "small-2x",
+          },
+        });
+
+        //trigger the run
+        const run = await engine.trigger(
+          {
+            number: 1,
+            friendlyId: "run_1234",
+            environment: authenticatedEnvironment,
+            taskIdentifier,
+            payload: "{}",
+            payloadType: "application/json",
+            context: {},
+            traceContext: {},
+            traceId: "t12345",
+            spanId: "s12345",
+            masterQueue: "main",
+            queueName: "task/test-task",
+            isTest: false,
+            tags: [],
+          },
+          prisma
+        );
+
+        //dequeue the run
+        const dequeued = await engine.dequeueFromMasterQueue({
+          consumerId: "test_12345",
+          masterQueue: run.masterQueue,
+          maxRunCount: 10,
+        });
+
+        //create first attempt
+        const attemptResult = await engine.startRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: dequeued[0].snapshot.id,
+        });
+
+        //fail the first attempt with an OOM error
+        const error = {
+          type: "INTERNAL_ERROR" as const,
+          code: "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE" as const,
+          message: "Process exited with code -1 after signal SIGKILL.",
+          stackTrace: "JavaScript heap out of memory",
+        };
+
+        const result = await engine.completeRunAttempt({
+          runId: dequeued[0].run.id,
+          snapshotId: attemptResult.snapshot.id,
+          completion: {
+            ok: false,
+            id: dequeued[0].run.id,
+            error,
+          },
+        });
+
+        // The run should be retried with a larger machine
+        expect(result.attemptStatus).toBe("RETRY_QUEUED");
+        expect(result.snapshot.executionStatus).toBe("QUEUED");
+        expect(result.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //state should be queued
+        const executionData = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(executionData);
+        expect(executionData.snapshot.executionStatus).toBe("QUEUED");
+        expect(executionData.run.attemptNumber).toBe(1);
+        expect(executionData.run.status).toBe("RETRYING_AFTER_FAILURE");
+
+        //wait for 1s
+        await setTimeout(1_000);
+
+        //dequeue again
+        const dequeued2 = await engine.dequeueFromMasterQueue({
+          consumerId: "test_12345",
+          masterQueue: run.masterQueue,
+          maxRunCount: 10,
+        });
+        expect(dequeued2.length).toBe(1);
+
+        //create second attempt
+        const attemptResult2 = await engine.startRunAttempt({
+          runId: dequeued2[0].run.id,
+          snapshotId: dequeued2[0].snapshot.id,
+        });
+        expect(attemptResult2.run.attemptNumber).toBe(2);
+
+        //fail the second attempt with the same OOM error
+        const result2 = await engine.completeRunAttempt({
+          runId: dequeued2[0].run.id,
+          snapshotId: attemptResult2.snapshot.id,
+          completion: {
+            ok: false,
+            id: dequeued2[0].run.id,
+            error,
+            retry: {
+              timestamp: Date.now(),
+              delay: 0,
+            },
+          },
+        });
+
+        // The run should fail after the second OOM
+        expect(result2.attemptStatus).toBe("RUN_FINISHED");
+        expect(result2.snapshot.executionStatus).toBe("FINISHED");
+        expect(result2.run.status).toBe("CRASHED");
+
+        //final state should be crashed
+        const finalExecutionData = await engine.getRunExecutionData({ runId: run.id });
+        assertNonNullable(finalExecutionData);
+        expect(finalExecutionData.snapshot.executionStatus).toBe("FINISHED");
+        expect(finalExecutionData.run.attemptNumber).toBe(2);
+        expect(finalExecutionData.run.status).toBe("CRASHED");
+      } finally {
+        engine.quit();
+      }
+    }
+  );
+});
diff --git a/internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts b/internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts
index 66392536548..d2869886d0c 100644
--- a/internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts
+++ b/internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts
@@ -5,7 +5,7 @@ import {
   setupBackgroundWorker,
 } from "@internal/testcontainers";
 import { trace } from "@opentelemetry/api";
-import { expect } from "vitest";
+import { expect, describe } from "vitest";
 import { RunEngine } from "../index.js";
 import { setTimeout } from "node:timers/promises";
 import { generateFriendlyId } from "@trigger.dev/core/v3/apps";
diff --git a/internal-packages/run-engine/src/engine/tests/heartbeats.test.ts b/internal-packages/run-engine/src/engine/tests/heartbeats.test.ts
index 1ff229cd7b4..b6068454905 100644
--- a/internal-packages/run-engine/src/engine/tests/heartbeats.test.ts
+++ b/internal-packages/run-engine/src/engine/tests/heartbeats.test.ts
@@ -5,7 +5,7 @@ import {
   assertNonNullable,
 } from "@internal/testcontainers";
 import { trace } from "@opentelemetry/api";
-import { expect } from "vitest";
+import { expect, describe } from "vitest";
 import { RunEngine } from "../index.js";
 import { setTimeout } from "timers/promises";
 
diff --git a/internal-packages/run-engine/src/engine/tests/trigger.test.ts b/internal-packages/run-engine/src/engine/tests/trigger.test.ts
index 69dbb66b08d..1eef5dd8382 100644
--- a/internal-packages/run-engine/src/engine/tests/trigger.test.ts
+++ b/internal-packages/run-engine/src/engine/tests/trigger.test.ts
@@ -330,161 +330,4 @@ describe("RunEngine trigger()", () => {
       engine.quit();
     }
   });
-
-  containerTest(
-    "Single run (retry attempt, then succeed)",
-    { timeout: 15_000 },
-    async ({ prisma, redisOptions }) => {
-      //create environment
-      const authenticatedEnvironment = await setupAuthenticatedEnvironment(prisma, "PRODUCTION");
-
-      const engine = new RunEngine({
-        prisma,
-        worker: {
-          redis: redisOptions,
-          workers: 1,
-          tasksPerWorker: 10,
-          pollIntervalMs: 100,
-        },
-        queue: {
-          redis: redisOptions,
-        },
-        runLock: {
-          redis: redisOptions,
-        },
-        machines: {
-          defaultMachine: "small-1x",
-          machines: {
-            "small-1x": {
-              name: "small-1x" as const,
-              cpu: 0.5,
-              memory: 0.5,
-              centsPerMs: 0.0001,
-            },
-          },
-          baseCostInCents: 0.0001,
-        },
-        tracer: trace.getTracer("test", "0.0.0"),
-      });
-
-      try {
-        const taskIdentifier = "test-task";
-
-        //create background worker
-        const backgroundWorker = await setupBackgroundWorker(
-          prisma,
-          authenticatedEnvironment,
-          taskIdentifier
-        );
-
-        //trigger the run
-        const run = await engine.trigger(
-          {
-            number: 1,
-            friendlyId: "run_1234",
-            environment: authenticatedEnvironment,
-            taskIdentifier,
-            payload: "{}",
-            payloadType: "application/json",
-            context: {},
-            traceContext: {},
-            traceId: "t12345",
-            spanId: "s12345",
-            masterQueue: "main",
-            queueName: "task/test-task",
-            isTest: false,
-            tags: [],
-          },
-          prisma
-        );
-
-        //dequeue the run
-        const dequeued = await engine.dequeueFromMasterQueue({
-          consumerId: "test_12345",
-          masterQueue: run.masterQueue,
-          maxRunCount: 10,
-        });
-
-        //create an attempt
-        const attemptResult = await engine.startRunAttempt({
-          runId: dequeued[0].run.id,
-          snapshotId: dequeued[0].snapshot.id,
-        });
-
-        //fail the attempt
-        const error = {
-          type: "BUILT_IN_ERROR" as const,
-          name: "UserError",
-          message: "This is a user error",
-          stackTrace: "Error: This is a user error\n    at <anonymous>:1:1",
-        };
-        const result = await engine.completeRunAttempt({
-          runId: dequeued[0].run.id,
-          snapshotId: attemptResult.snapshot.id,
-          completion: {
-            ok: false,
-            id: dequeued[0].run.id,
-            error,
-            retry: {
-              timestamp: Date.now(),
-              delay: 0,
-            },
-          },
-        });
-        expect(result.attemptStatus).toBe("RETRY_IMMEDIATELY");
-        expect(result.snapshot.executionStatus).toBe("PENDING_EXECUTING");
-        expect(result.run.status).toBe("RETRYING_AFTER_FAILURE");
-
-        //state should be completed
-        const executionData3 = await engine.getRunExecutionData({ runId: run.id });
-        assertNonNullable(executionData3);
-        expect(executionData3.snapshot.executionStatus).toBe("PENDING_EXECUTING");
-        //only when the new attempt is created, should the attempt be increased
-        expect(executionData3.run.attemptNumber).toBe(1);
-        expect(executionData3.run.status).toBe("RETRYING_AFTER_FAILURE");
-
-        //create a second attempt
-        const attemptResult2 = await engine.startRunAttempt({
-          runId: dequeued[0].run.id,
-          snapshotId: executionData3.snapshot.id,
-        });
-        expect(attemptResult2.run.attemptNumber).toBe(2);
-
-        //now complete it successfully
-        const result2 = await engine.completeRunAttempt({
-          runId: dequeued[0].run.id,
-          snapshotId: attemptResult2.snapshot.id,
-          completion: {
-            ok: true,
-            id: dequeued[0].run.id,
-            output: `{"foo":"bar"}`,
-            outputType: "application/json",
-          },
-        });
-        expect(result2.snapshot.executionStatus).toBe("FINISHED");
-        expect(result2.run.attemptNumber).toBe(2);
-        expect(result2.run.status).toBe("COMPLETED_SUCCESSFULLY");
-
-        //waitpoint should have been completed, with the output
-        const runWaitpointAfter = await prisma.waitpoint.findMany({
-          where: {
-            completedByTaskRunId: run.id,
-          },
-        });
-        expect(runWaitpointAfter.length).toBe(1);
-        expect(runWaitpointAfter[0].type).toBe("RUN");
-        expect(runWaitpointAfter[0].output).toBe(`{"foo":"bar"}`);
-        expect(runWaitpointAfter[0].outputIsError).toBe(false);
-
-        //state should be completed
-        const executionData4 = await engine.getRunExecutionData({ runId: run.id });
-        assertNonNullable(executionData4);
-        expect(executionData4.snapshot.executionStatus).toBe("FINISHED");
-        expect(executionData4.run.attemptNumber).toBe(2);
-        expect(executionData4.run.status).toBe("COMPLETED_SUCCESSFULLY");
-      } finally {
-        engine.quit();
-      }
-    }
-  );
 });
diff --git a/internal-packages/testcontainers/src/setup.ts b/internal-packages/testcontainers/src/setup.ts
index f79f2f1ef7d..ffd9d86e1b0 100644
--- a/internal-packages/testcontainers/src/setup.ts
+++ b/internal-packages/testcontainers/src/setup.ts
@@ -68,7 +68,8 @@ export async function setupBackgroundWorker(
   prisma: PrismaClient,
   environment: AuthenticatedEnvironment,
   taskIdentifier: string | string[],
-  machineConfig?: MachineConfig
+  machineConfig?: MachineConfig,
+  retryOptions?: RetryOptions
 ) {
   const worker = await prisma.backgroundWorker.create({
     data: {
@@ -86,7 +87,7 @@ export async function setupBackgroundWorker(
   const tasks: BackgroundWorkerTask[] = [];
 
   for (const identifier of taskIdentifiers) {
-    const retryConfig: RetryOptions = {
+    const retryConfig: RetryOptions = retryOptions ?? {
       maxAttempts: 3,
       factor: 1,
       minTimeoutInMs: 100,
diff --git a/packages/cli-v3/src/executions/taskRunProcess.ts b/packages/cli-v3/src/executions/taskRunProcess.ts
index 59643b5db2d..052da5bce9b 100644
--- a/packages/cli-v3/src/executions/taskRunProcess.ts
+++ b/packages/cli-v3/src/executions/taskRunProcess.ts
@@ -19,6 +19,7 @@ import { ChildProcess, fork } from "node:child_process";
 import { chalkError, chalkGrey, chalkRun, prettyPrintDate } from "../utilities/cliOutput.js";
 
 import { execOptionsForRuntime, execPathForRuntime } from "@trigger.dev/core/v3/build";
+import { nodeOptionsWithMaxOldSpaceSize } from "@trigger.dev/core/v3/machines";
 import { InferSocketMessageSchema } from "@trigger.dev/core/v3/zodSocket";
 import { logger } from "../utilities/logger.js";
 import {
@@ -117,14 +118,16 @@ export class TaskRunProcess {
   }
 
   async initialize() {
-    const { env: $env, workerManifest, cwd, messageId } = this.options;
+    const { env: $env, workerManifest, cwd, messageId, payload } = this.options;
+
+    const maxOldSpaceSize = nodeOptionsWithMaxOldSpaceSize(undefined, payload.execution.machine);
 
     const fullEnv = {
       ...(this.isTest ? { TRIGGER_LOG_LEVEL: "debug" } : {}),
       ...$env,
       OTEL_IMPORT_HOOK_INCLUDES: workerManifest.otelImportHook?.include?.join(","),
       // TODO: this will probably need to use something different for bun (maybe --preload?)
-      NODE_OPTIONS: execOptionsForRuntime(workerManifest.runtime, workerManifest),
+      NODE_OPTIONS: execOptionsForRuntime(workerManifest.runtime, workerManifest, maxOldSpaceSize),
       PATH: process.env.PATH,
       TRIGGER_PROCESS_FORK_START_TIME: String(Date.now()),
     };
diff --git a/packages/core/package.json b/packages/core/package.json
index 31900084f84..d3f2d38f2ae 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -59,6 +59,7 @@
       "./v3/workers": "./src/v3/workers/index.ts",
       "./v3/schemas": "./src/v3/schemas/index.ts",
       "./v3/runEngineWorker": "./src/v3/runEngineWorker/index.ts",
+      "./v3/machines": "./src/v3/machines/index.ts",
       "./v3/serverOnly": "./src/v3/serverOnly/index.ts"
     },
     "sourceDialects": [
@@ -178,6 +179,9 @@
       "v3/runEngineWorker": [
         "dist/commonjs/v3/runEngineWorker/index.d.ts"
       ],
+      "v3/machines": [
+        "dist/commonjs/v3/machines/index.d.ts"
+      ],
       "v3/serverOnly": [
         "dist/commonjs/v3/serverOnly/index.d.ts"
       ]
@@ -663,6 +667,17 @@
         "default": "./dist/commonjs/v3/runEngineWorker/index.js"
       }
     },
+    "./v3/machines": {
+      "import": {
+        "@triggerdotdev/source": "./src/v3/machines/index.ts",
+        "types": "./dist/esm/v3/machines/index.d.ts",
+        "default": "./dist/esm/v3/machines/index.js"
+      },
+      "require": {
+        "types": "./dist/commonjs/v3/machines/index.d.ts",
+        "default": "./dist/commonjs/v3/machines/index.js"
+      }
+    },
     "./v3/serverOnly": {
       "import": {
         "@triggerdotdev/source": "./src/v3/serverOnly/index.ts",
diff --git a/packages/core/src/v3/build/flags.test.ts b/packages/core/src/v3/build/flags.test.ts
new file mode 100644
index 00000000000..aaa3149bea7
--- /dev/null
+++ b/packages/core/src/v3/build/flags.test.ts
@@ -0,0 +1,53 @@
+import { describe, it, expect } from "vitest";
+import { dedupFlags } from "./flags.js";
+
+describe("dedupFlags", () => {
+  it("should keep single flags unchanged", () => {
+    expect(dedupFlags("--verbose")).toBe("--verbose");
+    expect(dedupFlags("-v")).toBe("-v");
+    expect(dedupFlags("--debug=true")).toBe("--debug=true");
+  });
+
+  it("should preserve multiple different flags", () => {
+    expect(dedupFlags("--quiet --verbose")).toBe("--quiet --verbose");
+    expect(dedupFlags("-v -q --log=debug")).toBe("-v -q --log=debug");
+  });
+
+  it("should use last value when flags are duplicated", () => {
+    expect(dedupFlags("--debug=false --debug=true")).toBe("--debug=true");
+    expect(dedupFlags("--log=info --log=warn --log=error")).toBe("--log=error");
+  });
+
+  it("should handle mix of flags with and without values", () => {
+    expect(dedupFlags("--debug=false -v --debug=true")).toBe("-v --debug=true");
+    expect(dedupFlags("-v --quiet -v")).toBe("--quiet -v");
+  });
+
+  // Edge cases
+  it("should handle empty string", () => {
+    expect(dedupFlags("")).toBe("");
+  });
+
+  it("should handle multiple spaces between flags", () => {
+    expect(dedupFlags("--debug=false    --verbose   --debug=true")).toBe("--verbose --debug=true");
+  });
+
+  it("should handle flags with empty values", () => {
+    expect(dedupFlags("--path= --path=foo")).toBe("--path=foo");
+    expect(dedupFlags("--path=foo --path=")).toBe("--path=");
+  });
+
+  it("should preserve values containing equals signs", () => {
+    expect(dedupFlags("--url=http://example.com?foo=bar")).toBe("--url=http://example.com?foo=bar");
+  });
+
+  it("should handle flags with special characters", () => {
+    expect(dedupFlags("--path=/usr/local --path=/home/user")).toBe("--path=/home/user");
+    expect(dedupFlags('--name="John Doe" --name="Jane Doe"')).toBe('--name="Jane Doe"');
+  });
+
+  it("should handle multiple flag variants", () => {
+    const input = "--env=dev -v --env=prod --quiet -v --env=staging";
+    expect(dedupFlags(input)).toBe("--quiet -v --env=staging");
+  });
+});
diff --git a/packages/core/src/v3/build/flags.ts b/packages/core/src/v3/build/flags.ts
new file mode 100644
index 00000000000..88448fa7304
--- /dev/null
+++ b/packages/core/src/v3/build/flags.ts
@@ -0,0 +1,47 @@
+/**
+ * Deduplicates command line flags by keeping only the last occurrence of each flag.
+ * Preserves the order of the last occurrence of each flag.
+ *
+ * @param flags - A space-separated string of command line flags
+ * @returns A space-separated string of deduplicated flags
+ *
+ * @example
+ * // Single flags are preserved
+ * dedupFlags("--quiet --verbose") // returns "--quiet --verbose"
+ *
+ * @example
+ * // For duplicate flags, the last value wins and maintains its position
+ * dedupFlags("--debug=false --log=info --debug=true") // returns "--log=info --debug=true"
+ *
+ * @example
+ * // Mixing flags with and without values
+ * dedupFlags("-v --log=debug -v") // returns "--log=debug -v"
+ */
+export function dedupFlags(flags: string): string {
+  const seen = new Set<string>();
+  const result: [string, string | boolean][] = [];
+
+  const pairs = flags
+    .split(" ")
+    .filter(Boolean) // Remove empty strings from multiple spaces
+    .map((flag): [string, string | boolean] => {
+      const equalIndex = flag.indexOf("=");
+      if (equalIndex !== -1) {
+        const key = flag.slice(0, equalIndex);
+        const value = flag.slice(equalIndex + 1);
+        return [key, value];
+      } else {
+        return [flag, true];
+      }
+    });
+
+  // Process in reverse to keep last occurrence
+  for (const [key, value] of pairs.reverse()) {
+    if (!seen.has(key)) {
+      seen.add(key);
+      result.unshift([key, value]);
+    }
+  }
+
+  return result.map(([key, value]) => (value === true ? key : `${key}=${value}`)).join(" ");
+}
diff --git a/packages/core/src/v3/build/index.ts b/packages/core/src/v3/build/index.ts
index 38245d4798d..c29a82da2ef 100644
--- a/packages/core/src/v3/build/index.ts
+++ b/packages/core/src/v3/build/index.ts
@@ -2,3 +2,4 @@ export * from "./extensions.js";
 export * from "./resolvedConfig.js";
 export * from "./runtime.js";
 export * from "./externals.js";
+export * from "./flags.js";
diff --git a/packages/core/src/v3/build/runtime.ts b/packages/core/src/v3/build/runtime.ts
index 94b3fac08c6..dc8ce19c6c3 100644
--- a/packages/core/src/v3/build/runtime.ts
+++ b/packages/core/src/v3/build/runtime.ts
@@ -1,6 +1,7 @@
 import { join } from "node:path";
 import { pathToFileURL } from "url";
 import { BuildRuntime } from "../schemas/build.js";
+import { dedupFlags } from "./flags.js";
 
 export const DEFAULT_RUNTIME = "node" satisfies BuildRuntime;
 
@@ -41,7 +42,11 @@ export type ExecOptions = {
   customConditions?: string[];
 };
 
-export function execOptionsForRuntime(runtime: BuildRuntime, options: ExecOptions): string {
+export function execOptionsForRuntime(
+  runtime: BuildRuntime,
+  options: ExecOptions,
+  additionalNodeOptions?: string
+): string {
   switch (runtime) {
     case "node":
     case "node-22": {
@@ -51,15 +56,19 @@ export function execOptionsForRuntime(runtime: BuildRuntime, options: ExecOption
 
       const conditions = options.customConditions?.map((condition) => `--conditions=${condition}`);
 
-      return [
+      //later flags will win (after the dedupe)
+      const flags = [
+        process.env.NODE_OPTIONS,
+        additionalNodeOptions,
         importEntryPoint,
         conditions,
-        process.env.NODE_OPTIONS,
         nodeRuntimeNeedsGlobalWebCryptoFlag() ? "--experimental-global-webcrypto" : undefined,
       ]
         .filter(Boolean)
         .flat()
         .join(" ");
+
+      return dedupFlags(flags);
     }
     case "bun": {
       return "";
diff --git a/packages/core/src/v3/errors.ts b/packages/core/src/v3/errors.ts
index 88c089025f9..a8e789f81d6 100644
--- a/packages/core/src/v3/errors.ts
+++ b/packages/core/src/v3/errors.ts
@@ -76,6 +76,58 @@ export function isManualOutOfMemoryError(error: TaskRunError) {
   return false;
 }
 
+export function isOOMRunError(error: TaskRunError) {
+  if (error.type === "INTERNAL_ERROR") {
+    if (
+      error.code === "TASK_PROCESS_OOM_KILLED" ||
+      error.code === "TASK_PROCESS_MAYBE_OOM_KILLED"
+    ) {
+      return true;
+    }
+
+    // For the purposes of retrying on a larger machine, we're going to treat this is an OOM error.
+    // This is what they look like if we're executing using k8s. They then get corrected later, but it's too late.
+    // {"code": "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE", "type": "INTERNAL_ERROR", "message": "Process exited with code -1 after signal SIGKILL."}
+    if (
+      error.code === "TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE" &&
+      error.message &&
+      error.message.includes("-1")
+    ) {
+      if (error.message.includes("SIGKILL")) {
+        return true;
+      }
+
+      if (error.message.includes("SIGABRT") && error.stackTrace) {
+        const oomIndicators = [
+          "JavaScript heap out of memory",
+          "Reached heap limit",
+          "FATAL ERROR: Reached heap limit Allocation failed",
+        ];
+
+        if (oomIndicators.some((indicator) => error.stackTrace!.includes(indicator))) {
+          return true;
+        }
+      }
+    }
+  }
+
+  if (error.type === "BUILT_IN_ERROR") {
+    // ffmpeg also does weird stuff
+    // { "name": "Error", "type": "BUILT_IN_ERROR", "message": "ffmpeg was killed with signal SIGKILL" }
+    if (error.message && error.message.includes("ffmpeg was killed with signal SIGKILL")) {
+      return true;
+    }
+  }
+
+  // Special `OutOfMemoryError` for doing a manual OOM kill.
+  // Useful if a native library does an OOM but doesn't actually crash the run and you want to manually
+  if (isManualOutOfMemoryError(error)) {
+    return true;
+  }
+
+  return false;
+}
+
 export class TaskPayloadParsedError extends Error {
   public readonly cause: unknown;
 
@@ -562,6 +614,8 @@ const findSignalInMessage = (message?: string, truncateLength = 100) => {
     return "SIGSEGV";
   } else if (trunc.includes("SIGKILL")) {
     return "SIGKILL";
+  } else if (trunc.includes("SIGABRT")) {
+    return "SIGABRT";
   } else {
     return;
   }
@@ -587,6 +641,10 @@ export function taskRunErrorEnhancer(error: TaskRunError): EnhanceError<TaskRunE
               return {
                 ...getPrettyTaskRunError("TASK_PROCESS_MAYBE_OOM_KILLED"),
               };
+            case "SIGABRT":
+              return {
+                ...getPrettyTaskRunError("TASK_PROCESS_MAYBE_OOM_KILLED"),
+              };
             default:
               return {
                 ...getPrettyTaskRunError("TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE"),
@@ -636,6 +694,10 @@ export function taskRunErrorEnhancer(error: TaskRunError): EnhanceError<TaskRunE
             return {
               ...getPrettyTaskRunError("TASK_PROCESS_MAYBE_OOM_KILLED"),
             };
+          case "SIGABRT":
+            return {
+              ...getPrettyTaskRunError("TASK_PROCESS_MAYBE_OOM_KILLED"),
+            };
           default: {
             return {
               ...getPrettyTaskRunError("TASK_PROCESS_EXITED_WITH_NON_ZERO_CODE"),
@@ -689,6 +751,11 @@ export function exceptionEventEnhancer(
               ...exception,
               ...getPrettyExceptionEvent("TASK_PROCESS_MAYBE_OOM_KILLED"),
             };
+          case "SIGABRT":
+            return {
+              ...exception,
+              ...getPrettyExceptionEvent("TASK_PROCESS_MAYBE_OOM_KILLED"),
+            };
           default:
             return exception;
         }
diff --git a/packages/core/src/v3/machines/index.ts b/packages/core/src/v3/machines/index.ts
new file mode 100644
index 00000000000..771f4345a44
--- /dev/null
+++ b/packages/core/src/v3/machines/index.ts
@@ -0,0 +1,57 @@
+import { MachinePreset } from "../schemas/common.js";
+
+/**
+ * Returns a value to be used for `--max-old-space-size`. It is in MiB.
+ * Setting this correctly means V8 spends more times running Garbage Collection (GC).
+ * It won't eliminate crashes but it will help avoid them.
+ * @param {MachinePreset} machine - The machine preset configuration containing memory specifications
+ * @param {number} [overhead=0.2] - The memory overhead factor (0.2 = 20% reserved for system operations)
+ * @returns {number} The calculated max old space size in MiB
+ */
+export function maxOldSpaceSizeForMachine(machine: MachinePreset, overhead: number = 0.2): number {
+  return Math.round(machine.memory * 1_024 * (1 - overhead));
+}
+
+/**
+ * Returns a flag to be used for `--max-old-space-size`. It is in MiB.
+ * Setting this correctly means V8 spends more times running Garbage Collection (GC).
+ * It won't eliminate crashes but it will help avoid them.
+ * @param {MachinePreset} machine - The machine preset configuration containing memory specifications
+ * @param {number} [overhead=0.2] - The memory overhead factor (0.2 = 20% reserved for system operations)
+ * @returns {string} The calculated max old space size flag
+ */
+export function maxOldSpaceSizeFlag(machine: MachinePreset, overhead: number = 0.2): string {
+  return `--max-old-space-size=${maxOldSpaceSizeForMachine(machine, overhead)}`;
+}
+
+/**
+ * Takes the existing NODE_OPTIONS value, removes any existing max-old-space-size flag, and adds a new one.
+ * @param {string | undefined} existingOptions - The existing NODE_OPTIONS value
+ * @param {MachinePreset} machine - The machine preset configuration containing memory specifications
+ * @param {number} [overhead=0.2] - The memory overhead factor (0.2 = 20% reserved for system operations)
+ * @returns {string} The updated NODE_OPTIONS value with the new max-old-space-size flag
+ */
+export function nodeOptionsWithMaxOldSpaceSize(
+  existingOptions: string | undefined,
+  machine: MachinePreset,
+  overhead: number = 0.2
+): string {
+  let options = existingOptions ?? "";
+
+  //remove existing max-old-space-size flag
+  options = options.replace(/--max-old-space-size=\d+/g, "").trim();
+
+  //get max-old-space-size flag
+  const flag = maxOldSpaceSizeFlag(machine, overhead);
+
+  return normalizeCommandLineFlags(options ? `${options} ${flag}` : flag);
+}
+
+/**
+ * Normalizes spaces in a string of command line flags, ensuring single spaces between flags
+ * @param {string} input - The string to normalize
+ * @returns {string} The normalized string with single spaces between flags
+ */
+function normalizeCommandLineFlags(input: string): string {
+  return input.split(/\s+/).filter(Boolean).join(" ");
+}
diff --git a/packages/core/src/v3/machines/max-old-space.test.ts b/packages/core/src/v3/machines/max-old-space.test.ts
new file mode 100644
index 00000000000..425b8197624
--- /dev/null
+++ b/packages/core/src/v3/machines/max-old-space.test.ts
@@ -0,0 +1,57 @@
+import { describe, it, expect } from "vitest";
+import { nodeOptionsWithMaxOldSpaceSize } from "./index.js";
+import { MachinePreset } from "../schemas/common.js";
+
+describe("nodeOptionsWithMaxOldSpaceSize", () => {
+  const testMachine: MachinePreset = {
+    name: "small-2x",
+    memory: 1, // 1GB = 1024 MiB
+    cpu: 1,
+    centsPerMs: 0,
+  };
+
+  // With default 0.2 overhead, max-old-space-size should be 819 (1024 * 0.8)
+  const expectedFlag = "--max-old-space-size=819";
+
+  it("handles undefined NODE_OPTIONS", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize(undefined, testMachine);
+    expect(result).toBe(expectedFlag);
+  });
+
+  it("handles empty string NODE_OPTIONS", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize("", testMachine);
+    expect(result).toBe(expectedFlag);
+  });
+
+  it("preserves existing flags while adding max-old-space-size", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize("--inspect --trace-warnings", testMachine);
+    expect(result).toBe(`--inspect --trace-warnings ${expectedFlag}`);
+  });
+
+  it("replaces existing max-old-space-size flag", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize(
+      "--max-old-space-size=4096 --inspect",
+      testMachine
+    );
+    expect(result).toBe(`--inspect ${expectedFlag}`);
+  });
+
+  it("handles multiple existing max-old-space-size flags", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize(
+      "--max-old-space-size=4096 --inspect --max-old-space-size=8192",
+      testMachine
+    );
+    expect(result).toBe(`--inspect ${expectedFlag}`);
+  });
+
+  it("handles extra spaces between flags", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize("--inspect    --trace-warnings", testMachine);
+    expect(result).toBe(`--inspect --trace-warnings ${expectedFlag}`);
+  });
+
+  it("uses custom overhead value", () => {
+    const result = nodeOptionsWithMaxOldSpaceSize("--inspect", testMachine, 0.5);
+    // With 0.5 overhead, max-old-space-size should be 512 (1024 * 0.5)
+    expect(result).toBe("--inspect --max-old-space-size=512");
+  });
+});
diff --git a/packages/core/src/v3/schemas/common.ts b/packages/core/src/v3/schemas/common.ts
index cf82719f76d..fbfd4e97e7f 100644
--- a/packages/core/src/v3/schemas/common.ts
+++ b/packages/core/src/v3/schemas/common.ts
@@ -308,7 +308,7 @@ export const TaskRunExecution = z.object({
   organization: TaskRunExecutionOrganization,
   project: TaskRunExecutionProject,
   batch: TaskRunExecutionBatch.optional(),
-  machine: MachinePreset.optional(),
+  machine: MachinePreset,
 });
 
 export type TaskRunExecution = z.infer<typeof TaskRunExecution>;
diff --git a/packages/core/vitest.config.ts b/packages/core/vitest.config.ts
index 758c27056e8..8db9c42ae0c 100644
--- a/packages/core/vitest.config.ts
+++ b/packages/core/vitest.config.ts
@@ -2,7 +2,7 @@ import { defineConfig } from "vitest/config";
 
 export default defineConfig({
   test: {
-    include: ["test/**/*.test.ts"],
+    include: ["test/**/*.test.ts", "src/v3/**/*.test.ts"],
     globals: true,
   },
 });


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 2**

<details>
<summary>🧹 Nitpick comments (19)</summary><blockquote>

<details>
<summary>.zed/tasks.json (1)</summary><blockquote>

`1-45`: **Well-structured task configuration for building packages**

This task configuration is well-defined and provides a clear build command for Trigger.dev packages. The configuration includes detailed comments explaining each option, which is helpful for maintainability.


One minor observation: You have an environment variable `foo: bar` defined on line 7, which appears to be a placeholder. Consider replacing it with actual environment variables needed for the build process if any are required.

</blockquote></details>
<details>
<summary>packages/core/src/v3/errors.ts (1)</summary><blockquote>

`117-118`: **Consider using optional chaining for error message check**

The static analysis tool suggests using optional chaining for the error message property check.


```diff
-    if (error.message && error.message.includes("ffmpeg was killed with signal SIGKILL")) {
+    if (error.message?.includes("ffmpeg was killed with signal SIGKILL")) {
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Biome (1.9.4)</summary>

[error] 117-117: Change to an optional chain.

Unsafe fix: Change to an optional chain.


(lint/complexity/useOptionalChain)

</details>

</details>

</blockquote></details>
<details>
<summary>packages/core/src/v3/build/runtime.ts (1)</summary><blockquote>

`59-72`: **Validate flag ordering and deduplication**  
Combining `process.env.NODE_OPTIONS` with `additionalNodeOptions` and then deduplicating is sensible. However, consider edge cases where flags conflict between the environment variable and manual additions. Ensure that critical flags aren’t inadvertently removed or overridden.

</blockquote></details>
<details>
<summary>internal-packages/testcontainers/src/setup.ts (1)</summary><blockquote>

`90-96`: **Reasonable default retry strategy**  
Providing a fallback `retryConfig` object when `retryOptions` is not passed helps maintain reliable behavior. You may consider logging or surfacing these defaults for debugging when diagnosing retry failures.

</blockquote></details>
<details>
<summary>internal-packages/run-engine/src/engine/index.ts (2)</summary><blockquote>

`2920-2934`: **Concise handling of “cancel_run” case**  
On “cancel_run,” you finalize the run immediately. This is straightforward, but watch out for states that might slip through if the run is partially updated in a different thread.  

Do you need further validation scripts to confirm that no concurrency collisions occur during cancellation?

---

`2946-2957`: **Dedicated branch for permanent failure**  
This branch calls `#permanentlyFailRun` and returns. Logic is clean and cohesive, but might be streamlined if “cancel_run” and “fail_run” share repeated code.

</blockquote></details>
<details>
<summary>packages/core/src/v3/machines/index.ts (2)</summary><blockquote>

`11-13`: **Validate overhead range for safer usage.**  
Consider adding safeguards for unexpected overhead values (e.g., negative or >1) which could lead to zero or negative results for `--max-old-space-size`.  

```diff
 export function maxOldSpaceSizeForMachine(machine: MachinePreset, overhead: number = 0.2): number {
-  return Math.round(machine.memory * 1_024 * (1 - overhead));
+  const safeOverhead = Math.min(Math.max(overhead, 0), 0.9);
+  return Math.max(
+    1,
+    Math.round(machine.memory * 1_024 * (1 - safeOverhead))
+  );
 }
```

---

`34-48`: **Consider additional checks for multiple or invalid overhead flags.**  
The function works well for typical usage. Optionally, you could add extra validation to warn if overhead might exceed expected bounds.

</blockquote></details>
<details>
<summary>internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts (3)</summary><blockquote>

`13-169`: **Additional partial success test scenario may help.**  
This test suite thoroughly covers user error retry scenarios. Consider adding a partial success scenario or a scenario with incremental error resolution for comprehensive coverage.

---

`395-501`: **Consider testing different memory configurations.**  
This OOM test is valuable; you could optionally expand coverage to include multiple memory presets to ensure consistent behavior.

---

`663-830`: **Additional scenario: partial up-scaling.**  
It might be insightful to test a scenario where memory is increased incrementally beyond two tiers, ensuring the logic scales repeatedly until the run either succeeds or fully crashes.

</blockquote></details>
<details>
<summary>packages/core/src/v3/machines/max-old-space.test.ts (2)</summary><blockquote>

`31-37`: **Correct replacement logic.**  
The test confirms only one `--max-old-space-size` is ultimately present.

---

`52-56`: **Tests custom overhead usage.**  
Optional improvement: test boundary overhead values (e.g., 0, 1, or invalid) to confirm resilience.

</blockquote></details>
<details>
<summary>internal-packages/run-engine/src/engine/retrying.ts (6)</summary><blockquote>

`13-13`: **Ensure clarity for relative import**  
The statement `import { ServiceValidationError } from ".";` can be confusing because `"."` usually references the current directory's `index` file. Consider explicitly referencing the correct file path for improved clarity and maintainability.

---

`15-21`: **Consider making `attemptNumber` mandatory or defaulted**  
`attemptNumber` is declared as `number | null`. In practice, you ensure correctness by defaulting it to 1 in the OOM logic. Alternatively, consider making this parameter always defined or handle the initialization more explicitly to avoid potential confusion.

---

`23-40`: **Review naming consistency**  
While the outcome strings such as `"cancel_run"`, `"fail_run"`, and `"retry"` are descriptive, consider aligning them further with your domain conventions (e.g., “canceled,” “failed,” “retrying”) for clarity. This is just a naming suggestion, but consistent domain terminology can make logs and error reporting more intuitive.

---

`53-73`: **Validate concurrency for OOM-based retry**  
When handling OOM errors, you query the database to determine if a bigger machine is available. In highly concurrent scenarios, another process might concurrently modify these values. If concurrency can occur, ensure that each task run is effectively locked or that you re-check machine availability post-update to avoid race conditions.

---

`92-99`: **Use `findUnique` for performance and clarity**  
When looking for a single record by primary key (`id` in this case), using `findUnique` instead of `findFirst` can be more semantically appropriate and might yield a minor performance benefit.

---

`122-176`: **Refine error handling strategy and logging**  
The `retryOOMOnMachine` function logs an error to the console if the database query fails or parsing fails. In production, consider a structured logging approach or a logger instead of `console.error`, especially if you’re relying on logs for debugging. This will help you capture context more effectively and manage error levels in a production environment.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between e297c7f7f0fdea952029353c8965281efdaad4ba and a5b6f0664da7f94d14d57ba8caca30ea5de94413.

</details>

<details>
<summary>📒 Files selected for processing (21)</summary>

* `.zed/tasks.json` (1 hunks)
* `apps/webapp/app/v3/services/alerts/deliverAlert.server.ts` (2 hunks)
* `apps/webapp/app/v3/services/completeAttempt.server.ts` (2 hunks)
* `internal-packages/run-engine/src/engine/index.ts` (2 hunks)
* `internal-packages/run-engine/src/engine/retrying.ts` (1 hunks)
* `internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts` (1 hunks)
* `internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts` (1 hunks)
* `internal-packages/run-engine/src/engine/tests/heartbeats.test.ts` (1 hunks)
* `internal-packages/run-engine/src/engine/tests/trigger.test.ts` (0 hunks)
* `internal-packages/testcontainers/src/setup.ts` (2 hunks)
* `packages/cli-v3/src/executions/taskRunProcess.ts` (2 hunks)
* `packages/core/package.json` (3 hunks)
* `packages/core/src/v3/build/flags.test.ts` (1 hunks)
* `packages/core/src/v3/build/flags.ts` (1 hunks)
* `packages/core/src/v3/build/index.ts` (1 hunks)
* `packages/core/src/v3/build/runtime.ts` (3 hunks)
* `packages/core/src/v3/errors.ts` (5 hunks)
* `packages/core/src/v3/machines/index.ts` (1 hunks)
* `packages/core/src/v3/machines/max-old-space.test.ts` (1 hunks)
* `packages/core/src/v3/schemas/common.ts` (1 hunks)
* `packages/core/vitest.config.ts` (1 hunks)

</details>

<details>
<summary>💤 Files with no reviewable changes (1)</summary>

* internal-packages/run-engine/src/engine/tests/trigger.test.ts

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 Biome (1.9.4)</summary>

<details>
<summary>packages/core/src/v3/errors.ts</summary>

[error] 117-117: Change to an optional chain.

Unsafe fix: Change to an optional chain.


(lint/complexity/useOptionalChain)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (7)</summary>

* GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
* GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
* GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
* GitHub Check: units / 🧪 Unit Tests
* GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
* GitHub Check: typecheck / typecheck
* GitHub Check: Analyze (javascript-typescript)

</details>

<details>
<summary>🔇 Additional comments (35)</summary><blockquote>

<details>
<summary>packages/core/src/v3/build/index.ts (1)</summary>

`5-5`: **Export addition looks good**

This change properly exposes the functionality from flags.js through the index file, making it accessible to consumers of this module.

</details>
<details>
<summary>packages/core/vitest.config.ts (1)</summary>

`5-5`: **Good addition to test configuration**

Adding the `src/v3/**/*.test.ts` pattern to the Vitest configuration ensures that tests located in the source directory will be executed alongside those in the dedicated test directory. This aligns with the approach of co-locating tests with their implementation in the v3 directory.

</details>
<details>
<summary>internal-packages/run-engine/src/engine/tests/heartbeats.test.ts (1)</summary>

`8-8`: **Fixed missing import**

Good catch adding the `describe` import, which was being used on line 12 but wasn't previously imported.

</details>
<details>
<summary>internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts (1)</summary>

`8-8`: **Fixed missing import**

Good catch adding the `describe` import, which was being used on line 13 but wasn't previously imported.

</details>
<details>
<summary>packages/core/src/v3/build/flags.ts (1)</summary>

`1-47`: **Well-implemented flag deduplication utility with good documentation**

The `dedupFlags` function is well-documented with clear examples and handles various flag formats correctly. The implementation efficiently processes the flags in reverse order to preserve the last occurrence of each flag.


The approach of using a set to track seen keys and unshifting to preserve order is clean and effective. The function handles both flags with values (`--key=value`) and boolean flags (`--verbose`) appropriately.

</details>
<details>
<summary>packages/core/src/v3/build/flags.test.ts (1)</summary>

`1-53`: **Comprehensive test suite for flag deduplication**

The test suite is thorough and covers a wide range of scenarios including:
- Simple flag handling
- Multiple flags
- Duplicate flags
- Mixed flag types
- Various edge cases

This comprehensive testing will help ensure the robustness of the flag handling, which is critical for proper memory management in the run engine.

</details>
<details>
<summary>apps/webapp/app/v3/services/alerts/deliverAlert.server.ts (2)</summary>

`16-16`: **Updated OOM error detection import**

The import has been updated to use the more specific `isOOMRunError` function from the core package instead of the previous `isOOMError` function.

---

`379-379`: **Improved OOM error detection**

The code now uses the more specific `isOOMRunError` function instead of the previous `isOOMError`. This provides better detection of out-of-memory conditions by checking for specific error patterns that indicate memory issues.

</details>
<details>
<summary>packages/cli-v3/src/executions/taskRunProcess.ts (3)</summary>

`22-22`: **Added machine-specific memory management import**

This new import brings in the function that handles calculating appropriate memory limits based on machine configurations.

---

`121-123`: **Enhanced memory management based on machine size**

Added code to extract the payload from options and calculate the appropriate Node.js memory limit based on the machine configuration specified in the execution.

---

`130-130`: **Applying dynamic memory limits to Node.js process**

The Node.js options now include the machine-specific memory limits calculated above, which helps prevent OOM errors by configuring the proper `--max-old-space-size` parameter.

</details>
<details>
<summary>apps/webapp/app/v3/services/completeAttempt.server.ts (2)</summary>

`14-14`: **Updated OOM error detection import**

The import has been updated to use the more specific `isOOMRunError` function from the core package instead of the previous `isOOMError` function.

---

`259-259`: **Improved OOM error detection**

The code now uses the more specific `isOOMRunError` function instead of the previous `isOOMError`. This provides better detection of out-of-memory conditions which is critical for the retry logic that may upgrade the machine size on OOM errors.

</details>
<details>
<summary>packages/core/src/v3/errors.ts (4)</summary>

`79-129`: **New comprehensive OOM error detection function**

This new `isOOMRunError` function provides a robust way to detect Out of Memory errors by checking for:

1. Specific OOM error codes
2. Error messages containing SIGKILL signals
3. SIGABRT signals with OOM-related stack traces
4. ffmpeg killed with SIGKILL (common in memory-intensive operations)
5. Manual OOM kills

The enhanced detection capabilities will improve the system's ability to identify and respond to memory issues appropriately.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Biome (1.9.4)</summary>

[error] 117-117: Change to an optional chain.

Unsafe fix: Change to an optional chain.


(lint/complexity/useOptionalChain)

</details>

</details>

---

`617-618`: **Added SIGABRT signal detection**

SIGABRT signals can also indicate memory issues, especially when combined with specific stack traces. Adding this check improves error detection capabilities.

---

`644-647`: **Enhanced error handling for SIGABRT signals**

SIGABRT signals are now properly categorized as potential OOM errors in the error enhancer, ensuring consistent error handling throughout the codebase.

---

`754-758`: **Extended exception handling for SIGABRT signals**

Similar to the changes above, SIGABRT signals are now properly handled in the exception event enhancer, ensuring consistent error categorization across different error handling paths.

</details>
<details>
<summary>packages/core/src/v3/build/runtime.ts (2)</summary>

`4-4`: **New import is consistent with code usage**  
The newly introduced `dedupFlags` import is properly referenced below. No issues found.

---

`45-49`: **Ensure optional parameter usage is well-tested**  
The function signature now optionally accepts `additionalNodeOptions`; confirm that all callers either supply the argument or handle its absence. Unit tests would help prevent runtime errors (e.g., passing an unexpected type).

</details>
<details>
<summary>internal-packages/testcontainers/src/setup.ts (1)</summary>

`71-73`: **Optional retry configurations**  
The addition of `retryOptions?: RetryOptions` after `machineConfig` is a practical extension. Be sure to handle scenarios where `machineConfig` is provided but `retryOptions` remains undefined, and vice versa.

</details>
<details>
<summary>packages/core/package.json (2)</summary>

`62-62`: **New export path acknowledged**  
Exporting `./v3/machines` aligns with existing patterns. No issues spotted here.

---

`681-690`: **Additional export mapping**  
Paths for both ESM and CommonJS under `./v3/machines` appear consistent. Ensuring these entries get tested in various bundler workflows (e.g., webpack, TS path resolution) would be beneficial.

</details>
<details>
<summary>internal-packages/run-engine/src/engine/index.ts (3)</summary>

`74-74`: **New import for retry outcome logic**  
Importing `retryOutcomeFromCompletion` clarifies how retries are centrally handled. Ensure versioning or compatibility issues don’t arise if this method evolves.

---

`2871-2898`: **Centralized retry logic and minimal run retrieval**  
Calling `retryOutcomeFromCompletion` is a clear way to consolidate retry handling, and fetching a “minimalRun” object for event emission is a clean approach. Verify that partial updates or concurrency locks don’t block subsequent changes.

---

`2971-3004`: **OOM handling and scheduled retries**  
The logic for detecting OOM errors and scheduling an immediate or delayed retry seems robust. Ensure that repeated OOMs eventually converge on a final failure to avoid infinite loops.

</details>
<details>
<summary>packages/core/src/v3/machines/index.ts (2)</summary>

`23-25`: **Looks solid.**  
Implementation is straightforward and aligns with the function description.

---

`55-57`: **Helper function is concise.**  
Implementation is correct and ensures single spacing.

</details>
<details>
<summary>internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts (3)</summary>

`171-280`: **All looks good.**  
The test for no remaining retries is well-structured and verifies expected status transitions.

---

`282-394`: **Robust test coverage.**  
Covers unretriable errors thoroughly; no major concerns noted.

---

`502-661`: **Thorough coverage of OOM retry logic.**  
The test effectively verifies scaling up to a larger machine after failing.

</details>
<details>
<summary>packages/core/src/v3/machines/max-old-space.test.ts (5)</summary>

`16-19`: **Handles undefined input well.**  
Ensures defaults are applied when `NODE_OPTIONS` is absent.

---

`21-24`: **Correct fallback for empty string.**  
Verifies a typical edge case successfully.

---

`26-29`: **Good test for preserving flags.**  
Ensures existing flags are retained and new flags are appended.

---

`39-45`: **Multiple flags scenario is well-covered.**  
Demonstrates robust handling of duplicates.

---

`47-50`: **Extra spaces are normalized.**  
Properly validated spacing logic.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T19:32:59Z
user: login: ericallam
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-07T09:07:22Z

Review comments:
diff_hunk: @@ -308,7 +308,7 @@ export const TaskRunExecution = z.object({
   organization: TaskRunExecutionOrganization,
   project: TaskRunExecutionProject,
   batch: TaskRunExecutionBatch.optional(),
-  machine: MachinePreset.optional(),
+  machine: MachinePreset,
path: packages/core/src/v3/schemas/common.ts
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>🧩 Analysis chain</summary>

**Breaking schema change: `machine` property now required**

You've changed the `machine` property in the `TaskRunExecution` schema from optional to required. This is a breaking change that aligns with the PR's focus on improving memory management and OOM retrying.

Make sure all instances of `TaskRunExecution` creation in the codebase include the `machine` property to avoid validation errors.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if there are any places creating TaskRunExecution objects that might be missing the machine property
rg -A 3 -B 3 "TaskRunExecution" --type typescript | grep -v "machine:"
```

Length of output: 103

---

Below is an updated shell script to re-run the verification using a filename filter instead of the unsupported `--type typescript`. Please run this script to check for any instances where the `TaskRunExecution` object might be created without the required `machine` property:



**Breaking Change: Mandatory 'machine' Property for TaskRunExecution Schema**

- The `machine` property in the `TaskRunExecution` schema (see `packages/core/src/v3/schemas/common.ts`, line 311) is now required.
- This is a breaking change meant to improve memory management and OOM retrying.
- **Attention:** Ensure that every instance of creating a `TaskRunExecution` object throughout the codebase includes the `machine` property to avoid validation errors.

Please re-run the updated script above or manually verify the affected parts of the codebase to confirm that all `TaskRunExecution` creations provide the `machine` property.



---

🏁 Script executed:

```shell

#!/bin/bash
# Re-run the search for TaskRunExecution usages in TS files,
# filtering on the "*.ts" file extension.
rg -A 3 -B 3 "TaskRunExecution" -g "*.ts" | grep -v "machine:"
```

Length of output: 65719

---



</details>

**Action Required: Update TaskRunExecution creations with the required `machine` property**

The schema in **packages/core/src/v3/schemas/common.ts** now mandates that every TaskRunExecution object includes a valid `machine` property. Our re-run search across the codebase revealed many references where TaskRunExecution objects are constructed without an obvious assignment for `machine`. For example, in:

- **apps/webapp/app/v3/services/createTaskRunAttempt.server.ts** – the object literal assigned to `const execution: TaskRunExecution = { ... }` does not appear to include a `machine` property.
- **apps/webapp/app/v3/services/completeAttempt.server.ts** and **apps/webapp/app/v3/services/resumeAttempt.server.ts** – similar patterns are visible where TaskRunExecution is referenced without the required `machine` field.

Please ensure that all locations where a TaskRunExecution object is created are updated to include the `machine` property (with a valid MachinePreset value) to prevent validation errors and potential runtime issues.

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-06T19:32:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,175 @@
+import {
+  isOOMRunError,
+  RetryOptions,
+  shouldRetryError,
+  TaskRunError,
+  TaskRunExecutionRetry,
+  taskRunErrorEnhancer,
+  sanitizeError,
+  calculateNextRetryDelay,
+} from "@trigger.dev/core/v3";
+import { PrismaClientOrTransaction, TaskRunStatus } from "@trigger.dev/database";
+import { MAX_TASK_RUN_ATTEMPTS } from "./consts";
+import { ServiceValidationError } from ".";
+
+type Params = {
+  runId: string;
+  attemptNumber: number | null;
+  error: TaskRunError;
+  retryUsingQueue: boolean;
+  retrySettings: TaskRunExecutionRetry | undefined;
+};
+
+export type RetryOutcome =
+  | {
+      outcome: "cancel_run";
+      reason?: string;
+    }
+  | {
+      outcome: "fail_run";
+      sanitizedError: TaskRunError;
+      wasOOMError?: boolean;
+    }
+  | {
+      outcome: "retry";
+      method: "queue" | "immediate";
+      settings: TaskRunExecutionRetry;
+      machine?: string;
+      wasOOMError?: boolean;
+    };
+
+export async function retryOutcomeFromCompletion(
+  prisma: PrismaClientOrTransaction,
+  { runId, attemptNumber, error, retryUsingQueue, retrySettings }: Params
+): Promise<RetryOutcome> {
+  // Canceled
+  if (error.type === "INTERNAL_ERROR" && error.code === "TASK_RUN_CANCELLED") {
+    return { outcome: "cancel_run", reason: error.message };
+  }
+
+  const sanitizedError = sanitizeError(error);
+
+  // OOM error (retry on a larger machine or fail)
+  if (isOOMRunError(error)) {
+    const oomResult = await retryOOMOnMachine(prisma, runId);
+    if (!oomResult) {
+      return { outcome: "fail_run", sanitizedError, wasOOMError: true };
+    }
+
+    const delay = calculateNextRetryDelay(oomResult.retrySettings, attemptNumber ?? 1);
+
+    if (!delay) {
+      //no more retries left
+      return { outcome: "fail_run", sanitizedError, wasOOMError: true };
+    }
+
+    return {
+      outcome: "retry",
+      method: "queue",
+      machine: oomResult.machine,
+      settings: { timestamp: Date.now() + delay, delay },
+      wasOOMError: true,
+    };
+  }
+
+  // No retry settings
+  if (!retrySettings) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // Not a retriable error: fail
+  const retriableError = shouldRetryError(taskRunErrorEnhancer(error));
+  if (!retriableError) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
+  // Exceeded global max attempts
+  if (attemptNumber !== null && attemptNumber > MAX_TASK_RUN_ATTEMPTS) {
+    return { outcome: "fail_run", sanitizedError };
+  }
+
path: internal-packages/run-engine/src/engine/retrying.ts
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Consolidate max attempt checks**  
You check global maximum attempts at lines 86–90 and run-specific max attempts at lines 110–113. Although logically correct, these conditions overlap (the run’s max attempts might be lower, equal to, or higher than the global `MAX_TASK_RUN_ATTEMPTS`). Consider combining or clearly documenting how these conditions apply together to simplify maintainability.  



Also applies to: 110-113

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-06T19:32:58Z
author_association: CONTRIBUTOR

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-03-06T19:29:03Z
body: ###  ⚠️  No Changeset found

Latest commit: 271eaa8e7010dc11b3052de8e391dfeb50c51632

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. **If these changes should result in a version bump, you need to add a changeset.**

<details><summary>This PR includes no changesets</summary>

  When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

</details>

[Click here to learn what changesets are, and how to add one](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add a changeset to this PR](https://github.com/triggerdotdev/trigger.dev/new/run-engine-2-retry-oom?filename=.changeset/fair-starfishes-work.md&value=---%0A%22trigger.dev%22%3A%20patch%0A%22%40trigger.dev%2Fcore%22%3A%20patch%0A%22%40internal%2Frun-engine%22%3A%20patch%0A%22%40internal%2Ftestcontainers%22%3A%20patch%0A%22docker-provider%22%3A%20patch%0A%22webapp%22%3A%20patch%0A---%0A%0ARun%20Engine%202%3A%20More%20robust%20attempt%20failing%2Fretrying%20(inc.%20OOM%20retrying)%0A)


author_association: NONE
user: login: coderabbitai[bot]
created_at: 2025-03-06T19:29:07Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough
This PR introduces a new task configuration for building packages and adjusts several error handling paths across the codebase. The error identification logic has been updated by replacing the general OOM check with a more specific one, and old functions have been removed. In addition, the run engine’s retry logic has been refactored into a centralized function, and new tests have been added or modified across multiple modules. Changes also include updates to task run initialization for memory management and expanded export mappings and schema validations in the core package.

## Changes

| File(s) | Change Summary |
|---------|----------------|
| `.zed/tasks.json` | Adds a new JSON configuration for the "Build packages" task with environment variable overrides and terminal behavior settings. |
| `apps/webapp/.../alerts/deliverAlert.server.ts`<br>`apps/webapp/.../completeAttempt.server.ts` | Updates error handling by replacing `isOOMError` with `isOOMRunError`; removes the obsolete `isOOMError` function in the CompleteAttempt service. |
| `internal-packages/.../engine/index.ts`<br>`internal-packages/.../engine/retrying.ts` | Refactors run engine error handling by introducing `retryOutcomeFromCompletion` and a helper (`retryOOMOnMachine`) for centralized retry outcome decisions. |
| `internal-packages/.../tests/attemptFailures.test.ts`<br>`internal-packages/.../tests/batchTriggerAndWait.test.ts`<br>`internal-packages/.../tests/heartbeats.test.ts`<br>`internal-packages/.../tests/trigger.test.ts` | Adds new tests for various failure scenarios, updates imports to include `describe`, and removes a specific test case (“Single run (retry attempt, then succeed)”). |
| `internal-packages/testcontainers/src/setup.ts` | Updates the `setupBackgroundWorker` function signature by adding an optional `retryOptions` parameter for enhanced task retry configuration. |
| `packages/cli-v3/.../taskRunProcess.ts` | Enhances the initialization process by including `payload` in option destructuring and adding memory management through `nodeOptionsWithMaxOldSpaceSize`. |
| `packages/core/package.json` | Adds a new export entry for the `./v3/machines` module, updating both module and type definitions for ESM and CommonJS formats. |
| `packages/core/src/v3/build/flags.ts`<br>`packages/core/src/v3/build/flags.test.ts`<br>`packages/core/src/v3/build/index.ts`<br>`packages/core/src/v3/build/runtime.ts` | Introduces the `dedupFlags` function with corresponding tests and export updates; modifies `execOptionsForRuntime` to accept additional Node options and deduplicate flags. |
| `packages/core/src/v3/errors.ts`<br>`packages/core/src/v3/machines/index.ts`<br>`packages/core/src/v3/machines/max-old-space.test.ts` | Adds the `isOOMRunError` function for improved OOM error identification; introduces new functions to compute and append max-old-space-size flags along with their tests. |
| `packages/core/src/v3/schemas/common.ts`<br>`packages/core/vitest.config.ts` | Updates the TaskRunExecution schema to require the `machine` property and expands Vitest’s file matching pattern to include tests in `src/v3`. |
| `apps/docker-provider/src/index.ts` | Updates the import source for `testDockerCheckpoint` from `@trigger.dev/core/v3/checkpoints` to `@trigger.dev/core/v3/serverOnly`. |

## Sequence Diagram(s)
```mermaid
sequenceDiagram
    participant Engine as RunEngine
    participant RetryFn as retryOutcomeFromCompletion
    participant DB as Prisma DB
    participant Worker as Background Worker

    Engine->>RetryFn: Pass runId, error, retry settings
    RetryFn->>DB: Query run settings & machine preset
    DB-->>RetryFn: Return settings
    alt Retry conditions met
        RetryFn-->>Engine: Return "retry" outcome
        Engine->>Worker: Schedule retry with updated machine config
    else No retry available
        RetryFn-->>Engine: Return "fail" outcome
        Engine->>Engine: Mark run as failed
    end
```

## Suggested reviewers
- ericallam

## Poem
> In the land of code, I hop with glee,  
> A rabbit’s cheer for changes, you see!  
> Memory flagged and errors fixed,  
> With retry logic so neatly mixed.  
> Carrots and commits in a joyful spree 🥕✨  
> Hopping forward, bug-free and free!  
>  
>  
> 🐰 Happy coding!

<!-- walkthrough_end -->


---

<details>
<summary>📜 Recent review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 6f5cfac73bc062fed6fa75ef9ba2f6674581448e and 271eaa8e7010dc11b3052de8e391dfeb50c51632.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `apps/docker-provider/src/index.ts` (1 hunks)

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* apps/docker-provider/src/index.ts

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (7)</summary>

* GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
* GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
* GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
* GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
* GitHub Check: typecheck / typecheck
* GitHub Check: units / 🧪 Unit Tests
* GitHub Check: Analyze (javascript-typescript)

</details>

</details>
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpADK2VmgqyjcEj2wAbb5AokAI7Y0riQ4rjedJAARDbYWACiGETw5JAATFwAsvgB/oLYiGHUNMzcYQBm2t5pRAD0AbiudZAAFGkMGpAA8j3Z/iTNsnUAlDGQAO5oyIjYAszqNPRykEWUkMylYNq4ANZk6Bj0qPBKGOKV8NEzkBhOAhsAjADsLwDM3R6b2liV+AwivxKuEEMhbChkARIGRYJgGJ5cLBPE1XJBvPhUgxIP8+NV4FF6KUSOVcMhJuoEFgkZ54kkUmlPBkcVQ2JM8nsNEYANIkeQMOEpaQoDAMbzYJSgzwkAAezTQYng+Cw+BBNMGw3RmPg2NxOJqN1wZQqyDS0LQkEQJG4LmoniuUQANFMEAL0PBmFD8JaPdxapV5Or8FIqL4FJLihRsGJsAFugBBWi0dRKjBoXyyZ3qiFmiiKaPC8iTcKhKFwsJMEOQCQuJVApjHFPK5AJJR8Sau2DoI0kk0/eT47zOshzCitJHUKWW2TFEmMTCQQW0KJSq2WhFpsf4ZAkSqVUTiKTeWRcgzxn1EDDwK4MTBhWHwknsSFTtBJpsqkFzbjcPJVPK9HgwKQNkJJ5PIbR9Nkowai0KTOh2OpdumGKTGWniIDOZThN6xKkrB1ymlgt5WuSyL5GwzDgQoGCRr8ZLoPkZBMAkNABLQ3QAGKxjSFBUQEWbItYdhUcmVzCuqAAGEBbDKYD4N4tBgIgNoIsp8AAF4kJJPC2mwbH8NSQnQDMex0lYeYIogiDOm+QhFOIKQoGEAgzNEypTlsAqMj6WlfGC6C0PZxRsBcMK0bGEnllOlQJIqyrpuo8ioA2iBIDQYUKnm1mQAI+BIpAShHvg3ChSUxw8HmtDRuIHlkBI8B5hgZWIKenEAfxnhKLgNQ2VO3A+H4ATBKE85YI86AMFZVpEmE6r/L4+AUk5ACqNgADJcAA2rARrcIgHD1PUqRIvMGhMMw9TNPARCkBQtD5UVV1jrdlAaE9A2+PUrwfAAum0u24Pth3HZSZ0Xc9N13Q9uBPddr0UO9JASPUn3eN9bzvKMp76MY4BQGQ9CqjgBDEGQyjLAorDsFwvD8MIB6SMKqxMG2KhqJo2i6GAhgmFAcCnMgC5oHghCkOQVCUxdZVcFQxZzM4aIs4oyiqOoWg6LjeOmAYGhabQV2ma1QiIMqHAGDElsGBYkDxgAkmTEt2vQCtbGixMCpgpCIEY55FpAABS9g9AAcjRVxELG1CpjiBL2gBPWIHsyBwjIJCHLm+YIrQzpKFcV5ORaid7OiKgkISsQAELYAS9CqXsaDexoEwC1CpkwjKoh4FFnjS5g9CSdwGClf4CS5TXincw6BkxAAAvDd1IyjABUEwQNPGwL29RWSf5vfKhHUe1VgnTikoO4YA1TVldWtYqCuwaUGO5/OipojXjqKHyFaRqtJJ/z4B0tCSSrkKC7xgEJYuL4f44VHifIyvdYwBDCmxRYaY/AUiRPgICJUyCtAtP7HqAgjj0AeoWfK6BFrFgbICCgyD7yd0BMfVqEDESUDQemXS6RHhwgagBFKB8bqRXoOabw0wZyDCkJw9UxcADkyAbTpH7uEFQCEEArhkewtI0iVBTAJH4DAFCJoICTIcbAv5qTtwun6IYqY96WmRGGdY9A9Syi7o5IgU4oGoBgdCdUmFZzMHkYVXcItvCaFtu+Y+X9BKeGwUDICP8PFC3yA2Q+7FYGIFgEtXK+UuwyPbrUJRFV1R9wqkUzwaQpzxIGpoIw5hLDxnCRTVMXopxKDFLaZhwFZS/goJTACA0BC1GxOwFM0gjBQBDiQYsUC3xKFoFwSSMRvBl28DELgMRq6110gwBu3sYg6SqbvfWhsk7G1NhgSSFsrYGAgGAIwaAfyIHqJMEgrkfz1CedweoEh3j1CtBQBqVkvlRH6S8pQtQQxNMoJoQFIYNBknNpbGI1tGkO3FhTaIrsXDyA9oKb27ghKeyFERCQCkpCeUUB/W8x9gILifgBJcRTPFsBJUgZgeikRVKkgAEXLkzCgML+n2EoMC7SjBVnWW6PYbg78bwxKnBU0Ek5eDIzrIgY8jBkR7OiHqeJ8lKhgEotRRlFAU63EeGY7gtBnb2LyDdbRfhlV4AJJpaI804p0skkgKCiQ6F5DaGa0YQDvTdS0eka8U4zVTFuNVREuEVR4ENcasCaIGzJmPv5VAqdcrpywAEP0CpoiYK7D6xAUE6T+rzBQINAaKAhrUUhdE8ADhasLQEK0FwhabDyBhOVDAaX8D4N8vMvB4B2m1aIEu+rk2qlTfxeQZqiJTgbDQOU9K4HZuQCS0glC2LoVjYGIS8KdSVPOJcQiJD5w0CIA6rSO563IACKsym0ITVoiQHMYU1UxxOSgW4phqZnS/gyuIL+6A9yMyctk4s6ZYUpM8JiyWNwKpdpcm5ImCDwiyDlT0+t4UWIXEoHQU8NsmlsWjs2WB6oOmrMlq0npMo+kDL4EMkZ4UIiEUmZAbioo6VpUvNQSKawbXOyWb6vAPRKigUXdWvIXBfX9Hk7W4NRysCSX5VCygwrcCiqBWenSnScrHO+S8t5HyfnfN+f809IL4PgvqJCwVumNDwremSHSgAkwkgOWnoUmZNptkCpxTFb+hVvrXWmtIaRS+a0y5sFemxWGclTMFdkkzOvPedZ6zfyAXJfs4liFAroWJbc2KjziBrkosmTrTLFmcufLy3Z6Q9RrFRBoPGHspJytAsq8iq25GMXkxQy7Jwbs8Ugl3RMgw3xpsrvVLwYMZxohiRBGkcl3hKXqlEjSqjn4YQEeZa0DEWIuVUinJJAAwiwGxnXusVH0+KozUqhYlKEgEKiNY/DEykkp7IKmdKxX43YhwA69sZliS29IE4whqr4UULVaWbrkBER96QPgwjEz88pyL7XbHKg0Gp2BOPshdeNLgHSuarUFutKs7O530DzjDMA/7EWa1tHx1mtT26RTJlpcKC0WTrxhCqbB/gc6jUfoglBGCy7GLnrGeJHOlpsCvWKOOaK6pTs6hxABcNfE0itCgjRTNjHqf5sI08uYr7og8qEv7Un7O8hA69XY32UTUyKvVGM/IpPAc4jdx5C3hxPuPxV4hN0a7fjYvBzeKdezkB6hrFuIEMbWaC4qmwayjdhQvudrAk3GaPwsNboHkHypglh++ygcotQoqTkWwEBHyAde6n4Re68IwnIm/l6gQxWohR8HIHQdyfAQ+0+qGIPtJwLjenVP7YH8UMD2pDChS0zQaoid+0JGNx2nKfd+ERHqYhnQCCAv4uPuvfRRDKvtkJPUCRJ470r7vnje9PsXGgSl02OJEt7gSj+t6IYqLhcFVAWHcDModjWjhnKsgABOmAZKUsqM0ApDiKhBeFeAqhcMeAANxSjfw37yCFr05xLpD6TZJEwggxorYXB7Z0qlpHB5IbDKLV6IhCQKT0AUGKCcaNTlwnj1Joq2zNIMbUZ+JCR0ZdKMbEy9J/hj5eDDK65jLiAzZQB8bL6DBfZ0BLKyH9Ll4aH/YqZRYKYwCmTO4Nrqa+b1bZafK5a2YFatZc4kDk69hwoVaIyeY3Kop3I6xmiUDoJgD1y54vJRgYBgBkCpDkAAoUAMD1ARGMj1BpBKAyiIoHReFCH2yOxYpjaKyTbapewzZzaAELZCSSRVoMjkAvZpaVQei1haobYUqeAWgBBT4EB/qeI77ShHb9wspahna/qa77x14E4HbNFjx4QVD2Lw4aq142J350rR6MhEiWiUBXoezKhm4JR+ACjTptIG5oKVIghjFYBZLYKTwTS3iijlx0DOiDg3HDoETRCuQzSGReK4ZxLUEEbMTYLEbsS85t5f5pyh507Fr0CMEWi3gs64SQHFhL7HzOiSSoiyD+a4AXQkCcR5jMA3bDHHySRNpR7sChjuqHodJICpjGpoB7CtBLbTSDBEAuDJj/ocF4BonATqihG5SYavHIE4kxxmqnjfD+yjr4AKhdhpR14fw9zoE5KrD76eJBgsksDCidFwLdgU6xZC6YJujFB2g3zPHuTYadqY4aixio4sgsC+ZIkolokYksDYl3apjgJl5wkxz7GMhkRDDkSwKXEIh+DskJC3E1DOTOgARInBlSgBCrDsTRiDFUw2IyhJRsm77dHHC9EZ5THN4zE7GJ5658BvyDrx4xq4DvFvb0DF7dK8KeA06aHh62TvaeAAmGLFgvprHYaRhb4ZLxKslLYNQLK5THqeCL5B4r4e6bHoLHhQ5ylJmeBoK1FDRjyCAiBiCAl5qHCRgkBoDMAVIq4jixia6N4O6iDSCIC4qFTUAWioD7ionIhYZaodjAnBB8Eq7KI+5SBhQkhfqMZ6h3GzQU4WqUrVl2QOQeregtFRDLkL5QHtkxj5Aql749F1Cng9Br6+BQ7zbhSCgIirr0aJnKJbBmi/AqBurFnTnhhVn8jNgKRnDRxOTwWpmtBmjQlpQpArgukeSsF0AxlMmDkPAbDrGNjRJ+ACBUCijIglFDEOkeRt6ngNLCGUbdLiFdSiD0b7bwHUHMZyFExsbzAcbKHcZ3IgSek8HzLaGWlDCuDWlKm2lYm3YdaphtDjqIBbDOgADecCdsO59azoSJK0zFRAAAiiECEN5eZbIKKr/CkH1BMbgCHLxXwAAL4xbHKIAxGNChWIWeE1Y+EPIGB+EUABFBHeyNAJDhEVEkDRGxHxFRFIkZVpE1ZDZZGjaq65HATzb/4wmxwriInpUpCpHqaoHVRWT6GCWJl6hbBpiRHyntwcmohXpVJC5YRziw5rDiC1APpM4WQcoWi2pEJuTdB2xhB5zulM6SRWB6RVawHxx8BpC1K6SshDCUClknU2ChWWVsBALvGwIdrSDPiLbbhpTDJxKKnZ6bphm4KiEr6sI/BVJsUYAIlWnA3omYn2l2XKh4mVQAjHlSlc4xzExFzTUBkP4RqtD3m8TemPhDj6gEghl8Bhn+njRckeQp5ZkbEl7OinwSitBFklkhWahJJ1BRX1l3BxX0oPYMRbBKAClCSw2xzmoVg6rJwoBqjJkwFJGfw0A9o+nXEiIE1w0mn5X4IKB0LSAWKMkdGI37VhAnlXjiDrU+4EbKLZmK2zq4AprS7tCy7QF5B9QNFUmFyLjlxyp4jDlmXDBQQ9AYDZCimMihpE2G6RpHGwR4pYAWj0Z3Q/DeSRoKL/XwCA37UggD5hmXkbkA1OjOT620RM6Djb4W2QB2wghhn82RUK53AUI9krYq7qCUKmyDA24MTzVU3ihxhQ0y1pZY0elk322q3PrmUTqA3Xqk1ek7ZoAJnMBODC3MCPB8B41i0WpAlYCygIij4q5NCmmtDV2wWI1CzIDkDTS4qnj12m4l6UIYF6jNGhVQ4y2zXIxSlQIclN1EBJ6YntLnn6nOhO3n0dQr0ejr3RVPWn0G1OQX1A2olKmmggi1CLAMQuAogbk7EcR10N2z2C75BB2LDWS51l1d0IOV3v2ahdlKmM7qjCnjqTrcFEhoZDDJJS2eCOJB3DUxzdVh39AR1R2Z0Srf1SCHp/0JDBJeRUieBqowIcWahpJCIQ0gYuB+WrqCKRwQ2wJukJ1M5zDqD3wznR3pB6g1W0UJnFAsIHVf6pnCigZjKcLy4DFMmoA0nHnOinaTVSicoqNn1OSti7hLFK1ThhlqN6P36nCXyJRaVrBpg1gEhmNIUoWU3zRxxW77Q+B2jPqFDFBe3j4IWFwVRrp5h+CVCv2hk9VTVJxwIyCM0nFX7YgMBjhsQTrXrRNHytIyUZEiGqU0YSHKVSHUYyEaX9LyHsZKF0EqE+wGXQCfUmWLK+bFl4ZnWsgXXHJ5UFUKj7KtahGlWRHlUpWVVlVpXDC1XXL8zLOmKrOSS6FzSfUvVh2I2WE+q/F7N7LBHFVhFVWnOpUAuXNwREB9UGXqF0orM6GTMlCYSigCMeQI2oNsDWUo0jEOVjhOVoBcCbXYtXb14XA9AUDQAiUnnL6uXuXPli2xWb2UDDheVJ2+V1CBUkDBVJ3hXJKQDxW4vnWjC4uYlIAkDACvMWWI16AfO7PpiBH7O/NHPAtnNxEXPWNgueFqEh3QtWHwu6gh1WkiOR0WMkCYtbUCtbUEvXBEsktksKjwlUtcCRhjACssBCvABuXyOMj2vXQpD4FImcsC1cCivIkVCMbxWQAAA+STR1qOErsWnzbE3zBzIRJVCrQLyrdT4LWV9yRgUr3gMrPzRV8rFzirwLGtZIXyYtnENQkUrUpbqRA2qKDVyGBeOK7sU2xR7VhCo0DongGWFbVbnaiKoQfVIoA1BYmtt2AQyItETMqu6gHxJYdjuZl25RJzVR1ktxAIRQrQwepTnizN2CSe/bGEG4tY24hUe5/67cgGrqhOUNpbLYq17ql2ixEs0AoQruFe1I3oMC5ikICkBe9UjUyoLUS7eDYpJ7W4w4EU7RKqc0pRK7Md/AQd+TkbGwFTCkFc5ZfT7V97CgVYa94S8ANi84pE7NooZ8dQ5sTw3Qy8y8gbrQvlGwKmiAtHXA3wVwctC7VtMDtuPa6wfAZqzoIYXe+5cH0oZVRtAQYg7axD0j7cyi5iO1Up/9PUuAQI0IcQiQ0ANgAAmgAPp2zZAA68p2zxjQCJDrQ6cTDKKafac6d2whwADien8YnE5nNgennE9s60a0iQMQXIGQNHy8lbbqTkjBIc3ouQ+QorhErHECPiEHSoXHh6ALwSPC3+So7YU7TOUCg45IlI8SjxfUvS0G8pJ6qn6n3oE0MQV2/QVg60WniQvKenAA6nbNAAABJ6eJA2A2A9A2D2D+cGCfCQC0cRdhExdmOQDMdxffC4dO2Lg5I+4SfMrCgJCzVTfLpQejiieRNjw6lqemgthiZvreg1c2Dxj2AddNdDcAAsQXKJwEsm1EkE/QMEM3y8XAoqKFyXPoBHk6RerNzCQnqxAYu3y3JzRXXi7c7FpcFA6d7rxSZZeQUnkQ8ginKH9NG+wmTTLxHkCpKLypytTRu9p4iQopXHJHlS5HEogu1ksKjG0I321FNA0azGB4TxJAvCmXpFZRCQyQq7r80YyEO6CtsZ17AmFXUVYtG9W9gtRMeAt16xOJHqRs9iuH6YPdu5+Qy1EPPkYoG5ldv7na2CMRzM/IKEf8ALGgj5uAbQSVKdvm+cX8OkwyAInIRgyFygqFMU2ToQ98SAYlTOeY5+RTpb59917IFAM6Cc8H/PZV6N2vu3Xdlxjjy4ng+7QIyD64ZAp7yARAVACIsUGY/T5GgzCl8+IznSENalHcLG0zOlszXGqhkAEX+6LSYhlfSl1fQzEz9fiTMzoyczV6byFEb41PoIqAa2XImbvhXz0rhVhzybRbqbJzV0pY9Qrk15pLUMOmxwLXuwg7xQdb6RjbI2zb42p5+KBRCzRRN/sWTey2fZa2Iom222Qku2CqdKKpvof42PZQz4e3GwiKbdtfMW/AUDvwRjxh9+h/Wtp5hlStNvcQkX/noRVKSRz47TXOhKhlqVBgGUkBqKWyAShAI+m5GZByBXKAV7mlCZUIAxWxs90oDFcoH/zQElcxAH7ZfLznmQfgX6S0Z/HwHVDrBeeGAscI8HYF0poQBfbBNwEGC24REo0EiMKFLRADKeJjVnnqCkgIdKiqWaVK329DME+An/dWozy759EdQtxYcolGLI00N8UYGCvO00RFMFBiGSqD9TCj24p+14SoKX0aTl9jB7SUZjXyYz98Hig/XgvMx4x2wmBehA7k+DCgY9lgEmKIWEDcqsCwgobXARaRiAEDQgMQXApK3n65tF+Sbf5iv3OZr972m/agBAJeh3RoBtAA/urDgEXUfMPqJIZABSHs9T8ISRAJgImjpDgGWQudsUFyH5D42C/WVgW2X5r9i2FzCoeANgCQDahMAxoUO0yq3Is2uVAoXm0TZ/NjmCRGYeUI37IgXAuAR4NQBrarC6qg2dFI1Qv4tVr+pKdquhWUHt1n+ng1/o0SpRiQjBHkH/m0JiEgdlBkkY4f0jOFkgj+mgTzJ1RIAID5Un8SHEqh8goCPEvmVIUDjwGlFshxQHSLUGEqnkJ87fDJKIh7oc1JQ6A6QL0O0icDPc7FKhHwKnCCC0BwgrAWIJjjKDS2s7GgPiS7CoB1m8IrVM4lgSSDje5cAvLhwICkBeIUHTCrGTyD0krwGke/BxTfBEVagJFFUhyKcHdA2++gxFugnyKkoHihg2lH4O1zagGADLGAnKWsHocqmGBHBi4PQwP8Ao0/QQmX3kqmiq+KlbpH300ohDG+Q/ZvgsygCRCWM//WIWEHiGmVWhYYjoQOjSHmlOUgw0tiMNjY5tthcrKYfsNX4JEKhII04RuXBFNDvMvmZEe0Lr4Hhc4FIkQZ4H6GZCsRuAVMTsy2FFDdhKbMobmKOEblQRhYi4cfzWHeENh6Y1sYW2mE5ioiFQreB4UuH1sMiw2J2JTBbZ5E2qs2CQu8NOJoQdGKvGsjXjxqU8FB4QdQBXBiD2A6gK4Dkm0DDLRVYkJxaMEfVoDjBExXiUaKoKurLt4+q7bQaXgCiai3IsaZAOfBRwgU/ueTVnkLkS7eh7y+QY4i+CnH3E8udZNHBnBnrXRsUd448sX2PD2I2CyPSUKSI9Qnohgv7NAZoO0hQ4px27ROqETQoBB78e41yHsmFEVQo+BwCgIhKnBTliYHJOBmRzFCc0nIeUAqNn2URzApox5XSEiBYSyo4RkJCcuXVLYESqyXPDLgBDgoSd7yjvbiTLzy7XpUAbsA4C4n4TUxkwk6JEhuyoRPF5AQudCdZGL5xlUaB2fxKIA2JqlXC9iMev0kYxKDsMv4tcMJ2Vxs94x2KKXpngV4JJsGYQTPrMB6jexSKHJbxuu3Cg7cmSB5cTic2CQxDwgZLZ+qP33jG0xAPDHcZwh3y8j5Bf4gVBwg1pThmeO1XGsTyTqmDsQyidQNfVNiDpm2qnAAjf25EKAfAJwcoDa2qQZN9xj8KgHunUFx8MAAvRDrDUsECF6k7ojvpXUUohIe+Po9SsEMGQBiwh+lfmGVLXA4SlkJ4s8SiDHiXjQqrk0kDeNVxiS6A4wSwg4Kto1w1BAEPnlNIT5piWxEwpfiULHEdiJxG/KcRCIzbrC5+Ywwod9JeSlsX2j1CqvljU7cAT+9VG4U20XGX9W2BowlKuJ7Y/xzElcfZkxPqEchKArIndvvV7TfDgJ+Eo4IFDHLFTg2WxO6qQLYjw1XqDM2iOjUjw8ja+6zHtoGx6DsyqsvOabDwM3GepP2sCBUAiAqCMAHIFpKJro16bUYNJKxCKp4l/YMS9ghMqYMTP4Fq8oasoBgU5BtD3UDIkkRHiQBuwYAI4OkQ/GkBbCiUCiKuc/GECbJ646RU4R3FaUFk6QTZzMjYMoJlqCY0wanOMEYBa6UhA5w5KHEI1cBWybZt8McFN37w5Ijc4GNatEC3bcVQ6FlH2UzP0gbBwSwtXwIHwUDwYeh27ZDm0XaCSQAA/LXMkjYx4uyANgJgDLCTgo0scoNswiOQ3026eYXsvcS7nxyboOkCkGGDzhhI5oiaemEuTCCME1UUbaIN9hCD0jzZK9FwqSCqwIlWiLuBEmgmgAegSA8SO2JHW3m+ZZIh8tgCfLPno1lEiJfuM6y0hOkAoIsh8FcUPTVNDZagNUXkXZIXSem+jdQUbEYC0TKY3kqcJrO1ksSNguMxGZ7wyZQ5jRQzTOINSaIdU/ZBcvEABHfkM5AF9+RCCuHwoXBfg4vWxiiNmm/zSKsNbwXJSWltJaMAQ3vhtL9FbTFCgY8ZMGMMpYIXYKOYTPkCjEPMnm6AbVnqNVl4yCZeYVsC111nGtsWprfFoS1wBQcr4wHGmLbDwBTtxAAuWgMkFUXNR2AWYUyB5SVzXAKAnrGDhGwdYpAtov0Z0BbJHlEBa5OQQ1o4od6livp+bVrNDJQIx5zUcMuBcOxaHCKZgsgBFkHKIncB8ZjEqRfv1kWOUtgCirYOa0MXhR9FMsDRTSDoI6K9FQHAxRcCMVJwTFdBcSOYpsGtArFXrIgLYvsWuLdGzikCHUutk3Reauc5hA0v5mCz3FcbfwuMK8VQzQgMM/xYq0CUDjasOVVsWKHgBgBmsQLRhDe1ohnIzICQCyJjWlRIpT+KM8/mjPuFtsb+Tw4oiO3AJYUCEUBMsQCOfB6hJIhiJQALOYQRykQUdGUD0EUiypi0p45+U+Kkjzwah28ZGG1j7Q2Z6gFsi6qJB8Awi662GT5imESifL2GvPEyEnHMiWRjya7PqIwugqhzKJazMEBoBKg9yKBluQRbAmpmDw0AsgDEG+HAR+woCqUHqAUovkr0XltAN5QiA+VkSXwS8+gFnLK49sblJAO5a0geWwAnlzK1lSQHZWky9aXMlRAcGQCSRQm+cOgDpHvk2gKVIpDiBLzsQWyVVKSSOC1HsSSQQ4PQXlIkD049ArA0AO2KHHsC+yx0sKZKFCswl6KdIi5A8C+EMFUyaeZI2SGKtUgSr3UuqmmVwMEqMR9Vv1EwY80YSCrmw7UCgHSHEDvU9RIVItIOmznTED2PoFik0Xh5OB2APDNcOhVwXsFqeMKtavfgBKrBOgeQPpDRVZRBYfgE1CMUh26T6ksMbPdxDHAtk9StgVkqDMuXFzqgIuktE2BjVpKrcW5DalCACH2y0KKM9C4Zt329HSEWFUzAfttL0ot9QIPCzAvws8CCKlkoShFmnInQZz7elhYzPKsRXLKMAqy6aB80mW1AZltmOZZ2ubBLLkVaymts0K1ZhLsQJ62FUa3cWXrfM16z9fetjaPrplsyyqvMuYQfqVlKK9ZRdTaArNfMaqylbQFjrnxN8dgomF0pn6gyJlkMgFQEFRjfSNAJsM2JsttjzjsizVCbK1XbbYzMZwoIEYv0o2XIgc2TVBRAVOXFhhFhJAcLHx7YaAgVIKnSGCqiDUixyIoPiefCZxbAfwkDPgIJLLTIiVVFUbqo+QCBGYgefTMHDJKQGDkoCQmy0FfilKmxYwWFUARNJnL5gVwtwXeHDLywgrEixwWUH1XYld0l5h6G0JJKXYyJPqUbZ+i+RGbVBjSAcb/GgHsCYCZZtmgCGpum72ABgyibElRAwBBxcyWwcEVDUki8yLqdm5LQMBDBpQPIv4M0CTjE3JhigcRJyuJsNYvIkinmjiJ5ibQaIhI6W5UFltK0xwKt3aKrU5nSgArWAyoE2A1vEZNaPNKRVrULMiR0yER/GzjGiCuWubGtvc2mcBKkgFaAAao9VaQ6QrQFLHgPgDNDjgTBTAY2ipFZr/pgtYTG2jHFAHKCutmW+wIVHSgiCFl3QL3qGEyZiV7N3wk0dRmLVfDwVwSYRbeBtA/zOFtkKhAxWnl5pZifojijprtzjQ8kFM8FdOECStRZK86wIStMkKBDfRa6/0ewp2kt9pkAm2Fstqsn3MlkYmtbZNofUkbLt5VDjVRquQGUqdSOvQkpu4CUdIABgSALoF8xzwpxMMJ6FZrN4bJYgYmxVkzoUZTbkiqRQ5MLtF3LICtsu/zkNtq3SBLoiu90u5pV2za1dIuqAMsknmY5td1W4bfrom1K7jdnmk2GrqmRQFhoNcMfsppSBcB1dFumIFrs2S27at0sMbS8kN3kBldLW1XdcnN1i6rd4SG3brtwAjaMt42iPa1ma0pEXdNzVvlAV237bgdYBOnQsl91x7lkGexALLq2gxAatKe0PRgHT3/I3NWe96Krt+gs7+lpG9nRRs53VYiNRgSZYCoV3/Jz8tceoNU0bh9jIRVwhtlsoXFoTdlrGhZjStmRdtsm/8VZIA2BlQiCRvG7OD1JfY9pXxwEBIF3Vw5FbyR1UbgJxC32FaQ6GQzlFJEn3b6TYkmhzRCrm6lgW6cozAO6m5XaN1QO2oYVUEj7kDHao0xOTMR6G58twbSbXiWsk7urKFiZbHAsnMS36p94CHDt/pqmTplqflVinfs0K/Akmv+exZjiI4rg1sJGMKC/ucFKMKsz5d7JOBVn0GW61+kZM7ChxSpIp6YEIC+CaAx4CGiYBbXJMemi9p0HE3dsBHYPglmJBXICCvOkC8SKOtFWgHuicE3SReMIUkN/CqV9QCO4gYjipGLRpxcAbyQ4PQfYlyHKQuhioPIGUPfj7QFgqhZeR4hL1QgJaOw+wZfatB8yp6/IoXwPTXonDsHOTWfGlDBBNemBFhAgu95/bgBT0udu6E9DekJ2XPEcEzC1T+SByeabnqpPqlX6MDd+qVZNGyg9oRKe6YmDdTwBPU6AmhtyLjoWk+CPRnffwWtJXV19WF2lcnZuq4U86ORJ+zVuSJ6E1i2gtezijfrv0xBnQ9vSAAAF49A7QjQKscSqd6dhbOlzaPongGx6DO+qrOkWyqD7Wdw+1KnljH2KQJ9d+pGdcNo23CdljGh4VjNX1iLij0xrA4SozhgEs4aO33l1SH1kaR9m/XY9can19VecMtKftICAkiIw0Uxrg6zzKT0BlU7BqMkMFIVORlQWqbXDMCxxTQkEzEedmB3QKNwXQ2TRgwZm4qNRh0bYS2pNGlnYNzNxaZSNaC6TBSYOyvcaiiZ8jsHbgtR8qPQBoY9p0DfodWhyexX7GR6IdBKaRCnACn+y5mtUSiK7qMU+ckgM4NgE4TWHOqnTLE0BEwBLo9D5SyKuxP51c0KeL+yWZAAOCyAZl/BxRtoDYkugSMpJwBudkK5hGHRzQQsTcBkD4AFIhvLKSEHsSinETbIhRJQFxBsAVgyUD0UySkVEB8k0tYg1UmbyPU4k90elklIvZldUppcIpgCFoTIIsKMhS01vpfAHBJivGJ1OIeloh06VtghiAT1urWLPEqwIQKdoLi8qQknB8Uy4jv0bs+IpQQYoEx/Bjok5NALVG2teKL1N4yIeQJWREVtSJ0lMJw3Ot8HtHGFnR8ZqutYwKFdKw/LdUZSJD07URNOmWqKcwOAM2g9BixY6xNNEANjvzLY8CcuN7GbjYy44wYEBOAtYiFx0E63o2XIz7jqMxfU8b2WPCWNIs/fZnlpkfg3GNOi5W4KhVAXCt2TDEDoudE9tfz2xkE7XB0jJhUe4EWEQWX5GZhy68yQ9MqijVhjl4T4nXfsZd15Cep06zcevieZAMLSz+m42/sx0aIquTRMSRQ3nqP7LsaF9/dVGk3NyKZ9BPksxn7h+btp8YKwHbFIpSbyKER/iT2bXTsAeZpRJi+hbLq8R3kS6DADKOzkoGSK0W1JvPWhC6iNL7c0XG0LEufMVd8At0a0YXWE6mF607o6TrYWHmgxPGHnSErPO0W/+9FsS4xd4tV68hkG040CfOM7Hx9Elo4xsNwtvnQToRRNTCOAt3HMiYFnIhBeX0HL7+sFpnP7HxVe4/AmCh6s6Yyw0j0Ew6gVT7KzCRrAMMa2iHGoTVHypVPUt2VLOtDNnYAAQR06bMeoihZwb4NkuyHsSw1gkAJPSdSgCnQhqZ3KG+lATqsGRlBjZmqHVMuz0GVVdCclRmXVSZqaG4Ym+MtSIsHgZOocs0sLBOt5F2DBIkgqCUZxLbmaU3TfVgZ6n4T5VsponJfA0DGrTV5qy1dapDi2qESIamqy1a6vnzlEuorDs2HmtOoK6V158G7Nknyqrzd+283fpizQgwz4pmKMQceC4hcGD10c7Dr1M9nxEbphg0hpeJBN8qpGUcghcW0dVzlnUm+FcvxufHXLYmwy1TktSW40N+ZggNIJVLdsyO/OOteEY2tiKU5xYV1BnLBKRyoVnVwWT1dYh9W9R6TBI2hUOUg7xZy+Oabzx1vMI9bdBJNTLVWBsX4dkssSTLOUSyn4Loa/lS2q8nqJxOyUvM2JyZscGET/ZvNFTbzStAaGHNlo3QoJ0mCidzC/y/udCH9GeM26ygrutDn7qTu0Y4RTLWtutJbbuVy8frbYBcBtkikXq2wBDKCyuAiQaNV0vvMpBnzRVV88lfwtXGcrR8oJeebDEF3G7NtvINXaNbd3y7kASu7QBHu13mE9dwe60lshNX0wCNwWQ0rbP8tHzrdxwmcf/MpWu7ZdvK4cagAZ3jK4VssW5UFvum6xSYkW7FabEaZMrHd98381ysgzBxOsR+7vbiJPpbjc+0C9svAtX9ILWM74C/1gs9ohyn7BEmzn571pOZnYOOgcXQZM5Ugb5UDWYVgc1oNt+eU7jTINTzp3ar3aCE/SzQyWZaTtFLgRl5ndMVYr8RARmC1D4A/aniPUBniIgK2URuvSBO3GmDIAqSvgVbAIehCA8BKC97QyItA2XduQenWwD0CuyJB7A9gc1f0D07cg7Y60Brryk00DxoAUjmR313keKO9OZOHTpXDNVQRVH6jzR+Agcaa9vQANxqbAlW7maZJxTTYMeWCLhG1aAuGmSbjShqd9smjTyYCHox3lsuU9DqB480O+L7ZvmewHbEc5qONHrqvgJJHieOd4wlcGwNADvnlMFah6HUnsiynFol2De3oP0HVO0pvap4UQ1zbrMuGJZxalLuQq5opllwrQKtaKBrV5BFbFDpdpcddpVItud1bRXkwoA4nskB00Ubg//iVBygJAJ85Hacj8OK4qwNJwk6SfrQX5DTjQnY+ZzeB5VSAKOvcHTAolpMz3VwAHnNBTQfi/4AwZgC1N+ATcKzuIyxpf57PPVh6f+EkVPFCZvAp80CDnlIBlG1r3q9BQtyuXpPMn2T8BNxH6TkROoMc4uOYWSCYUSZ16KNYydTCJA3yuAVF4+DAR6jnBxK6EM48grFh1nGTrJzk6p681gmniFhpOfE5ov6AMabPCeVikTSfEfC7wEbd+0m37+vSMLV0StEyHIdqozhTILFHegRHY5J6r6AHmIGC+DznCuqM+IwFC0f4FTS47IvYhg5mvDc20eWkJ3fLXRp5g3z6NHmuFJ9086Xr7t/8oTYWbIOYWMJlKwNmDvIN0s/tArl0b98ZSca73t2v7Le6bb/bnEPHAHGMlca8aWZypYtY4eLdkz30/G0FaOScL5pWI/dYaM9WQbAnGqNxYy0kCknJE4LKR/V6kT5Vab1AtXON6AH8OGbRt5bfVry/1eyrjViMFGZRyEiE8x4f9oGa9TlJwWZNYU0oWkWLNkHgCVxOSePR3q8OiCSQO3jICyD9Upzhx1GSoiqAuGquMzRpxwlxDazyBViIthHf9N6AAAMGgQLk26ZUtv3l7qa82UbAvjtnAxwMAKiYrNtnAtfb1euvSHemGR3T7Hlf1EzIHstUXKw23lv5WI3hVoq292ysDViKoxzTo2Z4iNUmrzHkNm1YdqqWKnq8+CDAEuhadOQi3skeSIpDLcsnR32Ais8om+SExYyRYLVN29Al6ot99JidVTw9I69d8RHjooLP7CI7FVSxBMEve5t8MNgsNBEoYmHMZz0t/cdaIyGvPny4m8m4CQgaVvEoWA3Jt97yeIMOjozxITWwVEIP9ozDeaCw5bhpDMB0bK4Bx91AND0Ag4vKAEFTBA6uJLT0c7ob0MGI0ntrk1tm1gHXOeW47QzHyzucrok6U7G6q1+nZPObahFF5kOs25ZWtv73eQRd+QDaAWyXF4jZdz/BDIhg93XAe4HSz4ALHIAF7jIBvZK9b0t7LyINw7qN1pXj7cXkY/naS83uUvd7rSNeay+GscvCjPL0MAK+UAivsvDYOV8q8b22zdX7vXhZDfuWj73CzO218S8SyoPgsmD51/FXsqg0vHxG83c8QRshPqOWpeIwG9LvO0w3/gIV43KrMavE3ir5e+m9VLZvDXyvU7pSJqteMGrcKzLWk9bBZPWn+T4p4JsCnDvL39om953uNfI9n3v19+e9cffSPpb/93lZnE0bCrAD4q0A9KtngOqHI0AevJLfke0fBxsW+TNgsR4EHR+4xs9PnZn65o3+orTLWuUqxoPlIWD11/g/Pz1e3+h0b/oVGZzADQkYA4T7APR8aHVYKKTn03BKh6RU5KSGDYw9WqsPaSvJTfG+t5072uBhJvgfrOfs0uykvhFl0OBofwbFq1X9DY20nf5CDKY022e265ny6Ce5syYNSHzuZIK9Mj0pDR8VvqPZJmc7DwCMBT3a+C0hzgcXZ7OFuy1Q2RriQZ6fSGV3vrGCWy5cDs5JH736j/LdUegcW+9ictUNMdwUPvmL3yT9985+EPfJ/IO9bwUo87rJ4ebXU4ovm26UpU2cIZOurYNodJFUlz0RnKUHiORZwkx/N55l+ffFHtSLn7dPXpZQ8oYd8zCGCWG/gNx22PuL/EqfIjcJ6sKDzyNWX5ARcwEMUAtK7u7vt8cUCQCrEZa6InDrXAb40Kp8gKYfUouP+z+Ueq/NHm5/dDPHzSftKEciftLCg6ntYiTsWRj/ijSueNQrDki1uaKz+Ghr3BNG4DIIjx0niC1IFAofLgCR6K6Pm7+MNbqOrbuldDg7AS7tNOpA6I5LHb46oXia7hetfOa7rqlrsFbc6+ei+L0+8XksjMijwBMYbe9ypz7beqXlpCzGkAKsYaAXrolZ/msPq1go+pPv6rk+TAWvqOCf4iMbqAExhx62+9AMr4Q2VvoNzOgwgaIGBuMPsj5Z+0gcWiyBbuvIEVgigWfZ28MQBx6uE+hjBwaBlvlDbaBQgasZ6BmxgYHN6jWsCpGBFfiYHFicgev5rgSgdYGUmUjMX7x+LDsQaEKTRO+BOQUgX4FT+7qIIG6B0PklbBu3gQkGT+6Pv2JLegxoAFsBpYtYG1+QAbx4/A5ftkH++bpikFuBaQeIGGBFQWT4BBZgUEGxBdrj6jWBHHkYZUG0oGUFZBfvlR5M2NQSIF1B83pkG+B2QaYF565gVTyFBHQSoH9+O4HKBUAC/uYbL+QwToG1BCVvoHpBEgS8j9BMgc0HTBrQXMHKBMQOsA7ocsoO63eM1sobDB7gS+aeBPgZNo+BjQYcGXC/eu/bEaOwfUG2YOxFsD1eWnoTj5Wf9lj4L6OPlG7Mad/EKCyW4kF846qGNEHQkUQIu65TScGo6Szy7qj0LIgWwE+KPAlcqGrQgHuk+SkWEOPU7Q4PbNl6NKuXld6aABAemD28joBT6rkxEIATb+68ud7Uhg3rSFMhnApuZYArcstLRQRfvbL0qpZiCCSQqIQ3ZvqVyJsAOQrdMWDUyFoByGduiIQ6rhGaSHkBpGK0gu6GsQ3qu7YhJIGgA9SGbv4i4a2dtejEhn2HpZLsFsmu4xMWaESgpQgBLMQ2sh6HgZ0oAJFcpSh6IWjS2QpoPPL6ICoeFC4gWFE3iuC4oZdgIhS2EiHyAHjJ4iGhuIR6HAYJ2mBinqWqFEBvg52sUzIOj/LGEvgzrD2Cz4RwI6o6kVxKAqFiuNHwA7UFoMmEeQMBnL7bghrt5bUBy6rubJ2FrkFacKPGKsr5h+rhaEHqjKpyEZeJAPqF4q7MgyHdKPoTKEliKoR6xchl3iu4fM04UBho06Vh/ZiBvyCAbnQujGG5n84IQxq4+K4nNiKyQCiJqQAYvqNDh8SDBL4lwBIp6rb+SoSnSieEkl2DqC6+lEDOGw6I6j6i4fjHAh+1wGm5Y4GADiYfhwoDlqZ0PZv5rxsYuqWz1AtHPBEHGhyA4hnEKwNTyqeBDN8DEqf4XVAKWxwGWC8iuEAc72OYLpTzdsLclUJUgUEcSD5UYusCYIRy8EhEvyzoQUSEiBTkMCyGccE5Z6IYYBcTNgK2BkjxhXHHUCw6qENq64cmFgXhAiI+oRZ8E0+GiDQgE0BLw3AGIJFR0CWCGuBAihAu9rEWrgM2Hx2HRm2ERee5p2FN83YdzpERRrgwpeiYzMZEdh9AV2HzMUwK6YS0MIuuE5UmWA9C6oFAIEQDyAkXDJoWs4vuH0aS4kxr7K0Fi6ENEW2OBErWb+PKb/CfNpcrnh+WqEDOe3kVdgK0/Wqu40KUNA6iREnCGWK8OloKbwM4rlj8q78iME9BbGeWE7RZRSNhVBd0BIiS72OLlsAySQ5UQjBLwc3s1juEEdMeDMRQsPyECeaAP2phAg6qUSlsaUaxIZR06FlFlGqAENb08HDNyqPs+sOdi7WtDqmGuMDDt46K2QuAgCVAoBKDpRAGUuaHb411MRgeaABpAG6WcoIVKdoWOPVIZ4JBrE5xQbIQX7kQOaoOTegyChXxKo5opaJMou7DaIoElTNKTFgECj7hlB+/rzxTRlANeoCy9CmioPERfrqImyj0f8ZuRFAfyHWRS6rZG0BsLKZEcK4QgZShif/NLpYUg4SlHFAcMRQAzReyHNFPi7UUDJVRgKjVGZRXZlCItCk0e7yUA9MXsCMxZUSzH/K1UfYR9YFAH1GyAHzJ5G8xPkXO4UAAUaG5fmWsHzDhQVBCTBiw2Pq57qKcsIeFKwFFGzBqwnMJrA8wBgKrHSw6gHpxnAiAHpyZkMyHQB6cOpHoQqx+MOgAAArAIAAAbJUBnunsZ7G3ctqC8CVAAAJwBxTwAHFuxLwK5AAAHLeC3g7wGe4bkbsUoChxt3E8DvA3MLzCux3sW7EMAU+B8ACADAL7EZA+4LQDexaAC8Buxu4MHGuQJcX7EvAt3G7HRx4cbdzRxngLjCqxGQC8BPAG5GgBtxLwGe5PAZ7rQAMATwE8ACACcW7EZASgG3HvAwcU8C0A+4AIBuxZ7gwBuxTwJ7HvAzIC7EQArnpbHWxtseqj2xtAHpyEwmcUAA== -->

<!-- internal state end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=triggerdotdev/trigger.dev&utm_content=1773):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR