Repository: Significant-Gravitas/AutoGPT
Pull request #9759: feat(backend/executor): Move execution queue + cancel mechanism to RabbitMQ
Author: login: majdyz
Labels: name: size/l
name: platform/backend
description: AutoGPT Platform - Back end
name: Review effort 3/5
Created: 2025-04-04T12:38:13Z | Merged: 2025-04-11T14:39:09Z

The graph execution queue is not disk-persisted; when the executor dies, the executions are lost.

The scope of this issue is migrating the execution queue from an inter-process queue to a RabbitMQ message queue. A sync client should be used for this.

- Resolves #9746
- Resolves #9714

### Changes 🏗️

Move the execution manager from multiprocess.Queue into persisted Rabbit-MQ.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Execute agents.

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
  - [ ] Import an agent from file upload, and confirm it executes correctly
  - [ ] Upload agent to marketplace
  - [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my changes
- [ ] I have included a list of my configuration changes in the PR description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>


Linked issue #9746: Convert cancel execution request API into a fan-out event
There is already an API to cancel a running execution, however, the current implementation won't be reliable when working on a multi-executor setup. This is caused by the API call that needs to identify which executor is running the agent, and send it to the right executor to make it work. Instead of doing this, we can fan out the cancelation event that each executor subscribes to. 

The change of execution status can be done on the Rest API layer.

Diff:
diff --git a/autogpt_platform/backend/backend/executor/manager.py b/autogpt_platform/backend/backend/executor/manager.py
index 9596434806ad..7f93a8436aaa 100644
--- a/autogpt_platform/backend/backend/executor/manager.py
+++ b/autogpt_platform/backend/backend/executor/manager.py
@@ -5,11 +5,15 @@
 import signal
 import sys
 import threading
+import time
 from concurrent.futures import Future, ProcessPoolExecutor
 from contextlib import contextmanager
 from multiprocessing.pool import AsyncResult, Pool
 from typing import TYPE_CHECKING, Any, Generator, Optional, TypeVar, cast
 
+from pika.adapters.blocking_connection import BlockingChannel
+from pika.spec import Basic
+from pydantic import BaseModel
 from redis.lock import Lock as RedisLock
 
 from backend.blocks.io import AgentOutputBlock
@@ -20,6 +24,13 @@
     NotificationEventDTO,
     NotificationType,
 )
+from backend.data.rabbitmq import (
+    Exchange,
+    ExchangeType,
+    Queue,
+    RabbitMQConfig,
+    SyncRabbitMQ,
+)
 from backend.util.exceptions import InsufficientBalanceError
 
 if TYPE_CHECKING:
@@ -454,6 +465,51 @@ def validate_exec(
     return data, node_block.name
 
 
+GRAPH_EXECUTION_EXCHANGE = Exchange(
+    name="graph_execution",
+    type=ExchangeType.DIRECT,
+    durable=True,
+    auto_delete=False,
+)
+GRAPH_EXECUTION_QUEUE_NAME = "graph_execution_queue"
+GRAPH_EXECUTION_ROUTING_KEY = "graph_execution.run"
+
+GRAPH_EXECUTION_CANCEL_EXCHANGE = Exchange(
+    name="graph_execution_cancel",
+    type=ExchangeType.FANOUT,
+    durable=True,
+    auto_delete=True,
+)
+GRAPH_EXECUTION_CANCEL_QUEUE_NAME = "graph_execution_cancel_queue"
+
+
+def create_execution_config() -> RabbitMQConfig:
+    """
+    Define two exchanges and queues:
+    - 'graph_execution' (DIRECT) for run tasks.
+    - 'graph_execution_cancel' (FANOUT) for cancel requests.
+    """
+    run_queue = Queue(
+        name=GRAPH_EXECUTION_QUEUE_NAME,
+        exchange=GRAPH_EXECUTION_EXCHANGE,
+        routing_key=GRAPH_EXECUTION_ROUTING_KEY,
+        durable=True,
+        auto_delete=False,
+    )
+    cancel_queue = Queue(
+        name=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+        exchange=GRAPH_EXECUTION_CANCEL_EXCHANGE,
+        routing_key="",  # not used for FANOUT
+        durable=True,
+        auto_delete=False,
+    )
+    return RabbitMQConfig(
+        vhost="/",
+        exchanges=[GRAPH_EXECUTION_EXCHANGE, GRAPH_EXECUTION_CANCEL_EXCHANGE],
+        queues=[run_queue, cancel_queue],
+    )
+
+
 class Executor:
     """
     This class contains event handlers for the process pool executor events.
@@ -927,12 +983,18 @@ def _handle_low_balance_notif(
         )
 
 
+class CancelExecutionEvent(BaseModel):
+    graph_exec_id: str
+
+
 class ExecutionManager(AppService):
     def __init__(self):
         super().__init__()
         self.pool_size = settings.config.num_graph_workers
-        self.queue = ExecutionQueue[GraphExecutionEntry]()
-        self.active_graph_runs: dict[str, tuple[Future, threading.Event]] = {}
+        self.rabbit_config = create_execution_config()
+        self.rabbitmq_service = SyncRabbitMQ(self.rabbit_config)
+        self.running = True
+        self.active_graph_runs: dict[str, tuple[Future, threading.Event, int]] = {}
 
     @classmethod
     def get_port(cls) -> int:
@@ -943,6 +1005,10 @@ def run_service(self):
 
         self.credentials_store = IntegrationCredentialsStore()
 
+        logger.info(f"[{self.service_name}] ⏳ Connecting to RabbitMQ...")
+        self.rabbitmq_service.connect()
+        channel = self.rabbitmq_service.get_channel()
+
         logger.info(f"[{self.service_name}] ⏳ Spawn max-{self.pool_size} workers...")
         self.executor = ProcessPoolExecutor(
             max_workers=self.pool_size,
@@ -952,25 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            request = CancelExecutionEvent.model_validate_json(body)
+            graph_exec_id = request.graph_exec_id
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(
+        self, channel: BlockingChannel, method_frame: Basic.GetOk, body: bytes
+    ):
+        delivery_tag = method_frame.delivery_tag
+        try:
+            graph_exec_entry = GraphExecutionEntry.model_validate_json(body)
+        except Exception as e:
+            logger.error(f"[{self.service_name}] Could not parse run message: {e}")
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        graph_exec_id = graph_exec_entry.graph_exec_id
+        logger.info(
+            f"[{self.service_name}] Received RUN for graph_exec_id={graph_exec_id}"
+        )
+        if graph_exec_id in self.active_graph_runs:
+            logger.warning(
+                f"[{self.service_name}] Graph {graph_exec_id} already running; rejecting duplicate run."
             )
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        cancel_event = multiprocessing.Manager().Event()
+        future = self.executor.submit(
+            Executor.on_graph_execution, graph_exec_entry, cancel_event
+        )
+        self.active_graph_runs[graph_exec_id] = (future, cancel_event, delivery_tag)
+
+        def _on_run_done(f: Future):
+            logger.info(f"[{self.service_name}] Run completed for {graph_exec_id}")
+            info = self.active_graph_runs.pop(graph_exec_id, None)
+            if not info:
+                return
+            _, _, delivery_tag = info
+            if future.exception():
+                logger.error(
+                    f"[{self.service_name}] Execution for {graph_exec_id} failed: {future.exception()}"
+                )
+                channel.basic_nack(delivery_tag, requeue=False)
+            else:
+                channel.basic_ack(delivery_tag)
+
+        future.add_done_callback(_on_run_done)
 
     def cleanup(self):
         super().cleanup()
 
+        logger.info(f"[{self.service_name}] ⏳ Shutting down service loop...")
+        self.running = False
+
+        logger.info(f"[{self.service_name}] ⏳ Shutting down graph executor pool...")
+        self.executor.shutdown(cancel_futures=True)
+
+        logger.info(f"[{self.service_name}] ⏳ Disconnecting Redis...")
+        redis.disconnect()
+
+        logger.info(f"[{self.service_name}] ⏳ Disconnecting RabbitMQ...")
+        self.rabbitmq_service.disconnect()
+
         logger.info(f"[{self.service_name}] ⏳ Shutting down graph executor pool...")
         self.executor.shutdown(cancel_futures=True)
 
@@ -1064,8 +1217,11 @@ def add_execution(
                 for node_exec in graph_exec.node_executions
             ],
         )
-        self.queue.add(graph_exec_entry)
-
+        self.rabbitmq_service.publish_message(
+            routing_key=GRAPH_EXECUTION_ROUTING_KEY,
+            message=graph_exec_entry.model_dump_json(),
+            exchange=GRAPH_EXECUTION_EXCHANGE,
+        )
         return graph_exec_entry
 
     @expose
@@ -1077,16 +1233,11 @@ def cancel_execution(self, graph_exec_id: str) -> None:
            reinitializes worker pool, and returns.
         3. Update execution statuses in DB and set `error` outputs to `"TERMINATED"`.
         """
-        if graph_exec_id not in self.active_graph_runs:
-            logger.warning(
-                f"Graph execution #{graph_exec_id} not active/running: "
-                "possibly already completed/cancelled."
-            )
-        else:
-            future, cancel_event = self.active_graph_runs[graph_exec_id]
-            if not cancel_event.is_set():
-                cancel_event.set()
-                future.result()
+        self.rabbitmq_service.publish_message(
+            routing_key="",
+            message=CancelExecutionEvent(graph_exec_id=graph_exec_id).model_dump_json(),
+            exchange=GRAPH_EXECUTION_CANCEL_EXCHANGE,
+        )
 
         # Update the status of the graph & node executions
         self.db_client.update_graph_execution_stats(
diff --git a/autogpt_platform/backend/backend/util/json.py b/autogpt_platform/backend/backend/util/json.py
index 106d79b99dfe..d71b61a246df 100644
--- a/autogpt_platform/backend/backend/util/json.py
+++ b/autogpt_platform/backend/backend/util/json.py
@@ -15,21 +15,25 @@ def to_dict(data) -> dict:
 
 
 def dumps(data) -> str:
-    return json.dumps(jsonable_encoder(data))
+    return json.dumps(to_dict(data))
 
 
 T = TypeVar("T")
 
 
 @overload
-def loads(data: str, *args, target_type: Type[T], **kwargs) -> T: ...
+def loads(data: str | bytes, *args, target_type: Type[T], **kwargs) -> T: ...
 
 
 @overload
-def loads(data: str, *args, **kwargs) -> Any: ...
+def loads(data: str | bytes, *args, **kwargs) -> Any: ...
 
 
-def loads(data: str, *args, target_type: Type[T] | None = None, **kwargs) -> Any:
+def loads(
+    data: str | bytes, *args, target_type: Type[T] | None = None, **kwargs
+) -> Any:
+    if isinstance(data, bytes):
+        data = data.decode("utf-8")
     parsed = json.loads(data, *args, **kwargs)
     if target_type:
         return type_match(parsed, target_type)
diff --git a/autogpt_platform/backend/backend/util/metrics.py b/autogpt_platform/backend/backend/util/metrics.py
index f6a2ab567678..06b0ce776e59 100644
--- a/autogpt_platform/backend/backend/util/metrics.py
+++ b/autogpt_platform/backend/backend/util/metrics.py
@@ -14,9 +14,7 @@ def sentry_init():
         traces_sample_rate=1.0,
         profiles_sample_rate=1.0,
         environment=f"app:{Settings().config.app_env.value}-behave:{Settings().config.behave_as.value}",
-        _experiments={
-            "enable_logs": True,
-        },
+        _experiments={"enable_logs": True},
         integrations=[
             LoggingIntegration(sentry_logs_level=logging.INFO),
             AnthropicIntegration(


Reviews:
user: login: Pwuts
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T13:54:31Z
user: login: Pwuts
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T13:55:06Z
user: login: Pwuts
body: - #7750
- #9746

Does this PR fix either/both of these issues? :)
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T13:59:03Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:15:24Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:15:37Z
user: login: Pwuts
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T11:40:25Z
user: login: Pwuts
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-04-07T11:41:55Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T16:57:11Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T16:57:55Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T16:58:53Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T17:03:05Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T17:03:36Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T19:52:28Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:06:39Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:07:40Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:08:29Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:31:21Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T14:00:25Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T14:01:20Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T14:02:03Z
user: login: ntindle
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-11T14:03:05Z

Review comments:
diff_hunk: @@ -952,31 +1011,110 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            graph_exec_id = json.loads(body).get("graph_exec_id")
path: autogpt_platform/backend/backend/executor/manager.py
user: login: Pwuts
body: Please define and use models for the message bodies
created_at: 2025-04-04T13:54:31Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1011,110 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            graph_exec_id = json.loads(body).get("graph_exec_id")
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(self, channel, method_frame, body: bytes):
path: autogpt_platform/backend/backend/executor/manager.py
user: login: Pwuts
body: untyped parameters
created_at: 2025-04-04T13:55:06Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1011,110 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            graph_exec_id = json.loads(body).get("graph_exec_id")
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: Updated
created_at: 2025-04-04T16:15:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -952,31 +1011,110 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            graph_exec_id = json.loads(body).get("graph_exec_id")
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(self, channel, method_frame, body: bytes):
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: Updated
created_at: 2025-04-04T16:15:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1064,8 +1211,11 @@ def add_execution(
                 for node_exec in graph_exec.node_executions
             ],
         )
-        self.queue.add(graph_exec_entry)
-
+        self.rabbitmq_service.publish_message(
+            routing_key=GRAPH_EXECUTION_ROUTING_KEY,
+            message=json.dumps(graph_exec_entry),
path: autogpt_platform/backend/backend/executor/manager.py
user: login: Pwuts
body: ```suggestion
            message=graph_exec_entry.model_dump_json(),
```
created_at: 2025-04-07T11:40:24Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: nit: document that auto_ack is a shittly named arg
created_at: 2025-04-07T16:57:11Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: are both channels using the saem frame?
created_at: 2025-04-07T16:57:55Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            request = CancelExecutionEvent.model_validate_json(body)
+            graph_exec_id = request.graph_exec_id
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(
+        self, channel: BlockingChannel, method_frame: Basic.GetOk, body: bytes
+    ):
+        delivery_tag = method_frame.delivery_tag
+        try:
+            graph_exec_entry = GraphExecutionEntry.model_validate_json(body)
+        except Exception as e:
+            logger.error(f"[{self.service_name}] Could not parse run message: {e}")
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        graph_exec_id = graph_exec_entry.graph_exec_id
+        logger.info(
+            f"[{self.service_name}] Received RUN for graph_exec_id={graph_exec_id}"
+        )
+        if graph_exec_id in self.active_graph_runs:
+            logger.warning(
+                f"[{self.service_name}] Graph {graph_exec_id} already running; rejecting duplicate run."
             )
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        cancel_event = multiprocessing.Manager().Event()
+        future = self.executor.submit(
+            Executor.on_graph_execution, graph_exec_entry, cancel_event
+        )
+        self.active_graph_runs[graph_exec_id] = (future, cancel_event, delivery_tag)
+
+        def _on_run_done(f: Future):
+            logger.info(f"[{self.service_name}] Run completed for {graph_exec_id}")
+            info = self.active_graph_runs.pop(graph_exec_id, None)
+            if not info:
+                return
+            _, _, delivery_tag = info
+            if future.exception():
+                logger.error(
+                    f"[{self.service_name}] Execution for {graph_exec_id} failed: {future.exception()}"
+                )
+                channel.basic_nack(delivery_tag, requeue=False)
+            else:
+                channel.basic_ack(delivery_tag)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: the queue could be dead here and your nack or ack could fail
created_at: 2025-04-07T16:58:53Z
author_association: MEMBER
diff_hunk: @@ -454,6 +465,51 @@ def validate_exec(
     return data, node_block.name
 
 
+GRAPH_EXECUTION_EXCHANGE = Exchange(
+    name="graph_execution",
+    type=ExchangeType.DIRECT,
+    durable=True,
+    auto_delete=False,
+)
+GRAPH_EXECUTION_QUEUE_NAME = "graph_execution_queue"
+GRAPH_EXECUTION_ROUTING_KEY = "graph_execution.run"
+
+GRAPH_EXECUTION_CANCEL_EXCHANGE = Exchange(
+    name="graph_execution_cancel",
+    type=ExchangeType.FANOUT,
+    durable=True,
+    auto_delete=True,
+)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: Heavily consider topic or header rather than direct. Changing exchange types later sucks

 fanout is good if you want a broadcast notification style. Not sure goal with that
created_at: 2025-04-07T17:03:05Z
author_association: MEMBER
diff_hunk: @@ -1077,16 +1227,11 @@ def cancel_execution(self, graph_exec_id: str) -> None:
            reinitializes worker pool, and returns.
         3. Update execution statuses in DB and set `error` outputs to `"TERMINATED"`.
         """
-        if graph_exec_id not in self.active_graph_runs:
-            logger.warning(
-                f"Graph execution #{graph_exec_id} not active/running: "
-                "possibly already completed/cancelled."
-            )
-        else:
-            future, cancel_event = self.active_graph_runs[graph_exec_id]
-            if not cancel_event.is_set():
-                cancel_event.set()
-                future.result()
+        self.rabbitmq_service.publish_message(
+            routing_key="",
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: is this a required param?
created_at: 2025-04-07T17:03:36Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: What's so shit about it? 🥺 
The name sounds self-explanatory to me.
created_at: 2025-04-08T19:52:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: no, it's just a variable used to read the data
created_at: 2025-04-08T20:06:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            request = CancelExecutionEvent.model_validate_json(body)
+            graph_exec_id = request.graph_exec_id
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(
+        self, channel: BlockingChannel, method_frame: Basic.GetOk, body: bytes
+    ):
+        delivery_tag = method_frame.delivery_tag
+        try:
+            graph_exec_entry = GraphExecutionEntry.model_validate_json(body)
+        except Exception as e:
+            logger.error(f"[{self.service_name}] Could not parse run message: {e}")
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        graph_exec_id = graph_exec_entry.graph_exec_id
+        logger.info(
+            f"[{self.service_name}] Received RUN for graph_exec_id={graph_exec_id}"
+        )
+        if graph_exec_id in self.active_graph_runs:
+            logger.warning(
+                f"[{self.service_name}] Graph {graph_exec_id} already running; rejecting duplicate run."
             )
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        cancel_event = multiprocessing.Manager().Event()
+        future = self.executor.submit(
+            Executor.on_graph_execution, graph_exec_entry, cancel_event
+        )
+        self.active_graph_runs[graph_exec_id] = (future, cancel_event, delivery_tag)
+
+        def _on_run_done(f: Future):
+            logger.info(f"[{self.service_name}] Run completed for {graph_exec_id}")
+            info = self.active_graph_runs.pop(graph_exec_id, None)
+            if not info:
+                return
+            _, _, delivery_tag = info
+            if future.exception():
+                logger.error(
+                    f"[{self.service_name}] Execution for {graph_exec_id} failed: {future.exception()}"
+                )
+                channel.basic_nack(delivery_tag, requeue=False)
+            else:
+                channel.basic_ack(delivery_tag)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: Why could it be dead and how can it be avoided? If a queue can be dead, then the executor breaks and it's the end of the world.
created_at: 2025-04-08T20:07:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1077,16 +1227,11 @@ def cancel_execution(self, graph_exec_id: str) -> None:
            reinitializes worker pool, and returns.
         3. Update execution statuses in DB and set `error` outputs to `"TERMINATED"`.
         """
-        if graph_exec_id not in self.active_graph_runs:
-            logger.warning(
-                f"Graph execution #{graph_exec_id} not active/running: "
-                "possibly already completed/cancelled."
-            )
-        else:
-            future, cancel_event = self.active_graph_runs[graph_exec_id]
-            if not cancel_event.is_set():
-                cancel_event.set()
-                future.result()
+        self.rabbitmq_service.publish_message(
+            routing_key="",
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: yes
created_at: 2025-04-08T20:08:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -454,6 +465,51 @@ def validate_exec(
     return data, node_block.name
 
 
+GRAPH_EXECUTION_EXCHANGE = Exchange(
+    name="graph_execution",
+    type=ExchangeType.DIRECT,
+    durable=True,
+    auto_delete=False,
+)
+GRAPH_EXECUTION_QUEUE_NAME = "graph_execution_queue"
+GRAPH_EXECUTION_ROUTING_KEY = "graph_execution.run"
+
+GRAPH_EXECUTION_CANCEL_EXCHANGE = Exchange(
+    name="graph_execution_cancel",
+    type=ExchangeType.FANOUT,
+    durable=True,
+    auto_delete=True,
+)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: I think for this use case we should avoid complicated filtering of the routing and should stick to having a queue of entries that is load balanced by executors. 

> Changing exchange types later sucks

Why is that so? Why can't we just introduce a new Exchange with different name & type then push ?

> fanout is good if you want a broadcast notification style. Not sure goal with that

It's obviously not the right use for execution queue, but intended for cancel execution, since all executors need to check if the execution is present on their side and cancel it accordingly.
created_at: 2025-04-08T20:31:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
             )
-            cancel_event = sync_manager.Event()
-            future = self.executor.submit(
-                Executor.on_graph_execution, graph_exec_data, cancel_event
+            if method_frame:
+                self._handle_cancel_message(body)
+
+            # start graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_QUEUE_NAME,
+                auto_ack=False,
             )
-            self.active_graph_runs[graph_exec_id] = (future, cancel_event)
-            future.add_done_callback(
-                lambda _: self.active_graph_runs.pop(graph_exec_id, None)
+            if method_frame:
+                self._handle_run_message(channel, method_frame, body)
+            else:
+                time.sleep(0.1)
+
+    def _handle_cancel_message(self, body: bytes):
+        try:
+            request = CancelExecutionEvent.model_validate_json(body)
+            graph_exec_id = request.graph_exec_id
+            if not graph_exec_id:
+                logger.warning(
+                    f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
+                )
+                return
+            if graph_exec_id not in self.active_graph_runs:
+                logger.debug(
+                    f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
+                )
+                return
+
+            _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
+            logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
+            if not cancel_event.is_set():
+                cancel_event.set()
+            else:
+                logger.debug(
+                    f"[{self.service_name}] Cancel already set for {graph_exec_id}"
+                )
+
+        except Exception as e:
+            logger.exception(f"Error handling cancel message: {e}")
+
+    def _handle_run_message(
+        self, channel: BlockingChannel, method_frame: Basic.GetOk, body: bytes
+    ):
+        delivery_tag = method_frame.delivery_tag
+        try:
+            graph_exec_entry = GraphExecutionEntry.model_validate_json(body)
+        except Exception as e:
+            logger.error(f"[{self.service_name}] Could not parse run message: {e}")
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        graph_exec_id = graph_exec_entry.graph_exec_id
+        logger.info(
+            f"[{self.service_name}] Received RUN for graph_exec_id={graph_exec_id}"
+        )
+        if graph_exec_id in self.active_graph_runs:
+            logger.warning(
+                f"[{self.service_name}] Graph {graph_exec_id} already running; rejecting duplicate run."
             )
+            channel.basic_nack(delivery_tag, requeue=False)
+            return
+
+        cancel_event = multiprocessing.Manager().Event()
+        future = self.executor.submit(
+            Executor.on_graph_execution, graph_exec_entry, cancel_event
+        )
+        self.active_graph_runs[graph_exec_id] = (future, cancel_event, delivery_tag)
+
+        def _on_run_done(f: Future):
+            logger.info(f"[{self.service_name}] Run completed for {graph_exec_id}")
+            info = self.active_graph_runs.pop(graph_exec_id, None)
+            if not info:
+                return
+            _, _, delivery_tag = info
+            if future.exception():
+                logger.error(
+                    f"[{self.service_name}] Execution for {graph_exec_id} failed: {future.exception()}"
+                )
+                channel.basic_nack(delivery_tag, requeue=False)
+            else:
+                channel.basic_ack(delivery_tag)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: The queue can be configured to be durable so it will come back up with the same content, just worth thinking about this could throw an exception 

the service could go down between when you pop you message and when you decide to keep or reject it and it’ll leave your message hanging 
created_at: 2025-04-11T14:00:25Z
author_association: MEMBER
diff_hunk: @@ -952,31 +1018,112 @@ def run_service(self):
         logger.info(f"[{self.service_name}] ⏳ Connecting to Redis...")
         redis.connect()
 
-        sync_manager = multiprocessing.Manager()
+        logger.info(f"[{self.service_name}] Ready to consume messages...")
         while True:
-            graph_exec_data = self.queue.get()
-            graph_exec_id = graph_exec_data.graph_exec_id
-            logger.debug(
-                f"[ExecutionManager] Dispatching graph execution {graph_exec_id}"
+            # cancel graph execution requests
+            method_frame, _, body = channel.basic_get(
+                queue=GRAPH_EXECUTION_CANCEL_QUEUE_NAME,
+                auto_ack=True,
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: It’s behavior is weird compared to the name acto_ack being true means you have to manually ack or nack which is crazy
created_at: 2025-04-11T14:01:20Z
author_association: MEMBER
diff_hunk: @@ -454,6 +465,51 @@ def validate_exec(
     return data, node_block.name
 
 
+GRAPH_EXECUTION_EXCHANGE = Exchange(
+    name="graph_execution",
+    type=ExchangeType.DIRECT,
+    durable=True,
+    auto_delete=False,
+)
+GRAPH_EXECUTION_QUEUE_NAME = "graph_execution_queue"
+GRAPH_EXECUTION_ROUTING_KEY = "graph_execution.run"
+
+GRAPH_EXECUTION_CANCEL_EXCHANGE = Exchange(
+    name="graph_execution_cancel",
+    type=ExchangeType.FANOUT,
+    durable=True,
+    auto_delete=True,
+)
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: Sounds good just want to make sure we like what we have
created_at: 2025-04-11T14:02:03Z
author_association: MEMBER

PR comments:
user: login: qodo-code-review[bot]
created_at: 2025-04-04T12:38:27Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 3 🔵🔵🔵⚪⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9759/files#diff-1b278ebf10a9da0fb5030010222b3a6df2b05a5463cad428cd6c38a1541b0f73R1082-R1096'><strong>Message Acknowledgment</strong></a>

The message acknowledgment logic in _handle_run_message could lead to message loss. If the system crashes after processing but before the _on_run_done callback executes, the message will neither be acknowledged nor requeued.
</summary>

```python
def _on_run_done(f: Future):
    logger.info(f"[{self.service_name}] Run completed for {graph_exec_id}")
    info = self.active_graph_runs.pop(graph_exec_id, None)
    if not info:
        return
    _, _, delivery_tag = info
    if future.exception():
        logger.error(
            f"[{self.service_name}] Execution for {graph_exec_id} failed: {future.exception()}"
        )
        channel.basic_nack(delivery_tag, requeue=False)
    else:
        channel.basic_ack(delivery_tag)

future.add_done_callback(_on_run_done)
```

</details>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9759/files#diff-1b278ebf10a9da0fb5030010222b3a6df2b05a5463cad428cd6c38a1541b0f73R1029-R1054'><strong>Error Handling</strong></a>

The _handle_cancel_message method silently continues execution when encountering malformed messages. Consider adding more robust error handling to log these issues properly.
</summary>

```python
def _handle_cancel_message(self, body: bytes):
    try:
        graph_exec_id = json.loads(body).get("graph_exec_id")
        if not graph_exec_id:
            logger.warning(
                f"[{self.service_name}] Cancel message missing 'graph_exec_id'"
            )
            return
        if graph_exec_id not in self.active_graph_runs:
            logger.debug(
                f"[{self.service_name}] Cancel received for {graph_exec_id} but not active."
            )
            return

        _, cancel_event, _ = self.active_graph_runs[graph_exec_id]
        logger.info(f"[{self.service_name}] Received cancel for {graph_exec_id}")
        if not cancel_event.is_set():
            cancel_event.set()
        else:
            logger.debug(
                f"[{self.service_name}] Cancel already set for {graph_exec_id}"
            )

    except Exception as e:
        logger.exception(f"Error handling cancel message: {e}")

```

</details>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9759/files#diff-0e7d1e8d0b8b0ec83fd6855473450d8cebf14ba717d017bb3f90be52ff77269fR17-R18'><strong>Potential Bug</strong></a>

The dumps function now calls to_dict but the implementation might not be equivalent to the previous jsonable_encoder call, which could cause serialization differences.
</summary>

```python
def dumps(data) -> str:
    return json.dumps(to_dict(data))
```

</details>

</td></tr>
</table>

author_association: NONE
user: login: deepsource-io[bot]
created_at: 2025-04-04T12:38:51Z
body: <div><p>Here's the code health analysis summary for commits <code>2ca18d7..573e9b9</code>. <a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/b64ed98d-69f8-4bdd-bc03-43aac6909e47/">View details on DeepSource</a>&nbsp;↗.</p><h3>Analysis Summary</h3><table><thead><tr><th>Analyzer</th><th>Status</th><th>Summary</th><th>Link</th></tr></thead><tbody><tr><td><img src="https://static.deepsource.com/analyzer_logos/javascript.svg" alt="DeepSource JavaScript Logo" width="16px" height="16px"/><strong>JavaScript</strong></td><td><span>✅&nbsp;</span><span>Success</span></td><td></td><td><a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/b64ed98d-69f8-4bdd-bc03-43aac6909e47/javascript/">View Check</a>&nbsp;↗</td></tr><tr><td><img src="https://static.deepsource.com/analyzer_logos/python.svg" alt="DeepSource Python Logo" width="16px" height="16px"/><strong>Python</strong></td><td><span>✅&nbsp;</span><span>Success</span></td><td><div>❗ 18 occurences introduced</div><div>🎯 3 occurences resolved</div></td><td><a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/b64ed98d-69f8-4bdd-bc03-43aac6909e47/python/">View Check</a>&nbsp;↗</td></tr></tbody></table><hr/><blockquote><div>💡 If you’re a repository administrator, you can configure the quality gates from the <a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/settings/reporting">settings</a>.</div></blockquote></div>

author_association: NONE
user: login: majdyz
created_at: 2025-04-04T16:16:09Z
body: @Pwuts idk about the first one, but it addresses the second one.
author_association: CONTRIBUTOR
user: login: Pwuts
created_at: 2025-04-07T11:34:38Z
body: Does the persistence of the queue mean existing unfinished graph executions will be picked back up when the system is restarted?
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-04-08T20:11:41Z
body: This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
author_association: CONTRIBUTOR
user: login: netlify[bot]
created_at: 2025-04-08T20:16:03Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *auto-gpt-docs-dev* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 573e9b9f4dcfb534192e5afd423764a7cf3c5da5 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/67f7ed8424cb5d0008278276 |
author_association: NONE
user: login: netlify[bot]
created_at: 2025-04-08T20:16:50Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *auto-gpt-docs* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 573e9b9f4dcfb534192e5afd423764a7cf3c5da5 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/auto-gpt-docs/deploys/67f7ed848f24190008e1ed2c |
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-08T20:17:34Z
body: Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
author_association: CONTRIBUTOR
user: login: majdyz
created_at: 2025-04-09T04:37:18Z
body: > Does the persistence of the queue mean existing unfinished graph executions will be picked back up when the system is restarted?

@Pwuts yes, it should be this way.
author_association: CONTRIBUTOR
user: login: Pwuts
created_at: 2025-04-09T10:01:55Z
body: So does it resume or try to start over? We should probably test this scenario.
author_association: MEMBER
user: login: majdyz
created_at: 2025-04-09T11:33:08Z
body: > So does it resume or try to start over? We should probably test this scenario.

Start over. Unless the message is acked/nacked, when the consuming worker is disconnected, the message will be turned back to the queue. Letting another worker pick it.

Do you think the testing would be a blocker to getting this in?
I can follow up with the test with my other change on removing RPC from the execution manager service. I don't expect this feature as the main objective of this PR, the main objective is to have the message persisted while our app keeps crashing.
author_association: CONTRIBUTOR
user: login: Pwuts
created_at: 2025-04-09T13:59:08Z
body: > Do you think the testing would be a blocker to getting this in?

No, not necessarily. For me it depends on the potential for data corruption or unexpected crashes if we try to re-start a graph execution that already has partially run. From scanning through the updated `executor/manager.py` it seems that it'll work but with a dangling uncompleted execution tree:
- Execution stats are reset
  - Cost incurred in earlier incomplete run is invisible to user
- Executor starts again with the starting nodes which branches into a new execution tree
- Old execution tree is still in DB and will be included in results queries (not great not terrible)
- Anything that was already done during the first run is done again (potentially problematic)

In any case, once this PR is merged I think it'll be fairly easy to properly implement execution resumption/recovery, which is good:
- #7750
author_association: MEMBER
user: login: ntindle
created_at: 2025-04-11T14:03:28Z
body: Make sure durable modes are enabled if desired
author_association: MEMBER