Repository: Significant-Gravitas/AutoGPT
Pull request #9534: feat(backend): low balance notiifcation
Author: login: ntindle
Labels: name: size/xl
name: platform/backend
description: AutoGPT Platform - Back end
name: platform/blocks
name: Review effort 4/5
Created: 2025-02-27T07:43:59Z | Merged: 2025-03-05T05:35:55Z

<!-- Clearly explain the need for these changes: -->
For emailing, we want the user to know when an agent stopped because their balance was too low. This is the first step of that. 

### Changes 🏗️
- Raise InsufficientBalanceError from credit system rather than value error when user runs out of money
- Handle when an agent output isn't hooked up well
- Fix the contents of the email for low balance to be a bit more aligned with the PRD
- expose the topup intent from the db manager
- objectify the execution stats so we can pass it around a bit more type safe
- extract the notification stuff in manager into a function
<!-- Concisely describe all of the changes made in this pull request: -->

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
  - [x] Set balance to $0.01
  - [x] Run an agent that costs something more than $0.01
  - [x] Check you get an email
  - [x] Check your top up link works



Diff:
diff --git a/autogpt_platform/backend/backend/blocks/llm.py b/autogpt_platform/backend/backend/blocks/llm.py
index ee63d42e7575..2ec54b6d6e83 100644
--- a/autogpt_platform/backend/backend/blocks/llm.py
+++ b/autogpt_platform/backend/backend/blocks/llm.py
@@ -8,6 +8,7 @@
 
 from pydantic import BaseModel, SecretStr
 
+from backend.data.model import NodeExecutionStats
 from backend.integrations.providers import ProviderName
 
 if TYPE_CHECKING:
@@ -711,10 +712,10 @@ def parse_response(resp: str) -> tuple[dict[str, Any], str | None]:
                 )
                 response_text = llm_response.response
                 self.merge_stats(
-                    {
-                        "input_token_count": llm_response.prompt_tokens,
-                        "output_token_count": llm_response.completion_tokens,
-                    }
+                    NodeExecutionStats(
+                        input_token_count=llm_response.prompt_tokens,
+                        output_token_count=llm_response.completion_tokens,
+                    )
                 )
                 logger.info(f"LLM attempt-{retry_count} response: {response_text}")
 
@@ -757,10 +758,10 @@ def parse_response(resp: str) -> tuple[dict[str, Any], str | None]:
                 retry_prompt = f"Error calling LLM: {e}"
             finally:
                 self.merge_stats(
-                    {
-                        "llm_call_count": retry_count + 1,
-                        "llm_retry_count": retry_count,
-                    }
+                    NodeExecutionStats(
+                        llm_call_count=retry_count + 1,
+                        llm_retry_count=retry_count,
+                    )
                 )
 
         raise RuntimeError(retry_prompt)
diff --git a/autogpt_platform/backend/backend/data/block.py b/autogpt_platform/backend/backend/data/block.py
index 2509dcd66476..941d459edf1d 100644
--- a/autogpt_platform/backend/backend/data/block.py
+++ b/autogpt_platform/backend/backend/data/block.py
@@ -19,6 +19,7 @@
 from prisma.models import AgentBlock
 from pydantic import BaseModel
 
+from backend.data.model import NodeExecutionStats
 from backend.util import json
 from backend.util.settings import Config
 
@@ -316,7 +317,7 @@ def __init__(
         self.static_output = static_output
         self.block_type = block_type
         self.webhook_config = webhook_config
-        self.execution_stats = {}
+        self.execution_stats: NodeExecutionStats = NodeExecutionStats()
 
         if self.webhook_config:
             if isinstance(self.webhook_config, BlockWebhookConfig):
@@ -394,18 +395,29 @@ def run_once(self, input_data: BlockSchemaInputType, output: str, **kwargs) -> A
                 return data
         raise ValueError(f"{self.name} did not produce any output for {output}")
 
-    def merge_stats(self, stats: dict[str, Any]) -> dict[str, Any]:
-        for key, value in stats.items():
-            if isinstance(value, dict):
-                self.execution_stats.setdefault(key, {}).update(value)
-            elif isinstance(value, (int, float)):
-                self.execution_stats.setdefault(key, 0)
-                self.execution_stats[key] += value
-            elif isinstance(value, list):
-                self.execution_stats.setdefault(key, [])
-                self.execution_stats[key].extend(value)
+    def merge_stats(self, stats: NodeExecutionStats) -> NodeExecutionStats:
+        stats_dict = stats.model_dump()
+        current_stats = self.execution_stats.model_dump()
+
+        for key, value in stats_dict.items():
+            if key not in current_stats:
+                # Field doesn't exist yet, just set it, but this will probably
+                # not happen, just in case though so we throw for invalid when
+                # converting back in
+                current_stats[key] = value
+            elif isinstance(value, dict) and isinstance(current_stats[key], dict):
+                current_stats[key].update(value)
+            elif isinstance(value, (int, float)) and isinstance(
+                current_stats[key], (int, float)
+            ):
+                current_stats[key] += value
+            elif isinstance(value, list) and isinstance(current_stats[key], list):
+                current_stats[key].extend(value)
             else:
-                self.execution_stats[key] = value
+                current_stats[key] = value
+
+        self.execution_stats = NodeExecutionStats(**current_stats)
+
         return self.execution_stats
 
     @property
diff --git a/autogpt_platform/backend/backend/data/credit.py b/autogpt_platform/backend/backend/data/credit.py
index db422531c41f..452b60d75079 100644
--- a/autogpt_platform/backend/backend/data/credit.py
+++ b/autogpt_platform/backend/backend/data/credit.py
@@ -32,6 +32,7 @@
 from backend.data.notifications import NotificationEventDTO, RefundRequestData
 from backend.data.user import get_user_by_id
 from backend.notifications import NotificationManager
+from backend.util.exceptions import InsufficientBalanceError
 from backend.util.service import get_service_client
 from backend.util.settings import Settings
 
@@ -313,9 +314,13 @@ async def _add_transaction(
 
             if amount < 0 and user_balance + amount < 0:
                 if fail_insufficient_credits:
-                    raise ValueError(
-                        f"Insufficient balance of ${user_balance/100}, where this will cost ${abs(amount)/100}"
+                    raise InsufficientBalanceError(
+                        message=f"Insufficient balance of ${user_balance/100}, where this will cost ${abs(amount)/100}",
+                        user_id=user_id,
+                        balance=user_balance,
+                        amount=amount,
                     )
+
                 amount = min(-user_balance, 0)
 
             # Create the transaction
diff --git a/autogpt_platform/backend/backend/data/execution.py b/autogpt_platform/backend/backend/data/execution.py
index 57dba68ab79e..ca6926ee1c97 100644
--- a/autogpt_platform/backend/backend/data/execution.py
+++ b/autogpt_platform/backend/backend/data/execution.py
@@ -15,6 +15,7 @@
 
 from backend.data.block import BlockData, BlockInput, CompletedBlockOutput
 from backend.data.includes import EXECUTION_RESULT_INCLUDE, GRAPH_EXECUTION_INCLUDE
+from backend.data.model import GraphExecutionStats, NodeExecutionStats
 from backend.data.queue import AsyncRedisEventBus, RedisEventBus
 from backend.server.v2.store.exceptions import DatabaseError
 from backend.util import mock, type
@@ -282,13 +283,16 @@ async def update_graph_execution_start_time(graph_exec_id: str) -> ExecutionResu
 async def update_graph_execution_stats(
     graph_exec_id: str,
     status: ExecutionStatus,
-    stats: dict[str, Any],
+    stats: GraphExecutionStats,
 ) -> ExecutionResult:
+    data = stats.model_dump()
+    if isinstance(data["error"], Exception):
+        data["error"] = str(data["error"])
     res = await AgentGraphExecution.prisma().update(
         where={"id": graph_exec_id},
         data={
             "executionStatus": status,
-            "stats": Json(stats),
+            "stats": Json(data),
         },
     )
     if not res:
@@ -297,10 +301,13 @@ async def update_graph_execution_stats(
     return ExecutionResult.from_graph(res)
 
 
-async def update_node_execution_stats(node_exec_id: str, stats: dict[str, Any]):
+async def update_node_execution_stats(node_exec_id: str, stats: NodeExecutionStats):
+    data = stats.model_dump()
+    if isinstance(data["error"], Exception):
+        data["error"] = str(data["error"])
     await AgentNodeExecution.prisma().update(
         where={"id": node_exec_id},
-        data={"stats": Json(stats)},
+        data={"stats": Json(data)},
     )
 
 
diff --git a/autogpt_platform/backend/backend/data/graph.py b/autogpt_platform/backend/backend/data/graph.py
index 18e0cb7ee62b..39fdd5b6ed12 100644
--- a/autogpt_platform/backend/backend/data/graph.py
+++ b/autogpt_platform/backend/backend/data/graph.py
@@ -186,7 +186,9 @@ def from_db(_graph_exec: AgentGraphExecution):
         outputs: dict[str, list] = defaultdict(list)
         for exec in node_executions:
             if exec.block_id == _OUTPUT_BLOCK_ID:
-                outputs[exec.input_data["name"]].append(exec.input_data["value"])
+                outputs[exec.input_data["name"]].append(
+                    exec.input_data.get("value", None)
+                )
 
         return GraphExecution(
             **{
diff --git a/autogpt_platform/backend/backend/data/model.py b/autogpt_platform/backend/backend/data/model.py
index b52b13b2f732..b1f3a74dc00b 100644
--- a/autogpt_platform/backend/backend/data/model.py
+++ b/autogpt_platform/backend/backend/data/model.py
@@ -402,3 +402,37 @@ class RefundRequest(BaseModel):
     status: str
     created_at: datetime
     updated_at: datetime
+
+
+class NodeExecutionStats(BaseModel):
+    """Execution statistics for a node execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception | str] = None
+    walltime: float = 0
+    cputime: float = 0
+    cost: float = 0
+    input_size: int = 0
+    output_size: int = 0
+    llm_call_count: int = 0
+    llm_retry_count: int = 0
+    input_token_count: int = 0
+    output_token_count: int = 0
+
+
+class GraphExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception | str] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
diff --git a/autogpt_platform/backend/backend/data/notifications.py b/autogpt_platform/backend/backend/data/notifications.py
index db519e103f7b..43134f0d3a99 100644
--- a/autogpt_platform/backend/backend/data/notifications.py
+++ b/autogpt_platform/backend/backend/data/notifications.py
@@ -49,10 +49,12 @@ class ZeroBalanceData(BaseNotificationData):
 
 
 class LowBalanceData(BaseNotificationData):
-    current_balance: float
-    threshold_amount: float
-    top_up_link: str
-    recent_usage: float = Field(..., description="Usage in the last 24 hours")
+    agent_name: str = Field(..., description="Name of the agent")
+    current_balance: float = Field(
+        ..., description="Current balance in credits (100 = $1)"
+    )
+    billing_page_link: str = Field(..., description="Link to billing page")
+    shortfall: float = Field(..., description="Amount of credits needed to continue")
 
 
 class BlockExecutionFailedData(BaseNotificationData):
@@ -197,7 +199,7 @@ def strategy(self) -> QueueType:
             NotificationType.AGENT_RUN: QueueType.IMMEDIATE,
             # These are batched by the notification service, but with a backoff strategy
             NotificationType.ZERO_BALANCE: QueueType.BACKOFF,
-            NotificationType.LOW_BALANCE: QueueType.BACKOFF,
+            NotificationType.LOW_BALANCE: QueueType.IMMEDIATE,
             NotificationType.BLOCK_EXECUTION_FAILED: QueueType.BACKOFF,
             NotificationType.CONTINUOUS_AGENT_ERROR: QueueType.BACKOFF,
             NotificationType.DAILY_SUMMARY: QueueType.DAILY,
diff --git a/autogpt_platform/backend/backend/executor/manager.py b/autogpt_platform/backend/backend/executor/manager.py
index 4b92903a9b55..2a3100166f42 100644
--- a/autogpt_platform/backend/backend/executor/manager.py
+++ b/autogpt_platform/backend/backend/executor/manager.py
@@ -13,11 +13,14 @@
 from redis.lock import Lock as RedisLock
 
 from backend.blocks.basic import AgentOutputBlock
+from backend.data.model import GraphExecutionStats, NodeExecutionStats
 from backend.data.notifications import (
     AgentRunData,
+    LowBalanceData,
     NotificationEventDTO,
     NotificationType,
 )
+from backend.util.exceptions import InsufficientBalanceError
 
 if TYPE_CHECKING:
     from backend.executor import DatabaseManager
@@ -117,7 +120,7 @@ def execute_node(
     db_client: "DatabaseManager",
     creds_manager: IntegrationCredentialsManager,
     data: NodeExecutionEntry,
-    execution_stats: dict[str, Any] | None = None,
+    execution_stats: NodeExecutionStats | None = None,
 ) -> ExecutionStream:
     """
     Execute a node in the graph. This will trigger a block execution on a node,
@@ -126,7 +129,6 @@ def execute_node(
     Args:
         db_client: The client to send execution updates to the server.
         creds_manager: The manager to acquire and release credentials.
-        notification_service: The service to send notifications.
         data: The execution data for executing the current node.
         execution_stats: The execution statistics to be updated.
 
@@ -256,10 +258,12 @@ def update_execution(status: ExecutionStatus) -> ExecutionResult:
 
         # Update execution stats
         if execution_stats is not None:
-            execution_stats.update(node_block.execution_stats)
-            execution_stats["input_size"] = input_size
-            execution_stats["output_size"] = output_size
-            execution_stats["cost"] = cost
+            execution_stats = execution_stats.model_copy(
+                update=node_block.execution_stats.model_dump()
+            )
+            execution_stats.input_size = input_size
+            execution_stats.output_size = output_size
+            execution_stats.cost = cost
 
 
 def _enqueue_next_nodes(
@@ -476,7 +480,6 @@ def on_node_executor_start(cls):
         cls.pid = os.getpid()
         cls.db_client = get_db_client()
         cls.creds_manager = IntegrationCredentialsManager()
-        cls.notification_service = get_notification_service()
 
         # Set up shutdown handlers
         cls.shutdown_lock = threading.Lock()
@@ -517,7 +520,7 @@ def on_node_execution(
         cls,
         q: ExecutionQueue[NodeExecutionEntry],
         node_exec: NodeExecutionEntry,
-    ) -> dict[str, Any]:
+    ) -> NodeExecutionStats:
         log_metadata = LogMetadata(
             user_id=node_exec.user_id,
             graph_eid=node_exec.graph_exec_id,
@@ -527,13 +530,15 @@ def on_node_execution(
             block_name="-",
         )
 
-        execution_stats = {}
+        execution_stats = NodeExecutionStats()
         timing_info, _ = cls._on_node_execution(
             q, node_exec, log_metadata, execution_stats
         )
-        execution_stats["walltime"] = timing_info.wall_time
-        execution_stats["cputime"] = timing_info.cpu_time
+        execution_stats.walltime = timing_info.wall_time
+        execution_stats.cputime = timing_info.cpu_time
 
+        if isinstance(execution_stats.error, Exception):
+            execution_stats.error = str(execution_stats.error)
         cls.db_client.update_node_execution_stats(
             node_exec.node_exec_id, execution_stats
         )
@@ -546,7 +551,7 @@ def _on_node_execution(
         q: ExecutionQueue[NodeExecutionEntry],
         node_exec: NodeExecutionEntry,
         log_metadata: LogMetadata,
-        stats: dict[str, Any] | None = None,
+        stats: NodeExecutionStats | None = None,
     ):
         try:
             log_metadata.info(f"Start node execution {node_exec.node_exec_id}")
@@ -569,6 +574,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats.error = e
+
     @classmethod
     def on_graph_executor_start(cls):
         configure_logging()
@@ -577,6 +585,7 @@ def on_graph_executor_start(cls):
         cls.db_client = get_db_client()
         cls.pool_size = settings.config.num_node_workers
         cls.pid = os.getpid()
+        cls.notification_service = get_notification_service()
         cls._init_node_executor_pool()
         logger.info(
             f"Graph executor {cls.pid} started with {cls.pool_size} node workers"
@@ -618,9 +627,12 @@ def on_graph_execution(
         timing_info, (exec_stats, status, error) = cls._on_graph_execution(
             graph_exec, cancel, log_metadata
         )
-        exec_stats["walltime"] = timing_info.wall_time
-        exec_stats["cputime"] = timing_info.cpu_time
-        exec_stats["error"] = str(error) if error else None
+        exec_stats.walltime = timing_info.wall_time
+        exec_stats.cputime = timing_info.cpu_time
+        exec_stats.error = error
+
+        if isinstance(exec_stats.error, Exception):
+            exec_stats.error = str(exec_stats.error)
         result = cls.db_client.update_graph_execution_stats(
             graph_exec_id=graph_exec.graph_exec_id,
             status=status,
@@ -628,48 +640,7 @@ def on_graph_execution(
         )
         cls.db_client.send_execution_update(result)
 
-        metadata = cls.db_client.get_graph_metadata(
-            graph_exec.graph_id, graph_exec.graph_version
-        )
-        assert metadata is not None
-        outputs = cls.db_client.get_execution_results(graph_exec.graph_exec_id)
-
-        # Collect named outputs as a list of dictionaries
-        named_outputs = []
-        for output in outputs:
-            if output.block_id == AgentOutputBlock().id:
-                # Create a dictionary for this named output
-                named_output = {
-                    # Include the name as a field in each output
-                    "name": (
-                        output.output_data["name"][0]
-                        if isinstance(output.output_data["name"], list)
-                        else output.output_data["name"]
-                    )
-                }
-
-                # Add all other fields
-                for key, value in output.output_data.items():
-                    if key != "name":
-                        named_output[key] = value
-
-                named_outputs.append(named_output)
-
-        event = NotificationEventDTO(
-            user_id=graph_exec.user_id,
-            type=NotificationType.AGENT_RUN,
-            data=AgentRunData(
-                outputs=named_outputs,
-                agent_name=metadata.name,
-                credits_used=exec_stats["cost"],
-                execution_time=timing_info.wall_time,
-                graph_id=graph_exec.graph_id,
-                node_count=exec_stats["node_count"],
-            ).model_dump(),
-        )
-
-        logger.info(f"Sending notification for {event}")
-        get_notification_service().queue_notification(event)
+        cls._handle_agent_run_notif(graph_exec, exec_stats)
 
     @classmethod
     @time_measured
@@ -678,7 +649,7 @@ def _on_graph_execution(
         graph_exec: GraphExecutionEntry,
         cancel: threading.Event,
         log_metadata: LogMetadata,
-    ) -> tuple[dict[str, Any], ExecutionStatus, Exception | None]:
+    ) -> tuple[GraphExecutionStats, ExecutionStatus, Exception | None]:
         """
         Returns:
             dict: The execution statistics of the graph execution.
@@ -686,12 +657,7 @@ def _on_graph_execution(
             Exception | None: The error that occurred during the execution, if any.
         """
         log_metadata.info(f"Start graph execution {graph_exec.graph_exec_id}")
-        exec_stats = {
-            "nodes_walltime": 0,
-            "nodes_cputime": 0,
-            "node_count": 0,
-            "cost": 0,
-        }
+        exec_stats = GraphExecutionStats()
         error = None
         finished = False
 
@@ -717,18 +683,26 @@ def cancel_handler():
                 queue.add(node_exec)
 
             running_executions: dict[str, AsyncResult] = {}
+            low_balance_error: Optional[InsufficientBalanceError] = None
 
             def make_exec_callback(exec_data: NodeExecutionEntry):
-                node_id = exec_data.node_id
 
                 def callback(result: object):
-                    running_executions.pop(node_id)
-                    nonlocal exec_stats
-                    if isinstance(result, dict):
-                        exec_stats["node_count"] += 1
-                        exec_stats["nodes_cputime"] += result.get("cputime", 0)
-                        exec_stats["nodes_walltime"] += result.get("walltime", 0)
-                        exec_stats["cost"] += result.get("cost", 0)
+                    running_executions.pop(exec_data.node_id)
+
+                    if not isinstance(result, NodeExecutionStats):
+                        return
+
+                    nonlocal exec_stats, low_balance_error
+                    exec_stats.node_count += 1
+                    exec_stats.nodes_cputime += result.cputime
+                    exec_stats.nodes_walltime += result.walltime
+                    exec_stats.cost += result.cost
+                    if (err := result.error) and isinstance(err, Exception):
+                        exec_stats.node_error_count += 1
+
+                        if isinstance(err, InsufficientBalanceError):
+                            low_balance_error = err
 
                 return callback
 
@@ -773,6 +747,16 @@ def callback(result: object):
                         execution.wait(3)
 
             log_metadata.info(f"Finished graph execution {graph_exec.graph_exec_id}")
+
+            if isinstance(low_balance_error, InsufficientBalanceError):
+                cls._handle_low_balance_notif(
+                    graph_exec.user_id,
+                    graph_exec.graph_id,
+                    exec_stats,
+                    low_balance_error,
+                )
+                raise low_balance_error
+
         except Exception as e:
             log_metadata.exception(
                 f"Failed graph execution {graph_exec.graph_exec_id}: {e}"
@@ -791,6 +775,67 @@ def callback(result: object):
             error,
         )
 
+    @classmethod
+    def _handle_agent_run_notif(
+        cls,
+        graph_exec: GraphExecutionEntry,
+        exec_stats: GraphExecutionStats,
+    ):
+        metadata = cls.db_client.get_graph_metadata(
+            graph_exec.graph_id, graph_exec.graph_version
+        )
+        outputs = cls.db_client.get_execution_results(graph_exec.graph_exec_id)
+
+        named_outputs = [
+            {
+                key: value[0] if key == "name" else value
+                for key, value in output.output_data.items()
+            }
+            for output in outputs
+            if output.block_id == AgentOutputBlock().id
+        ]
+
+        event = NotificationEventDTO(
+            user_id=graph_exec.user_id,
+            type=NotificationType.AGENT_RUN,
+            data=AgentRunData(
+                outputs=named_outputs,
+                agent_name=metadata.name if metadata else "Unknown Agent",
+                credits_used=exec_stats.cost,
+                execution_time=exec_stats.walltime,
+                graph_id=graph_exec.graph_id,
+                node_count=exec_stats.node_count,
+            ).model_dump(),
+        )
+
+        cls.notification_service.queue_notification(event)
+
+    @classmethod
+    def _handle_low_balance_notif(
+        cls,
+        user_id: str,
+        graph_id: str,
+        exec_stats: GraphExecutionStats,
+        e: InsufficientBalanceError,
+    ):
+        shortfall = e.balance - e.amount
+        metadata = cls.db_client.get_graph_metadata(graph_id)
+        base_url = (
+            settings.config.frontend_base_url or settings.config.platform_base_url
+        )
+        cls.notification_service.queue_notification(
+            NotificationEventDTO(
+                user_id=user_id,
+                type=NotificationType.LOW_BALANCE,
+                data=LowBalanceData(
+                    current_balance=exec_stats.cost,
+                    billing_page_link=f"{base_url}/profile/credits",
+                    shortfall=shortfall,
+                    agent_name=metadata.name if metadata else "Unknown Agent",
+                ).model_dump(),
+            )
+        )
+
 
 class ExecutionManager(AppService):
     def __init__(self):
diff --git a/autogpt_platform/backend/backend/notifications/templates/low_balance.html.jinja2 b/autogpt_platform/backend/backend/notifications/templates/low_balance.html.jinja2
new file mode 100644
index 000000000000..d39d17cb148e
--- /dev/null
+++ b/autogpt_platform/backend/backend/notifications/templates/low_balance.html.jinja2
@@ -0,0 +1,114 @@
+{# Low Balance Notification Email Template #}
+{# Template variables:
+data.agent_name: the name of the agent
+data.current_balance: the current balance of the user
+data.billing_page_link: the link to the billing page
+data.shortfall: the shortfall amount
+#}
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+  <strong>Low Balance Warning</strong>
+</p>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 20px;
+  ">
+  Your agent "<strong>{{ data.agent_name }}</strong>" has been stopped due to low balance.
+</p>
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #5D23BB;
+    background-color: #f8f8ff;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Current Balance:</strong> ${{ "{:.2f}".format((data.current_balance|float)/100) }}
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Shortfall:</strong> ${{ "{:.2f}".format((data.shortfall|float)/100) }}
+  </p>
+</div>
+
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #FF6B6B;
+    background-color: #FFF0F0;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Low Balance:</strong>
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 5px;
+  ">
+    Your agent "<strong>{{ data.agent_name }}</strong>" requires additional credits to continue running. The current operation has been canceled until your balance is replenished.
+  </p>
+</div>
+
+<div style="
+    text-align: center;
+    margin: 30px 0;
+  ">
+  <a href="{{ data.billing_page_link }}" style="
+    font-family: 'Poppins', sans-serif;
+    background-color: #5D23BB;
+    color: white;
+    padding: 12px 24px;
+    text-decoration: none;
+    border-radius: 4px;
+    font-weight: 500;
+    display: inline-block;
+  ">
+    Manage Billing
+  </a>
+</div>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 150%;
+    margin-top: 30px;
+    margin-bottom: 10px;
+    font-style: italic;
+  ">
+  This is an automated notification. Your agent is stopped and will need manually restarted unless set to trigger automatically.
+</p>
diff --git a/autogpt_platform/backend/backend/util/exceptions.py b/autogpt_platform/backend/backend/util/exceptions.py
index 4bb3a08d9548..eb6e42bda769 100644
--- a/autogpt_platform/backend/backend/util/exceptions.py
+++ b/autogpt_platform/backend/backend/util/exceptions.py
@@ -4,3 +4,22 @@ class MissingConfigError(Exception):
 
 class NeedConfirmation(Exception):
     """The user must explicitly confirm that they want to proceed"""
+
+
+class InsufficientBalanceError(ValueError):
+    user_id: str
+    message: str
+    balance: float
+    amount: float
+
+    def __init__(self, message: str, user_id: str, balance: float, amount: float):
+        super().__init__(message)
+        self.args = (message, user_id, balance, amount)
+        self.message = message
+        self.user_id = user_id
+        self.balance = balance
+        self.amount = amount
+
+    def __str__(self):
+        """Used to display the error message in the frontend, because we str() the error when sending the execution update"""
+        return self.message
diff --git a/autogpt_platform/backend/backend/util/service.py b/autogpt_platform/backend/backend/util/service.py
index e29a9a5d1929..716d1b13f2e5 100644
--- a/autogpt_platform/backend/backend/util/service.py
+++ b/autogpt_platform/backend/backend/util/service.py
@@ -42,6 +42,7 @@
 from Pyro5 import config as pyro_config
 
 from backend.data import db, rabbitmq, redis
+from backend.util.exceptions import InsufficientBalanceError
 from backend.util.json import to_dict
 from backend.util.process import AppProcess
 from backend.util.retry import conn_retry
@@ -251,6 +252,7 @@ class RemoteCallError(BaseModel):
         ValueError,
         TimeoutError,
         ConnectionError,
+        InsufficientBalanceError,
     ]
 }
 
@@ -441,6 +443,7 @@ def _call_method(self, method_name: str, **kwargs) -> Any:
             except httpx.HTTPStatusError as e:
                 logger.error(f"HTTP error in {method_name}: {e.response.text}")
                 error = RemoteCallError.model_validate(e.response.json(), strict=False)
+                # DEBUG HELP: if you made a custom exception, make sure you override self.args to be how to make your exception
                 raise EXCEPTION_MAPPING.get(error.type, Exception)(
                     *(error.args or [str(e)])
                 )
diff --git a/autogpt_platform/backend/backend/util/text.py b/autogpt_platform/backend/backend/util/text.py
index e118325f7e2e..0c1e22edc0af 100644
--- a/autogpt_platform/backend/backend/util/text.py
+++ b/autogpt_platform/backend/backend/util/text.py
@@ -21,46 +21,83 @@ def __init__(self):
         self.env.globals.clear()
 
         # Instead of clearing all filters, just remove potentially unsafe ones
-        unsafe_filters = ["pprint", "urlize", "xmlattr", "tojson"]
+        unsafe_filters = ["pprint", "tojson", "urlize", "xmlattr"]
         for f in unsafe_filters:
             if f in self.env.filters:
                 del self.env.filters[f]
 
         self.env.filters["format"] = format_filter_for_jinja2
 
-        # Define allowed CSS properties
+        # Define allowed CSS properties (sorted alphabetically, if you add more)
         allowed_css_properties = [
-            "font-family",
+            "background-color",
+            "border",
+            "border-bottom",
+            "border-color",
+            "border-left",
+            "border-radius",
+            "border-right",
+            "border-style",
+            "border-top",
+            "border-width",
+            "bottom",
+            "box-shadow",
+            "clear",
             "color",
+            "display",
+            "float",
+            "font-family",
             "font-size",
+            "font-weight",
+            "height",
+            "left",
+            "letter-spacing",
             "line-height",
-            "margin-top",
             "margin-bottom",
             "margin-left",
             "margin-right",
-            "background-color",
+            "margin-top",
             "padding",
-            "border-radius",
-            "font-weight",
+            "position",
+            "right",
             "text-align",
+            "text-decoration",
+            "text-shadow",
+            "text-transform",
+            "top",
+            "width",
         ]
 
         self.css_sanitizer = CSSSanitizer(allowed_css_properties=allowed_css_properties)
 
+        # Define allowed tags (sorted alphabetically, if you add more)
         self.allowed_tags = [
-            "p",
+            "a",
             "b",
-            "i",
-            "u",
-            "ul",
-            "li",
             "br",
-            "strong",
-            "em",
             "div",
+            "em",
+            "h1",
+            "h2",
+            "h3",
+            "h4",
+            "h5",
+            "i",
+            "img",
+            "li",
+            "p",
             "span",
+            "strong",
+            "u",
+            "ul",
         ]
-        self.allowed_attributes = {"*": ["style", "class"]}
+
+        # Define allowed attributes to be used on specific tags
+        self.allowed_attributes = {
+            "*": ["class", "style"],
+            "a": ["href"],
+            "img": ["src"],
+        }
 
     def format_string(self, template_str: str, values=None, **kwargs) -> str:
         """Regular template rendering with escaping"""


Reviews:
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T22:20:09Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T22:21:45Z
user: login: majdyz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T09:20:16Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T11:46:20Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T11:46:47Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T11:47:46Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T11:52:41Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T15:52:37Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T15:56:13Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T19:51:05Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T23:58:18Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T23:58:31Z
user: login: majdyz
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T02:17:15Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T02:33:05Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T04:25:05Z
user: login: ntindle
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T05:04:53Z
user: login: majdyz
body: Thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T05:16:20Z

Review comments:
diff_hunk: @@ -594,6 +581,14 @@ def _on_node_execution(
                 log_metadata.info(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
+                if (
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: this will trigger multiple times
created_at: 2025-02-27T22:20:09Z
author_association: MEMBER
diff_hunk: @@ -763,6 +762,14 @@ def callback(result: object):
                 f"Failed graph execution {graph_exec.graph_exec_id}: {e}"
             )
             error = e
+            if (
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: this one never triggers :(
created_at: 2025-02-27T22:21:45Z
author_association: MEMBER
diff_hunk: @@ -4,3 +4,18 @@ class MissingConfigError(Exception):
 
 class NeedConfirmation(Exception):
     """The user must explicitly confirm that they want to proceed"""
+
+
+class InsufficientBalanceError(ValueError):
+    user_id: str
+    message: str
+    balance: float
+    amount: float
+
+    def __init__(self, message: str, user_id: str, balance: float, amount: float):
+        super().__init__(message)
+        self.args = (message, user_id, balance, amount)
+        self.message = message
+        self.user_id = user_id
+        self.balance = balance
+        self.amount = amount
path: autogpt_platform/backend/backend/util/exceptions.py
user: login: majdyz
body: What does it look like on the block error output?
Does it only show the message only?
What about when printed as an exception?

created_at: 2025-03-04T09:07:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: majdyz
body: Why can't I extend the above class and override the error field
created_at: 2025-03-04T09:08:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: majdyz
body: Also, do we really need two versions of this? Can we just stick to this class only ?
created_at: 2025-03-04T09:09:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: majdyz
body: GraphExecutionStats ? 
created_at: 2025-03-04T09:09:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,6 +78,7 @@ def send_execution_update(self, execution_result: ExecutionResult):
 
     # Credits
     spend_credits = exposed_run_and_wait(_spend_credits)
+    top_up_intent = exposed_run_and_wait(_top_up_intent)
path: autogpt_platform/backend/backend/executor/database.py
user: login: majdyz
body: I don't see where this function is used.
created_at: 2025-03-04T09:11:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -569,6 +570,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats["error"] = e
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: Request: can we make sure the error being persisted here in the db (`AgentNodeExecution` table) is just a message, not a full stacktrace ?
created_at: 2025-03-04T09:13:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -569,6 +570,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats["error"] = e
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: if it's a full stacktrace we should exclude it from the stats dict when persisting.
created_at: 2025-03-04T09:16:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -791,6 +772,72 @@ def callback(result: object):
             error,
         )
 
+    @classmethod
+    def _handle_agent_run_notif(
+        cls,
+        graph_exec: GraphExecutionEntry,
+        exec_stats: ExecutionStats,
+    ):
+        metadata = cls.db_client.get_graph_metadata(
+            graph_exec.graph_id, graph_exec.graph_version
+        )
+        assert metadata is not None
+        outputs = cls.db_client.get_execution_results(graph_exec.graph_exec_id)
+
+        named_outputs = [
+            {
+                key: value[0] if key == "name" else value
+                for key, value in output.output_data.items()
+            }
+            for output in outputs
+            if output.block_id == AgentOutputBlock().id
+        ]
+
+        event = NotificationEventDTO(
+            user_id=graph_exec.user_id,
+            type=NotificationType.AGENT_RUN,
+            data=AgentRunData(
+                outputs=named_outputs,
+                agent_name=metadata.name,
+                credits_used=exec_stats.cost,
+                execution_time=exec_stats.walltime,
+                graph_id=graph_exec.graph_id,
+                node_count=exec_stats.node_count,
+            ).model_dump(),
+        )
+
+        logger.info(
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: we've been logging at `queue_notification`
created_at: 2025-03-04T09:17:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -773,6 +744,16 @@ def callback(result: object):
                         execution.wait(3)
 
             log_metadata.info(f"Finished graph execution {graph_exec.graph_exec_id}")
+
+            if isinstance(low_balance_error, InsufficientBalanceError):
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: Shoudn't this is not None check only ?
created_at: 2025-03-04T09:18:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,114 @@
+{# Low Balance Notification Email Template #}
+{# Template variables:
+data.agent_name: the name of the agent
+data.current_balance: the current balance of the user
+data.billing_page_link: the link to the billing page
+data.shortfall: the shortfall amount
+#}
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+  <strong>Low Balance Warning</strong>
+</p>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 20px;
+  ">
+  Your agent "<strong>{{ data.agent_name }}</strong>" has been stopped due to low balance.
+</p>
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #5D23BB;
+    background-color: #f8f8ff;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Current Balance:</strong> ${{ "{:.2f}".format((data.current_balance|float)/100) }}
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Shortfall:</strong> ${{ "{:.2f}".format((data.shortfall|float)/100) }}
+  </p>
+</div>
+
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #FF6B6B;
+    background-color: #FFF0F0;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Low Balance:</strong>
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 5px;
+  ">
+    Your agent "<strong>{{ data.agent_name }}</strong>" requires additional credits to continue running. The current operation has been canceled until your balance is replenished.
+  </p>
+</div>
+
+<div style="
+    text-align: center;
+    margin: 30px 0;
+  ">
+  <a href="{{ data.billing_page_link }}" style="
+    font-family: 'Poppins', sans-serif;
+    background-color: #5D23BB;
+    color: white;
+    padding: 12px 24px;
+    text-decoration: none;
+    border-radius: 4px;
+    font-weight: 500;
+    display: inline-block;
+  ">
+    Manage Billing
+  </a>
+</div>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 150%;
+    margin-top: 30px;
+    margin-bottom: 10px;
+    font-style: italic;
+  ">
+  This is an automated notification. Your agent is stopped and will need manually restarted unless set to trigger automatically.
+</p>
path: autogpt_platform/backend/backend/notifications/templates/low_balance.html.jinja2
user: login: majdyz
body: new line
created_at: 2025-03-04T09:19:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: ntindle
body: Exception isn't serializable and pydantic will complain about it, figured this was the easiest way without building and registering searlizers while still keeping the data I need for it for handling the errors
created_at: 2025-03-04T11:46:20Z
author_association: MEMBER
diff_hunk: @@ -74,6 +78,7 @@ def send_execution_update(self, execution_result: ExecutionResult):
 
     # Credits
     spend_credits = exposed_run_and_wait(_spend_credits)
+    top_up_intent = exposed_run_and_wait(_top_up_intent)
path: autogpt_platform/backend/backend/executor/database.py
user: login: ntindle
body: Twas removed because the button was swapped to billing. Good spot
created_at: 2025-03-04T11:46:47Z
author_association: MEMBER
diff_hunk: @@ -569,6 +570,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats["error"] = e
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: Why not the whole thing sorry?
created_at: 2025-03-04T11:47:46Z
author_association: MEMBER
diff_hunk: @@ -618,58 +623,21 @@ def on_graph_execution(
         timing_info, (exec_stats, status, error) = cls._on_graph_execution(
             graph_exec, cancel, log_metadata
         )
-        exec_stats["walltime"] = timing_info.wall_time
-        exec_stats["cputime"] = timing_info.cpu_time
-        exec_stats["error"] = str(error) if error else None
+        exec_stats.walltime = timing_info.wall_time
+        exec_stats.cputime = timing_info.cpu_time
+        exec_stats.error = error
+        str_stats = exec_stats.model_dump()
+        str_stats["error"] = str(error) if error else None
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: So like error.message here?
created_at: 2025-03-04T11:49:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,114 @@
+{# Low Balance Notification Email Template #}
+{# Template variables:
+data.agent_name: the name of the agent
+data.current_balance: the current balance of the user
+data.billing_page_link: the link to the billing page
+data.shortfall: the shortfall amount
+#}
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+  <strong>Low Balance Warning</strong>
+</p>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 165%;
+    margin-top: 0;
+    margin-bottom: 20px;
+  ">
+  Your agent "<strong>{{ data.agent_name }}</strong>" has been stopped due to low balance.
+</p>
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #5D23BB;
+    background-color: #f8f8ff;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Current Balance:</strong> ${{ "{:.2f}".format((data.current_balance|float)/100) }}
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Shortfall:</strong> ${{ "{:.2f}".format((data.shortfall|float)/100) }}
+  </p>
+</div>
+
+
+<div style="
+    margin-left: 15px;
+    margin-bottom: 20px;
+    padding: 15px;
+    border-left: 4px solid #FF6B6B;
+    background-color: #FFF0F0;
+  ">
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 10px;
+  ">
+    <strong>Low Balance:</strong>
+  </p>
+  <p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    margin-top: 0;
+    margin-bottom: 5px;
+  ">
+    Your agent "<strong>{{ data.agent_name }}</strong>" requires additional credits to continue running. The current operation has been canceled until your balance is replenished.
+  </p>
+</div>
+
+<div style="
+    text-align: center;
+    margin: 30px 0;
+  ">
+  <a href="{{ data.billing_page_link }}" style="
+    font-family: 'Poppins', sans-serif;
+    background-color: #5D23BB;
+    color: white;
+    padding: 12px 24px;
+    text-decoration: none;
+    border-radius: 4px;
+    font-weight: 500;
+    display: inline-block;
+  ">
+    Manage Billing
+  </a>
+</div>
+
+<p style="
+    font-family: 'Poppins', sans-serif;
+    color: #070629;
+    font-size: 16px;
+    line-height: 150%;
+    margin-top: 30px;
+    margin-bottom: 10px;
+    font-style: italic;
+  ">
+  This is an automated notification. Your agent is stopped and will need manually restarted unless set to trigger automatically.
+</p>
path: autogpt_platform/backend/backend/notifications/templates/low_balance.html.jinja2
user: login: ntindle
body: This will have the file inserted into the email with an empty line just so you're aware 
created_at: 2025-03-04T11:52:09Z
author_association: MEMBER
diff_hunk: @@ -4,3 +4,18 @@ class MissingConfigError(Exception):
 
 class NeedConfirmation(Exception):
     """The user must explicitly confirm that they want to proceed"""
+
+
+class InsufficientBalanceError(ValueError):
+    user_id: str
+    message: str
+    balance: float
+    amount: float
+
+    def __init__(self, message: str, user_id: str, balance: float, amount: float):
+        super().__init__(message)
+        self.args = (message, user_id, balance, amount)
+        self.message = message
+        self.user_id = user_id
+        self.balance = balance
+        self.amount = amount
path: autogpt_platform/backend/backend/util/exceptions.py
user: login: ntindle
body: Looks like this ('Insufficient balance of $0.01, where this will cost $0.04', '8c18fc64-d4d3-42d9-9f10-c4d2f4657f9a', 1, -4)
![image](https://github.com/user-attachments/assets/6c734b83-a14c-4486-8054-e72e8d70623b)

created_at: 2025-03-04T15:52:36Z
author_association: MEMBER
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: ntindle
body: ```python
/Users/ntindle/code/agpt/AutoGPT/autogpt_platform/backend/backend/data/execution.py
  /Users/ntindle/code/agpt/AutoGPT/autogpt_platform/backend/backend/data/execution.py:43:5 - error: "error" overrides symbol of same name in class "ExecutionStats"
    Variable is mutable so its type is invariant
      Override type "str | None" is not the same as base type "Exception | None" (reportIncompatibleVariableOverride)
```
    
We have a rule against it, and pydantic will serialize a shared object (was also tested) given the opportunity so we'd be doing fucky stuff with `del attr` or set some fields to `None` again
created_at: 2025-03-04T15:56:13Z
author_association: MEMBER
diff_hunk: @@ -4,3 +4,18 @@ class MissingConfigError(Exception):
 
 class NeedConfirmation(Exception):
     """The user must explicitly confirm that they want to proceed"""
+
+
+class InsufficientBalanceError(ValueError):
+    user_id: str
+    message: str
+    balance: float
+    amount: float
+
+    def __init__(self, message: str, user_id: str, balance: float, amount: float):
+        super().__init__(message)
+        self.args = (message, user_id, balance, amount)
+        self.message = message
+        self.user_id = user_id
+        self.balance = balance
+        self.amount = amount
path: autogpt_platform/backend/backend/util/exceptions.py
user: login: ntindle
body: ![image](https://github.com/user-attachments/assets/4466b628-93e2-4214-8e07-d9f9ddddbe91)
fixed it to this
created_at: 2025-03-04T19:51:04Z
author_association: MEMBER
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: ntindle
body: can do optional[whatever] but then you just end up with a bunch of null fields in the db
created_at: 2025-03-04T23:58:18Z
author_association: MEMBER
diff_hunk: @@ -773,6 +744,16 @@ def callback(result: object):
                         execution.wait(3)
 
             log_metadata.info(f"Finished graph execution {graph_exec.graph_exec_id}")
+
+            if isinstance(low_balance_error, InsufficientBalanceError):
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: imo no harm in this
created_at: 2025-03-04T23:58:31Z
author_association: MEMBER
diff_hunk: @@ -402,3 +402,96 @@ class RefundRequest(BaseModel):
     status: str
     created_at: datetime
     updated_at: datetime
+
+
+class NodeExecutionStats(BaseModel):
+    """Execution statistics for a node execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    cost: float = 0
+    input_size: int = 0
+    output_size: int = 0
+    llm_call_count: int = 0
+    llm_retry_count: int = 0
+    input_token_count: int = 0
+    output_token_count: int = 0
+
+
+class NodeExecutionStatsStrError(BaseModel):
+    """Execution statistics for a node execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[str] = None
+    walltime: float = 0
+    cputime: float = 0
+    cost: float = 0
+    input_size: int = 0
+    output_size: int = 0
+    llm_call_count: int = 0
+    llm_retry_count: int = 0
+    input_token_count: int = 0
+    output_token_count: int = 0
+
+
+def convert_node_execution_stats(
path: autogpt_platform/backend/backend/data/model.py
user: login: majdyz
body: nit: instead of creating this generic function out of context, why don't you put this as a class method and use self, instead of stats. e.g: convert_to_str_err or something
created_at: 2025-03-05T02:10:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -402,3 +402,96 @@ class RefundRequest(BaseModel):
     status: str
     created_at: datetime
     updated_at: datetime
+
+
+class NodeExecutionStats(BaseModel):
+    """Execution statistics for a node execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    cost: float = 0
+    input_size: int = 0
+    output_size: int = 0
+    llm_call_count: int = 0
+    llm_retry_count: int = 0
+    input_token_count: int = 0
+    output_token_count: int = 0
+
+
+class NodeExecutionStatsStrError(BaseModel):
+    """Execution statistics for a node execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[str] = None
+    walltime: float = 0
+    cputime: float = 0
+    cost: float = 0
+    input_size: int = 0
+    output_size: int = 0
+    llm_call_count: int = 0
+    llm_retry_count: int = 0
+    input_token_count: int = 0
+    output_token_count: int = 0
+
+
+def convert_node_execution_stats(
+    stats: NodeExecutionStats,
+) -> NodeExecutionStatsStrError:
+    return NodeExecutionStatsStrError(
+        error=str(stats.error) if stats.error else None,
+        walltime=stats.walltime,
+        cputime=stats.cputime,
+        cost=stats.cost,
+        input_size=stats.input_size,
+        output_size=stats.output_size,
+    )
+
+
+class GraphExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class GraphExecutionStatsStrErr(BaseModel):
+    """Execution statistics for a graph execution with the error stringified."""
+
+    error: Optional[str] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+def convert_graph_execution_stats(
path: autogpt_platform/backend/backend/data/model.py
user: login: majdyz
body: ditto
created_at: 2025-03-05T02:10:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -256,10 +263,12 @@ def update_execution(status: ExecutionStatus) -> ExecutionResult:
 
         # Update execution stats
         if execution_stats is not None:
-            execution_stats.update(node_block.execution_stats)
-            execution_stats["input_size"] = input_size
-            execution_stats["output_size"] = output_size
-            execution_stats["cost"] = cost
+            execution_stats = execution_stats.model_copy(
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: why do we need to copy?
created_at: 2025-03-05T02:11:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -34,20 +34,49 @@ def __init__(self):
             "color",
             "font-size",
             "line-height",
+            "letter-spacing",
             "margin-top",
             "margin-bottom",
             "margin-left",
             "margin-right",
             "background-color",
             "padding",
-            "border-radius",
             "font-weight",
             "text-align",
+            "width",
+            "height",
+            "text-decoration",
+            "text-transform",
+            "text-shadow",
+            "box-shadow",
+            "border",
+            "border-top",
+            "border-bottom",
+            "border-left",
+            "border-right",
+            "border-style",
+            "border-width",
+            "border-color",
+            "border-radius",
+            "display",
+            "float",
+            "clear",
+            "position",
+            "top",
+            "right",
+            "bottom",
+            "left",
path: autogpt_platform/backend/backend/util/text.py
user: login: majdyz
body: nit: you might want to ask llm to sort this, otherwise duplicates will keep being added into the list.
created_at: 2025-03-05T02:13:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,35 @@
 from backend.util.settings import Config
 
 
+class ExecutionStats(BaseModel):
+    """Execution statistics for a graph execution."""
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    error: Optional[Exception] = None
+    walltime: float = 0
+    cputime: float = 0
+    nodes_walltime: float = 0
+    nodes_cputime: float = 0
+    node_count: int = 0
+    node_error_count: int = 0
+    cost: float = 0
+
+
+class ExecutionStatsStrErr(BaseModel):
path: autogpt_platform/backend/backend/data/execution.py
user: login: majdyz
body: IMO, having one none field seems fine compared to complicating the code with more class duplications.

TBF you have introduced more null fields in the db by introducing Pydantic class for the dynamic dict stats, now we are forcing all blocks persisting `llm_token_count` and other non-specific metrics with null/zero values for node execution.
created_at: 2025-03-05T02:16:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -569,6 +570,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats["error"] = e
path: autogpt_platform/backend/backend/executor/manager.py
user: login: majdyz
body: We should avoid persisting long text in our row-based db at all cost, plus our ORM will keep loading this multiple times throughout the node execution.
created_at: 2025-03-05T02:17:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -256,10 +263,12 @@ def update_execution(status: ExecutionStatus) -> ExecutionResult:
 
         # Update execution stats
         if execution_stats is not None:
-            execution_stats.update(node_block.execution_stats)
-            execution_stats["input_size"] = input_size
-            execution_stats["output_size"] = output_size
-            execution_stats["cost"] = cost
+            execution_stats = execution_stats.model_copy(
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: I'm copying over the existing behavior of updating the previous with the new data. This is how you do it in pydantic as far as I know
created_at: 2025-03-05T02:33:05Z
author_association: MEMBER
diff_hunk: @@ -34,20 +34,49 @@ def __init__(self):
             "color",
             "font-size",
             "line-height",
+            "letter-spacing",
             "margin-top",
             "margin-bottom",
             "margin-left",
             "margin-right",
             "background-color",
             "padding",
-            "border-radius",
             "font-weight",
             "text-align",
+            "width",
+            "height",
+            "text-decoration",
+            "text-transform",
+            "text-shadow",
+            "box-shadow",
+            "border",
+            "border-top",
+            "border-bottom",
+            "border-left",
+            "border-right",
+            "border-style",
+            "border-width",
+            "border-color",
+            "border-radius",
+            "display",
+            "float",
+            "clear",
+            "position",
+            "top",
+            "right",
+            "bottom",
+            "left",
path: autogpt_platform/backend/backend/util/text.py
user: login: ntindle
body: done
created_at: 2025-03-05T04:25:04Z
author_association: MEMBER
diff_hunk: @@ -569,6 +570,9 @@ def _on_node_execution(
                     f"Failed node execution {node_exec.node_exec_id}: {e}"
                 )
 
+            if stats is not None:
+                stats["error"] = e
path: autogpt_platform/backend/backend/executor/manager.py
user: login: ntindle
body: okay its chill i belive
created_at: 2025-03-05T05:04:53Z
author_association: MEMBER

PR comments:
user: login: deepsource-io[bot]
created_at: 2025-02-27T07:44:39Z
body: <div><p>Here's the code health analysis summary for commits <code>27a5635..21230c4</code>. <a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/97e03137-84ac-4c2b-8bdc-ab3e788f6921/">View details on DeepSource</a>&nbsp;↗.</p><h3>Analysis Summary</h3><table><thead><tr><th>Analyzer</th><th>Status</th><th>Summary</th><th>Link</th></tr></thead><tbody><tr><td><img src="https://static.deepsource.com/analyzer_logos/javascript.svg" alt="DeepSource JavaScript Logo" width="16px" height="16px"/><strong>JavaScript</strong></td><td><span>✅&nbsp;</span><span>Success</span></td><td></td><td><a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/97e03137-84ac-4c2b-8bdc-ab3e788f6921/javascript/">View Check</a>&nbsp;↗</td></tr><tr><td><img src="https://static.deepsource.com/analyzer_logos/python.svg" alt="DeepSource Python Logo" width="16px" height="16px"/><strong>Python</strong></td><td><span>✅&nbsp;</span><span>Success</span></td><td><div>❗ 82 occurences introduced</div><div>🎯 78 occurences resolved</div></td><td><a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/run/97e03137-84ac-4c2b-8bdc-ab3e788f6921/python/">View Check</a>&nbsp;↗</td></tr></tbody></table><hr/><blockquote><div>💡 If you’re a repository administrator, you can configure the quality gates from the <a href="https://app.deepsource.com/gh/Significant-Gravitas/AutoGPT/settings/reporting">settings</a>.</div></blockquote></div>

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-02-27T22:19:52Z
body: This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
author_association: CONTRIBUTOR
user: login: netlify[bot]
created_at: 2025-02-27T22:19:55Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *auto-gpt-docs-dev* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 21230c4e6fdafbe057b7916f9b31482bccf28d51 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/67c7dbf23ad67a0008eac4ef |
author_association: NONE
user: login: netlify[bot]
created_at: 2025-02-27T22:20:24Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *auto-gpt-docs* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 21230c4e6fdafbe057b7916f9b31482bccf28d51 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/auto-gpt-docs/deploys/67c7dbf27d9058000893c663 |
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-02-28T10:05:05Z
body: Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-01T12:14:55Z
body: This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-03T20:40:33Z
body: Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
author_association: CONTRIBUTOR
user: login: qodo-code-review[bot]
created_at: 2025-03-03T22:29:42Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 4 🔵🔵🔵🔵⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9534/files#diff-1b278ebf10a9da0fb5030010222b3a6df2b05a5463cad428cd6c38a1541b0f73R748-R755'><strong>Error Handling</strong></a>

The low balance error handling and notification logic could lead to race conditions or duplicate notifications if multiple nodes fail simultaneously due to insufficient balance.
</summary>

```python
if isinstance(low_balance_error, InsufficientBalanceError):
    cls._handle_low_balance_notif(
        graph_exec.user_id,
        graph_exec.graph_id,
        exec_stats,
        low_balance_error,
    )
    raise low_balance_error
```

</details>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9534/files#diff-fe699074de9055342cbf5c2aef8eba0ae08ef41429c2f012789c20eb9dec532cR189-R191'><strong>Null Safety</strong></a>

The output value handling change could lead to unexpected behavior if the input_data dictionary doesn't contain the expected keys or contains null values.
</summary>

```python
outputs[exec.input_data["name"]].append(
    exec.input_data.get("value", None)
)
```

</details>

<details><summary><a href='https://github.com/Significant-Gravitas/AutoGPT/pull/9534/files#diff-1b278ebf10a9da0fb5030010222b3a6df2b05a5463cad428cd6c38a1541b0f73R682-R683'><strong>Resource Cleanup</strong></a>

The running_executions dictionary items are only removed in the callback, which could lead to memory leaks if callbacks fail to execute.
</summary>

```python
running_executions: dict[str, AsyncResult] = {}
low_balance_error: Optional[InsufficientBalanceError] = None
```

</details>

</td></tr>
</table>

author_association: NONE