Skill: weixuanjiang/agent-guru (version 1.0.0)

--- name: production-agent-design description: Use when building, designing, or reviewing a multi-agent system for production — routing agents, orchestrating subagents, guarding tools with permissions, managing memory and context windows, adding observability and cost tracking, handling errors, or setting up session persistence. compatibility: Designed for agentic frameworks (LangGraph, Strands, or similar). Examples use LangGraph. pip install langgraph langgraph-supervisor. metadata: version: "1.0" --- # Production Agent Design ## Core Principle > **The LLM is the reasoning engine. Your code is the execution engine. The loop is the contract between them.** Every production concern — safety, cost, retries, logging, permissions — lives in the harness, not the prompt. A prompt that says "be careful with deletions" is a suggestion. A `GuardedToolNode` that intercepts `delete_*` calls is a guarantee. ## When to Use This Skill - Designing a new multi-agent system from scratch - Adding safety, cost controls, or observability to an existing agent - Debugging runaway cost, infinite loops, or context window exhaustion - Choosing between single-agent vs multi-agent topology - Implementing human-in-the-loop (HITL) for irreversible actions - Setting up session persistence and resumption ## Architecture at a Glance ``` INGRESS (HTTP / CLI / Webhook / Schedule) │ ROUTER LAYER — classify intent, dispatch cheaply │ ORCHESTRATOR — decompose tasks, delegate to specialists ├── Agent A (scoped tools) └── Agent B (scoped tools) │ TOOL LAYER — validate schema → check permission → execute → truncate │ CROSS-CUTTING CONCERNS ├── MEMORY (short-term / working / long-term) ├── OBSERVABILITY (traces, cost, session replay) └── RESILIENCE (retry, circuit breaker, loop guard) │ PERSISTENCE — checkpoints (Redis / Postgres) + audit log ``` ## Single Agent vs Multi-Agent ``` Task scoped to ONE domain? YES → Single ReAct agent with appropriate tools NO → Independent subtasks? YES → Parallel multi-agent (supervisor + specialists) NO → Sequential / hierarchical orchestrator │ Any irreversible step requiring human review? YES → Plan-then-execute with HITL interrupt NO → Orchestrator with auto-delegation ``` **Rule:** Start with a single agent. Add multi-agent complexity only when you hit a concrete limit — context window size, tool set sprawl, latency, or accuracy. ## Framework Selection | Need | Use | |------|-----| | Complex branching, HITL, durable persistence, fine-grained control | **LangGraph** | | Simple loop, minimal boilerplate, rapid prototype, leaf agents | **Strands** | | Orchestration graph + simple leaf agents | **LangGraph + Strands hybrid** | ## Reference Files Load these **on demand** using the triggers listed below. Do not load all of them upfront. | File | Load when... | |------|-------------| | [references/router-layer.md](references/router-layer.md) | Designing intent routing, building a classifier node, handling misrouting | | [references/orchestrator-layer.md](references/orchestrator-layer.md) | Decomposing tasks, spawning subagents, implementing plan-then-execute | | [references/tool-safety-layer.md](references/tool-safety-layer.md) | Designing tools, adding permission rules, implementing HITL or killswitch | | [references/memory-layer.md](references/memory-layer.md) | Context window approaching limit, adding long-term memory, injecting project context | | [references/observability-layer.md](references/observability-layer.md) | Adding tracing, tracking token cost, debugging agent behavior, setting up alerts | | [references/resilience-layer.md](references/resilience-layer.md) | Adding retry logic, circuit breakers, preventing infinite loops | | [references/persistence-layer.md](references/persistence-layer.md) | Choosing a checkpointer, implementing session resume, session branching | | [references/production-checklist.md](references/production-checklist.md) | Before deploying to production — full ~40-point readiness checklist | ## Quick Reference | Pattern | Key implementation | Reference | |---------|--------------------|-----------| | Intent routing | `conditional_edges` + confidence threshold | `router-layer.md` | | Scoped subagents | `create_react_agent` with tool subset | `orchestrator-layer.md` | | Plan-then-execute | Two nodes, read-only tools in plan phase | `orchestrator-layer.md` | | Tool schema | `args_schema=PydanticModel` on `@tool` | `tool-safety-layer.md` | | Permission guard | `GuardedToolNode` with `PermissionRule` list | `tool-safety-layer.md` | | HITL interrupt | `interrupt()` + `Command(resume=...)` | `tool-safety-layer.md` | | Runtime concurrency | `is_concurrency_safe(input)` per tool call | `tool-safety-layer.md` | | Abort hierarchy | Query-level abort + sibling-level child abort | `tool-safety-layer.md` | | Tiered compaction | budget → snip → microcompact → autocompact | `memory-layer.md` | | Auto-compaction | Summarization node at 80% context | `memory-layer.md` | | Context injection | `AGENT.md` loaded into system prompt | `memory-layer.md` | | Full trace | `BaseCallbackHandler` + structured events | `observability-layer.md` | | Cost tracking | Per-turn token accounting in callback | `observability-layer.md` | | Config snapshot | Freeze all feature flags at query entry | `observability-layer.md` | | Diminishing returns | Track token deltas; stop if delta < 500 × 2 | `resilience-layer.md` | | Output limit escalation | Escalate to 64k tokens before compaction | `resilience-layer.md` | | Streaming cleanup | Tombstone partial messages on fallback | `resilience-layer.md` | | Error-as-observation | `try/except` → `ToolMessage` | `resilience-layer.md` | | Circuit breaker | State machine wrapping tool fn | `resilience-layer.md` | | Session resume | Checkpointer + stable `thread_id` | `persistence-layer.md` | ## Gotchas - **Safety rules must be code, not prompts.** A prompt saying "don't delete production data" is not a safety control. - **Never dump the full parent message history into a subagent.** Pass only the specific task and relevant data — context pollution degrades performance and wastes tokens. - **`InMemorySaver` is for development only.** Use Redis or Postgres checkpointers in production. - **`interrupt()` pauses the graph.** Resume it by calling `graph.invoke(Command(resume=...), config=config)` — forgetting this leaves the agent stuck. - **Tool result truncation is mandatory.** Large tool outputs (file reads, search results) will exhaust the context window if not truncated before returning. - **Always set `max_iterations`.** Without a loop guard, a miscalibrated agent runs indefinitely and incurs unbounded cost. - **Apply compaction in tiers.** Budget tool results → snip → microcompact → autocompact. Jumping straight to full summarization wastes tokens when a cheaper step would suffice. - **Track diminishing returns, not just token budget.** An agent can burn through its iteration budget producing nearly empty continuations. Stop when the last 2 deltas are both below ~500 tokens. - **Snapshot config at query entry.** Never re-read feature flags or env vars mid-turn — a remote config change during a 30-second response causes inconsistent behavior within a single turn. - **Concurrency safety must be checked at runtime.** Schema metadata cannot determine if a bash command is safe — inspect the actual input string at call time. Fail conservatively (serial) if parsing fails.

path: references/persistence-layer.md
content: # Persistence Layer

## Rationale

Production agents are long-running. They must survive process restarts, support resumption after interrupts, and maintain consistency across turns that may span hours or days. Ephemeral in-memory state is a development convenience, not a production architecture.

## Checkpoint Backends by Environment

```bash
# Install the appropriate package for your environment:
pip install langgraph                        # includes InMemorySaver
pip install langgraph-checkpoint-sqlite      # SQLite (lightweight, single-server)
pip install langgraph-checkpoint-redis       # Redis (staging)
pip install langgraph-checkpoint-postgres    # Postgres (production)
```

```python
import os
import sqlite3

# Development — in-process, no external dependency
from langgraph.checkpoint.memory import InMemorySaver

# Lightweight staging / single-server — SQLite file on disk
from langgraph.checkpoint.sqlite import SqliteSaver

# Staging — Redis
from langgraph.checkpoint.redis import RedisSaver

# Production — Postgres
from langgraph.checkpoint.postgres import PostgresSaver


def get_checkpointer(env: str):
    if env == "development":
        return InMemorySaver()
    elif env == "sqlite":
        conn = sqlite3.connect("agent_sessions.db", check_same_thread=False)
        return SqliteSaver(conn)
    elif env == "staging":
        import redis
        r = redis.Redis.from_url("redis://redis:6379")
        return RedisSaver(r)
    else:  # production
        import psycopg
        conn = psycopg.connect("postgresql://user:pass@db:5432/agents")
        return PostgresSaver(conn)


checkpointer = get_checkpointer(os.getenv("ENV", "development"))
app = graph.compile(checkpointer=checkpointer)
```

## Session ID Pattern

```python
import uuid
from datetime import datetime
from langchain_core.messages import HumanMessage

def create_session(user_id: str, task_description: str) -> dict:
    """Create a new agent session with stable ID."""
    return {
        "session_id": str(uuid.uuid4()),
        "user_id": user_id,
        "task": task_description,
        "created_at": datetime.utcnow().isoformat(),
        "thread_id": str(uuid.uuid4()),   # LangGraph checkpoint key
    }

def resume_session(app, session_id: str, new_message: str) -> dict:
    """Resume an existing session — LangGraph loads checkpoint automatically."""
    config = {"configurable": {"thread_id": session_id}}
    return app.invoke(
        {"messages": [HumanMessage(new_message)]},
        config=config
    )

# Start a session
session = create_session("user-123", "refactor the auth module")
result = app.invoke(
    {"messages": [HumanMessage(session["task"])]},
    config={"configurable": {"thread_id": session["session_id"]}}
)

# Resume later — picks up exactly where it left off
result = resume_session(app, session["session_id"], "also add rate limiting")
```

## Session Branching (Fork)

Experiment with a different approach from a known-good checkpoint without losing the original.

```python
def fork_session(app, parent_session_id: str) -> str:
    """Fork a session from its current state into a new independent session."""
    parent_state = app.get_state(
        config={"configurable": {"thread_id": parent_session_id}}
    )
    fork_id = str(uuid.uuid4())
    app.update_state(
        config={"configurable": {"thread_id": fork_id}},
        values=parent_state.values
    )
    return fork_id
```

## Async Background Agent + Polling

```python
import asyncio
from fastapi import FastAPI

api = FastAPI()

@api.post("/agent/run")
async def start_agent(request: dict):
    session_id = str(uuid.uuid4())
    config = {"configurable": {"thread_id": session_id}}
    asyncio.create_task(
        app.ainvoke(
            {"messages": [HumanMessage(request["task"])]},
            config=config
        )
    )
    return {"status": "accepted", "session_id": session_id}

async def poll_for_result(app, session_id: str, timeout: float = 300.0) -> dict:
    import asyncio
    start = asyncio.get_event_loop().time()
    while asyncio.get_event_loop().time() - start < timeout:
        state = app.get_state({"configurable": {"thread_id": session_id}})
        if state.next == ():    # no next nodes = finished
            return state.values
        await asyncio.sleep(5.0)
    raise TimeoutError(f"Session {session_id} did not complete within {timeout}s")
```

## Common Mistakes

| Mistake | Fix |
|---------|-----|
| Using `InMemorySaver` in production | State is lost on restart; use Redis or Postgres |
| Session ID not indexed | Store session ID → user ID mapping in a DB for lookup |
| No TTL on old sessions | Define cleanup policy; stale checkpoints grow storage unboundedly |
| Resume without validation | Verify session belongs to the requesting user before loading |
| `thread_id` reuse across users | Always generate a fresh UUID per session; never share thread IDs |


path: references/resilience-layer.md
content: # Resilience Layer

## Rationale

LLMs are non-deterministic. Tools fail. Networks time out. Rate limits hit. An agent that crashes on first error is not production-grade. **Design for failure as the expected case, not the exception.**

## Error-as-Observation

The most powerful pattern: feed errors back to the LLM as tool results. The LLM can reason about the failure and adapt — try a different approach, ask for clarification, or gracefully degrade.

```python
from langchain_core.messages import ToolMessage
from langgraph.prebuilt import ToolNode

class ResilientToolNode(ToolNode):
    """
    Returns tool execution errors as ToolMessages instead of crashing.
    The LLM observes the error and can reason about next steps.
    """
    def __call__(self, state):
        try:
            return super().__call__(state)
        except Exception as e:
            tool_call_id = state["messages"][-1].tool_calls[0]["id"]
            return {"messages": [ToolMessage(
                content=(
                    f"Tool execution failed: {type(e).__name__}: {str(e)}\n"
                    f"You can try a different approach or ask for clarification."
                ),
                tool_call_id=tool_call_id
            )]}
```

## Retry with Exponential Backoff

```python
import asyncio
import random
from typing import TypeVar, Callable, Awaitable

T = TypeVar("T")

async def with_retry(
    fn: Callable[[], Awaitable[T]],
    max_attempts: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    retryable_exceptions: tuple = (Exception,)
) -> T:
    """Exponential backoff with jitter — safe for rate-limited APIs."""
    for attempt in range(max_attempts):
        try:
            return await fn()
        except retryable_exceptions as e:
            if attempt == max_attempts - 1:
                raise
            delay = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)
            await asyncio.sleep(delay)
```

## Circuit Breaker

Prevents cascading failures when a downstream tool is consistently failing.

```python
from enum import Enum
import time

class CircuitState(Enum):
    CLOSED    = "closed"     # normal operation
    OPEN      = "open"       # failing — reject calls fast
    HALF_OPEN = "half_open"  # testing if service recovered

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 60.0):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.last_failure_time = None

    def call(self, fn, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
            else:
                raise RuntimeError(
                    f"Circuit OPEN — service unavailable. "
                    f"Retry after {self.recovery_timeout}s."
                )
        try:
            result = fn(*args, **kwargs)
            self.failure_count = 0
            self.state = CircuitState.CLOSED
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
            raise

# Usage
search_breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30.0)

from langchain_core.tools import tool

@tool
def web_search(query: str) -> str:
    """Search the web."""
    return search_breaker.call(_do_web_search, query)
```

## Max Iteration Guard

Prevents infinite loops — the most common production failure mode for agents.

```python
from typing import TypedDict
from langgraph.graph import END
from langchain_core.messages import AIMessage

class AgentState(TypedDict):
    messages: list
    iteration_count: int
    max_iterations: int

def should_continue(state: AgentState) -> str:
    if state["iteration_count"] >= state["max_iterations"]:
        return "force_stop"
    last = state["messages"][-1]
    return "tools" if getattr(last, "tool_calls", None) else "end"

def increment_counter(state: AgentState) -> AgentState:
    return {**state, "iteration_count": state.get("iteration_count", 0) + 1}

def force_stop_node(state: AgentState) -> AgentState:
    return {**state, "messages": state["messages"] + [AIMessage(
        "Maximum iterations reached. Here is what I accomplished so far..."
    )]}

# In graph:
# graph.add_node("counter", increment_counter)
# graph.add_node("force_stop", force_stop_node)
# graph.add_conditional_edges("agent", should_continue, {
#     "tools": "tools", "force_stop": "force_stop", "end": END
# })
```

## Graceful Degradation

```python
@tool
def search_with_fallback(query: str) -> str:
    """Search with automatic fallback chain."""
    try:
        return premium_search_api(query)      # rich results, paid API
    except Exception:
        try:
            return free_search_api(query)     # basic results, free tier
        except Exception:
            return cached_search_results(query)  # stale but available
```

## Diminishing Returns Detection

Stop a turn not just at an absolute token budget, but when the model starts producing diminishing output per continuation — a "zombie turn" generating only a few tokens per iteration.

```python
from dataclasses import dataclass, field

COMPLETION_THRESHOLD  = 0.90   # stop at 90% of budget consumed
DIMINISHING_THRESHOLD = 500    # token delta below this = diminishing
MIN_CONTINUATIONS     = 3      # apply check only after >= 3 continuations

@dataclass
class TurnBudgetTracker:
    total_budget: int
    tokens_used: int = 0
    continuation_count: int = 0
    recent_deltas: list = field(default_factory=list)

    def record_continuation(self, tokens_this_turn: int) -> None:
        self.tokens_used += tokens_this_turn
        self.continuation_count += 1
        self.recent_deltas.append(tokens_this_turn)
        if len(self.recent_deltas) > 3:
            self.recent_deltas.pop(0)

    def should_stop(self) -> tuple[bool, str]:
        # Hard stop: 90% of budget consumed
        if self.tokens_used >= self.total_budget * COMPLETION_THRESHOLD:
            return True, "budget_threshold"

        # Diminishing returns: >= 3 continuations with tiny deltas
        if (self.continuation_count >= MIN_CONTINUATIONS
                and len(self.recent_deltas) >= 2
                and all(d < DIMINISHING_THRESHOLD for d in self.recent_deltas[-2:])):
            return True, "diminishing_returns"

        return False, ""
```

**Why this matters:** Without diminishing returns detection, an agent can continue a turn indefinitely, producing single-digit token responses per iteration until it hits the absolute limit — accruing input token costs with no useful output.

## Max Output Tokens Escalation Order

When the model hits `max_output_tokens`, escalate before reaching for expensive compaction:

```
1. Escalate output limit → retry with 64k output tokens (same messages, just higher limit)
2. Reactive compact      → compact ONLY if a real 413 API error occurs
3. Context collapse      → staged removal of older message groups
4. Multi-turn recovery   → split response across multiple turns as last resort
```

```python
async def handle_output_limit_error(messages, llm, max_output_tokens=8192):
    """Try escalating output tokens before invoking compaction."""
    escalated_limit = 64_000

    if max_output_tokens < escalated_limit:
        # Step 1: Try same messages with higher limit
        try:
            response = await llm.ainvoke(
                messages,
                max_tokens=escalated_limit
            )
            return response, escalated_limit
        except Exception:
            pass  # fall through to compaction

    # Step 2: Reactive compact (only here, not proactively)
    messages = compact_messages(messages, llm)
    return await llm.ainvoke(messages, max_tokens=escalated_limit), escalated_limit
```

## Streaming Fallback + Orphan Message Cleanup

When streaming fails mid-turn, stale partial state must be cleaned up before retrying non-streaming. Skipping this step causes orphan `tool_results` with stale IDs that poison the fallback response.

```python
from dataclasses import dataclass
from typing import Optional

@dataclass
class StreamingState:
    partial_messages: list = None
    pending_tool_results: list = None
    executor: object = None

    def tombstone_and_reset(self) -> "StreamingState":
        """
        Mark all partial messages as abandoned, discard pending tool results.
        Call this before falling back to non-streaming retry.
        """
        # Mark partial assistant messages as tombstoned
        # (prevents 'thinking blocks cannot be modified' API error)
        for msg in (self.partial_messages or []):
            msg.metadata = {**(msg.metadata or {}), "tombstoned": True}

        # Discard pending tool results (they have IDs tied to the failed stream)
        # Return a clean slate
        return StreamingState(
            partial_messages=[],
            pending_tool_results=[],
            executor=None
        )

def retry_with_non_streaming(messages: list, llm, streaming_state: StreamingState):
    """Fallback from streaming to non-streaming after a stream error."""
    clean_state = streaming_state.tombstone_and_reset()
    # Now safe to retry: no orphan tool_results, no partial assistant messages
    return llm.invoke(messages)
```

## Unattended Retry with Heartbeats

For long-running agents in automated pipelines that may wait on rate limits (429/529):

```python
import asyncio
import time
from typing import AsyncGenerator

MAX_BACKOFF_SECONDS    = 5 * 60       # 5 minutes max between retries
BACKOFF_RESET_SECONDS  = 6 * 60 * 60  # reset cap after 6 hours
HEARTBEAT_INTERVAL_SEC = 30           # yield keep-alive every 30s

async def retry_with_heartbeat(
    fn,
    is_unattended: bool = False,
) -> AsyncGenerator:
    """
    Retry indefinitely with exponential backoff + keep-alive heartbeats.
    Use only for truly unattended contexts — not interactive sessions.
    """
    if not is_unattended:
        yield await fn()
        return

    delay = 1.0
    session_start = time.monotonic()
    while True:
        try:
            yield await fn()
            return
        except RateLimitError:
            elapsed = time.monotonic() - session_start
            if elapsed > BACKOFF_RESET_SECONDS:
                delay = 1.0  # reset after 6 hours

            wait_until = time.monotonic() + delay
            while time.monotonic() < wait_until:
                await asyncio.sleep(min(HEARTBEAT_INTERVAL_SEC, wait_until - time.monotonic()))
                yield {"type": "heartbeat", "waiting_for_rate_limit": True}

            delay = min(delay * 2, MAX_BACKOFF_SECONDS)
```

## Common Mistakes

| Mistake | Fix |
|---------|-----|
| Raising exceptions from tools | Return errors as `ToolMessage` — LLM observes and adapts |
| No max iteration limit | Always set `max_iterations`; log and summarize on force-stop |
| No diminishing returns check | Track token deltas per continuation — stop when output is shrinking |
| Jumping to compaction on output limit | Escalate `max_output_tokens` first — same messages, just higher limit |
| Skipping streaming cleanup on fallback | Tombstone partial messages before retrying non-streaming |
| Retry without jitter | Add `random.uniform(0, 1)` to avoid thundering herd |
| Circuit breaker threshold too high | 3-5 failures is typical; tune to API's normal error rate |
| No fallback chain for critical tools | Every tool that can fail needs a degraded alternative |
| Heartbeat retry in interactive sessions | Unattended retry must be gated behind a flag — confuses interactive users |


path: references/production-checklist.md
content: # Production Readiness Checklist

Use this before deploying any agent to production.

---

### Router Layer
- [ ] Intent classifier handles unknown/ambiguous inputs gracefully
- [ ] Low-confidence routes fall back to clarification, not random dispatch
- [ ] Routing logic is tested against representative real-world inputs
- [ ] Router is stateless — it only classifies and dispatches, no side effects

### Orchestrator Agent
- [ ] Subagents receive only the context they need (no full parent history dump)
- [ ] Each subagent has a scoped tool set — not the full tool list
- [ ] Plan-then-execute used for any workflow with irreversible steps
- [ ] Result aggregation handles partial failures gracefully

### Tool Layer
- [ ] Every tool has a typed JSON schema (Pydantic `args_schema`)
- [ ] Tools are single-responsibility and named explicitly (no generic `execute`, `run`)
- [ ] Tool results are truncated before being fed back to the LLM
- [ ] Read-only tools are clearly separated from write/mutate tools
- [ ] No tool can affect systems outside its intended scope
- [ ] Concurrency safety checked at runtime (not just schema) — inspect actual input content
- [ ] Abort controller hierarchy implemented: query-level cancel + sibling-level peer abort

### Permission & Safety Layer
- [ ] Deny rules exist for all known-dangerous patterns (regex-tested)
- [ ] Irreversible operations require human approval via `interrupt()`, resumed with `Command(resume=...)`
- [ ] Every tool call is audit-logged (tool name, args, decision, timestamp)
- [ ] Bypass/auto modes have remote killswitch capability
- [ ] Permission rules are code (not prompts) — tested independently

### Memory Layer
- [ ] Auto-compaction configured at 80–85% context window threshold
- [ ] Compaction uses tiered strategy: tool result budgeting → snip → microcompact → autocompact
- [ ] Autocompact circuit breaker: stops retrying after 3 consecutive failures
- [ ] Long-term memory retrieval tested for relevance and speed
- [ ] Working state schema is typed (`TypedDict` or Pydantic)
- [ ] No PII stored in vector stores without appropriate encryption/retention policy

### Observability Layer
- [ ] Every tool call emits a structured trace event (name, args, result, latency)
- [ ] Token usage tracked per turn, per session, per agent
- [ ] Cost tracked and alertable (alert if session exceeds N USD)
- [ ] Session ID propagated to all subagents for trace correlation
- [ ] Session replay possible from stored traces
- [ ] Config snapshotted at query entry — no mid-turn re-reads of feature flags or env vars
- [ ] Continuation analytics tracked (token deltas per continuation) to detect diminishing returns

### Error Handling & Resilience
- [ ] Tool errors returned as `ToolMessage` observations, not process crashes
- [ ] Retry with exponential backoff on transient failures
- [ ] Circuit breaker on unreliable external tools/APIs
- [ ] Max iteration guard prevents infinite loops (tested)
- [ ] Diminishing returns detection: stop turn when token deltas shrink below threshold
- [ ] Max output tokens escalation: try larger limit before triggering compaction
- [ ] Streaming fallback cleanup: tombstone partial messages before non-streaming retry
- [ ] Unattended retry with heartbeats (gated behind feature flag — not for interactive sessions)
- [ ] Graceful degradation path exists for every critical tool

### Session & State Persistence
- [ ] Production uses Redis or Postgres checkpointer (not `InMemorySaver`)
- [ ] Session IDs are stable and indexed in a lookup store
- [ ] Resume flow tested end-to-end (kill process mid-session, verify resume works)
- [ ] Old sessions have a TTL cleanup policy defined

### Operations
- [ ] Per-tenant rate limiting and cost caps enforced in code
- [ ] Model version is pinned (no silent upgrades mid-production)
- [ ] Agent can be disabled per-tenant without redeployment
- [ ] Runbook exists for: agent stuck, cost explosion, tool consistently failing


path: references/router-layer.md
content: # Router Layer

## Rationale

A single generalist agent accumulates too many tools, grows an unwieldy system prompt, and makes routing decisions on every turn instead of once. **Route early, route cheaply, route at the boundary.** The router's job is narrow: classify intent and dispatch. It must not perform the task itself.

## Routing Strategies

| Strategy | When to use | Cost | Latency |
|----------|------------|------|---------|
| **Regex / keyword** | Structured inputs (commands, IDs) | Near zero | ~0ms |
| **Embedding similarity** | Semantic routing across many intents | Low | ~50ms |
| **Structured LLM output** | Ambiguous natural language | Medium | ~500ms |
| **Supervisor agent** | Context needed to decide | High | ~1-2s |

Use the cheapest strategy that maintains acceptable accuracy. Reserve LLM-based routing for genuinely ambiguous inputs.

## Failure Modes to Design For

- **Misrouting** → add confidence threshold; route low-confidence to clarification
- **Ambiguous intent** → route to a clarification node, not a random specialist
- **Unknown intent** → always have a catch-all that asks or gracefully rejects

## Real-World Example

Customer support triage: *"my invoice is wrong"* → billing agent, *"the app crashes"* → technical agent, *"I want to cancel"* → cancellation agent, *"help me"* → clarification. A misroute is costly — wrong agent, wrong tone, wrong tools.

## Core Pattern (LangGraph)

```python
from typing import Literal
from pydantic import BaseModel
from langchain_core.language_models import BaseChatModel
from langgraph.graph import StateGraph, END
from typing import TypedDict

# --- State ---
class RouterState(TypedDict):
    user_message: str
    route: str
    confidence: float
    messages: list

# --- Structured output schema ---
class RouteDecision(BaseModel):
    route: Literal["billing", "technical", "cancellation", "clarification"]
    confidence: float  # 0.0 - 1.0
    reason: str

# --- Router node ---
# llm = your LLM instance (any provider)
router_llm = llm.with_structured_output(RouteDecision)

ROUTER_PROMPT = """You are a support triage router. Classify the user's intent.

Routes:
- billing: invoice, payment, charge, refund, pricing
- technical: bug, crash, error, not working, broken
- cancellation: cancel, unsubscribe, quit, close account
- clarification: unclear, could be multiple things, needs more info

Be conservative with confidence — if unsure, route to clarification."""

def router_node(state: RouterState) -> RouterState:
    decision = router_llm.invoke([
        {"role": "system", "content": ROUTER_PROMPT},
        {"role": "user", "content": state["user_message"]}
    ])
    return {
        **state,
        # Confidence threshold (0.7) — below it, always clarify
        "route": decision.route if decision.confidence >= 0.7 else "clarification",
        "confidence": decision.confidence
    }

def route_decision(state: RouterState) -> str:
    return state["route"]

# --- Build graph ---
graph = StateGraph(RouterState)
graph.add_node("router",        router_node)
graph.add_node("billing",       billing_agent_node)
graph.add_node("technical",     technical_agent_node)
graph.add_node("cancellation",  cancellation_agent_node)
graph.add_node("clarification", clarification_node)

graph.set_entry_point("router")
graph.add_conditional_edges("router", route_decision, {
    "billing":       "billing",
    "technical":     "technical",
    "cancellation":  "cancellation",
    "clarification": "clarification",
})
for node in ["billing", "technical", "cancellation", "clarification"]:
    graph.add_edge(node, END)

app = graph.compile()
```

## Common Mistakes

| Mistake | Fix |
|---------|-----|
| Router also performs the task | Router classifies only — delegate execution to specialist nodes |
| No confidence threshold | Low-confidence routes → always route to clarification |
| No catch-all route | Add `"clarification"` or `"fallback"` as the default |
| Router holds conversation history | Router is stateless — it reads the current message only |
| Hallucinated routes | Use structured output with `Literal` type — invalid routes become validation errors |


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)