Skill: anderskev/agent-architecture-analysis (version 1.0.2)

--- name: agent-architecture-analysis description: "Use when auditing an agent codebase against the 12-Factor Agents methodology, reviewing LLM-powered system architecture, or assessing agentic app compliance. Triggers on \"analyze agent architecture\", \"12-factor audit\", \"how compliant is this agent\", or \"evaluate this LLM app\". Also applies when comparing frameworks or planning agent improvements. Not for quick checklists \u2014 this performs deep per-factor codebase analysis with file-level evidence." --- # 12-Factor Agents Compliance Analysis > Reference: [12-Factor Agents](https://github.com/humanlayer/12-factor-agents) ## Input Parameters | Parameter | Description | Required | |-----------|-------------|----------| | `docs_path` | Path to documentation directory (for existing analyses) | Optional | | `codebase_path` | Root path of the codebase to analyze | Required | ## Analysis Framework ### Factor 1: Natural Language to Tool Calls **Principle:** Convert natural language inputs into structured, deterministic tool calls using schema-validated outputs. **Search Patterns:** ```bash # Look for Pydantic schemas grep -r "class.*BaseModel" --include="*.py" grep -r "TaskDAG\|TaskResponse\|ToolCall" --include="*.py" # Look for JSON schema generation grep -r "model_json_schema\|json_schema" --include="*.py" # Look for structured output generation grep -r "output_type\|response_model" --include="*.py" ``` **File Patterns:** `**/agents/*.py`, `**/schemas/*.py`, `**/models/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | All LLM outputs use Pydantic/dataclass schemas with validators | | **Partial** | Some outputs typed, but dict returns or unvalidated strings exist | | **Weak** | LLM returns raw strings parsed manually or with regex | **Anti-patterns:** - `json.loads(llm_response)` without schema validation - `output.split()` or regex parsing of LLM responses - `dict[str, Any]` return types from agents - No validation between LLM output and handler execution --- ### Factor 2: Own Your Prompts **Principle:** Treat prompts as first-class code you control, version, and iterate on. **Search Patterns:** ```bash # Look for embedded prompts grep -r "SYSTEM_PROMPT\|system_prompt" --include="*.py" grep -r '""".*You are' --include="*.py" # Look for template systems grep -r "jinja\|Jinja\|render_template" --include="*.py" find . -name "*.jinja2" -o -name "*.j2" # Look for prompt directories find . -type d -name "prompts" ``` **File Patterns:** `**/prompts/**`, `**/templates/**`, `**/agents/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Prompts in separate files, templated (Jinja2), versioned | | **Partial** | Prompts as module constants, some parameterization | | **Weak** | Prompts hardcoded inline in functions, f-strings only | **Anti-patterns:** - `f"You are a {role}..."` inline in agent methods - Prompts mixed with business logic - No way to iterate on prompts without code changes - No prompt versioning or A/B testing capability --- ### Factor 3: Own Your Context Window **Principle:** Control how history, state, and tool results are formatted for the LLM. **Search Patterns:** ```bash # Look for context/message management grep -r "AgentMessage\|ChatMessage\|messages" --include="*.py" grep -r "context_window\|context_compiler" --include="*.py" # Look for custom serialization grep -r "to_xml\|to_context\|serialize" --include="*.py" # Look for token management grep -r "token_count\|max_tokens\|truncate" --include="*.py" ``` **File Patterns:** `**/context/*.py`, `**/state/*.py`, `**/core/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Custom context format, token optimization, typed events, compaction | | **Partial** | Basic message history with some structure | | **Weak** | Raw message accumulation, standard OpenAI format only | **Anti-patterns:** - Unbounded message accumulation - Large artifacts embedded inline (diffs, files) - No agent-specific context filtering - Same context for all agent types --- ### Factor 4: Tools Are Structured Outputs **Principle:** Tools produce schema-validated JSON that triggers deterministic code, not magic function calls. **Search Patterns:** ```bash # Look for tool/response schemas grep -r "class.*Response.*BaseModel" --include="*.py" grep -r "ToolResult\|ToolOutput" --include="*.py" # Look for deterministic handlers grep -r "def handle_\|def execute_" --include="*.py" # Look for validation layer grep -r "model_validate\|parse_obj" --include="*.py" ``` **File Patterns:** `**/tools/*.py`, `**/handlers/*.py`, `**/agents/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | All tool outputs schema-validated, handlers type-safe | | **Partial** | Most tools typed, some loose dict returns | | **Weak** | Tools return arbitrary dicts, no validation layer | **Anti-patterns:** - Tool handlers that directly execute LLM output - `eval()` or `exec()` on LLM-generated code - No separation between decision (LLM) and execution (code) - Magic method dispatch based on string matching --- ### Factor 5: Unify Execution State **Principle:** Merge execution state (step, retries) with business state (messages, results). **Search Patterns:** ```bash # Look for state models grep -r "ExecutionState\|WorkflowState\|Thread" --include="*.py" # Look for dual state systems grep -r "checkpoint\|MemorySaver" --include="*.py" grep -r "sqlite\|database\|repository" --include="*.py" # Look for state reconstruction grep -r "load_state\|restore\|reconstruct" --include="*.py" ``` **File Patterns:** `**/state/*.py`, `**/models/*.py`, `**/database/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Single serializable state object with all execution metadata | | **Partial** | State exists but split across systems (memory + DB) | | **Weak** | Execution state scattered, requires multiple queries to reconstruct | **Anti-patterns:** - Retry count stored separately from task state - Error history in logs but not in state - LangGraph checkpoints + separate database storage - No unified event thread --- ### Factor 6: Launch/Pause/Resume **Principle:** Agents support simple APIs for launching, pausing at any point, and resuming. **Search Patterns:** ```bash # Look for REST endpoints grep -r "@router.post\|@app.post" --include="*.py" grep -r "start_workflow\|pause\|resume" --include="*.py" # Look for interrupt mechanisms grep -r "interrupt_before\|interrupt_after" --include="*.py" # Look for webhook handlers grep -r "webhook\|callback" --include="*.py" ``` **File Patterns:** `**/routes/*.py`, `**/api/*.py`, `**/orchestrator/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | REST API + webhook resume, pause at any point including mid-tool | | **Partial** | Launch/pause/resume exists but only at coarse-grained points | | **Weak** | CLI-only launch, no pause/resume capability | **Anti-patterns:** - Blocking `input()` or `confirm()` calls - No way to resume after process restart - Approval only at plan level, not per-tool - No webhook-based resume from external systems --- ### Factor 7: Contact Humans with Tools **Principle:** Human contact is a tool call with question, options, and urgency. **Search Patterns:** ```bash # Look for human input mechanisms grep -r "typer.confirm\|input(\|prompt(" --include="*.py" grep -r "request_human_input\|human_contact" --include="*.py" # Look for approval patterns grep -r "approval\|approve\|reject" --include="*.py" # Look for structured question formats grep -r "question.*options\|HumanInputRequest" --include="*.py" ``` **File Patterns:** `**/agents/*.py`, `**/tools/*.py`, `**/orchestrator/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | `request_human_input` tool with question/options/urgency/format | | **Partial** | Approval gates exist but hardcoded in graph structure | | **Weak** | Blocking CLI prompts, no tool-based human contact | **Anti-patterns:** - `typer.confirm()` in agent code - Human contact hardcoded at specific graph nodes - No way for agents to ask clarifying questions - Single response format (yes/no only) --- ### Factor 8: Own Your Control Flow **Principle:** Custom control flow, not framework defaults. Full control over routing, retries, compaction. **Search Patterns:** ```bash # Look for routing logic grep -r "add_conditional_edges\|route_\|should_continue" --include="*.py" # Look for custom loops grep -r "while True\|for.*in.*range" --include="*.py" | grep -v test # Look for execution mode control grep -r "execution_mode\|agentic\|structured" --include="*.py" ``` **File Patterns:** `**/orchestrator/*.py`, `**/graph/*.py`, `**/core/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Custom routing functions, conditional edges, execution mode control | | **Partial** | Framework control flow with some customization | | **Weak** | Default framework loop with no custom routing | **Anti-patterns:** - Single path through graph with no branching - No distinction between tool types (all treated same) - Framework-default error handling only - No rate limiting or resource management --- ### Factor 9: Compact Errors into Context **Principle:** Errors in context enable self-healing. Track consecutive errors, escalate after threshold. **Search Patterns:** ```bash # Look for error handling grep -r "except.*Exception\|error_history\|consecutive_errors" --include="*.py" # Look for retry logic grep -r "retry\|backoff\|max_attempts" --include="*.py" # Look for escalation grep -r "escalate\|human_escalation" --include="*.py" ``` **File Patterns:** `**/agents/*.py`, `**/orchestrator/*.py`, `**/core/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Errors in context, retry with threshold, automatic escalation | | **Partial** | Errors logged and returned, no automatic retry loop | | **Weak** | Errors logged only, not fed back to LLM, task fails immediately | **Anti-patterns:** - `logger.error()` without adding to context - No retry mechanism (fail immediately) - No consecutive error tracking - No escalation to humans after repeated failures --- ### Factor 10: Small, Focused Agents **Principle:** Each agent has narrow responsibility, 3-10 steps max. **Search Patterns:** ```bash # Look for agent classes grep -r "class.*Agent\|class.*Architect\|class.*Developer" --include="*.py" # Look for step definitions grep -r "steps\|tasks" --include="*.py" | head -20 # Count methods per agent grep -r "async def\|def " agents/*.py 2>/dev/null | wc -l ``` **File Patterns:** `**/agents/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | 3+ specialized agents, each with single responsibility, step limits | | **Partial** | Multiple agents but some have broad scope | | **Weak** | Single "god" agent that handles everything | **Anti-patterns:** - Single agent with 20+ tools - Agent with unbounded step count - Mixed responsibilities (planning + execution + review) - No step or time limits on agent execution --- ### Factor 11: Trigger from Anywhere **Principle:** Workflows triggerable from CLI, REST, WebSocket, Slack, webhooks, etc. **Search Patterns:** ```bash # Look for entry points grep -r "@cli.command\|@router.post\|@app.post" --include="*.py" # Look for WebSocket support grep -r "WebSocket\|websocket" --include="*.py" # Look for external integrations grep -r "slack\|discord\|webhook" --include="*.py" -i ``` **File Patterns:** `**/routes/*.py`, `**/cli/*.py`, `**/main.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | CLI + REST + WebSocket + webhooks + chat integrations | | **Partial** | CLI + REST API available | | **Weak** | CLI only, no programmatic access | **Anti-patterns:** - Only `if __name__ == "__main__"` entry point - No REST API for external systems - No event streaming for real-time updates - Trigger logic tightly coupled to execution --- ### Factor 12: Stateless Reducer **Principle:** Agents as pure functions: (state, input) -> (state, output). No side effects in agent logic. **Search Patterns:** ```bash # Look for state mutation patterns grep -r "\.status = \|\.field = " --include="*.py" # Look for immutable updates grep -r "model_copy\|\.copy(\|with_" --include="*.py" # Look for side effects in agents grep -r "write_file\|subprocess\|requests\." agents/*.py 2>/dev/null ``` **File Patterns:** `**/agents/*.py`, `**/nodes/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Immutable state updates, side effects isolated to tools/handlers | | **Partial** | Mostly immutable, some in-place mutations | | **Weak** | State mutated in place, side effects mixed with agent logic | **Anti-patterns:** - `state.field = new_value` (mutation) - File writes inside agent methods - HTTP calls inside agent decision logic - Shared mutable state between agents --- ### Factor 13: Pre-fetch Context **Principle:** Fetch likely-needed data upfront rather than mid-workflow. **Search Patterns:** ```bash # Look for context pre-fetching grep -r "pre_fetch\|prefetch\|fetch_context" --include="*.py" # Look for RAG/embedding systems grep -r "embedding\|vector\|semantic_search" --include="*.py" # Look for related file discovery grep -r "related_tests\|similar_\|find_relevant" --include="*.py" ``` **File Patterns:** `**/context/*.py`, `**/retrieval/*.py`, `**/rag/*.py` **Compliance Criteria:** | Level | Criteria | |-------|----------| | **Strong** | Automatic pre-fetch of related tests, files, docs before planning | | **Partial** | Manual context passing, design doc support | | **Weak** | No pre-fetching, LLM must request all context via tools | **Anti-patterns:** - Architect starts with issue only, no codebase context - No semantic search for similar past work - Related tests/files discovered only during execution - No RAG or document retrieval system --- ## Output Format **Gate order:** Do not assign Strong / Partial / Weak or treat recommendations as observed facts until **Hard gates** (after [Analysis Workflow](#analysis-workflow)) are satisfied for the factors in scope. ### Executive Summary Table ```markdown | Factor | Status | Notes | |--------|--------|-------| | 1. Natural Language -> Tool Calls | **Strong/Partial/Weak** | [Key finding] | | 2. Own Your Prompts | **Strong/Partial/Weak** | [Key finding] | | ... | ... | ... | | 13. Pre-fetch Context | **Strong/Partial/Weak** | [Key finding] | **Overall**: X Strong, Y Partial, Z Weak ``` ### Per-Factor Analysis For each factor, provide: 1. **Current Implementation** - Evidence with file:line references - Code snippets showing patterns 2. **Compliance Level** - Strong/Partial/Weak with justification 3. **Gaps** - What's missing vs. 12-Factor ideal 4. **Recommendations** - Actionable improvements with code examples --- ## Analysis Workflow 1. **Initial Scan** - Run search patterns for all factors - Identify key files for each factor - Note any existing compliance documentation 2. **Deep Dive** (per factor) - Read identified files - Evaluate against compliance criteria - Document evidence with file paths 3. **Gap Analysis** - Compare current vs. 12-Factor ideal - Identify anti-patterns present - Prioritize by impact 4. **Recommendations** - Provide actionable improvements - Include before/after code examples - Reference roadmap if exists 5. **Summary** - Compile executive summary table - Highlight strengths and critical gaps - Suggest priority order for improvements --- ## Hard gates (evidence before scores) Run these in order. Do not skip ahead: each **Pass** is an objective condition you can check (paths on disk, citations present), not internal certainty. 1. **Scan gate** — After the initial scan (workflow step 1), **Pass:** for every factor (1–13) you have either (a) ≥1 repo-relative path or glob hit to inspect, or (b) a one-line note with rationale (e.g. search command/output, or “no matches — codebase may omit this concern”). Empty hand-waving (“looks fine”) fails this gate. 2. **Evidence gate (per factor)** — Before writing Strong / Partial / Weak for that factor, **Pass:** “Current Implementation” includes ≥1 citation with **file path** plus **line range or short quoted snippet** from `codebase_path`, or an explicit **no evidence located** statement after targeted reads. If evidence is missing after search, default that factor to **Weak** unless the criterion is clearly N/A (say why). 3. **Synthesis gate** — Executive summary table and per-factor analysis sections, **Pass:** only after gates 1–2 are satisfied for the factors in scope. Recommendations may name new files or patterns only as proposals; they must not be presented as observed facts without matching citations from step 2. --- ## Quick Reference: Compliance Scoring | Score | Meaning | Action | |-------|---------|--------| | **Strong** | Fully implements principle | Maintain, minor optimizations | | **Partial** | Some implementation, significant gaps | Planned improvements | | **Weak** | Minimal or no implementation | High priority for roadmap | ## When to Use This Skill - Evaluating new LLM-powered systems - Reviewing agent architecture decisions - Auditing production agentic applications - Planning improvements to existing agents - Comparing frameworks or implementations


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)