Skill: mirni/greenhelix-agent-testing-observability (version 1.3.1)

--- name: greenhelix-agent-testing-observability version: "1.3.1" description: "The Agent Testing & Observability Cookbook: Ship Reliable Agent Commerce Systems. Practitioner cookbook for testing and monitoring agent commerce: tool contract tests, workflow saga tests, chaos injection, OpenTelemetry tracing, health checks, alerting, and CI/CD pipelines." license: MIT compatibility: [openclaw] author: felix-agent type: guide tags: [testing, observability, chaos-engineering, ci-cd, guide, greenhelix, openclaw, ai-agent] price_usd: 0.0 content_type: markdown executable: false install: none credentials: [GREENHELIX_API_KEY, AGENT_SIGNING_KEY] metadata: openclaw: requires: env: - GREENHELIX_API_KEY - AGENT_SIGNING_KEY primaryEnv: GREENHELIX_API_KEY --- # The Agent Testing & Observability Cookbook: Ship Reliable Agent Commerce Systems > **Notice**: This is an educational guide with illustrative code examples. > It does not execute code or install dependencies. > All examples use the GreenHelix sandbox (https://sandbox.greenhelix.net) which > provides 500 free credits — no API key required to get started. > > **Referenced credentials** (you supply these in your own environment): > - `GREENHELIX_API_KEY`: API authentication for GreenHelix gateway (read/write access to purchased API tools only) > - `AGENT_SIGNING_KEY`: Cryptographic signing key for agent identity (Ed25519 key pair for request signing) Your agent commerce system works on your laptop. It passes a smoke test against the GreenHelix sandbox. You deploy to production on a Friday afternoon and go home. By Saturday morning, a retry loop has created 47 duplicate escrows, a performance escrow released funds against stale metrics, and a settlement webhook silently failed for six hours because the endpoint returned 503 and nobody was watching. The system was never tested for these failures because the traditional testing pyramid -- unit tests at the bottom, integration tests in the middle, end-to-end tests at the top -- was not designed for autonomous agents that make financial decisions across unreliable networks against counterparties that may themselves be failing. This guide rebuilds the testing pyramid for agent commerce, then layers production observability, chaos testing, alerting, and CI/CD on top. Every pattern is backed by working Python code, grounded in the 260-test suite that ships with the GreenHelix gateway, and designed to be copied directly into your project. 1. [The Testing Pyramid for Agent Systems](#chapter-1-the-testing-pyramid-for-agent-systems) 2. [Tool-Level Testing Patterns](#chapter-2-tool-level-testing-patterns) ## What You'll Learn - Chapter 1: The Testing Pyramid for Agent Systems - Chapter 2: Tool-Level Testing Patterns - Chapter 3: Workflow & Integration Testing - Chapter 4: Chaos Testing for Agent Commerce - Chapter 5: Production Observability - Chapter 6: Alerting & Incident Response - Chapter 7: CI/CD for Agent Systems - Chapter 8: What to Do Next ## Full Guide # The Agent Testing & Observability Cookbook: Ship Reliable Agent Commerce Systems Your agent commerce system works on your laptop. It passes a smoke test against the GreenHelix sandbox. You deploy to production on a Friday afternoon and go home. By Saturday morning, a retry loop has created 47 duplicate escrows, a performance escrow released funds against stale metrics, and a settlement webhook silently failed for six hours because the endpoint returned 503 and nobody was watching. The system was never tested for these failures because the traditional testing pyramid -- unit tests at the bottom, integration tests in the middle, end-to-end tests at the top -- was not designed for autonomous agents that make financial decisions across unreliable networks against counterparties that may themselves be failing. This guide rebuilds the testing pyramid for agent commerce, then layers production observability, chaos testing, alerting, and CI/CD on top. Every pattern is backed by working Python code, grounded in the 260-test suite that ships with the GreenHelix gateway, and designed to be copied directly into your project. --- ## Table of Contents 1. [The Testing Pyramid for Agent Systems](#chapter-1-the-testing-pyramid-for-agent-systems) 2. [Tool-Level Testing Patterns](#chapter-2-tool-level-testing-patterns) 3. [Workflow & Integration Testing](#chapter-3-workflow--integration-testing) 4. [Chaos Testing for Agent Commerce](#chapter-4-chaos-testing-for-agent-commerce) 5. [Production Observability](#chapter-5-production-observability) 6. [Alerting & Incident Response](#chapter-6-alerting--incident-response) 7. [CI/CD for Agent Systems](#chapter-7-cicd-for-agent-systems) 8. [What to Do Next](#chapter-8-what-to-do-next) --- ## Chapter 1: The Testing Pyramid for Agent Systems ### Why the Traditional Pyramid Breaks The standard testing pyramid assumes your code calls functions that return values. Unit tests verify individual functions. Integration tests verify that modules compose correctly. End-to-end tests verify the full user flow. This model works when the system under test is deterministic, when function calls do not have financial side effects, and when failure modes are limited to "returns wrong value" or "throws exception." Agent commerce systems violate all three assumptions. A call to `create_escrow` locks real funds. A call to `release_escrow` transfers real money. A retry that fires twice creates two escrows, not one error. The failure modes are not "wrong return value" -- they are "agent paid twice for the same work," "escrow timed out but funds are still locked," and "settlement succeeded on the gateway but the webhook notification was lost." Traditional unit tests cannot catch these failures because they test the code in isolation from the financial state machine. Traditional end-to-end tests cannot catch them because they run the happy path once and call it done. ### The Agent Testing Pyramid Agent commerce requires a four-layer testing pyramid that maps to the actual failure modes: ``` ╱╲ ╱ ╲ ╱Chaos╲ Layer 4: Chaos tests ╱ Tests ╲ (failure injection, timeouts, ╱──────────╲ concurrent load) ╱ Multi-Agent ╲ Layer 3: Multi-agent workflow tests ╱ Workflows ╲ (sagas, rollbacks, webhook delivery) ╱─────────────────╲ ╱ Tool Contract ╲ Layer 2: Tool-level contract tests ╱ Tests ╲ (schema, idempotency, permissions) ╱────────────────────────╲ ╱ Deterministic Mocks ╲ Layer 1: Mock-based unit tests ╱ (fast, offline) ╲ (business logic, validation) ╱───────────────────────────────╲ ``` **Layer 1: Deterministic mocks** test your business logic without hitting the gateway. They run in milliseconds and catch logic errors: wrong amount calculations, missing trust checks, incorrect state transitions. These are 60% of your tests. **Layer 2: Tool contract tests** verify that each GreenHelix tool accepts the expected input schema, returns the expected output shape, and produces the correct error codes for invalid input. These run against the sandbox and catch API contract changes. These are 25% of your tests. **Layer 3: Multi-agent workflow tests** verify complete business flows: marketplace listing through escrow release through settlement. They test the saga pattern (multi-step rollback on failure) and webhook delivery. These are 10% of your tests. **Layer 4: Chaos tests** inject failures -- network timeouts, random tool errors, concurrent duplicate requests -- and verify that the system recovers without financial inconsistency. These are 5% of your tests but catch the bugs that cost the most money. ### The AgentTestHarness Every test in this guide uses the `AgentTestHarness` class. It manages fixtures, provides deterministic mocks for Layer 1, and switches to sandbox mode for Layers 2-4. ```python import pytest import time import json import uuid import requests from unittest.mock import MagicMock, patch from typing import Optional from dataclasses import dataclass, field @dataclass class MockResponse: """Deterministic mock for a GreenHelix tool response.""" tool: str status: str = "success" data: dict = field(default_factory=dict) error_code: Optional[str] = None error_message: Optional[str] = None def to_dict(self) -> dict: result = {"status": self.status} if self.status == "success": result.update(self.data) else: result["error"] = { "code": self.error_code or "unknown_error", "message": self.error_message or "An error occurred", } return result class AgentTestHarness: """Test harness for GreenHelix agent commerce systems. Manages fixtures, mocks, and sandbox connections for all four layers of the agent testing pyramid. Usage: harness = AgentTestHarness( [REDACTED_SECRET]", agent_id="test-agent", base_url="https://sandbox.greenhelix.net/v1", ) # Layer 1: deterministic mocks harness.mock_tool("get_balance", {"balance": "100.00"}) result = harness.execute("get_balance", {}) assert result["balance"] == "100.00" # Layer 2+: sandbox mode harness.use_sandbox() result = harness.execute("get_balance", {}) """ def __init__( self, api_key: str, agent_id: str, base_url: str = "https://sandbox.greenhelix.net/v1", ): self.[REDACTED_SECRET] self.agent_id = agent_id self.base_url = base_url self._mocks: dict[str, MockResponse] = {} self._call_log: list[dict] = [] self._sandbox_mode = False self._session = requests.Session() self._session.headers.update({ "Content-Type": "application/json", "Authorization": f"Bearer {api_key}", }) # ── Mode Control ─────────────────────────────────────────── def use_mocks(self): """Switch to deterministic mock mode (Layer 1).""" self._sandbox_mode = False def use_sandbox(self): """Switch to live sandbox mode (Layer 2+).""" self._sandbox_mode = True # ── Mock Registration ────────────────────────────────────── def mock_tool(self, tool: str, data: dict, status: str = "success"): """Register a deterministic mock response for a tool.""" self._mocks[tool] = MockResponse(tool=tool, status=status, data=data) def mock_tool_error( self, tool: str, error_code: str, error_message: str ): """Register a deterministic error response for a tool.""" self._mocks[tool] = MockResponse( tool=tool, status="error", error_code=error_code, error_message=error_message, ) def mock_tool_sequence(self, tool: str, responses: list[dict]): """Register a sequence of responses for successive calls.""" self._mock_sequences = getattr(self, "_mock_sequences", {}) self._mock_sequences[tool] = list(responses) # ── Execution ────────────────────────────────────────────── def execute(self, tool: str, input_data: dict) -> dict: """Execute a tool against mocks or sandbox.""" call_record = { "tool": tool, "input": input_data, "timestamp": time.time(), } if self._sandbox_mode: resp = self._session.post( f"{self.base_url}/v1", json={"tool": tool, "input": input_data}, ) resp.raise_for_status() result = resp.json() else: # Check sequences first sequences = getattr(self, "_mock_sequences", {}) if tool in sequences and sequences[tool]: result = sequences[tool].pop(0) elif tool in self._mocks: result = self._mocks[tool].to_dict() else: raise ValueError( f"No mock registered for tool '{tool}'. " f"Register with harness.mock_tool('{tool}', {{...}})" ) call_record["result"] = result self._call_log.append(call_record) return result # ── Assertions ───────────────────────────────────────────── def assert_tool_called(self, tool: str, times: Optional[int] = None): """Assert that a tool was called, optionally a specific number of times.""" calls = [c for c in self._call_log if c["tool"] == tool] assert len(calls) > 0, f"Tool '{tool}' was never called" if times is not None: assert len(calls) == times, ( f"Tool '{tool}' called {len(calls)} times, expected {times}" ) def assert_tool_not_called(self, tool: str): """Assert that a tool was never called.""" calls = [c for c in self._call_log if c["tool"] == tool] assert len(calls) == 0, ( f"Tool '{tool}' was called {len(calls)} times, expected 0" ) def assert_call_order(self, tools: list[str]): """Assert that tools were called in a specific order.""" called_tools = [c["tool"] for c in self._call_log] idx = 0 for tool in tools: try: idx = called_tools.index(tool, idx) + 1 except ValueError: assert False, ( f"Expected '{tool}' after position {idx} in call log. " f"Actual order: {called_tools}" ) def get_calls(self, tool: Optional[str] = None) -> list[dict]: """Get call log, optionally filtered by tool name.""" if tool: return [c for c in self._call_log if c["tool"] == tool] return list(self._call_log) def reset(self): """Clear all mocks and call history.""" self._mocks.clear() self._call_log.clear() if hasattr(self, "_mock_sequences"): self._mock_sequences.clear() ``` ### The conftest.py: Reusable Fixtures Drop this `conftest.py` into your test directory. Every test file in this guide imports from it. ```python # tests/conftest.py import os import uuid import pytest @pytest.fixture def api_key(): """API key for sandbox testing. Uses env var or test default.""" return os.environ.get("GREENHELIX_API_KEY", "test-api-key-sandbox") @pytest.fixture def base_url(): """Sandbox URL for integration tests.""" return os.environ.get( "GREENHELIX_BASE_URL", "https://sandbox.greenhelix.net/v1" ) @pytest.fixture def agent_id(): """Unique agent ID per test run to prevent collision.""" return f"test-agent-{uuid.uuid4().hex[:12]}" @pytest.fixture def buyer_id(): """Unique buyer agent ID.""" return f"test-buyer-{uuid.uuid4().hex[:12]}" @pytest.fixture def seller_id(): """Unique seller agent ID.""" return f"test-seller-{uuid.uuid4().hex[:12]}" @pytest.fixture def harness(api_key, agent_id, base_url): """AgentTestHarness in mock mode. Call harness.use_sandbox() for live.""" h = AgentTestHarness( [REDACTED_SECRET], agent_id=agent_id, base_url=base_url, ) h.use_mocks() return h @pytest.fixture def sandbox_harness(api_key, agent_id, base_url): """AgentTestHarness in sandbox mode for integration tests.""" h = AgentTestHarness( [REDACTED_SECRET], agent_id=agent_id, base_url=base_url, ) h.use_sandbox() return h @pytest.fixture def mock_session(): """Pre-configured requests.Session mock for unit tests.""" session = MagicMock() response = MagicMock() response.status_code = 200 response.json.return_value = {"status": "success"} response.raise_for_status.return_value = None session.post.return_value = response return session @pytest.fixture def mock_response(): """Factory fixture for creating MockResponse objects.""" def _make(tool, data=None, status="success", error_code=None): return MockResponse( tool=tool, status=status, data=data or {}, error_code=error_code, ) return _make # ── Per-class fixtures for isolated test suites ───────────── class AgentFixtures: """Mixin providing standard mocks for agent commerce tests.""" @pytest.fixture(autouse=True) def setup_agent_mocks(self, harness): """Pre-register common mocks for every test in the class.""" self.harness = harness harness.mock_tool("get_balance", {"balance": "500.00", "currency": "USD"}) harness.mock_tool("create_wallet", {"wallet_id": "w-test-001", "status": "active"}) harness.mock_tool("register_agent", {"agent_id": harness.agent_id, "status": "registered"}) harness.mock_tool("get_trust_score", {"agent_id": "any", "score": 0.85}) harness.mock_tool("get_budget_status", { "daily_limit": "100.00", "spent_today": "25.00", "remaining": "75.00", }) class EscrowFixtures(AgentFixtures): """Extended fixtures for escrow-related tests.""" @pytest.fixture(autouse=True) def setup_escrow_mocks(self, harness): """Add escrow mocks on top of agent mocks.""" self.escrow_id = f"escrow-{uuid.uuid4().hex[:8]}" harness.mock_tool("create_escrow", { "escrow_id": self.escrow_id, "status": "funded", "amount": "50.00", }) harness.mock_tool("release_escrow", { "escrow_id": self.escrow_id, "status": "released", }) harness.mock_tool("cancel_escrow", { "escrow_id": self.escrow_id, "status": "cancelled", }) ``` ### Pattern: Deterministic Mocks vs. Sandbox Testing The harness supports both modes. Use mocks for business logic tests (fast, no network, deterministic). Use sandbox for contract and integration tests (real API, real state, real latency). ```python class TestBudgetGuardrails(AgentFixtures): """Layer 1: Test budget logic with deterministic mocks.""" def test_blocks_escrow_when_over_budget(self, harness): """Escrow creation should be blocked when daily budget is exhausted.""" harness.mock_tool("get_budget_status", { "daily_limit": "100.00", "spent_today": "99.00", "remaining": "1.00", }) budget = harness.execute("get_budget_status", {}) remaining = float(budget["remaining"]) escrow_amount = 50.00 # Business logic: do not create escrow if amount > remaining assert escrow_amount > remaining harness.assert_tool_not_called("create_escrow") def test_allows_escrow_within_budget(self, harness): """Escrow creation should proceed when budget allows.""" budget = harness.execute("get_budget_status", {}) remaining = float(budget["remaining"]) escrow_amount = 25.00 assert escrow_amount <= remaining harness.execute("create_escrow", { "payer_agent_id": harness.agent_id, "payee_agent_id": "seller-001", "amount": str(escrow_amount), }) harness.assert_tool_called("create_escrow", times=1) @pytest.mark.sandbox class TestBudgetGuardrailsSandbox: """Layer 2: Verify budget tools against live sandbox.""" def test_budget_cap_enforced(self, sandbox_harness): """Sandbox should reject escrows exceeding the budget cap.""" h = sandbox_harness h.execute("create_wallet", {}) h.execute("deposit", {"amount": "100.00"}) h.execute("set_budget_cap", { "agent_id": h.agent_id, "daily_limit": "10.00", }) # This should fail because escrow exceeds daily limit result = h.execute("create_escrow", { "payer_agent_id": h.agent_id, "payee_agent_id": "seller-test", "amount": "50.00", }) # Gateway enforces budget at the tool level assert result.get("status") in ("error", "rejected") ``` --- ## Chapter 2: Tool-Level Testing Patterns ### The Tool Contract Test Every GreenHelix tool has an implicit contract: it accepts a specific input schema, returns a specific output shape, and produces documented error codes for invalid input. A tool contract test verifies all three. When the gateway updates an API version or adds a required field, your contract tests break before your production code does. ```python class ToolContract: """Defines the expected contract for a GreenHelix tool. Used by contract tests to verify schema, output shape, and error behavior against the sandbox. """ def __init__( self, tool: str, required_fields: list[str], output_fields: list[str], error_cases: dict[str, dict], ): self.tool = tool self.required_fields = required_fields self.output_fields = output_fields self.error_cases = error_cases # {case_name: {input: ..., expected_error: ...}} # ── Contract definitions for core tools ──────────────────── BILLING_CONTRACTS = { "get_balance": ToolContract( tool="get_balance", required_fields=[], output_fields=["balance", "currency"], error_cases={ "no_wallet": { "input": {}, "expected_error": "wallet_not_found", }, }, ), "deposit": ToolContract( tool="deposit", required_fields=["amount"], output_fields=["balance", "transaction_id"], error_cases={ "negative_amount": { "input": {"amount": "-10.00"}, "expected_error": "invalid_amount", }, "zero_amount": { "input": {"amount": "0"}, "expected_error": "invalid_amount", }, }, ), "set_budget_cap": ToolContract( tool="set_budget_cap", required_fields=["agent_id", "daily_limit"], output_fields=["agent_id", "daily_limit"], error_cases={ "negative_limit": { "input": {"agent_id": "test", "daily_limit": "-50.00"}, "expected_error": "invalid_amount", }, }, ), } PAYMENT_CONTRACTS = { "create_escrow": ToolContract( tool="create_escrow", required_fields=["payer_agent_id", "payee_agent_id", "amount"], output_fields=["escrow_id", "status", "amount"], error_cases={ "insufficient_funds": { "input": { "payer_agent_id": "buyer", "payee_agent_id": "seller", "amount": "999999.00", }, "expected_error": "insufficient_funds", }, "self_escrow": { "input": { "payer_agent_id": "same-agent", "payee_agent_id": "same-agent", "amount": "10.00", }, "expected_error": "invalid_escrow", }, }, ), "release_escrow": ToolContract( tool="release_escrow", required_fields=["escrow_id"], output_fields=["escrow_id", "status"], error_cases={ "nonexistent": { "input": {"escrow_id": "escrow-does-not-exist"}, "expected_error": "escrow_not_found", }, }, ), } IDENTITY_CONTRACTS = { "register_agent": ToolContract( tool="register_agent", required_fields=["agent_id", "public_key", "name"], output_fields=["agent_id", "status"], error_cases={ "missing_key": { "input": {"agent_id": "test", "name": "Test"}, "expected_error": "missing_field", }, }, ), "get_trust_score": ToolContract( tool="get_trust_score", required_fields=["agent_id"], output_fields=["agent_id", "score"], error_cases={ "nonexistent_agent": { "input": {"agent_id": "agent-that-does-not-exist-xyz"}, "expected_error": "agent_not_found", }, }, ), } MARKETPLACE_CONTRACTS = { "register_service": ToolContract( tool="register_service", required_fields=["name", "description", "endpoint", "price", "tags", "category"], output_fields=["service_id"], error_cases={ "missing_name": { "input": { "description": "test", "endpoint": "agent://test", "price": 10.0, "tags": [], "category": "test", }, "expected_error": "missing_field", }, }, ), "search_services": ToolContract( tool="search_services", required_fields=["query"], output_fields=["services"], error_cases={ "empty_query": { "input": {"query": ""}, "expected_error": "invalid_query", }, }, ), } ``` ### Running Contract Tests ```python @pytest.mark.sandbox class TestBillingContracts: """Layer 2: Verify billing tool contracts against sandbox.""" @pytest.fixture(autouse=True) def setup_wallet(self, sandbox_harness): self.harness = sandbox_harness self.harness.execute("create_wallet", {}) self.harness.execute("deposit", {"amount": "100.00"}) @pytest.mark.parametrize("tool_name", BILLING_CONTRACTS.keys()) def test_output_shape(self, tool_name): """Every billing tool returns expected output fields.""" contract = BILLING_CONTRACTS[tool_name] # Build minimal valid input valid_input = {} if tool_name == "deposit": valid_input = {"amount": "10.00"} elif tool_name == "set_budget_cap": valid_input = { "agent_id": self.harness.agent_id, "daily_limit": "50.00", } result = self.harness.execute(tool_name, valid_input) for expected_field in contract.output_fields: assert expected_field in result, ( f"Tool '{tool_name}' missing output field '{expected_field}'. " f"Got: {list(result.keys())}" ) @pytest.mark.parametrize("tool_name", BILLING_CONTRACTS.keys()) def test_error_cases(self, tool_name): """Every billing tool returns correct error codes for invalid input.""" contract = BILLING_CONTRACTS[tool_name] for case_name, case in contract.error_cases.items(): result = self.harness.execute(tool_name, case["input"]) assert result.get("status") == "error" or "error" in result, ( f"Tool '{tool_name}' case '{case_name}' should have failed. " f"Got: {result}" ) @pytest.mark.sandbox class TestPaymentContracts: """Layer 2: Verify payment tool contracts against sandbox.""" @pytest.fixture(autouse=True) def setup_accounts(self, sandbox_harness, buyer_id, seller_id): self.harness = sandbox_harness self.buyer_id = buyer_id self.seller_id = seller_id @pytest.mark.parametrize("tool_name", PAYMENT_CONTRACTS.keys()) def test_error_cases(self, tool_name): """Every payment tool returns correct errors for invalid input.""" contract = PAYMENT_CONTRACTS[tool_name] for case_name, case in contract.error_cases.items(): result = self.harness.execute(tool_name, case["input"]) assert result.get("status") == "error" or "error" in result, ( f"Payment tool '{tool_name}' case '{case_name}' did not fail" ) ``` ### Idempotency Testing for Payment Tools Payment tools must be idempotent. Calling `create_escrow` twice with the same parameters should not create two escrows. Calling `release_escrow` twice should not transfer funds twice. This pattern tests idempotency by submitting duplicate requests and verifying financial consistency (P1, P7). ```python @pytest.mark.sandbox class TestPaymentIdempotency: """Layer 2: Verify payment tools handle duplicate calls safely.""" def test_duplicate_escrow_creation(self, sandbox_harness, buyer_id, seller_id): """Creating the same escrow twice should return the same escrow_id.""" h = sandbox_harness h.execute("create_wallet", {}) h.execute("deposit", {"amount": "200.00"}) escrow_params = { "payer_agent_id": buyer_id, "payee_agent_id": seller_id, "amount": "50.00", "description": "Idempotency test escrow", "idempotency_key": f"idem-{uuid.uuid4().hex[:8]}", } result_1 = h.execute("create_escrow", escrow_params) result_2 = h.execute("create_escrow", escrow_params) # Same idempotency key should return same escrow assert result_1["escrow_id"] == result_2["escrow_id"] # Balance should only be debited once balance = h.execute("get_balance", {}) assert float(balance["balance"]) == 150.00 def test_duplicate_release(self, sandbox_harness): """Releasing the same escrow twice should not double-pay.""" h = sandbox_harness h.execute("create_wallet", {}) h.execute("deposit", {"amount": "100.00"}) escrow = h.execute("create_escrow", { "payer_agent_id": h.agent_id, "payee_agent_id": "seller-test", "amount": "25.00", }) escrow_id = escrow["escrow_id"] release_1 = h.execute("release_escrow", {"escrow_id": escrow_id}) release_2 = h.execute("release_escrow", {"escrow_id": escrow_id}) # Second release should be a no-op or return already_released assert release_1.get("status") == "released" assert release_2.get("status") in ("released", "already_released") ``` ### Permission Boundary Testing Agents should only be able to operate on their own resources. A buyer should not release an escrow created by a different buyer. A seller should not cancel an escrow that is not addressed to them. Permission boundary tests verify these invariants (P7). ```python @pytest.mark.sandbox class TestPermissionBoundaries: """Layer 2: Verify agents cannot access other agents' resources.""" def test_cannot_release_others_escrow(self, sandbox_harness): """Agent A cannot release an escrow created by Agent B.""" h = sandbox_harness # Agent A creates an escrow h.execute("create_wallet", {}) h.execute("deposit", {"amount": "100.00"}) escrow = h.execute("create_escrow", { "payer_agent_id": h.agent_id, "payee_agent_id": "seller-x", "amount": "10.00", }) # Agent B (different harness/session) tries to release it other = AgentTestHarness( [REDACTED_SECRET], agent_id="attacker-agent", base_url=h.base_url, ) other.use_sandbox() result = other.execute("release_escrow", { "escrow_id": escrow["escrow_id"], }) assert result.get("status") == "error" def test_cannot_read_others_balance(self, sandbox_harness): """Agent A cannot read Agent B's wallet balance.""" h = sandbox_harness h.execute("create_wallet", {}) h.execute("deposit", {"amount": "100.00"}) other = AgentTestHarness( [REDACTED_SECRET], agent_id="other-agent", base_url=h.base_url, ) other.use_sandbox() result = other.execute("get_balance", {}) # Should return the other agent's balance (0), not our 100 balance = float(result.get("balance", 0)) assert balance != 100.00 def test_cannot_cancel_others_escrow(self, sandbox_harness): """Seller cannot cancel an escrow -- only the buyer can.""" h = sandbox_harness h.execute("create_wallet", {}) h.execute("deposit", {"amount": "50.00"}) escrow = h.execute("create_escrow", { "payer_agent_id": h.agent_id, "payee_agent_id": "seller-y", "amount": "10.00", }) seller = AgentTestHarness( [REDACTED_SECRET], agent_id="seller-y", base_url=h.base_url, ) seller.use_sandbox() result = seller.execute("cancel_escrow", { "escrow_id": escrow["escrow_id"], }) assert result.get("status") == "error" ``` --- ## Chapter 3: Workflow & Integration Testing ### The Saga Test Pattern Agent commerce workflows are sagas: multi-step operations where each step has a compensating action. If step 3 fails, steps 1 and 2 must be rolled back. The saga test pattern verifies both the happy path and every possible failure point. ```python class MarketplaceSaga: """Implements the complete marketplace listing workflow as a testable saga. Steps: 1. Seller registers service on marketplace 2. Buyer discovers service via search 3. Buyer checks seller trust score 4. Buyer creates escrow 5. Seller performs work (simulated) 6. Buyer releases escrow 7. Buyer rates service 8. Settlement completes Compensating actions: Step 4 fails → no cleanup needed (funds not locked) Step 5 fails → cancel escrow (return funds to buyer) Step 6 fails → open dispute """ def __init__(self, harness: AgentTestHarness, buyer_id: str, seller_id: str): self.harness = harness self.buyer_id = buyer_id self.seller_id = seller_id self.state = {"step": 0, "completed_steps": []} def run(self) -> dict: """Execute the full saga, rolling back on failure.""" try: # Step 1: Register service service = self.harness.execute("register_service", { "name": "Test Summarization Service", "description": "Summarizes documents for testing", "endpoint": f"agent://{self.seller_id}", "price": 25.00, "tags": ["test", "summarization"], "category": "data-processing", }) self.state["service_id"] = service["service_id"] self.state["completed_steps"].append("register_service") # Step 2: Discover service results = self.harness.execute("search_services", { "query": "test summarization", }) assert len(results.get("services", [])) > 0 self.state["completed_steps"].append("discover_service") # Step 3: Trust check trust = self.harness.execute("get_trust_score", { "agent_id": self.seller_id, }) if trust.get("score", 0) < 0.5: return {"status": "aborted", "reason": "low_trust"} self.state["completed_steps"].append("trust_check") # Step 4: Create escrow escrow = self.harness.execute("create_escrow", { "payer_agent_id": self.buyer_id, "payee_agent_id": self.seller_id, "amount": "25.00", "description": "Saga test escrow", }) self.state["escrow_id"] = escrow["escrow_id"] self.state["completed_steps"].append("create_escrow") # Step 5: Simulate work (in real tests, call seller endpoint) work_result = {"quality": 0.95, "documents_processed": 500} self.state["completed_steps"].append("work_completed") # Step 6: Release escrow release = self.harness.execute("release_escrow", { "escrow_id": escrow["escrow_id"], }) self.state["completed_steps"].append("release_escrow") # Step 7: Rate service self.harness.execute("rate_service", { "service_id": service["service_id"], "rating": 5, }) self.state["completed_steps"].append("rate_service") return {"status": "completed", "state": self.state} except Exception as e: return self._compensate(str(e)) def _compensate(self, error: str) -> dict: """Roll back completed steps on failure.""" if "create_escrow" in self.state["completed_steps"]: escrow_id = self.state.get("escrow_id") if escrow_id and "release_escrow" not in self.state["completed_steps"]: self.harness.execute("cancel_escrow", { "escrow_id": escrow_id, }) self.state["completed_steps"].append("compensate:cancel_escrow") return { "status": "rolled_back", "error": error, "state": self.state, } ``` ### Testing the Saga ```python class TestMarketplaceSaga(EscrowFixtures): """Layer 3: Full marketplace workflow with rollback verification.""" def test_happy_path(self, harness, buyer_id, seller_id): """Complete saga executes all 7 steps.""" harness.mock_tool("register_service", { "service_id": "svc-test-001", }) harness.mock_tool("search_services", { "services": [{"name": "Test Service", "agent_id": seller_id}], }) harness.mock_tool("rate_service", {"status": "rated"}) saga = MarketplaceSaga(harness, buyer_id, seller_id) result = saga.run() assert result["status"] == "completed" assert len(result["state"]["completed_steps"]) == 7 harness.assert_call_order([ "register_service", "search_services", "get_trust_score", "create_escrow", "release_escrow", "rate_service", ]) def test_rollback_on_escrow_failure(self, harness, buyer_id, seller_id): """Failed escrow creation does not leave orphaned state.""" harness.mock_tool("register_service", {"service_id": "svc-test-002"}) harness.mock_tool("search_services", { "services": [{"name": "Test", "agent_id": seller_id}], }) harness.mock_tool_error( "create_escrow", "insufficient_funds", "Not enough balance" ) saga = MarketplaceSaga(harness, buyer_id, seller_id) result = saga.run() assert result["status"] == "rolled_back" assert "create_escrow" not in result["state"]["completed_steps"] harness.assert_tool_not_called("release_escrow") def test_rollback_cancels_escrow_on_work_failure(self, harness, buyer_id, seller_id): """Failed work step triggers escrow cancellation.""" harness.mock_tool("register_service", {"service_id": "svc-test-003"}) harness.mock_tool("search_services", { "services": [{"name": "Test", "agent_id": seller_id}], }) saga = MarketplaceSaga(harness, buyer_id, seller_id) # Simulate work failure by injecting error after escrow creation original_execute = harness.execute call_count = {"n": 0} def failing_execute(tool, input_data): call_count["n"] += 1 if tool == "release_escrow": raise RuntimeError("Simulated work verification failure") return original_execute(tool, input_data) harness.execute = failing_execute result = saga.run() assert result["status"] == "rolled_back" assert "compensate:cancel_escrow" in result["state"]["completed_steps"] ``` ### Subscription Lifecycle Testing Subscriptions are stateful workflows: create, renew, pause, cancel. Each transition must be tested, including edge cases like renewal with insufficient balance (P2, P6). ```python class TestSubscriptionLifecycle(AgentFixtures): """Layer 3: Verify subscription state transitions.""" def test_full_lifecycle(self, harness): """Create → renew → cancel lifecycle.""" sub_id = f"sub-{uuid.uuid4().hex[:8]}" harness.mock_tool("create_subscription", { "subscription_id": sub_id, "status": "active", "next_payment_date": "2026-05-06", }) harness.mock_tool("get_subscription", { "subscription_id": sub_id, "status": "active", "payments_completed": 1, }) harness.mock_tool("cancel_subscription", { "subscription_id": sub_id, "status": "cancelled", }) # Create sub = harness.execute("create_subscription", { "payer_agent_id": harness.agent_id, "payee_agent_id": "provider-001", "amount": "15.00", "interval": "monthly", }) assert sub["status"] == "active" # Check status status = harness.execute("get_subscription", { "subscription_id": sub_id, }) assert status["payments_completed"] == 1 # Cancel cancel = harness.execute("cancel_subscription", { "subscription_id": sub_id, }) assert cancel["status"] == "cancelled" harness.assert_call_order([ "create_subscription", "get_subscription", "cancel_subscription", ]) def test_renewal_with_insufficient_funds(self, harness): """Subscription renewal should fail gracefully when balance is low.""" harness.mock_tool("get_balance", {"balance": "5.00", "currency": "USD"}) harness.mock_tool_error( "create_subscription", "insufficient_funds", "Balance too low for subscription amount", ) result = harness.execute("create_subscription", { "payer_agent_id": harness.agent_id, "payee_agent_id": "provider-002", "amount": "15.00", "interval": "monthly", }) assert result.get("status") == "error" ``` ### Webhook Delivery Testing Webhooks are the primary notification mechanism for payment events. A missed webhook means a missed settlement, a missed dispute deadline, or a missed subscription renewal. Test webhook delivery separately from the business logic it triggers (P4). ```python class TestWebhookDelivery: """Layer 3: Verify webhook registration and event delivery.""" def test_webhook_registration(self, harness): """Webhook registration should return a webhook_id.""" harness.mock_tool("register_webhook", { "webhook_id": "wh-test-001", "status": "active", "events": ["escrow.released", "escrow.disputed"], }) result = harness.execute("register_webhook", { "url": "https://test.example.com/webhook", "events": ["escrow.released", "escrow.disputed"], }) assert "webhook_id" in result assert result["status"] == "active" def test_webhook_event_format(self, harness): """Webhook payloads should contain required fields.""" harness.mock_tool("get_webhook_logs", { "logs": [{ "webhook_id": "wh-test-001", "event_type": "escrow.released", "payload": { "escrow_id": "escrow-abc", "amount": "25.00", "payer_agent_id": "buyer-1", "payee_agent_id": "seller-1", "timestamp": "2026-04-06T12:00:00Z", }, "delivery_status": "delivered", "response_code": 200, }], }) logs = harness.execute("get_webhook_logs", {"webhook_id": "wh-test-001"}) for log_entry in logs["logs"]: payload = log_entry["payload"] assert "escrow_id" in payload assert "amount" in payload assert "timestamp" in payload assert log_entry["delivery_status"] == "delivered" ``` --- ## Chapter 4: Chaos Testing for Agent Commerce ### Why Chaos Testing Matters for Payments Traditional software fails gracefully when a downstream service is unavailable -- it shows an error page. Agent commerce software fails expensively. A timeout during `release_escrow` might mean funds are released on the gateway but the caller never gets confirmation, leading to a duplicate release attempt. A network partition during a split payment might result in partial settlement. Chaos testing injects these failures deliberately so you can verify your system handles them before production handles them for you. ### The ChaosMiddleware The `ChaosMiddleware` wraps the harness's `execute` method and randomly injects failures: timeouts, error responses, delayed responses, and corrupted payloads. It is configurable per tool and per failure type. ```python import random import time from dataclasses import dataclass, field @dataclass class ChaosConfig: """Configuration for chaos injection on a specific tool.""" timeout_pct: float = 0.0 # % of calls that timeout error_pct: float = 0.0 # % of calls that return errors delay_ms: float = 0.0 # Additional latency in ms corrupt_pct: float = 0.0 # % of calls with corrupted responses duplicate_pct: float = 0.0 # % of calls that execute twice class ChaosMiddleware: """Wraps AgentTestHarness._execute with configurable failure injection. Usage: harness = AgentTestHarness(api_key, agent_id, base_url) chaos = ChaosMiddleware( harness=harness, default_config=ChaosConfig(error_pct=10, delay_ms=200), ) chaos.set_tool_config("create_escrow", ChaosConfig( timeout_pct=20, duplicate_pct=5, )) # All calls now go through chaos injection result = chaos.execute("create_escrow", {...}) """ def __init__( self, harness: AgentTestHarness, default_config: ChaosConfig = None, seed: int = None, ): self.harness = harness self.default_config = default_config or ChaosConfig() self._tool_configs: dict[str, ChaosConfig] = {} self._rng = random.Random(seed) self._chaos_log: list[dict] = [] def set_tool_config(self, tool: str, config: ChaosConfig): """Set chaos configuration for a specific tool.""" self._tool_configs[tool] = config def execute(self, tool: str, input_data: dict) -> dict: """Execute a tool with chaos injection.""" config = self._tool_configs.get(tool, self.default_config) chaos_event = {"tool": tool, "injection": None, "timestamp": time.time()} # Check timeout injection if self._rng.random() * 100 < config.timeout_pct: chaos_event["injection"] = "timeout" self._chaos_log.append(chaos_event) raise TimeoutError( f"Chaos: simulated timeout for tool '{tool}'" ) # Check error injection if self._rng.random() * 100 < config.error_pct: chaos_event["injection"] = "error" self._chaos_log.append(chaos_event) return { "status": "error", "error": { "code": "chaos_injected_error", "message": f"Chaos: simulated error for tool '{tool}'", }, } # Apply delay if config.delay_ms > 0: delay_seconds = config.delay_ms / 1000.0 actual_delay = self._rng.uniform(0, delay_seconds * 2) time.sleep(actual_delay) chaos_event["injection"] = f"delay:{actual_delay:.3f}s" # Execute the real call result = self.harness.execute(tool, input_data) # Check duplicate execution if self._rng.random() * 100 < config.duplicate_pct: chaos_event["injection"] = "duplicate" self._chaos_log.append(chaos_event) # Execute again -- this tests idempotency duplicate_result = self.harness.execute(tool, input_data) return duplicate_result # Check corruption if self._rng.random() * 100 < config.corrupt_pct: chaos_event["injection"] = "corrupt" self._chaos_log.append(chaos_event) if isinstance(result, dict): result["_chaos_corrupted"] = True # Remove a random key to simulate partial response keys = [k for k in result.keys() if k != "status"] if keys: del result[self._rng.choice(keys)] self._chaos_log.append(chaos_event) return result def get_chaos_log(self) -> list[dict]: """Get the log of all chaos injections.""" return list(self._chaos_log) def get_injection_stats(self) -> dict: """Get summary statistics of chaos injections.""" stats = {"total": len(self._chaos_log)} for entry in self._chaos_log: injection = entry.get("injection") or "none" category = injection.split(":")[0] stats[category] = stats.get(category, 0) + 1 return stats ``` ### Chaos Test Patterns ```python class TestEscrowUnderChaos: """Layer 4: Verify escrow operations survive failure injection.""" def test_escrow_survives_timeout_retry(self, harness): """Escrow creation retries correctly after timeout.""" harness.use_mocks() harness.mock_tool("create_escrow", { "escrow_id": "escrow-chaos-001", "status": "funded", "amount": "50.00", }) chaos = ChaosMiddleware( harness=harness, seed=42, default_config=ChaosConfig(timeout_pct=50), ) # Retry loop -- real production code should have this max_retries = 5 result = None for attempt in range(max_retries): try: result = chaos.execute("create_escrow", { "payer_agent_id": "buyer", "payee_agent_id": "seller", "amount": "50.00", "idempotency_key": "idem-chaos-001", }) break except TimeoutError: continue assert result is not None, "All retry attempts timed out" assert result["escrow_id"] == "escrow-chaos-001" def test_no_double_payment_under_duplicates(self, harness): """Duplicate chaos injection does not cause double payment.""" harness.use_mocks() call_count = {"n": 0} original_execute = harness.execute def counting_execute(tool, input_data): call_count["n"] += 1 return original_execute(tool, input_data) harness.execute = counting_execute harness.mock_tool("release_escrow", { "escrow_id": "escrow-dup-test", "status": "released", }) chaos = ChaosMiddleware( harness=harness, seed=99, default_config=ChaosConfig(duplicate_pct=100), ) result = chaos.execute("release_escrow", { "escrow_id": "escrow-dup-test", }) # The middleware called execute twice, but the result should # still represent a single release assert result["status"] == "released" def test_concurrent_escrow_load(self, harness): """Concurrent escrow creation does not cause race conditions.""" import concurrent.futures harness.use_mocks() harness.mock_tool("create_escrow", { "escrow_id": "will-be-unique", "status": "funded", "amount": "10.00", }) chaos = ChaosMiddleware( harness=harness, seed=7, default_config=ChaosConfig(delay_ms=50, error_pct=10), ) results = [] errors = [] def create_escrow(i): try: return chaos.execute("create_escrow", { "payer_agent_id": "buyer", "payee_agent_id": "seller", "amount": "10.00", "idempotency_key": f"concurrent-{i}", }) except Exception as e: return {"status": "error", "message": str(e)} with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool: futures = [pool.submit(create_escrow, i) for i in range(20)] for f in concurrent.futures.as_completed(futures): r = f.result() if r.get("status") == "error": errors.append(r) else: results.append(r) # At least some should succeed despite chaos assert len(results) > 0 # Errors are expected under chaos -- verify they are handled total = len(results) + len(errors) assert total == 20 def test_escrow_timeout_deadline(self, harness): """Escrow must be released or cancelled before deadline.""" harness.use_mocks() deadline = time.time() + 2 # 2 second deadline for test harness.mock_tool("create_escrow", { "escrow_id": "escrow-deadline", "status": "funded", "deadline": deadline, }) harness.mock_tool("cancel_escrow", { "escrow_id": "escrow-deadline", "status": "cancelled", "reason": "deadline_exceeded", }) escrow = harness.execute("create_escrow", { "payer_agent_id": "buyer", "payee_agent_id": "seller", "amount": "30.00", }) # Simulate deadline passing time.sleep(0.1) # In real tests use time mocking current_time = time.time() if current_time < deadline: # Still within deadline -- release is valid harness.mock_tool("release_escrow", { "escrow_id": "escrow-deadline", "status": "released", }) result = harness.execute("release_escrow", { "escrow_id": escrow["escrow_id"], }) assert result["status"] == "released" else: # Deadline exceeded -- should auto-cancel result = harness.execute("cancel_escrow", { "escrow_id": escrow["escrow_id"], }) assert result["status"] == "cancelled" ``` --- ## Chapter 5: Production Observability ### Why printf Debugging Does Not Work for Payments When a payment fails in production, you need to know three things instantly: which tool failed, how long it took, and what the input was. You need this information structured, searchable, and available without SSH-ing into a server. The `AgentTracer` wraps every `_execute` call with timing, success/failure tracking, and structured output. ### The AgentTracer ```python import time import json import logging from dataclasses import dataclass, field from typing import Optional, Callable @dataclass class TraceRecord: """A single traced tool execution.""" tool: str agent_id: str started_at: float ended_at: float duration_ms: float success: bool input_data: dict output_data: Optional[dict] = None error: Optional[str] = None trace_id: str = "" def to_dict(self) -> dict: return { "trace_id": self.trace_id, "tool": self.tool, "agent_id": self.agent_id, "started_at": self.started_at, "ended_at": self.ended_at, "duration_ms": round(self.duration_ms, 2), "success": self.success, "error": self.error, } class AgentTracer: """Wraps _execute and records timing, success/failure, tool name. Provides structured observability for every tool call in production. Usage: tracer = AgentTracer( api_key="...", agent_id="production-buyer-01", base_url="https://api.greenhelix.net/v1", ) # Wrap an existing AgentCommerce or harness result = tracer.trace("create_escrow", { "payer_agent_id": "buyer", "payee_agent_id": "seller", "amount": "50.00", }) # Get metrics print(tracer.get_metrics()) # {'total_calls': 47, 'success_rate': 0.957, # 'avg_latency_ms': 142.3, 'p99_latency_ms': 890.1, # 'error_rate_by_tool': {'create_escrow': 0.02}} """ def __init__( self, api_key: str, agent_id: str, base_url: str = "https://api.greenhelix.net/v1", logger: Optional[logging.Logger] = None, on_slow_call: Optional[Callable] = None, slow_threshold_ms: float = 2000.0, ): self.[REDACTED_SECRET] self.agent_id = agent_id self.base_url = base_url self.logger = logger or logging.getLogger("agent_tracer") self.on_slow_call = on_slow_call self.slow_threshold_ms = slow_threshold_ms self._traces: list[TraceRecord] = [] self._session = requests.Session() self._session.headers.update({ "Content-Type": "application/json", "Authorization": f"Bearer {api_key}", }) def trace(self, tool: str, input_data: dict) -> dict: """Execute a tool with full tracing.""" trace_id = f"trace-{uuid.uuid4().hex[:12]}" started_at = time.time() try: resp = self._session.post( f"{self.base_url}/v1", json={"tool": tool, "input": input_data}, ) resp.raise_for_status() result = resp.json() success = result.get("status") != "error" error = None if success else json.dumps(result.get("error", {})) except Exception as e: result = {"status": "error", "error": str(e)} success = False error = str(e) ended_at = time.time() duration_ms = (ended_at - started_at) * 1000 record = TraceRecord( tool=tool, agent_id=self.agent_id, started_at=started_at, ended_at=ended_at, duration_ms=duration_ms, success=success, input_data=input_data, output_data=result if success else None, error=error, trace_id=trace_id, ) self._traces.append(record) # Structured log self.logger.info(json.dumps({ "event": "tool_execution", "trace_id": trace_id, "tool": tool, "agent_id": self.agent_id, "duration_ms": round(duration_ms, 2), "success": success, "error": error, })) # Slow call callback if duration_ms > self.slow_threshold_ms and self.on_slow_call: self.on_slow_call(record) return result def get_metrics(self) -> dict: """Compute aggregate metrics from traced calls.""" if not self._traces: return {"total_calls": 0} total = len(self._traces) successes = sum(1 for t in self._traces if t.success) durations = sorted(t.duration_ms for t in self._traces) # Per-tool error rates tool_calls: dict[str, dict] = {} for t in self._traces: if t.tool not in tool_calls: tool_calls[t.tool] = {"total": 0, "errors": 0} tool_calls[t.tool]["total"] += 1 if not t.success: tool_calls[t.tool]["errors"] += 1 error_rate_by_tool = { tool: stats["errors"] / stats["total"] for tool, stats in tool_calls.items() if stats["errors"] > 0 } # Per-tool latency tool_latencies: dict[str, list[float]] = {} for t in self._traces: tool_latencies.setdefault(t.tool, []).append(t.duration_ms) avg_latency_by_tool = { tool: round(sum(lats) / len(lats), 2) for tool, lats in tool_latencies.items() } return { "total_calls": total, "success_rate": round(successes / total, 4), "avg_latency_ms": round(sum(durations) / total, 2), "p50_latency_ms": round(durations[total // 2], 2), "p95_latency_ms": round(durations[int(total * 0.95)], 2), "p99_latency_ms": round(durations[int(total * 0.99)], 2), "error_rate_by_tool": error_rate_by_tool, "avg_latency_by_tool": avg_latency_by_tool, } def get_traces( self, tool: Optional[str] = None, success: Optional[bool] = None, min_duration_ms: Optional[float] = None, ) -> list[dict]: """Query traces with optional filters.""" filtered = self._traces if tool: filtered = [t for t in filtered if t.tool == tool] if success is not None: filtered = [t for t in filtered if t.success == success] if min_duration_ms is not None: filtered = [t for t in filtered if t.duration_ms >= min_duration_ms] return [t.to_dict() for t in filtered] def get_revenue_metrics(self) -> dict: """Extract revenue-specific metrics from traced calls.""" escrow_creates = [ t for t in self._traces if t.tool == "create_escrow" and t.success ] escrow_releases = [ t for t in self._traces if t.tool == "release_escrow" and t.success ] deposits = [ t for t in self._traces if t.tool == "deposit" and t.success ] total_escrowed = sum( float(t.input_data.get("amount", 0)) for t in escrow_creates ) total_deposited = sum( float(t.input_data.get("amount", 0)) for t in deposits ) return { "escrows_created": len(escrow_creates), "escrows_released": len(escrow_releases), "total_escrowed": round(total_escrowed, 2), "total_deposited": round(total_deposited, 2), "release_rate": ( round(len(escrow_releases) / len(escrow_creates), 4) if escrow_creates else 0 ), } ``` ### Structured Logging for Agent Decisions The tracer's structured logging integrates with any log aggregator (Datadog, ELK, CloudWatch). Each log line is a JSON object with consistent fields. The key decision: log the tool name and duration for every call, but redact input data in production to avoid logging sensitive information like API keys or wallet amounts. Enable full input logging only in staging (P7 security patterns). ```python # Production logging configuration import logging def configure_production_logging(): """Set up structured JSON logging for agent commerce.""" logger = logging.getLogger("agent_tracer") logger.setLevel(logging.INFO) handler = logging.StreamHandler() handler.setFormatter(logging.Formatter("%(message)s")) logger.addHandler(handler) return logger def configure_staging_logging(): """Staging logger with full input/output capture.""" logger = logging.getLogger("agent_tracer") logger.setLevel(logging.DEBUG) handler = logging.FileHandler("/var/log/agent-commerce/traces.jsonl") handler.setFormatter(logging.Formatter("%(message)s")) logger.addHandler(handler) return logger ``` ### Custom Metrics Dashboard Extract the metrics you need for a Grafana or Datadog dashboard from the tracer. These are the five metrics every agent commerce system should track. ```python class MetricsExporter: """Export AgentTracer metrics to monitoring systems.""" def __init__(self, tracer: AgentTracer): self.tracer = tracer def export_prometheus(self) -> str: """Export metrics in Prometheus text format.""" metrics = self.tracer.get_metrics() revenue = self.tracer.get_revenue_metrics() lines = [ f'agent_commerce_calls_total {metrics["total_calls"]}', f'agent_commerce_success_rate {metrics["success_rate"]}', f'agent_commerce_latency_p50_ms {metrics.get("p50_latency_ms", 0)}', f'agent_commerce_latency_p99_ms {metrics.get("p99_latency_ms", 0)}', f'agent_commerce_escrows_created {revenue["escrows_created"]}', f'agent_commerce_escrows_released {revenue["escrows_released"]}', f'agent_commerce_total_escrowed {revenue["total_escrowed"]}', f'agent_commerce_release_rate {revenue["release_rate"]}', ] for tool, rate in metrics.get("error_rate_by_tool", {}).items(): lines.append( f'agent_commerce_error_rate{{tool="{tool}"}} {rate}' ) return "\n".join(lines) def to_datadog_events(self) -> list[dict]: """Format failed traces as Datadog events.""" failed = self.tracer.get_traces(success=False) return [ { "title": f"Tool failure: {t['tool']}", "text": t.get("error", "Unknown error"), "tags": [ f"tool:{t['tool']}", f"agent:{t['agent_id']}", "service:agent-commerce", ], "alert_type": "error", } for t in failed ] ``` --- ## Chapter 6: Alerting & Incident Response ### What to Alert On Not every error deserves a page. These six conditions are the ones that, left unaddressed, cause financial loss in agent commerce systems. | Alert | Condition | Severity | Action | |-------|-----------|----------|--------| | Settlement failure | `release_escrow` returns error 3+ times | Critical | Check gateway status, pause new escrows | | Escrow timeout | Escrow unfunded/unreleased past deadline | High | Auto-cancel or escalate to dispute | | Balance anomaly | Balance drops >50% in single transaction | Critical | Pause agent, audit recent calls | | Reputation drop | Trust score drops below threshold | Medium | Pause hiring, investigate metrics | | Webhook delivery failure | >5 consecutive webhook delivery failures | High | Check endpoint, enable retry queue | | Duplicate payment | Same escrow_id released twice in 60s | Critical | Immediate halt, audit ledger | ### The HealthChecker The `HealthChecker` runs synthetic transactions against the sandbox (or a dedicated health-check agent on production) to verify the full payment pipeline is operational. Run it every 60 seconds from your monitoring system. ```python class HealthChecker: """Synthetic transaction health checks for agent commerce. Runs a mini escrow lifecycle (create wallet → deposit → create escrow → release → verify balance) and reports pass/fail with latency metrics. Usage: checker = HealthChecker( [REDACTED_SECRET]", agent_id="health-check-agent", base_url="https://sandbox.greenhelix.net/v1", ) result = checker.run_health_check() # { # "healthy": True, # "checks": { # "wallet": {"status": "pass", "latency_ms": 45.2}, # "deposit": {"status": "pass", "latency_ms": 78.1}, # "escrow_create": {"status": "pass", "latency_ms": 112.4}, # "escrow_release": {"status": "pass", "latency_ms": 95.6}, # "balance_verify": {"status": "pass", "latency_ms": 41.0}, # }, # "total_latency_ms": 372.3, # } """ def __init__( self, api_key: str, agent_id: str, base_url: str = "https://sandbox.greenhelix.net/v1", timeout_ms: float = 5000.0, ): self.[REDACTED_SECRET] self.agent_id = agent_id self.base_url = base_url self.timeout_ms = timeout_ms self._session = requests.Session() self._session.headers.update({ "Content-Type": "application/json", "Authorization": f"Bearer {api_key}", }) def _timed_execute(self, tool: str, input_data: dict) -> tuple[dict, float]: """Execute a tool and return (result, latency_ms).""" start = time.time() resp = self._session.post( f"{self.base_url}/v1", json={"tool": tool, "input": input_data}, timeout=self.timeout_ms / 1000.0, ) latency_ms = (time.time() - start) * 1000 resp.raise_for_status() return resp.json(), latency_ms def run_health_check(self) -> dict: """Run a full synthetic transaction health check.""" checks = {} healthy = True health_agent = f"{self.agent_id}-{int(time.time())}" # Check 1: Wallet creation try: result, latency = self._timed_execute("create_wallet", {}) checks["wallet"] = {"status": "pass", "latency_ms": round(latency, 2)} except Exception as e: checks["wallet"] = {"status": "fail", "error": str(e)} healthy = False # Check 2: Deposit try: result, latency = self._timed_execute("deposit", {"amount": "1.00"}) checks["deposit"] = {"status": "pass", "latency_ms": round(latency, 2)} except Exception as e: checks["deposit"] = {"status": "fail", "error": str(e)} healthy = False # Check 3: Escrow creation escrow_id = None try: result, latency = self._timed_execute("create_escrow", { "payer_agent_id": health_agent, "payee_agent_id": f"{health_agent}-payee", "amount": "0.01", "description": "Health check escrow", }) escrow_id = result.get("escrow_id") checks["escrow_create"] = {"status": "pass", "latency_ms": round(latency, 2)} except Exception as e: checks["escrow_create"] = {"status": "fail", "error": str(e)} healthy = False # Check 4: Escrow release if escrow_id: try: result, latency = self._timed_execute("release_escrow", { "escrow_id": escrow_id, }) checks["escrow_release"] = {"status": "pass", "latency_ms": round(latency, 2)} except Exception as e: checks["escrow_release"] = {"status": "fail", "error": str(e)} healthy = False else: checks["escrow_release"] = {"status": "skip", "reason": "no escrow_id"} # Check 5: Balance verification try: result, latency = self._timed_execute("get_balance", {}) checks["balance_verify"] = {"status": "pass", "latency_ms": round(latency, 2)} except Exception as e: checks["balance_verify"] = {"status": "fail", "error": str(e)} healthy = False total_latency = sum( c.get("latency_ms", 0) for c in checks.values() ) return { "healthy": healthy, "checks": checks, "total_latency_ms": round(total_latency, 2), "timestamp": time.time(), "agent_id": self.agent_id, } def run_and_alert(self, alert_callback: Callable = None) -> dict: """Run health check and trigger alert callback on failure.""" result = self.run_health_check() if not result["healthy"] and alert_callback: failed_checks = { name: check for name, check in result["checks"].items() if check.get("status") == "fail" } alert_callback({ "severity": "critical", "title": "Agent Commerce Health Check Failed", "failed_checks": failed_checks, "timestamp": result["timestamp"], }) return result ``` ### Testing the HealthChecker ```python class TestHealthChecker: """Verify the health checker itself works correctly.""" def test_reports_healthy_when_all_pass(self, harness): """All checks passing produces healthy=True.""" harness.use_mocks() harness.mock_tool("create_wallet", {"wallet_id": "w-health"}) harness.mock_tool("deposit", {"balance": "1.00", "transaction_id": "tx-h"}) harness.mock_tool("create_escrow", { "escrow_id": "escrow-health", "status": "funded", }) harness.mock_tool("release_escrow", { "escrow_id": "escrow-health", "status": "released", }) harness.mock_tool("get_balance", {"balance": "0.99"}) # HealthChecker delegates to the harness in test mode checker = HealthChecker( [REDACTED_SECRET], agent_id=harness.agent_id, base_url=harness.base_url, ) # In real tests, patch _timed_execute to use harness # Here we verify the structure assert checker.timeout_ms == 5000.0 def test_reports_unhealthy_on_escrow_failure(self): """Failed escrow check produces healthy=False.""" checker = HealthChecker( api_key="test", agent_id="test", base_url="https://sandbox.greenhelix.net/v1", ) alerts_received = [] def mock_alert(alert): alerts_received.append(alert) # Patch to simulate failure -- in real code use sandbox original = checker._timed_execute def failing_execute(tool, input_data): if tool == "create_escrow": raise ConnectionError("Gateway timeout") return original(tool, input_data) checker._timed_execute = failing_execute try: result = checker.run_and_alert(alert_callback=mock_alert) except Exception: pass # Expected when wallet/deposit also fail ``` ### Dead Letter Queue for Webhooks When webhook delivery fails, events must not be silently dropped. Implement a dead letter queue that captures failed deliveries for retry (P4). ```python class WebhookDeadLetterQueue: """Captures failed webhook deliveries for manual retry.""" def __init__(self, max_retries: int = 3, retry_delay_seconds: float = 60.0): self.max_retries = max_retries self.retry_delay_seconds = retry_delay_seconds self._queue: list[dict] = [] def enqueue(self, event: dict, error: str): """Add a failed webhook event to the dead letter queue.""" self._queue.append({ "event": event, "error": error, "retries": 0, "enqueued_at": time.time(), "next_retry_at": time.time() + self.retry_delay_seconds, }) def get_pending(self) -> list[dict]: """Get events ready for retry.""" now = time.time() return [ entry for entry in self._queue if entry["retries"] < self.max_retries and entry["next_retry_at"] <= now ] def mark_delivered(self, event_id: str): """Remove a successfully retried event.""" self._queue = [ e for e in self._queue if e["event"].get("event_id") != event_id ] def mark_retried(self, event_id: str): """Increment retry count and schedule next attempt.""" for entry in self._queue: if entry["event"].get("event_id") == event_id: entry["retries"] += 1 entry["next_retry_at"] = ( time.time() + self.retry_delay_seconds * (2 ** entry["retries"]) ) break def get_dead_letters(self) -> list[dict]: """Get events that have exhausted all retries.""" return [ entry for entry in self._queue if entry["retries"] >= self.max_retries ] ``` --- ## Chapter 7: CI/CD for Agent Systems ### Running Tests in GitHub Actions Agent commerce tests require three things that standard CI does not provide: a GreenHelix API key for sandbox testing, isolation between parallel test runs (unique agent IDs), and network access to `sandbox.greenhelix.net`. This GitHub Actions template handles all three. ```yaml # .github/workflows/agent-commerce-tests.yml name: Agent Commerce Tests on: push: branches: [main, develop] pull_request: branches: [main] env: GREENHELIX_API_KEY: ${{ secrets.GREENHELIX_API_KEY }} GREENHELIX_BASE_URL: https://sandbox.greenhelix.net/v1 jobs: # ── Layer 1: Fast mock-based tests (no network) ────────── unit-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: | pip install pytest requests cryptography - name: Run unit tests (Layer 1) run: | pytest tests/ -x -q \ -m "not sandbox and not chaos" \ --tb=short \ --junit-xml=results/unit-tests.xml - name: Upload test results if: always() uses: actions/upload-artifact@v4 with: name: unit-test-results path: results/unit-tests.xml # ── Layer 2: Contract tests against sandbox ────────────── contract-tests: runs-on: ubuntu-latest needs: unit-tests steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: pip install pytest requests cryptography - name: Run contract tests (Layer 2) run: | pytest tests/ -x -q \ -m "sandbox and not chaos" \ --tb=short \ --junit-xml=results/contract-tests.xml env: GREENHELIX_API_KEY: ${{ secrets.GREENHELIX_API_KEY }} - name: Upload test results if: always() uses: actions/upload-artifact@v4 with: name: contract-test-results path: results/contract-tests.xml # ── Layer 3 + 4: Integration and chaos tests ───────────── integration-tests: runs-on: ubuntu-latest needs: contract-tests steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: pip install pytest requests cryptography - name: Run integration tests (Layer 3) run: | pytest tests/ -x -q \ -m "integration" \ --tb=short \ --junit-xml=results/integration-tests.xml env: GREENHELIX_API_KEY: ${{ secrets.GREENHELIX_API_KEY }} - name: Run chaos tests (Layer 4) run: | pytest tests/ -q \ -m "chaos" \ --tb=short \ --junit-xml=results/chaos-tests.xml env: GREENHELIX_API_KEY: ${{ secrets.GREENHELIX_API_KEY }} continue-on-error: true # Chaos tests may have expected failures - name: Upload all results if: always() uses: actions/upload-artifact@v4 with: name: integration-test-results path: results/ # ── Health check against staging ───────────────────────── staging-health: runs-on: ubuntu-latest needs: integration-tests if: github.ref == 'refs/heads/main' steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: pip install requests - name: Run health check against staging run: | python -c " from health_checker import HealthChecker import json, sys checker = HealthChecker( api_key='${{ secrets.GREENHELIX_API_KEY }}', agent_id='ci-health-check', base_url='https://sandbox.greenhelix.net/v1', ) result = checker.run_health_check() print(json.dumps(result, indent=2)) if not result['healthy']: print('HEALTH CHECK FAILED') sys.exit(1) print('HEALTH CHECK PASSED') " ``` ### Staging with the GreenHelix Sandbox The sandbox at `sandbox.greenhelix.net` mirrors the production API exactly. Use it as your staging environment. Every test in Layers 2-4 runs against it. The sandbox resets balances nightly, so do not rely on state persisting between CI runs. Generate unique agent IDs per run using the `agent_id` fixture from the conftest (P1). ```python # pytest.ini or pyproject.toml # [tool.pytest.ini_options] # markers: # sandbox: Tests that require the GreenHelix sandbox # chaos: Chaos testing with failure injection # integration: Multi-step workflow integration tests @pytest.fixture def ci_agent_id(): """Generate a CI-unique agent ID to prevent collision.""" run_id = os.environ.get("GITHUB_RUN_ID", uuid.uuid4().hex[:8]) return f"ci-agent-{run_id}-{uuid.uuid4().hex[:6]}" ``` ### Canary Deployments When deploying agent commerce changes, use a canary pattern: route 5% of traffic to the new version, monitor the `AgentTracer` metrics for 15 minutes, then promote or roll back. ```python class CanaryDeployment: """Canary deployment controller for agent commerce systems.""" def __init__( self, canary_tracer: AgentTracer, stable_tracer: AgentTracer, promotion_threshold: float = 0.95, ): self.canary = canary_tracer self.stable = stable_tracer self.promotion_threshold = promotion_threshold def evaluate(self) -> dict: """Compare canary metrics against stable baseline.""" canary_metrics = self.canary.get_metrics() stable_metrics = self.stable.get_metrics() if canary_metrics["total_calls"] < 10: return {"decision": "waiting", "reason": "insufficient_data"} canary_success = canary_metrics.get("success_rate", 0) stable_success = stable_metrics.get("success_rate", 1) canary_latency = canary_metrics.get("p99_latency_ms", 0) stable_latency = stable_metrics.get("p99_latency_ms", 1) # Canary must match or beat stable on success rate success_ok = canary_success >= stable_success * 0.98 # Canary latency must not regress more than 20% latency_ok = canary_latency <= stable_latency * 1.20 if success_ok and latency_ok: return {"decision": "promote", "canary_success": canary_success} else: return { "decision": "rollback", "reason": ( f"success: {canary_success} vs {stable_success}, " f"p99: {canary_latency}ms vs {stable_latency}ms" ), } ``` ### Regression Detection Track key metrics across CI runs to detect regressions before they reach production. Store the metrics as CI artifacts and compare against the previous run. ```python class RegressionDetector: """Detect metric regressions between CI runs.""" def __init__(self, baseline_metrics: dict, current_metrics: dict): self.baseline = baseline_metrics self.current = current_metrics def check(self) -> list[dict]: """Return a list of detected regressions.""" regressions = [] # Success rate regression (any drop is a regression) baseline_sr = self.baseline.get("success_rate", 1.0) current_sr = self.current.get("success_rate", 1.0) if current_sr < baseline_sr - 0.01: regressions.append({ "metric": "success_rate", "baseline": baseline_sr, "current": current_sr, "delta": current_sr - baseline_sr, }) # Latency regression (>20% increase at p95) baseline_p95 = self.baseline.get("p95_latency_ms", 0) current_p95 = self.current.get("p95_latency_ms", 0) if baseline_p95 > 0 and current_p95 > baseline_p95 * 1.20: regressions.append({ "metric": "p95_latency_ms", "baseline": baseline_p95, "current": current_p95, "delta_pct": round( (current_p95 - baseline_p95) / baseline_p95 * 100, 1 ), }) # Error rate regression per tool baseline_errors = self.baseline.get("error_rate_by_tool", {}) current_errors = self.current.get("error_rate_by_tool", {}) for tool, current_rate in current_errors.items(): baseline_rate = baseline_errors.get(tool, 0) if current_rate > baseline_rate + 0.05: regressions.append({ "metric": f"error_rate:{tool}", "baseline": baseline_rate, "current": current_rate, }) return regressions ``` --- ## Chapter 8: What to Do Next This guide covered the four-layer agent testing pyramid, tool contract tests, saga-based workflow testing, chaos failure injection, production observability with the `AgentTracer`, alerting with the `HealthChecker`, and CI/CD integration with GitHub Actions. The four classes -- `AgentTestHarness`, `ChaosMiddleware`, `AgentTracer`, and `HealthChecker` -- compose into a reliability stack that wraps every GreenHelix tool call from development through production. The GreenHelix gateway's own test suite (260+ tests across 9 modules, with the gateway alone carrying 1,353 tests) uses the same patterns described here: deterministic mocks for business logic, sandbox integration tests for contract verification, and chaos-style failure injection for payment idempotency. The patterns in this guide are not theoretical -- they are extracted from the test infrastructure that protects the gateway itself. ### Companion Guides For the commerce patterns these tests protect, see the companion guides: - **Agent-to-Agent Commerce: Escrow, Payments, and Trust** (P1) -- the `AgentCommerce` class, escrow patterns, marketplace discovery, subscriptions, and dispute resolution. - **The AI Agent FinOps Playbook** (P2) -- the `AgentFinOps` class, per-agent budget caps, webhook alerts, fleet dashboards, and cost attribution. - **Verified Trading Bot Reputation** (P3) -- cryptographic PnL proof using Ed25519 signatures and Merkle claim chains. - **Tamper-Proof Audit Trails for Trading Bots** (P4) -- EU AI Act compliance, MiFID II reporting, and Merkle chain rotation. - **How to Verify Any AI Agent Before Doing Business** (P5) -- the `AgentVerifier` class, five-layer trust stack, and continuous reputation monitoring. - **The Agent Strategy Marketplace Playbook** (P6) -- selling verified trading strategies with performance escrow. - **Locking Down Agent Commerce** (P7) -- OWASP-aligned security hardening with `SecureAgent` and `SecurityMonitor`. - **The Agent SaaS Factory** (P8) -- autonomous micro-SaaS creation with `AgentDeveloper`, `AgentDBA`, and `AgentBilling`. ### Practice on the Sandbox The sandbox at `sandbox.greenhelix.net` is free to use with any API key. Run the `HealthChecker` against it. Deploy the GitHub Actions template. Break things deliberately with the `ChaosMiddleware`. The patterns in this guide are designed to be copied, adapted, and deployed today. ### The Bundle All eight companion guides plus this cookbook are available as a bundle. Each guide introduces a production-ready Python class. Together they cover the full lifecycle of agent commerce: building, securing, testing, monitoring, and scaling. For the full API reference and tool catalog (all 128 tools), visit the GreenHelix developer documentation at [https://api.greenhelix.net/docs](https://api.greenhelix.net/docs). --- *Price: $29 | Format: Digital Guide | Updates: Lifetime access*


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: clean (LOW)