Skill: tensorlink-dev/codebase-guide (version 1.0.0)

--- name: codebase-guide description: "Use for every task involving this project. Covers running Ganglion, its CLI commands, HTTP bridge API, pipeline execution, knowledge queries, configuration, and operational workflows. Trigger phrases: 'run the pipeline', 'start the server', 'check status', 'query knowledge', 'configure', 'call the API', 'scaffold a project', 'check metrics', 'rollback', 'swap policy'." homepage: https://github.com/TensorLink-AI/ganglion metadata: {"openclaw": {"emoji": "📘", "requires": {"bins": ["python3", "ganglion"], "env": ["OPENAI_API_KEY"]}, "always": true}} --- # Ganglion — Operator's Manual Ganglion is a domain-specific execution engine for Bittensor subnet mining. It provides a pipeline framework for orchestrating autonomous mining agents that search for optimal model configurations. It exposes a CLI, an HTTP bridge API, and a Python library. Ganglion is search infrastructure — it doesn't know what a good model looks like, it knows how to search for one. ## Quick Reference ```bash # Scaffold a new project ganglion init ./my-subnet --subnet sn9 --netuid 9 # Check state (local mode) ganglion status ./my-subnet ganglion tools ./my-subnet ganglion agents ./my-subnet ganglion knowledge ./my-subnet --capability training --max-entries 10 ganglion pipeline ./my-subnet # Run (local mode) ganglion run ./my-subnet ganglion run ./my-subnet --stage plan ganglion run ./my-subnet --overrides '{"target_metric":"accuracy"}' # Start HTTP bridge (remote mode) ganglion serve ./my-subnet --bot-id alpha --port 8899 # Check state (remote mode) curl -s "$GANGLION_URL/v1/status" | jq .data ``` ## Mode Detection Ganglion supports two modes. **Always check which mode applies before running commands.** - **Local mode**: No `GANGLION_URL` set, or `GANGLION_PROJECT` is set. Use `ganglion <command> <project_dir>` directly. - **Remote mode**: `GANGLION_URL` is set. Use `curl` against the HTTP bridge. ```bash if [ -n "$GANGLION_PROJECT" ] || [ -z "$GANGLION_URL" ]; then echo "local" else echo "remote" fi ``` ## Response Format All HTTP bridge endpoints (except health probes) return responses in a standard envelope: - **Success**: `{"data": <payload>}` — use `jq .data` to extract - **Error**: `{"detail": {"error": {"code": "ERROR_CODE", "message": "..."}}}` Health probes (`/healthz`, `/readyz`) return raw JSON without the envelope. Interactive API docs: `$GANGLION_URL/v1/docs` (Swagger UI). > **Note:** Unversioned routes (e.g. `/status`) still work but are deprecated. Always use `/v1/`. ## How to Run **Prerequisites:** Python >= 3.11, `OPENAI_API_KEY` set (used by the LLM runtime). **Install:** `pip install ganglion` **Scaffold a project:** ```bash ganglion init ./my-subnet --subnet sn9 --netuid 9 ``` This creates `config.py`, `tools/`, `agents/`, and `skill/` in the target directory. **Start in local mode:** ```bash export GANGLION_PROJECT=./my-subnet ganglion status $GANGLION_PROJECT ``` **Start in remote mode:** ```bash ganglion serve ./my-subnet --bot-id alpha --port 8899 export GANGLION_URL=http://127.0.0.1:8899 ``` The project directory must contain a `config.py` that defines `subnet_config` (SubnetConfig) and `pipeline` (PipelineDef). See `{baseDir}/references/configuration.md` for full config details. ## Key Features ### Observe State Query the current framework state — registered tools, agents, pipeline definition, knowledge, metrics, and run history. Local mode uses CLI commands; remote mode uses GET endpoints. Full reference: `{baseDir}/references/commands.md` ### Execute Pipelines Run the full pipeline or a single stage. The orchestrator executes stages in dependency order, applies retry policies, injects accumulated knowledge into agent prompts, and records outcomes. ```bash # Local ganglion run ./my-subnet ganglion run ./my-subnet --stage plan # Remote curl -s -X POST "$GANGLION_URL/v1/run/pipeline" -H "Content-Type: application/json" -d '{}' | jq .data curl -s -X POST "$GANGLION_URL/v1/run/stage/plan" -H "Content-Type: application/json" -d '{}' | jq .data ``` ### Mutate at Runtime (Remote Only) Register new tools, agents, and components; patch the pipeline; swap retry policies; update prompts. All mutations are validated, audited, and reversible. ```bash # Register a tool curl -s -X POST "$GANGLION_URL/v1/tools" -H "Content-Type: application/json" \ -d '{"name":"my_tool","code":"<code>","category":"training"}' | jq .data # Patch pipeline curl -s -X PATCH "$GANGLION_URL/v1/pipeline" -H "Content-Type: application/json" \ -d '{"operations":[{"op":"add_stage","stage":{"name":"validate","agent":"Validator","depends_on":["train"]}}]}' | jq .data ``` Pipeline operations: `add_stage`, `remove_stage`, `update_stage`. See `{baseDir}/references/commands.md` for all mutation endpoints. ### Knowledge Store Cross-run strategic memory that compounds over time. Records patterns (what worked) and antipatterns (what failed), then automatically injects relevant history into agent prompts. Knowledge is queried by capability and filtered by bot_id for multi-bot setups. ```bash # Local ganglion knowledge ./my-subnet --bot-id alpha --capability training # Remote curl -s "$GANGLION_URL/v1/knowledge?capability=training&max_entries=10" | jq ``` ### Rollback Undo any mutation. Every mutation is recorded in an audit log with rollback data. ```bash curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq # undo ALL mutations ``` ### Multi-Bot Workflows Multiple OpenClaw sessions share a knowledge pool via `--bot-id`. Each bot's discoveries flow into the shared pool. Cooperation emerges from shared knowledge, not explicit coordination. ```bash # Two local sessions ganglion run ./my-subnet --bot-id alpha ganglion run ./my-subnet --bot-id beta # Two remote servers ganglion serve ./my-subnet --bot-id alpha --port 8899 ganglion serve ./my-subnet --bot-id beta --port 8900 ``` ### MCP Integration Connect to external MCP servers to add tools to the agent's repertoire at runtime. Tools from MCP servers appear as regular Ganglion tools with a prefix. ```bash # Static: add to config.py # from ganglion.mcp.config import MCPClientConfig # mcp_clients = [MCPClientConfig(name="weather", transport="stdio", command=["python", "-m", "weather_server"])] # Dynamic: add at runtime via API curl -s -X POST "$GANGLION_URL/v1/mcp/servers" -H "Content-Type: application/json" \ -d '{"name":"weather","transport":"stdio","command":["python","-m","weather_server"]}' | jq .data # Check connected MCP servers curl -s "$GANGLION_URL/v1/mcp" | jq .data # Disconnect curl -s -X DELETE "$GANGLION_URL/v1/mcp/servers/weather" | jq .data # Expose Ganglion tools as MCP server (for Claude Desktop etc.) ganglion mcp-serve ./my-subnet --transport stdio ``` ## Common Workflows See `{baseDir}/examples/common-workflows.md` for full step-by-step guides. 1. **First run**: `ganglion init` → edit `config.py` → `ganglion run` 2. **Iterative mining**: check status → review knowledge → run pipeline → check metrics → repeat 3. **Dynamic mutation**: observe tools/agents → register new tool via API → patch pipeline → run 4. **Multi-bot setup**: start multiple servers with different `--bot-id` values on the same project 5. **MCP integration**: connect external tool servers → tools appear in registry → agents can use them ## When Things Go Wrong | Symptom | Likely Cause | Fix | |---------|-------------|-----| | `FileNotFoundError: No config.py` | Wrong project path | Verify path contains `config.py` | | `OPENAI_API_KEY` errors | Missing or invalid API key | `export OPENAI_API_KEY=sk-...` | | `ConcurrentMutationError` | Mutating during a pipeline run | Wait for the run to finish | | `PipelineValidationError` | Invalid pipeline DAG (cycles, missing deps) | Check `ganglion pipeline` output | | Agent stuck / max turns reached | Agent cannot make progress | Review knowledge, swap retry policy, adjust prompts | Full troubleshooting: `{baseDir}/references/troubleshooting.md` ## Retry Policies Four built-in policies control how stages retry on failure: - **NoRetry** — single attempt - **FixedRetry** — retry N times (default: 3) - **EscalatingRetry** — increase temperature per attempt, optional stall detection - **ModelEscalationRetry** — climb a model cost ladder (cheap → expensive) Three presets: `SN50_PRESET` (escalating + stall detection), `SIMPLE_PRESET` (fixed), `AGGRESSIVE_PRESET` (model escalation). ## Additional Resources - Full CLI & API reference: `{baseDir}/references/commands.md` - Configuration guide: `{baseDir}/references/configuration.md` - Operational procedures: `{baseDir}/references/operations.md` - Troubleshooting: `{baseDir}/references/troubleshooting.md` - Workflow examples: `{baseDir}/examples/common-workflows.md` - Sample API requests: `{baseDir}/examples/sample-requests.md` - Health check script: `{baseDir}/scripts/healthcheck.sh`

path: examples/common-workflows.md
content: # Common Workflows

## Workflow 1: First-Time Project Setup and Run

**Goal:** Scaffold a new subnet project, configure it, and run the first pipeline.

### Steps

1. **Scaffold:**
   ```bash
   ganglion init ./sn9-pretrain --subnet sn9 --netuid 9
   ```

2. **Verify the project was created:**
   ```bash
   ls ./sn9-pretrain/
   # Expected: config.py  tools/  agents/  skill/
   ```

3. **Set your API key:**
   ```bash
   export OPENAI_API_KEY=sk-...
   ```

4. **Verify it loads:**
   ```bash
   ganglion status ./sn9-pretrain
   ```

5. **Run the pipeline:**
   ```bash
   ganglion run ./sn9-pretrain
   ```

6. **Check the result:** The output is JSON with `success`, `failed_stage`, and per-stage `results`.

---

## Workflow 2: Iterative Mining Loop (Local Mode)

**Goal:** Run repeated mining cycles, using accumulated knowledge to improve each run.

### Steps

1. **Set the project path:**
   ```bash
   export GANGLION_PROJECT=./sn9-pretrain
   ```

2. **Check prior knowledge:**
   ```bash
   ganglion knowledge $GANGLION_PROJECT --capability training
   ```

3. **Review what tools and agents are available:**
   ```bash
   ganglion tools $GANGLION_PROJECT
   ganglion agents $GANGLION_PROJECT
   ```

4. **Run the pipeline:**
   ```bash
   ganglion run $GANGLION_PROJECT
   ```

5. **Check knowledge after the run:** New patterns and antipatterns are recorded automatically.
   ```bash
   ganglion knowledge $GANGLION_PROJECT
   ```

6. **Repeat** steps 2-5. Each run benefits from knowledge accumulated in prior runs.

---

## Workflow 3: Remote Mode with Dynamic Mutation

**Goal:** Start the HTTP bridge, observe state, register new tools/agents, modify the pipeline, and run.

### Steps

1. **Start the server:**
   ```bash
   ganglion serve ./sn9-pretrain --bot-id alpha --port 8899
   ```

2. **Set the URL:**
   ```bash
   export GANGLION_URL=http://127.0.0.1:8899
   ```

3. **Observe current state:**
   ```bash
   curl -s "$GANGLION_URL/v1/status" | jq
   curl -s "$GANGLION_URL/v1/tools" | jq '.data[].name'
   curl -s "$GANGLION_URL/v1/pipeline" | jq '.data.stages[].name'
   ```

4. **Register a new tool:**
   ```bash
   curl -s -X POST "$GANGLION_URL/v1/tools" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "evaluate_model",
       "code": "from ganglion.composition.tool_registry import tool\nfrom ganglion.composition.tool_returns import ExperimentResult\n\n@tool(\"evaluate_model\", category=\"evaluation\")\ndef evaluate_model(model_path: str, dataset: str = \"default\") -> ExperimentResult:\n    \"\"\"Evaluate a trained model.\"\"\"\n    return ExperimentResult(content=\"Evaluated\", metrics={\"score\": 0.9})",
       "category": "evaluation"
     }' | jq
   ```

5. **Add a new stage to the pipeline:**
   ```bash
   curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
     -H "Content-Type: application/json" \
     -d '{
       "operations": [{
         "op": "add_stage",
         "stage": {
           "name": "final_eval",
           "agent": "Explorer",
           "depends_on": ["experiment"],
           "input_keys": ["experiment_result"],
           "output_keys": ["final_score"]
         }
       }]
     }' | jq
   ```

6. **Verify the pipeline:**
   ```bash
   curl -s "$GANGLION_URL/v1/pipeline" | jq '.data.stages[] | {name, depends_on}'
   ```

7. **Run the pipeline:**
   ```bash
   curl -s -X POST "$GANGLION_URL/v1/run/pipeline" \
     -H "Content-Type: application/json" \
     -d '{}' | jq
   ```

8. **Check metrics:**
   ```bash
   curl -s "$GANGLION_URL/v1/metrics" | jq
   ```

---

## Workflow 4: Multi-Bot Knowledge Sharing

**Goal:** Run multiple mining sessions that share knowledge.

### Same Machine Setup

1. **Terminal 1 — Start bot alpha:**
   ```bash
   ganglion serve ./sn9-pretrain --bot-id alpha --port 8899
   ```

2. **Terminal 2 — Start bot beta:**
   ```bash
   ganglion serve ./sn9-pretrain --bot-id beta --port 8900
   ```

3. **Run alpha:**
   ```bash
   curl -s -X POST "http://127.0.0.1:8899/v1/run/pipeline" -H "Content-Type: application/json" -d '{}' | jq .data.success
   ```

4. **Run beta:**
   ```bash
   curl -s -X POST "http://127.0.0.1:8900/v1/run/pipeline" -H "Content-Type: application/json" -d '{}' | jq .data.success
   ```

5. **Check shared knowledge from either bot:**
   ```bash
   curl -s "http://127.0.0.1:8899/v1/knowledge" | jq '.data.patterns | length'
   curl -s "http://127.0.0.1:8900/v1/knowledge" | jq '.data.patterns | length'
   ```

Both bots see patterns from all bots. The knowledge store tags entries with `source_bot` so you can tell who discovered what.

### Local Mode Alternative

```bash
# Terminal 1
ganglion run ./sn9-pretrain --bot-id alpha

# Terminal 2 (after alpha finishes)
ganglion run ./sn9-pretrain --bot-id beta
```

---

## Workflow 5: Recovering from a Bad Mutation

**Goal:** A mutation broke the pipeline — undo it.

### Steps

1. **Check mutation count:**
   ```bash
   curl -s "$GANGLION_URL/v1/status" | jq .data.mutations
   ```

2. **Undo the last mutation:**
   ```bash
   curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq
   ```

3. **Verify the pipeline is valid again:**
   ```bash
   curl -s "$GANGLION_URL/v1/pipeline" | jq
   ```

4. **If multiple bad mutations, undo them all:**
   ```bash
   # Undo everything back to mutation index 0
   curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq
   ```

---

## Workflow 6: Running a Quick Experiment Without the Full Pipeline

**Goal:** Test a specific configuration directly without going through all pipeline stages.

### Steps (Remote Only)

1. **Run the experiment:**
   ```bash
   curl -s -X POST "$GANGLION_URL/v1/run/experiment" \
     -H "Content-Type: application/json" \
     -d '{"config": {"learning_rate": 0.001, "epochs": 10, "architecture": "resnet18"}}' | jq
   ```

2. **Check the result:** The response includes `success`, `content`, and `metrics`.

Note: This calls the registered `run_experiment` tool directly. If no such tool exists, you'll get an error.

---

## Workflow 7: Swapping Retry Policy After Poor Results

**Goal:** A stage keeps failing — switch to a more aggressive retry strategy.

### Steps

1. **Check current pipeline and retry policies:**
   ```bash
   curl -s "$GANGLION_URL/v1/pipeline" | jq '.data.stages[] | {name, retry}'
   ```

2. **Swap to escalating retry with stall detection:**
   ```bash
   curl -s -X PUT "$GANGLION_URL/v1/policies/train" \
     -H "Content-Type: application/json" \
     -d '{"retry_policy": {"type": "escalating", "max_attempts": 5, "temperature_step": 0.1}}' | jq
   ```

3. **Re-run the pipeline:**
   ```bash
   curl -s -X POST "$GANGLION_URL/v1/run/pipeline" -H "Content-Type: application/json" -d '{}' | jq
   ```


path: examples/sample-requests.md
content: # Sample API Requests & Responses

All examples use `GANGLION_URL=http://127.0.0.1:8899`.

## Observation

### Get Full Status
```bash
curl -s "$GANGLION_URL/v1/status" | jq
```
```json
{
  "data": {
    "subnet": {
      "netuid": 9,
    "name": "SN9 Pretrain",
    "metrics": [{"name": "loss", "direction": "minimize", "weight": 1.0, "description": "Cross-entropy loss"}],
    "tasks": {"pretrain": {"name": "pretrain", "weight": 1.0, "metadata": {}}},
    "output_spec": {"format": "huggingface_model", "shape_constraints": {}, "description": "A pretrained language model"},
    "constraints": {"max_gpu_hours": 24}
  },
  "pipeline": {
    "name": "sn9-pipeline",
    "stages": [
      {"name": "plan", "agent": "Planner", "depends_on": [], "input_keys": [], "output_keys": ["plan"]},
      {"name": "train", "agent": "Trainer", "depends_on": ["plan"], "input_keys": ["plan"], "output_keys": ["model_path", "metrics"]}
    ],
    "default_retry": "FixedRetry(max_attempts=3)"
  },
  "tools": [
    {"name": "run_experiment", "description": "Run a training experiment", "category": "training"}
  ],
  "agents": [
    {"name": "Planner", "module": "planner"},
    {"name": "Trainer", "module": "trainer"}
  ],
  "knowledge": {"patterns": 12, "antipatterns": 5},
    "mutations": 0,
    "running": false
  }
}
```

### List Tools (Filtered)
```bash
curl -s "$GANGLION_URL/v1/tools?category=training" | jq
```
```json
{
  "data": [
    {
      "name": "run_experiment",
      "description": "Run a training experiment with the given config.",
      "category": "training",
      "parameters": {
        "type": "object",
        "properties": {
          "config": {"type": "object"}
        },
        "required": ["config"]
      }
    }
  ]
}
```

### Get Knowledge (Filtered)
```bash
curl -s "$GANGLION_URL/v1/knowledge?capability=training&max_entries=3" | jq
```
```json
{
  "data": {
    "patterns": [
      {
        "capability": "training",
      "description": "Conv+Gaussian head achieved best CRPS",
      "config": {"architecture": "conv_gaussian", "lr": 0.001},
      "metric_value": 0.85,
      "metric_name": "crps",
      "stage": "train",
      "timestamp": "2026-03-04T10:30:00",
      "source_bot": "alpha"
    }
  ],
  "antipatterns": [
    {
      "capability": "training",
      "error_summary": "LSTM diverged after 100 epochs",
      "config": {"architecture": "lstm", "lr": 0.01},
      "failure_mode": "numerical_instability",
      "stage": "train",
      "timestamp": "2026-03-04T09:15:00",
      "source_bot": "alpha"
    }
  ],
    "summary": {"patterns": 12, "antipatterns": 5}
  }
}
```

### Read Source File
```bash
curl -s "$GANGLION_URL/v1/source/tools/run_experiment.py" | jq .data.content -r
```
```python
from ganglion.composition.tool_registry import tool
from ganglion.composition.tool_returns import ExperimentResult

@tool("run_experiment", category="training")
def run_experiment(config: dict) -> ExperimentResult:
    """Run a training experiment with the given config."""
    return ExperimentResult(
        content=f"Experiment completed with config: {config}",
        experiment_id="exp-001",
        metrics={"score": 0.0},
    )
```

### Get Run History
```bash
curl -s "$GANGLION_URL/v1/runs?n=3" | jq
```
```json
{
  "data": [
    {
      "success": true,
      "failed_stage": null,
      "results": {"plan": {"success": true, "attempts": 1}, "train": {"success": true, "attempts": 2}}
    }
  ]
}
```

---

## Execution

### Run Full Pipeline
```bash
curl -s -X POST "$GANGLION_URL/v1/run/pipeline" \
  -H "Content-Type: application/json" \
  -d '{"overrides": {"target_metric": "accuracy"}}' | jq
```
```json
{
  "data": {
    "success": true,
    "failed_stage": null,
    "reason": null,
    "results": {
      "plan": {"success": true, "attempts": 1, "error": null, "structured": {"plan": "..."}},
      "train": {"success": true, "attempts": 2, "error": null, "structured": {"metrics": {"accuracy": 0.92}}}
    }
  }
}
```

### Run Single Stage
```bash
curl -s -X POST "$GANGLION_URL/v1/run/stage/plan" \
  -H "Content-Type: application/json" \
  -d '{"context": {"model_family": "transformer"}}' | jq
```
```json
{
  "data": {
    "success": true,
    "attempts": 1,
    "error": null,
    "structured": {"plan": "Try transformer with attention heads 4, 8, 16"}
  }
}
```

### Run Direct Experiment
```bash
curl -s -X POST "$GANGLION_URL/v1/run/experiment" \
  -H "Content-Type: application/json" \
  -d '{"config": {"learning_rate": 0.001, "epochs": 10}}' | jq
```
```json
{
  "data": {
    "success": true,
    "content": "Experiment completed with config: {'learning_rate': 0.001, 'epochs': 10}",
    "structured": null,
    "metrics": null
  }
}
```

---

## Mutation

### Register a Tool
```bash
curl -s -X POST "$GANGLION_URL/v1/tools" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "analyze_results",
    "code": "from ganglion.composition.tool_registry import tool\nfrom ganglion.composition.tool_returns import ToolOutput\n\n@tool(\"analyze_results\", category=\"analysis\")\ndef analyze_results(experiment_id: str) -> ToolOutput:\n    \"\"\"Analyze results of a past experiment.\"\"\"\n    return ToolOutput(content=f\"Analysis of {experiment_id}\")",
    "category": "analysis"
  }' | jq
```
```json
{"data": {"path": "/home/user/sn9-pretrain/tools/analyze_results.py"}}
```

### Register a Tool (Validation Failure)
```bash
curl -s -X POST "$GANGLION_URL/v1/tools" \
  -H "Content-Type: application/json" \
  -d '{"name": "bad_tool", "code": "def bad_tool(x):\n    pass", "category": "misc"}' | jq
```
```json
{"detail": {"error": {"code": "VALIDATION_FAILED", "message": "No @tool decorator found; Function 'bad_tool' missing docstring"}}}
```

### Patch Pipeline
```bash
curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [
      {"op": "add_stage", "stage": {"name": "analyze", "agent": "Explorer", "depends_on": ["train"], "input_keys": ["metrics"], "output_keys": ["analysis"]}},
      {"op": "update_stage", "stage_name": "train", "updates": {"is_optional": true}}
    ]
  }' | jq
```
```json
{
  "data": {
    "pipeline": {
      "name": "sn9-pipeline",
      "stages": [
        {"name": "plan", "agent": "Planner", "depends_on": []},
        {"name": "train", "agent": "Trainer", "depends_on": ["plan"], "is_optional": true},
        {"name": "analyze", "agent": "Explorer", "depends_on": ["train"]}
      ]
    }
  }
}
```

### Update a Prompt
```bash
curl -s -X POST "$GANGLION_URL/v1/prompts" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "Trainer",
    "prompt_section": "strategy",
    "content": "Prioritize transformer architectures. Start with small models (< 100M params) and scale up only if results are promising."
  }' | jq
```
```json
{"data": {"path": "/home/user/sn9-pretrain/prompts/trainer.py"}}
```

---

## Rollback

### Undo Last Mutation
```bash
curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq
```
```json
{"data": null}
```

### Undo All Mutations
```bash
curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq
```
```json
{"data": null}
```

### Rollback When No Mutations Exist
```bash
curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq
```
```json
{"detail": {"error": {"code": "ROLLBACK_ERROR", "message": "No mutations to rollback"}}}
```


path: references/commands.md
content: # Commands & API Reference

## CLI Commands

All CLI commands are invoked as `ganglion <command> [args]`.

### ganglion init

Scaffold a new subnet project directory.

```bash
ganglion init <target_dir> [--subnet <name>] [--netuid <id>]
```

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `target_dir` | Yes | — | Directory to create |
| `--subnet` | No | `generic` | Subnet name or built-in template |
| `--netuid` | No | `0` | Subnet network UID |

**Creates:**
- `config.py` — subnet configuration
- `tools/run_experiment.py` — starter tool
- `agents/explorer.py` — starter agent
- `skill/SKILL.md` — subnet-specific OpenClaw skill

**Errors:**
- Refuses to overwrite if `config.py` already exists in target

---

### ganglion serve

Start the HTTP bridge server for remote mode.

```bash
ganglion serve <project_dir> [--bot-id <id>] [--host <addr>] [--port <port>]
```

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `project_dir` | Yes | — | Path to project directory (must contain `config.py`) |
| `--bot-id` | No | `None` | Bot identifier for multi-bot knowledge |
| `--host` | No | `127.0.0.1` | Host to bind |
| `--port` | No | `8899` | Port to bind |

**Startup output:** Project root, pipeline name, tool count, agent count, bot ID, endpoint URL.

---

### ganglion status

Show full framework state snapshot.

```bash
ganglion status <project_dir> [--bot-id <id>]
```

**Output (JSON):**
```json
{
  "subnet": {"netuid": 9, "name": "...", "metrics": [...]},
  "pipeline": {"name": "...", "stages": [...]},
  "tools": [...],
  "agents": [...],
  "knowledge": {"patterns": 0, "antipatterns": 0},
  "mutations": 0,
  "running": false
}
```

---

### ganglion tools

List registered tools.

```bash
ganglion tools <project_dir> [--category <category>]
```

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `project_dir` | Yes | — | Path to project directory |
| `--category` | No | `None` | Filter by category (e.g. `training`) |

---

### ganglion agents

List registered agents.

```bash
ganglion agents <project_dir>
```

---

### ganglion knowledge

Show knowledge store contents.

```bash
ganglion knowledge <project_dir> [--bot-id <id>] [--capability <cap>] [--max-entries <n>]
```

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `project_dir` | Yes | — | Path to project directory |
| `--bot-id` | No | `None` | Bot identifier for filtering |
| `--capability` | No | `None` | Filter by capability |
| `--max-entries` | No | `20` | Max entries per type |

**Output (JSON):**
```json
{
  "patterns": [{"capability": "...", "description": "...", "metric_value": 0.85, ...}],
  "antipatterns": [{"capability": "...", "error_summary": "...", "failure_mode": "...", ...}],
  "summary": {"patterns": 5, "antipatterns": 3}
}
```

---

### ganglion pipeline

Show current pipeline definition.

```bash
ganglion pipeline <project_dir>
```

**Output (JSON):**
```json
{
  "name": "my-pipeline",
  "stages": [
    {"name": "plan", "agent": "Explorer", "depends_on": [], "input_keys": [], "output_keys": ["plan"], "retry": "FixedRetry(max_attempts=2)"}
  ],
  "default_retry": null
}
```

---

### ganglion run

Run the pipeline or a single stage.

```bash
ganglion run <project_dir> [--bot-id <id>] [--stage <name>] [--overrides '<json>']
```

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `project_dir` | Yes | — | Path to project directory |
| `--bot-id` | No | `None` | Bot identifier |
| `--stage` | No | `None` | Run only this stage |
| `--overrides` | No | `None` | JSON string of TaskContext overrides |

**Output (JSON):**
```json
{
  "success": true,
  "failed_stage": null,
  "reason": null,
  "results": {
    "plan": {"success": true, "attempts": 1, "error": null, "structured": {...}},
    "experiment": {"success": true, "attempts": 2, "error": null, "structured": {...}}
  }
}
```

---

## HTTP Bridge API

Base URL: `http://<host>:<port>` (default `http://127.0.0.1:8899`).

All endpoints are under the `/v1/` prefix. Unversioned routes (e.g. `/status`) still work for backward compatibility but are deprecated.

**Response envelope:** All successful responses are wrapped in `{"data": <payload>}`. Errors return `{"detail": {"error": {"code": "ERROR_CODE", "message": "..."}}}` with an appropriate HTTP status code (400, 403, 404, 500).

Interactive API docs are available at `/v1/docs` (Swagger UI).

### Health Endpoints

#### GET /healthz
Liveness probe — returns 200 if the process is alive.

**Response:** `{"status": "ok"}`

#### GET /readyz
Readiness probe — returns 200 if the bridge is configured and ready. Returns 503 if not yet configured.

**Response:** `{"status": "ready"}`

---

### Observation Endpoints

#### GET /v1/status
Full framework state snapshot. Same output as `ganglion status`.

#### GET /v1/pipeline
Current pipeline definition.

#### GET /v1/tools
Registered tools.

| Query Param | Default | Description |
|-------------|---------|-------------|
| `category` | — | Filter by category |

#### GET /v1/agents
Registered agents.

#### GET /v1/runs
Past pipeline runs.

| Query Param | Default | Description |
|-------------|---------|-------------|
| `n` | `10` | Number of runs to return |

Returns `[]` if no persistence backend configured.

#### GET /v1/metrics
Experiment metrics.

| Query Param | Default | Description |
|-------------|---------|-------------|
| `experiment_id` | — | Filter by experiment ID |

Returns `[]` if no persistence backend configured.

#### GET /v1/leaderboard
Current Bittensor subnet leaderboard. Returns `[]` if no subnet client configured.

#### GET /v1/knowledge
Knowledge store contents.

| Query Param | Default | Description |
|-------------|---------|-------------|
| `capability` | — | Filter by capability |
| `max_entries` | `20` | Max entries per type |

#### GET /v1/source/{path}
Read any file in the project directory. Path traversal (`..`) is blocked.

| Path Param | Description |
|------------|-------------|
| `path` | Relative path within project root |

**Response:** `{"data": {"path": "tools/train.py", "content": "..."}}`

Returns 404 if file not found.

#### GET /v1/components
Available model components. Returns `[]` if no training framework configured.

---

### Mutation Endpoints

All mutations are blocked while a pipeline is running (`ConcurrentMutationError`).

#### POST /v1/tools
Write and register a new tool.

**Request:**
```json
{
  "name": "my_tool",
  "code": "from ganglion.composition.tool_registry import tool\nfrom ganglion.composition.tool_returns import ExperimentResult\n\n@tool(\"my_tool\", category=\"training\")\ndef my_tool(param: str) -> ExperimentResult:\n    \"\"\"Docstring required.\"\"\"\n    return ExperimentResult(content=\"done\")",
  "category": "training",
  "test_code": "assert 1 + 1 == 2"
}
```

**Validation rules:**
- Valid Python syntax
- Must have `@tool` decorator
- All parameters must have type hints
- Docstring required
- No blocked imports: `subprocess`, `os.system`, `shutil.rmtree`, `socket`, `http.server`

**Success (201):** `{"data": {"path": "/path/to/tools/my_tool.py"}}`

**Failure (400):** `{"detail": {"error": {"code": "VALIDATION_FAILED", "message": "No @tool decorator found; Parameter 'x' missing type hint"}}}`

#### POST /v1/agents
Write and register a new agent.

**Request:**
```json
{
  "name": "MyAgent",
  "code": "from ganglion.composition.base_agent import BaseAgentWrapper\nfrom ganglion.composition.tool_registry import build_toolset\n\nclass MyAgent(BaseAgentWrapper):\n    def build_system_prompt(self, task):\n        return \"You are an agent.\"\n    def build_tools(self, task):\n        return build_toolset(\"run_experiment\", \"finish\")",
  "test_task": null
}
```

**Validation rules:**
- Valid Python syntax
- Must inherit from `BaseAgentWrapper`
- Must implement `build_system_prompt` and `build_tools`
- No blocked imports

**Success (201):** `{"data": {"path": "/path/to/agents/myagent.py"}}`

#### POST /v1/components
Write a model component.

**Request:**
```json
{
  "name": "my_backbone",
  "code": "class MyBackbone: ...",
  "component_type": "backbone"
}
```

**Success (201):** `{"data": {"path": "/path/to/components/my_backbone.py"}}`

#### POST /v1/prompts
Write or replace a prompt section for an agent.

**Request:**
```json
{
  "agent_name": "Planner",
  "prompt_section": "strategy",
  "content": "Focus on low parameter count architectures."
}
```

**Success:** `{"data": {"path": "/path/to/prompts/planner.py"}}`

Creates/updates `prompts/{agent_name}.py` with section markers.

#### PATCH /v1/pipeline
Apply pipeline modifications atomically.

**Request:**
```json
{
  "operations": [
    {"op": "add_stage", "stage": {"name": "validate", "agent": "Validator", "depends_on": ["train"], "input_keys": ["model_path"], "output_keys": ["validation"]}},
    {"op": "remove_stage", "stage_name": "old_stage"},
    {"op": "update_stage", "stage_name": "train", "updates": {"is_optional": true, "agent": "NewTrainer"}}
  ]
}
```

**Operations:**

| Operation | Required Fields | Description |
|-----------|----------------|-------------|
| `add_stage` | `stage` (object with `name`, `agent`) | Add a new stage to the pipeline |
| `remove_stage` | `stage_name` | Remove a stage and clean up dependency refs |
| `update_stage` | `stage_name`, `updates` (dict) | Update fields on an existing stage |

**Validation:** Pipeline must remain a valid DAG with all agents registered.

**Success:** `{"data": {"pipeline": {...}}}`

#### PUT /v1/policies/{stage_name}
Swap retry policy for a stage. Use `"default"` as stage_name for the pipeline default.

**Request:**
```json
{
  "retry_policy": {"type": "escalating", "max_attempts": 5, "temperature_step": 0.1}
}
```

**Success:** `{"data": null}`

---

### Execution Endpoints

#### POST /v1/run/pipeline
Execute the full pipeline.

**Request (optional):**
```json
{
  "overrides": {"target_metric": "accuracy"}
}
```

**Response:**
```json
{
  "data": {
    "success": true,
    "failed_stage": null,
    "reason": null,
    "results": {
      "plan": {"success": true, "attempts": 1, "error": null, "structured": {...}}
    }
  }
}
```

#### POST /v1/run/stage/{stage_name}
Execute a single pipeline stage in isolation.

**Request (optional):**
```json
{
  "context": {"model": "resnet18"}
}
```

**Response:**
```json
{
  "data": {
    "success": true,
    "attempts": 1,
    "error": null,
    "structured": {...}
  }
}
```

#### POST /v1/run/experiment
Run a single experiment directly (bypasses the pipeline). Calls the registered `run_experiment` tool.

**Request:**
```json
{
  "config": {"learning_rate": 0.001, "epochs": 10}
}
```

**Response:**
```json
{
  "success": true,
  "content": "Experiment completed. Accuracy: 0.85",
  "structured": {...},
  "metrics": {"accuracy": 0.85}
}
```

Returns an error response if no `run_experiment` tool is registered.

---

### Rollback Endpoints

#### POST /v1/rollback/last
Undo the most recent mutation.

**Response:** `{"data": null}`

Returns 400 with `{"detail": {"error": {"code": "ROLLBACK_ERROR", "message": "No mutations to rollback"}}}` if no mutations exist.

#### POST /v1/rollback/{index}
Undo all mutations back to the given index (0 = undo everything).

**Response:** `{"data": null}`

**What gets rolled back per mutation type:**
- `write_tool` / `write_agent` — file restored or deleted, unregistered from registry
- `patch_pipeline` — previous pipeline definition restored
- `swap_policy` — previous retry policy restored
- `write_prompt` — prompt file restored or deleted

---

## Retry Policy Reference

### NoRetry
Single attempt, no retries.

### FixedRetry
```
FixedRetry(max_attempts=3)
```
Retries up to `max_attempts` times with the same configuration.

### EscalatingRetry
```
EscalatingRetry(max_attempts=5, base_temp=0.1, temp_step=0.1, stall_detector=None)
```
Temperature increases per attempt: `base_temp + (attempt * temp_step)`. If a `stall_detector` is provided and detects repeated outputs, a divergence prompt is injected.

### ModelEscalationRetry
```
ModelEscalationRetry(model_ladder=["haiku-3.5", "sonnet-3.5", "opus-3"], attempts_per_model=2)
```
Tries each model in the ladder for `attempts_per_model` attempts before escalating.

### Presets
- **SN50_PRESET** — EscalatingRetry with ConfigComparisonStallDetector
- **SIMPLE_PRESET** — FixedRetry(max_attempts=3)
- **AGGRESSIVE_PRESET** — ModelEscalationRetry with haiku → sonnet → opus ladder

### Stall Detectors
- **ConfigComparisonStallDetector** — detects when the agent produces the same experiment config twice
- **OutputHashStallDetector** — detects when the agent output text repeats (configurable `max_repeats`, default 2)


path: references/configuration.md
content: # Configuration Reference

## Project Directory Structure

```
my-subnet/
├── config.py              # Subnet configuration (REQUIRED)
├── tools/                 # Tool functions (auto-discovered)
│   └── run_experiment.py
├── agents/                # Agent classes (auto-discovered)
│   └── explorer.py
├── prompts/               # Prompt sections (created by update_prompt mutations)
│   └── explorer.py
├── components/            # Model components (optional)
│   └── my_backbone.py
├── knowledge/             # Knowledge store data (created automatically)
│   ├── patterns.json
│   └── antipatterns.json
└── skill/                 # OpenClaw skill documentation
    └── SKILL.md
```

## config.py

The main configuration file. Must define two required objects and may define two optional ones.

### Required Exports

#### subnet_config (SubnetConfig)

```python
from ganglion.orchestration.task_context import SubnetConfig, MetricDef, OutputSpec, TaskDef

subnet_config = SubnetConfig(
    netuid=9,
    name="My Subnet",
    metrics=[
        MetricDef(name="accuracy", direction="maximize", weight=1.0, description="Primary metric"),
    ],
    tasks={
        "main": TaskDef(name="main", weight=1.0, metadata={}),
    },
    output_spec=OutputSpec(
        format="pytorch_model",
        description="A trained classifier model",
    ),
    constraints={"max_params": 10_000_000},
)
```

**SubnetConfig fields:**

| Field | Type | Description |
|-------|------|-------------|
| `netuid` | `int` | Bittensor subnet network UID |
| `name` | `str` | Human-readable subnet name |
| `metrics` | `list[MetricDef]` | Scoring metrics (name, direction, weight, description) |
| `tasks` | `dict[str, TaskDef]` | Named tasks (name, weight, metadata) |
| `output_spec` | `OutputSpec` | Expected output format and constraints |
| `constraints` | `dict` | Arbitrary constraints passed to agents |

**MetricDef fields:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | `str` | — | Metric identifier |
| `direction` | `"minimize"` or `"maximize"` | — | Optimization direction |
| `weight` | `float` | `1.0` | Relative importance |
| `description` | `str` | `""` | Human-readable description |

#### pipeline (PipelineDef)

```python
from ganglion.orchestration.pipeline import PipelineDef, StageDef
from ganglion.policies.retry import EscalatingRetry, FixedRetry

pipeline = PipelineDef(
    name="my-pipeline",
    stages=[
        StageDef(
            name="plan",
            agent="Explorer",
            output_keys=["plan"],
            retry=FixedRetry(max_attempts=2),
        ),
        StageDef(
            name="experiment",
            agent="Explorer",
            depends_on=["plan"],
            input_keys=["plan"],
            output_keys=["experiment_result"],
            retry=EscalatingRetry(max_attempts=3, base_temp=0.2, temp_step=0.15),
        ),
        StageDef(
            name="evaluate",
            agent="Explorer",
            depends_on=["experiment"],
            input_keys=["experiment_result"],
            output_keys=["evaluation"],
            is_optional=True,
        ),
    ],
    default_retry=FixedRetry(max_attempts=3),
)
```

**StageDef fields:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | `str` | — | Unique stage identifier |
| `agent` | `str` | — | Name of the registered agent class |
| `retry` | `RetryPolicy` | `None` | Stage-level retry policy (falls back to pipeline default) |
| `is_optional` | `bool` | `False` | If true, pipeline continues even if this stage fails |
| `depends_on` | `list[str]` | `[]` | Stages that must complete first |
| `input_keys` | `list[str]` | `[]` | Keys this stage reads from TaskContext |
| `output_keys` | `list[str]` | `[]` | Keys this stage writes to TaskContext |

**Pipeline validation rules:**
- No duplicate stage names
- No cycles in the dependency graph
- All `depends_on` references must exist
- All `input_keys` must be produced by an upstream stage's `output_keys`

### Optional Exports

#### persistence (PersistenceBackend)

Enables run history, metrics queries, and checkpointing. Must implement the `PersistenceBackend` protocol:

```python
class PersistenceBackend(Protocol):
    async def save_checkpoint(self, stage, context_snapshot, result) -> None: ...
    async def load_checkpoint(self, stage) -> tuple[dict, Any] | None: ...
    async def save_run(self, pipeline_result) -> None: ...
    async def load_run_history(self, n=10) -> list[Any]: ...
    async def query_metrics(self, experiment_id=None) -> list[dict]: ...
```

Without persistence, `GET /v1/runs` and `GET /v1/metrics` return empty lists.

#### knowledge (KnowledgeStore)

Enables cross-run strategic memory.

```python
from ganglion.knowledge.store import KnowledgeStore
from ganglion.knowledge.backends.json_backend import JsonKnowledgeBackend

knowledge = KnowledgeStore(
    backend=JsonKnowledgeBackend("./knowledge/"),
    max_patterns=500,
    max_antipatterns=500,
)
```

**KnowledgeStore parameters:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `backend` | `KnowledgeBackend` | — | Storage backend |
| `max_patterns` | `int` | `500` | Max patterns before trimming |
| `max_antipatterns` | `int` | `500` | Max antipatterns before trimming |
| `bot_id` | `str` | `None` | Set automatically by `--bot-id` CLI flag |

**Available backends:**

| Backend | Import | Storage |
|---------|--------|---------|
| `JsonKnowledgeBackend(dir)` | `ganglion.knowledge.backends.json_backend` | JSON files in directory |
| `SqliteKnowledgeBackend(path)` | `ganglion.knowledge.backends.sqlite_backend` | SQLite database file |
| `FederatedKnowledgeBackend(local, peers)` | `ganglion.knowledge.backends.federated` | Write local, read from all peers |

**Federated setup (multi-bot on same filesystem):**
```python
from ganglion.knowledge.backends.federated import FederatedKnowledgeBackend, FilesystemPeerDiscovery

backend = FederatedKnowledgeBackend(
    local=JsonKnowledgeBackend("./shared/alpha/"),
    peers=FilesystemPeerDiscovery(base_dir="./shared/", local_bot_id="alpha"),
)
knowledge = KnowledgeStore(backend=backend)
```

## Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `OPENAI_API_KEY` | Yes | OpenAI API key (read by the openai SDK for LLM calls) |
| `GANGLION_PROJECT` | No | Convention for local mode — set to project directory path |
| `GANGLION_URL` | No | Convention for remote mode — set to HTTP bridge URL |

`GANGLION_PROJECT` and `GANGLION_URL` are conventions used in skill mode detection, not read by the Ganglion code itself. All configuration is file-based via `config.py`.

The `OPENAI_API_KEY` is read by the `openai` Python SDK (a dependency) and is required for any pipeline execution involving LLM agents.

## LLM Client Configuration

The LLM client defaults can be overridden programmatically in config.py or via agent kwargs:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `api_key` | `None` (falls back to `OPENAI_API_KEY`) | API key |
| `base_url` | `None` (OpenAI default) | Base URL for OpenAI-compatible APIs |
| `model` | `gpt-4o` | Default model |
| `request_timeout` | `120.0` | Request timeout in seconds |
| `max_retries` | `5` | Max LLM API retries |
| `base_delay` | `1.0` | Initial retry backoff (seconds) |
| `max_delay` | `60.0` | Maximum retry backoff (seconds) |

## Minimal Working Config

```python
from ganglion.orchestration.task_context import SubnetConfig, MetricDef, OutputSpec, TaskDef
from ganglion.orchestration.pipeline import PipelineDef, StageDef

subnet_config = SubnetConfig(
    netuid=0,
    name="My Subnet",
    metrics=[MetricDef("score", "maximize")],
    tasks={"default": TaskDef("default")},
    output_spec=OutputSpec(format="model_weights"),
)

pipeline = PipelineDef(
    name="minimal",
    stages=[
        StageDef(name="run", agent="Explorer", output_keys=["result"]),
    ],
)
```

## Full Production Config

```python
from ganglion.orchestration.task_context import SubnetConfig, MetricDef, OutputSpec, TaskDef
from ganglion.orchestration.pipeline import PipelineDef, StageDef
from ganglion.policies.retry import EscalatingRetry, FixedRetry, ModelEscalationRetry
from ganglion.policies.stall import ConfigComparisonStallDetector
from ganglion.knowledge.store import KnowledgeStore
from ganglion.knowledge.backends.json_backend import JsonKnowledgeBackend

subnet_config = SubnetConfig(
    netuid=9,
    name="SN9 Pretrain",
    metrics=[
        MetricDef("loss", "minimize", weight=1.0, description="Cross-entropy loss"),
        MetricDef("perplexity", "minimize", weight=0.5, description="Perplexity"),
    ],
    tasks={
        "pretrain": TaskDef("pretrain", weight=1.0, metadata={"max_steps": 10000}),
    },
    output_spec=OutputSpec(
        format="huggingface_model",
        description="A pretrained language model",
        shape_constraints={"max_params": 1_000_000_000},
    ),
    constraints={
        "max_gpu_hours": 24,
        "min_dataset_size": 100_000,
    },
)

stall_detector = ConfigComparisonStallDetector(
    extract_config=lambda r: r.structured.get("config", {})
    if isinstance(r.structured, dict) else {}
)

pipeline = PipelineDef(
    name="sn9-pipeline",
    stages=[
        StageDef(
            name="plan",
            agent="Planner",
            output_keys=["plan", "search_space"],
            retry=FixedRetry(max_attempts=2),
        ),
        StageDef(
            name="screen",
            agent="Screener",
            depends_on=["plan"],
            input_keys=["plan", "search_space"],
            output_keys=["candidates"],
            retry=FixedRetry(max_attempts=3),
            is_optional=True,
        ),
        StageDef(
            name="train",
            agent="Trainer",
            depends_on=["plan"],
            input_keys=["plan"],
            output_keys=["model_path", "metrics"],
            retry=EscalatingRetry(
                max_attempts=5,
                base_temp=0.1,
                temp_step=0.1,
                stall_detector=stall_detector,
            ),
        ),
        StageDef(
            name="evaluate",
            agent="Evaluator",
            depends_on=["train"],
            input_keys=["model_path", "metrics"],
            output_keys=["evaluation", "submission"],
        ),
    ],
    default_retry=FixedRetry(max_attempts=3),
)

knowledge = KnowledgeStore(
    backend=JsonKnowledgeBackend("./knowledge/"),
    max_patterns=500,
    max_antipatterns=500,
)
```


path: references/operations.md
content: # Operational Procedures

## First-Time Setup

1. **Install Ganglion:**
   ```bash
   pip install ganglion
   ```

2. **Set your OpenAI API key:**
   ```bash
   export OPENAI_API_KEY=sk-...
   ```

3. **Scaffold a project:**
   ```bash
   ganglion init ./my-subnet --subnet sn9 --netuid 9
   ```

4. **Edit config.py** with your subnet's metrics, tasks, output spec, and constraints.

5. **Verify the project loads:**
   ```bash
   ganglion status ./my-subnet
   ```

6. **Run the pipeline:**
   ```bash
   ganglion run ./my-subnet
   ```

## Starting and Stopping

### Local Mode
Local mode requires no persistent server. Each command loads the project, executes, and exits.

```bash
# Set convenience variable
export GANGLION_PROJECT=./my-subnet

# Run commands directly
ganglion status $GANGLION_PROJECT
ganglion run $GANGLION_PROJECT
```

### Remote Mode (HTTP Bridge)

**Start:**
```bash
ganglion serve ./my-subnet --bot-id alpha --port 8899 &
export GANGLION_URL=http://127.0.0.1:8899
```

**Verify it's running:**
```bash
# Health check (liveness)
curl -s "$GANGLION_URL/healthz" | jq

# Readiness check
curl -s "$GANGLION_URL/readyz" | jq

# Full status
curl -s "$GANGLION_URL/v1/status" | jq .data.running
```

**Stop:**
Kill the server process (Ctrl+C or `kill` the PID). There is no graceful shutdown endpoint — if a pipeline is running, it will be interrupted.

## Checking Health and Status

### Local Mode
```bash
ganglion status ./my-subnet
```

### Remote Mode
```bash
# Full status
curl -s "$GANGLION_URL/v1/status" | jq

# Is a pipeline currently running?
curl -s "$GANGLION_URL/v1/status" | jq .data.running

# How many tools/agents are registered?
curl -s "$GANGLION_URL/v1/status" | jq '{tools: (.data.tools | length), agents: (.data.agents | length)}'

# Knowledge summary
curl -s "$GANGLION_URL/v1/knowledge" | jq .data.summary
```

### Using the Healthcheck Script
```bash
bash skills/codebase-guide/scripts/healthcheck.sh
```

## Viewing Run History and Metrics

```bash
# Last 5 runs (remote only — requires persistence backend)
curl -s "$GANGLION_URL/v1/runs?n=5" | jq

# All metrics
curl -s "$GANGLION_URL/v1/metrics" | jq

# Metrics for a specific experiment
curl -s "$GANGLION_URL/v1/metrics?experiment_id=exp_001" | jq

# Leaderboard
curl -s "$GANGLION_URL/v1/leaderboard" | jq
```

## Running Pipelines

### Full Pipeline
```bash
# Local
ganglion run ./my-subnet

# With overrides
ganglion run ./my-subnet --overrides '{"target_metric": "accuracy", "max_epochs": 50}'

# Remote
curl -s -X POST "$GANGLION_URL/v1/run/pipeline" \
  -H "Content-Type: application/json" \
  -d '{"overrides": {"target_metric": "accuracy"}}' | jq
```

### Single Stage
```bash
# Local
ganglion run ./my-subnet --stage plan

# Remote
curl -s -X POST "$GANGLION_URL/v1/run/stage/plan" \
  -H "Content-Type: application/json" \
  -d '{"context": {"model": "resnet18"}}' | jq
```

### Direct Experiment (Remote Only)
Bypass the pipeline and call the `run_experiment` tool directly:

```bash
curl -s -X POST "$GANGLION_URL/v1/run/experiment" \
  -H "Content-Type: application/json" \
  -d '{"config": {"learning_rate": 0.001, "epochs": 10}}' | jq
```

## Querying Knowledge

### View All Knowledge
```bash
# Local
ganglion knowledge ./my-subnet

# Remote
curl -s "$GANGLION_URL/v1/knowledge" | jq
```

### Filter by Capability
```bash
# Local
ganglion knowledge ./my-subnet --capability training --max-entries 10

# Remote
curl -s "$GANGLION_URL/v1/knowledge?capability=training&max_entries=10" | jq
```

### Multi-Bot Knowledge
```bash
# View knowledge tagged by a specific bot
ganglion knowledge ./my-subnet --bot-id alpha --capability training
```

## Registering Tools and Agents (Remote Only)

### Register a New Tool
```bash
curl -s -X POST "$GANGLION_URL/v1/tools" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my_tool",
    "code": "from ganglion.composition.tool_registry import tool\nfrom ganglion.composition.tool_returns import ExperimentResult\n\n@tool(\"my_tool\", category=\"training\")\ndef my_tool(config: dict) -> ExperimentResult:\n    \"\"\"Run my custom experiment.\"\"\"\n    return ExperimentResult(content=\"done\", metrics={\"score\": 0.5})",
    "category": "training"
  }' | jq
```

### Register a New Agent
```bash
curl -s -X POST "$GANGLION_URL/v1/agents" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "MyAgent",
    "code": "from ganglion.composition.base_agent import BaseAgentWrapper\nfrom ganglion.composition.tool_registry import build_toolset\n\nclass MyAgent(BaseAgentWrapper):\n    def build_system_prompt(self, task):\n        return \"You are a training agent.\"\n    def build_tools(self, task):\n        return build_toolset(\"my_tool\", \"finish\")"
  }' | jq
```

### Verify Registration
```bash
curl -s "$GANGLION_URL/v1/tools" | jq '.data[].name'
curl -s "$GANGLION_URL/v1/agents" | jq '.data[].name'
```

## Modifying the Pipeline (Remote Only)

### Add a Stage
```bash
curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [{
      "op": "add_stage",
      "stage": {
        "name": "validate",
        "agent": "Validator",
        "depends_on": ["train"],
        "input_keys": ["model_path"],
        "output_keys": ["validation"]
      }
    }]
  }' | jq
```

### Remove a Stage
```bash
curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
  -H "Content-Type: application/json" \
  -d '{"operations": [{"op": "remove_stage", "stage_name": "validate"}]}' | jq
```

### Update a Stage
```bash
curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [{
      "op": "update_stage",
      "stage_name": "train",
      "updates": {"agent": "NewTrainer", "is_optional": true}
    }]
  }' | jq
```

## Swapping Retry Policies (Remote Only)

```bash
# Swap policy for a specific stage
curl -s -X PUT "$GANGLION_URL/v1/policies/train" \
  -H "Content-Type: application/json" \
  -d '{"retry_policy": {"type": "escalating", "max_attempts": 5, "temperature_step": 0.1}}' | jq

# Swap the pipeline default policy
curl -s -X PUT "$GANGLION_URL/v1/policies/default" \
  -H "Content-Type: application/json" \
  -d '{"retry_policy": {"type": "fixed", "max_attempts": 3}}' | jq
```

## Rolling Back Mutations

```bash
# Undo the last mutation
curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq

# Undo all mutations (back to index 0)
curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq

# Check how many mutations exist
curl -s "$GANGLION_URL/v1/status" | jq .data.mutations
```

## Reading Project Source Files (Remote Only)

```bash
# Read a specific tool
curl -s "$GANGLION_URL/v1/source/tools/train.py" | jq .data.content -r

# Read config
curl -s "$GANGLION_URL/v1/source/config.py" | jq .data.content -r

# Read an agent
curl -s "$GANGLION_URL/v1/source/agents/explorer.py" | jq .data.content -r
```

## Multi-Bot Setup

### Same Machine (Filesystem Peer Discovery)

```bash
# Terminal 1
ganglion serve ./my-subnet --bot-id alpha --port 8899

# Terminal 2
ganglion serve ./my-subnet --bot-id beta --port 8900
```

Or local mode:
```bash
# Terminal 1
ganglion run ./my-subnet --bot-id alpha

# Terminal 2
ganglion run ./my-subnet --bot-id beta
```

Both bots read from and write to the same knowledge directory. Each bot's entries are tagged with its `bot_id`.

### Federated Setup (Different Hosts)

Configure the `FederatedKnowledgeBackend` in `config.py` — see configuration reference for details.

## Updating Prompts (Remote Only)

```bash
curl -s -X POST "$GANGLION_URL/v1/prompts" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_name": "Planner",
    "prompt_section": "constraints",
    "content": "Focus on architectures with fewer than 10M parameters."
  }' | jq
```


path: references/troubleshooting.md
content: # Troubleshooting

## Startup Errors

### FileNotFoundError: No config.py found

**Cause:** The project directory path is wrong or `config.py` doesn't exist.

**Fix:**
```bash
# Verify the path
ls -la ./my-subnet/config.py

# If no project exists, scaffold one
ganglion init ./my-subnet --subnet generic
```

### ValueError: config.py must define 'subnet_config'

**Cause:** `config.py` exists but doesn't export a `subnet_config` variable.

**Fix:** Ensure `config.py` contains:
```python
subnet_config = SubnetConfig(...)
```

### ValueError: config.py must define 'pipeline'

**Cause:** `config.py` exists but doesn't export a `pipeline` variable.

**Fix:** Ensure `config.py` contains:
```python
pipeline = PipelineDef(...)
```

### ImportError: Could not load config

**Cause:** `config.py` has a syntax error or imports a missing module.

**Fix:**
```bash
python3 -c "import importlib.util; spec = importlib.util.spec_from_file_location('c', './my-subnet/config.py'); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)"
```
This will show the actual import/syntax error.

### ImportError: openai package is required

**Cause:** The `openai` Python package is not installed.

**Fix:**
```bash
pip install openai>=1.0
# or reinstall ganglion with all deps
pip install ganglion
```

---

## API Key & LLM Errors

### OpenAI AuthenticationError / Invalid API key

**Cause:** `OPENAI_API_KEY` is not set, empty, or invalid.

**Fix:**
```bash
export OPENAI_API_KEY=sk-...
# Verify it's set
echo $OPENAI_API_KEY | head -c 10
```

### RateLimitError during pipeline execution

**Cause:** OpenAI API rate limit exceeded. The LLM client retries automatically with exponential backoff (up to 5 retries, max 60s delay).

**If retries exhaust:**
- Wait a few minutes and retry
- Switch to a model with higher rate limits
- Reduce pipeline concurrency

### APIConnectionError

**Cause:** Network issue reaching the LLM API.

**Fix:**
- Check network connectivity
- If using a custom `base_url`, verify it's reachable
- The client retries automatically — transient issues usually resolve

---

## Pipeline Errors

### PipelineValidationError: Dependency graph contains a cycle

**Cause:** Stage dependencies form a circular chain (e.g., A depends on B, B depends on A).

**Fix:**
```bash
# View the pipeline to find the cycle
ganglion pipeline ./my-subnet
# or
curl -s "$GANGLION_URL/v1/pipeline" | jq '.data.stages[] | {name, depends_on}'
```
Remove or restructure the circular dependency.

### PipelineValidationError: Stage 'X' depends on 'Y' which does not exist

**Cause:** A stage's `depends_on` references a non-existent stage name.

**Fix:** Check stage names and dependencies in `config.py` or use `PATCH /pipeline` to fix.

### PipelineValidationError: input_key 'X' not produced by any upstream stage

**Cause:** A stage declares `input_keys` that no upstream stage produces in its `output_keys`.

**Fix:** Add the missing key to an upstream stage's `output_keys`, or remove it from `input_keys`.

### Stage failed: Agent 'X' not found in registry

**Cause:** A pipeline stage references an agent class that isn't registered. Either the file is missing from `agents/`, has a syntax error, or the class name doesn't match.

**Fix:**
```bash
# Check registered agents
ganglion agents ./my-subnet

# Check the agents directory
ls ./my-subnet/agents/
```

### Max turns reached without calling finish()

**Cause:** The agent looped for 50 turns without completing. It may be stuck, confused, or the tools aren't returning useful results.

**Fix:**
- Review the agent's system prompt — is it clear about when to call `finish()`?
- Review the tools — are they returning actionable information?
- Increase `max_turns` if the task genuinely needs more iterations
- Use EscalatingRetry to inject stall-breaking prompts

---

## Mutation Errors

### ConcurrentMutationError: Cannot mutate during a pipeline run

**Cause:** You tried to register a tool, agent, or patch the pipeline while a pipeline is executing.

**Fix:** Wait for the current run to complete, then retry the mutation.

```bash
# Check if a run is in progress
curl -s "$GANGLION_URL/v1/status" | jq .data.running
```

### Tool validation failed: No @tool decorator found

**Cause:** The tool code submitted via `POST /tools` doesn't contain a `@tool(...)` decorator.

**Fix:** Ensure the code follows the required pattern:
```python
from ganglion.composition.tool_registry import tool

@tool("my_tool", category="training")
def my_tool(param: str) -> ...:
    """Docstring is required."""
    ...
```

### Tool validation failed: Parameter 'X' missing type hint

**Cause:** A tool function parameter doesn't have a type annotation.

**Fix:** Add type hints to all parameters: `def my_tool(x: int, y: str = "default")`.

### Tool validation failed: Blocked import

**Cause:** The tool code uses a blocked import (`subprocess`, `os.system`, `shutil.rmtree`, `socket`, `http.server`).

**Fix:** Remove the blocked import. These are blocked for safety — tools run in the same process.

### Agent validation failed: Missing build_system_prompt / build_tools

**Cause:** The agent class doesn't implement the required methods.

**Fix:** Ensure the agent class inherits from `BaseAgentWrapper` and implements both `build_system_prompt(self, task)` and `build_tools(self, task)`.

### PipelineOperationError: Stage 'X' already exists

**Cause:** Tried to add a stage with a name that's already in the pipeline.

**Fix:** Use `update_stage` instead, or choose a different name.

---

## Knowledge Store Errors

### Empty knowledge (no patterns or antipatterns)

**Cause:** Either no runs have completed yet, or the `knowledge` object isn't defined in `config.py`.

**Fix:**
```bash
# Check if knowledge is configured
ganglion status ./my-subnet | python3 -c "import sys,json; d=json.load(sys.stdin); print('knowledge:', d.get('knowledge'))"
```
If `null`, add a `knowledge = KnowledgeStore(...)` to `config.py`.

### Knowledge not compounding across runs

**Cause:** The knowledge store is configured but the backend directory doesn't persist between runs, or different paths are used.

**Fix:** Verify the knowledge backend directory exists and is the same path across runs:
```bash
ls ./my-subnet/knowledge/
cat ./my-subnet/knowledge/patterns.json | python3 -m json.tool | head -20
```

---

## Rollback Errors

### No mutations to rollback

**Cause:** The mutation log is empty — either no mutations have been made, or the server was restarted (mutations are in-memory only).

**Fix:** This is expected after a server restart. Mutation history does not persist across restarts.

---

## Connection & Network Errors

### Connection refused when calling the HTTP bridge

**Cause:** The bridge server isn't running, or it's on a different host/port.

**Fix:**
```bash
# Check if the server is running
curl -s http://127.0.0.1:8899/v1/status | jq

# Try with the correct host/port
curl -s http://<host>:<port>/v1/status | jq
```

### Server returns 500: Bridge not configured

**Cause:** Internal error — `configure()` was not called before the server started handling requests. This shouldn't happen with normal `ganglion serve` usage.

**Fix:** This indicates a bug or a custom startup script that skipped configuration. Use `ganglion serve` directly.

---

## Tool Loading Warnings

### "Failed to load tool from X.py"

**Cause:** A Python file in `tools/` has a syntax error, missing import, or doesn't contain a `@tool` decorated function. Logged as a warning — other tools still load.

**Fix:**
```bash
# Test the file directly
python3 -c "exec(open('./my-subnet/tools/problem_file.py').read())"
```

### "Failed to load agent from X.py"

**Cause:** A Python file in `agents/` has a syntax error or doesn't contain a `BaseAgentWrapper` subclass.

**Fix:** Same approach — test the file directly with Python.

---

## Getting Diagnostic Information

When reporting issues, gather:

```bash
# Python version
python3 --version

# Ganglion version
pip show ganglion

# Full status dump
ganglion status ./my-subnet 2>&1

# List of files in project
ls -R ./my-subnet/

# Config file contents
cat ./my-subnet/config.py

# Recent knowledge
ganglion knowledge ./my-subnet --max-entries 5 2>&1
```


path: scripts/healthcheck.sh
content: #!/usr/bin/env bash
set -euo pipefail

# Ganglion Health Check
# Usage: bash healthcheck.sh [project_dir] [ganglion_url]
#
# Checks both local and remote mode depending on what's available.

PROJECT_DIR="${1:-${GANGLION_PROJECT:-}}"
GANGLION_URL="${2:-${GANGLION_URL:-}}"

PASS=0
FAIL=0

pass() { echo "  [PASS] $1"; ((PASS++)); }
fail() { echo "  [FAIL] $1"; ((FAIL++)); }

echo "=== Checking prerequisites ==="

# Python
if command -v python3 &>/dev/null; then
    PYVER=$(python3 --version 2>&1)
    pass "Python installed: $PYVER"
else
    fail "python3 not found"
fi

# Ganglion CLI
if command -v ganglion &>/dev/null; then
    pass "ganglion CLI installed"
else
    fail "ganglion CLI not found (pip install ganglion)"
fi

# OpenAI API key
if [ -n "${OPENAI_API_KEY:-}" ]; then
    pass "OPENAI_API_KEY is set"
else
    fail "OPENAI_API_KEY is not set"
fi

echo ""
echo "=== Checking local mode ==="

if [ -n "$PROJECT_DIR" ]; then
    if [ -f "$PROJECT_DIR/config.py" ]; then
        pass "config.py found at $PROJECT_DIR/config.py"
    else
        fail "config.py not found at $PROJECT_DIR/config.py"
    fi

    if [ -d "$PROJECT_DIR/tools" ]; then
        TOOL_COUNT=$(find "$PROJECT_DIR/tools" -name "*.py" ! -name "_*" 2>/dev/null | wc -l)
        pass "tools/ directory exists ($TOOL_COUNT tool files)"
    else
        fail "tools/ directory not found"
    fi

    if [ -d "$PROJECT_DIR/agents" ]; then
        AGENT_COUNT=$(find "$PROJECT_DIR/agents" -name "*.py" ! -name "_*" 2>/dev/null | wc -l)
        pass "agents/ directory exists ($AGENT_COUNT agent files)"
    else
        fail "agents/ directory not found"
    fi

    # Try loading status
    if command -v ganglion &>/dev/null; then
        if ganglion status "$PROJECT_DIR" &>/dev/null; then
            pass "ganglion status loads successfully"
        else
            fail "ganglion status failed (check config.py for errors)"
        fi
    fi
else
    echo "  [SKIP] No project directory specified (set GANGLION_PROJECT or pass as argument)"
fi

echo ""
echo "=== Checking remote mode ==="

if [ -n "$GANGLION_URL" ]; then
    if command -v curl &>/dev/null; then
        pass "curl installed"
    else
        fail "curl not found (needed for remote mode)"
    fi

    # Check liveness
    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$GANGLION_URL/healthz" 2>/dev/null || echo "000")
    if [ "$HTTP_CODE" = "200" ]; then
        pass "Liveness probe OK at $GANGLION_URL/healthz"
    elif [ "$HTTP_CODE" = "000" ]; then
        fail "Cannot connect to $GANGLION_URL (connection refused)"
    else
        fail "Liveness probe returned HTTP $HTTP_CODE"
    fi

    # Check readiness
    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$GANGLION_URL/readyz" 2>/dev/null || echo "000")
    if [ "$HTTP_CODE" = "200" ]; then
        pass "Readiness probe OK at $GANGLION_URL/readyz"

        # Check if running
        RUNNING=$(curl -s "$GANGLION_URL/v1/status" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('data', {}).get('running', 'unknown'))" 2>/dev/null || echo "unknown")
        if [ "$RUNNING" = "False" ]; then
            pass "No pipeline currently running"
        elif [ "$RUNNING" = "True" ]; then
            pass "Pipeline currently running"
        else
            fail "Could not determine pipeline status"
        fi

        # Check tool count
        TOOL_COUNT=$(curl -s "$GANGLION_URL/v1/tools" 2>/dev/null | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('data', [])))" 2>/dev/null || echo "?")
        pass "Tools registered: $TOOL_COUNT"

        # Check agent count
        AGENT_COUNT=$(curl -s "$GANGLION_URL/v1/agents" 2>/dev/null | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('data', [])))" 2>/dev/null || echo "?")
        pass "Agents registered: $AGENT_COUNT"

    elif [ "$HTTP_CODE" = "503" ]; then
        fail "Readiness probe returned 503 — bridge not yet configured"
    else
        fail "Readiness probe returned HTTP $HTTP_CODE"
    fi
else
    echo "  [SKIP] No GANGLION_URL set (start server with: ganglion serve <project_dir>)"
fi

echo ""
echo "=== Results ==="
echo "  Passed: $PASS"
echo "  Failed: $FAIL"

if [ "$FAIL" -gt 0 ]; then
    echo ""
    echo "Some checks failed. See troubleshooting: skills/codebase-guide/references/troubleshooting.md"
    exit 1
else
    echo ""
    echo "All checks passed."
    exit 0
fi


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)