Skill: xiaoxing9/openclaw-skill-eval (version 1.1.1)

--- name: skill-eval description: "Skill evaluation framework. Use when: testing trigger rate, quality compare (with/without skill), or model comparison. Runs via sessions_spawn + sessions_history. Trigger words: evaluate skill, benchmark, trigger rate, A/B compare. NOT for: debugging conversations, general testing unrelated to skill evaluation." metadata: { "openclaw": { "emoji": "🔬" } } --- # openclaw-eval-skill Evaluation framework for any OpenClaw skill. No claude CLI dependency — all agent execution runs through `sessions_spawn` + `sessions_history`. **Scope**: Works with CLI tool skills, conversational skills, and API integration skills. --- ## Runtime Actions Disclosure This skill performs the following actions during evaluation: | Action | Purpose | When | |--------|---------|------| | Read `~/.openclaw/openclaw.json` | Find skill directories (extraDirs) | Path resolution | | Write to `eval-workspace/` | Store evaluation results | Every eval run | | Call `sessions_spawn` | Run test queries in isolated sessions | Trigger & quality tests | | Call `sessions_history` | Collect conversation data for analysis | After each spawn | | Persist `cleanup="keep"` sessions | Required for trigger detection | Trigger rate tests | **NOT performed automatically**: Gateway restart, config modification, skill installation. These require manual user action (see "Bundled Test Skill" section). --- ## Quick Eval Just say: ``` evaluate weather ``` The agent will: 1. Run `scripts/resolve_paths.py weather` to find all paths 2. Execute trigger rate + quality compare with detected evals 3. Output results to `eval-workspace/weather/iter-N/` **Options**: - `evaluate weather trigger` — trigger rate only - `evaluate weather quality` — quality compare only - `evaluate github --mode all` — explicit mode **What gets auto-detected**: - Skill path: from OpenClaw built-in skills or registered extraDirs - Evals: from `evals/{skill-name}/` or fallback to `evals/example-*.json` - Output: next available `iter-N` directory **First step for agent**: Run the resolver to get paths: ```bash python scripts/resolve_paths.py {skill-name} --mode {trigger|quality|all} ``` Use the JSON output to fill in paths for the workflows below. --- ## Bundled Test Skill: fake-tool A test skill (`test-skills/fake-tool/`) is included for validating trigger rate detection. It simulates a fictional "Zephyr API" that models cannot know from training. **Manual setup required**: The agent will NOT automatically install fake-tool or restart your gateway. If you want to test with fake-tool: 1. **Copy fake-tool** to your skills directory: ```bash cp -r test-skills/fake-tool ~/.openclaw/workspace/skills/ ``` 2. **Restart OpenClaw gateway** (from terminal): ```bash openclaw gateway restart ``` 3. **Verify registration**: ```bash python scripts/resolve_paths.py fake-tool ``` If step 3 returns a valid path, fake-tool is ready. If "not found", check that your `~/.openclaw/openclaw.json` includes the skills directory in `skills.load.extraDirs`. --- ## Evaluation Scenarios ### Tier 1: Core (Always Run) | Scenario | What It Tests | Output | |----------|---------------|--------| | **Trigger Rate** | Does description trigger SKILL.md reads at the right times? Includes positive (should trigger) AND negative (should NOT trigger) cases. | recall, specificity, precision, F1 | | **Quality Compare** | Does skill improve output vs no-skill baseline? | quality_score, assertion pass rate | | **Description Diagnosis** | Why did triggers fail? Analyzes both false negatives AND false positives. | gap analysis, recommendations | ### Tier 2: Optional (Run When Needed) | Scenario | What It Tests | When to Use | |----------|---------------|-------------| | **Model Comparison** | Quality + speed across haiku/sonnet/opus | Before deployment: which model is enough? | | **Efficiency Profile** | Response time + retry patterns | When skill feels slow: is agent walking wrong paths? | ### Tier 3: Future (Roadmap) | Scenario | What It Tests | Status | |----------|---------------|--------| | **Cross-skill Conflict** | Two skills with overlapping descriptions | Planned | | **Error Recovery** | Does agent recover when CLI fails? | Planned | --- ## How This Skill Works **Two-layer architecture**: ``` Layer 1: Agent (main OpenClaw session) — YOU ARE HERE → Reads evals.json → Calls sessions_spawn to run subagents → Calls sessions_history to collect results → Writes raw data to workspace/ Layer 2: Python analysis scripts (run via exec) → Read the raw data from workspace/ → Compute statistics → Generate reports ``` Python scripts (`analyze_*.py`) are data processors — they cannot call `sessions_spawn`. The agent drives the workflow. --- ## Usage **Follow `USAGE.md` for all workflows.** Quick reference: | Workflow | What It Tests | USAGE.md Section | |----------|---------------|------------------| | Trigger Rate | Does `description` trigger SKILL.md reads at the right times? | Workflow 1 | | Quality Compare | Does skill improve output vs no-skill baseline? | Workflow 2 | | Model Comparison | Quality + Speed across haiku/sonnet/opus | Workflow 3 | | Latency Profile | Response time p50/p90 | Workflow 4 | Each workflow follows the same pattern: 1. Agent spawns subagents using `sessions_spawn` 2. Agent collects histories using `sessions_history` 3. Agent writes raw data to `workspace/{skill}/iter-{n}/raw/` 4. Agent runs analysis script via `exec` --- ## Core Principles 1. **Never modify the evaluated skill** — observe only, give recommendations 2. **Keep eval records in workspace** — output goes to `eval-workspace/<skill-name>/iteration-N/` 3. **Keep full records** — save `full_history.json` (including tool_use + tool_result) --- ## agents/ Reference | File | Purpose | When to Use | |------|---------|-------------| | `grader.md` | Check assertions, record behavior anomalies, give priority recommendations | Required for every Quality Compare eval | | `comparator.md` | Blind A/B comparison without assertions | When unbiased comparison is needed | | `analyzer.md` | Analyze cross-eval patterns after all evals complete | Post-analysis | --- ## Directory Structure ``` eval-workspace/<skill-name>/ ├── evals.json ← Eval definition (shared across iterations) └── iteration-1/ ├── raw/ │ ├── histories/ ← Trigger test session histories │ └── transcripts/ ← Quality compare transcripts ├── trigger_results.json ← analyze_triggers output ├── quality_results.json ← analyze_quality output └── diagnostics/ └── RECOMMENDATIONS.md ``` --- ## evals.json Format **Quality Compare** (prompt + assertions): ```json { "skill_name": "my-skill", "evals": [ { "id": 1, "name": "onboarding-fresh", "prompt": "Check the weather in Tokyo", "context": "Clean machine, no prior setup. For grader only.", "expected_output": "Install → configure → verify profile", "assertions": [ { "id": "a1-1", "description": "Install command executed", "type": "output_contains", "value": "pip install" }, { "id": "a1-2", "description": "Profile verified after setup", "type": "output_contains", "value": "profile current", "priority": true } ] } ] } ``` **Trigger Rate** (query + expected): ```json { "id": 1, "name": "direct-weather", "query": "What's the weather in Singapore?", "expected": true, "category": "positive" } ``` --- ## Assertion Types | Type | Detection | |------|-----------| | `output_contains` | Value appears in conversation or tool output | | `output_not_contains` | Value does not appear | | `output_count_max` | Occurrences ≤ max | | `tool_called` | Specific tool called at least once | | `tool_not_called` | Specific tool not called | | `conversation_contains` | Value appears anywhere in conversation | | `conversation_contains_any` | At least one value appears | **Priority assertions** (`"priority": true`): any failure → overall=FAIL. **Gap assertions** (`"note": "Best practice..."`): failure = skill design gap. --- ## Issue Priority (grader output) ``` 🔴 P0 Critical — Core functionality broken 🟠 P1 High — Significantly impacts usability 🟡 P2 Medium — Room for improvement 🟢 P3 Low — Minor polish ``` --- ## Behavior Anomaly Tracking Grader records these signals beyond assertions: | Field | Trigger | |-------|---------| | `path_corrections` | Wrong path then self-corrected | | `retry_count` | Same command executed multiple times | | `missing_file_reads` | Attempted to read non-existent files | | `skipped_steps` | Steps required by skill were not executed | | `hallucinations` | Fabricated non-existent commands/APIs | --- ## Key Constraints - **`sandbox="inherit"`** — subagents inherit skill registration environment - **`cleanup="keep"`** — history must be retained for trigger detection - Skill must be in a real directory under `skills.load.extraDirs` (symlinks rejected)

path: scripts/legacy/README.md
content: # Legacy Scripts (v1)

These scripts were the original implementation before the two-layer architecture.

**Not recommended for new evaluations.** Use the agent-driven workflows in `USAGE.md` instead.

Kept for reference only.


path: tests/__init__.py
content: # skill-eval test suite


path: tests/conftest.py
content: """
Pytest configuration and fixtures for skill-eval tests.
"""
import pytest


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)