Skill: alannjaf/karpathy-autoresearch (version 1.0.0)

--- name: autoresearch description: "Autonomously optimize any OpenClaw skill by running it repeatedly, scoring outputs against binary evals, mutating the prompt, and keeping improvements. Based on Karpathy's autoresearch methodology. Use when: optimize this skill, improve this skill, run autoresearch on, make this skill better, self-improve skill, benchmark skill, eval my skill, run evals on." --- # autoresearch Autonomously optimize any OpenClaw skill by running it repeatedly, scoring outputs against binary evals, mutating the prompt, and keeping improvements. Based on Karpathy's autoresearch methodology. ## Triggers Use when: optimize this skill, improve this skill, run autoresearch on, make this skill better, self-improve skill, benchmark skill, eval my skill, run evals on. ## Description Autonomous prompt/strategy optimization using Karpathy's autoresearch pattern. Mutate → evaluate → keep improvements. Works on anything with a measurable score: trading strategies, content scripts, thumbnails, ad copy, email subjects. ## How It Works ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ 1. BASELINE │────▶│ 2. MUTATE │────▶│ 3. EVALUATE │────▶│ 4. DECIDE │ │ Score the │ │ Change one │ │ Run scoring │ │ Better? │ │ current │ │ thing │ │ function │ │ Keep : Revert│ │ version │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └──────┬───────┘ │ Loop back to 2 ``` ## Instructions ### Step 1: Identify the Mutable File The **mutable file** is the thing you're optimizing. It can be: - A SKILL.md prompt/instructions - A trading strategy config (thresholds, parameters) - A content template (YouTube script format, ad copy structure) - Any text file where changes produce measurable differences Create or identify this file. Example: ``` my-skill/ ├── SKILL.md ← this is your mutable file ├── eval/ │ ├── test_cases.json │ └── score.py ``` ### Step 2: Create an Evaluation Function Your eval function must: 1. **Take the current mutable file as input** 2. **Run it against test cases** 3. **Return a numeric score** (higher = better) The eval can be anything: - **LLM-as-judge**: Send output to an LLM, ask it to score 1-100 - **Backtest**: Run a strategy against historical data, measure Sharpe/returns - **A/B metrics**: CTR, engagement, conversion rate - **Binary pass/fail**: Count how many test cases pass out of N Template eval function (customize for your domain): ```python # eval/score.py import json import sys def evaluate(mutable_file_path: str, test_cases_path: str) -> float: """ Score the current version of the mutable file. Returns a float — higher is better. """ with open(mutable_file_path) as f: current_version = f.read() with open(test_cases_path) as f: test_cases = json.load(f) scores = [] for case in test_cases: # YOUR SCORING LOGIC HERE # Example: run the prompt, compare output to expected score = run_and_score(current_version, case) scores.append(score) return sum(scores) / len(scores) if __name__ == "__main__": score = evaluate(sys.argv[1], sys.argv[2]) print(f"SCORE: {score}") ``` ### Step 3: Run the Autoresearch Loop The loop follows this exact pattern: ``` 1. Git init (if not already) — every experiment is a commit 2. Run eval on current version → get BASELINE score 3. For each experiment (1..N): a. Read the current mutable file b. Generate a MUTATION (change one thing — a threshold, a phrase, a rule) c. Write the mutated version d. Run eval → get NEW score e. If NEW > BASELINE: - Git commit with message: "exp-{N}: {description} | score: {baseline} → {new}" - Update BASELINE = NEW - Log: "✅ KEPT — improvement" f. If NEW <= BASELINE: - Git checkout the mutable file (revert) - Log: "❌ REVERTED — no improvement" 4. Print final summary: experiments run, improvements found, final score ``` #### Agent Instructions for Running the Loop When the user says "run autoresearch on X", follow this procedure: 1. **Locate the mutable file** — ask the user or infer from context 2. **Locate or create the eval function** — the user must have a way to score 3. **Initialize git tracking** in the project directory 4. **Run baseline eval** — record the starting score 5. **Begin experiment loop:** - Read the mutable file - Think about what single change might improve the score - Make the change (be specific — change ONE thing per experiment) - Run eval - Keep or revert based on score - Log the result 6. **Continue for N experiments** (default: 20, or until user stops) 7. **Report results:** - Starting score → Final score - Number of experiments run - Number of improvements kept - Summary of what changes worked #### Mutation Strategy Good mutations change ONE thing at a time: - **Numeric parameters**: Adjust thresholds, weights, window sizes - **Prompt wording**: Rephrase instructions, add/remove constraints - **Structure**: Reorder sections, add examples, remove redundancy - **Rules**: Add a new rule, tighten an existing one, relax a constraint Bad mutations change everything at once — you can't learn what worked. ### Step 4: Git Tracking Every experiment MUST be tracked in git: ```bash # Before starting git init git add -A git commit -m "baseline: score {X}" # After each successful mutation git add -A git commit -m "exp-{N}: {what changed} | {old_score} → {new_score}" # After each failed mutation git checkout -- {mutable_file} ``` This gives you: - Full history of every experiment - Ability to diff any two versions - Easy rollback if something breaks - A log of what mutations worked vs didn't ## Proven Results ### Case Study 1: Gold Trading Strategy - **Task**: Optimize XAUUSD trading parameters - **Mutable file**: Strategy config (EMA periods, momentum threshold, position sizing) - **Eval function**: Backtest on historical data → Sharpe ratio - **Baseline**: Sharpe 5.80 - **Experiments**: 86 in 25 minutes - **Final**: Sharpe 12.23 (+111%) - **Key discoveries**: Momentum threshold 0.003→0, EMA 8/24→5/11, position sizing optimization - See: `references/gold-results.md` ### Case Study 2: YouTube Shorts Scripts - **Task**: Optimize script-writing prompt for higher quality scores - **Mutable file**: SKILL.md prompt instructions - **Eval function**: LLM judge scoring 1-100 - **Baseline**: 94.3/100 - **Experiments**: 11 - **Final**: 96.7/100 (+2.5%) - **Key discoveries**: Atomic sentences, strict 40-50 word range, stronger negative examples - See: `references/youtube-results.md` ## Example Usage **User**: "Run autoresearch on my email subject line skill" **Agent workflow**: 1. Read the skill's SKILL.md (mutable file) 2. Create eval: generate 20 test emails → score subject lines with LLM judge (1-100 on open-rate prediction) 3. Baseline: 72.4/100 4. Experiment 1: Add "use numbers in subject lines" → 74.1 ✅ KEPT 5. Experiment 2: Add "max 6 words" → 71.8 ❌ REVERTED 6. Experiment 3: Add "start with a verb" → 75.3 ✅ KEPT 7. ... continue for 20 experiments 8. Final: 79.2/100 (+9.4%) **User**: "Optimize my trading strategy config" **Agent workflow**: 1. Read strategy.json (mutable file) 2. Eval: run backtest script → Sharpe ratio 3. Baseline: Sharpe 2.1 4. Experiment 1: Lower stop-loss from 2% to 1.5% → Sharpe 2.3 ✅ 5. Experiment 2: Increase EMA fast period 12→15 → Sharpe 1.9 ❌ 6. ... continue 7. Final: Sharpe 3.8 (+81%)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)