Skill: jose-compu/self-improving-ai (version 1.1.0)

--- name: self-improving-ai description: "Captures learnings about GenAI/LLM configuration, model selection, inference optimization, fine-tuning, RAG pipelines, prompt engineering, multimodal processing, and cost management. Use when: (1) Model response quality degrades after a provider update or version change, (2) Inference latency exceeds acceptable thresholds, (3) Fine-tuned model regresses on evaluation benchmarks, (4) RAG retrieval returns irrelevant or stale chunks, (5) Token costs exceed budget projections, (6) Hallucination rate increases on factual queries, (7) Context window overflows cause critical information truncation, (8) Multimodal pipeline fails on specific input types (image, audio, video, PDF), (9) A better model or configuration is discovered for a task, (10) Guardrails block valid output or miss harmful content." --- # Self-Improving AI Skill Log AI/LLM-specific learnings, model issues, and feature requests to markdown files for continuous improvement. Captures model selection insights, prompt optimization patterns, inference tuning, fine-tuning regressions, RAG pipeline improvements, embedding management, multimodal processing failures, evaluation findings, and guardrail adjustments. Important learnings get promoted to model selection matrices, prompt libraries, fine-tuning runbooks, RAG architecture docs, inference optimization checklists, evaluation benchmarks, or guardrail policies. ## First-Use Initialisation Before logging anything, ensure the `.learnings/` directory and files exist in the project or workspace root. If any are missing, create them: ```bash mkdir -p .learnings [ -f .learnings/LEARNINGS.md ] || printf "# AI / LLM Learnings\n\nModel selection insights, prompt optimization patterns, inference tuning, fine-tuning lessons, RAG pipeline improvements, embedding management, multimodal processing, evaluation findings, and guardrail adjustments.\n\n**Categories**: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency\n**Areas**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails\n\n---\n" > .learnings/LEARNINGS.md [ -f .learnings/MODEL_ISSUES.md ] || printf "# Model Issues Log\n\nInference failures, model regressions, RAG retrieval problems, embedding drift, multimodal pipeline errors, and guardrail misfires.\n\n---\n" > .learnings/MODEL_ISSUES.md [ -f .learnings/FEATURE_REQUESTS.md ] || printf "# Feature Requests\n\nCapabilities needed for model selection, inference optimization, fine-tuning, RAG pipelines, multimodal processing, and evaluation.\n\n---\n" > .learnings/FEATURE_REQUESTS.md ``` Never overwrite existing files. This is a no-op if `.learnings/` is already initialised. Do not log API keys, model access tokens, customer data, or PII. Prefer redacted excerpts over raw model outputs. Mask user inputs and sensitive content in all entries. If you want automatic reminders, use the opt-in hook workflow described in [Hook Integration](#hook-integration). ## Quick Reference | Situation | Action | |-----------|--------| | Model quality drops after provider update | Log to `.learnings/MODEL_ISSUES.md` with model version details | | Latency spike on inference | Log to `.learnings/MODEL_ISSUES.md` with latency measurement | | Fine-tuned model regresses on eval | Log to `.learnings/LEARNINGS.md` with `fine_tune_regression` | | RAG returns wrong or stale chunks | Log to `.learnings/MODEL_ISSUES.md` with retrieval details | | Token cost over budget | Log to `.learnings/LEARNINGS.md` with `cost_efficiency` | | Hallucination detected | Log to `.learnings/LEARNINGS.md` with `hallucination_rate` | | Context window overflow | Log to `.learnings/LEARNINGS.md` with `context_management` | | Better model discovered for task | Log to `.learnings/LEARNINGS.md` with `model_selection` | | Prompt tweak improves output significantly | Log to `.learnings/LEARNINGS.md` with `prompt_optimization` | | Multimodal input fails (image/audio/video) | Log to `.learnings/MODEL_ISSUES.md` with modality details | | Embedding quality degrades | Log to `.learnings/MODEL_ISSUES.md` with similarity metrics | | Guardrail false positive | Log to `.learnings/LEARNINGS.md` with guardrails note | | New AI capability needed | Log to `.learnings/FEATURE_REQUESTS.md` | ## OpenClaw Setup (Recommended) OpenClaw is the primary platform for this skill. It uses workspace-based prompt injection with automatic skill loading. ### Installation **Via ClawdHub (recommended):** ```bash clawdhub install self-improving-ai ``` **Manual:** ```bash git clone https://github.com/jose-compu/self-improving-ai.git ~/.openclaw/skills/self-improving-ai ``` ### Workspace Structure OpenClaw injects these files into every session: ``` ~/.openclaw/workspace/ ├── AGENTS.md # Multi-agent workflows, model routing ├── SOUL.md # Model behavior guidelines, personality ├── TOOLS.md # Model/tool configuration, parameters ├── MEMORY.md # Long-term memory (main session only) ├── memory/ # Daily memory files │ └── YYYY-MM-DD.md └── .learnings/ # This skill's log files ├── LEARNINGS.md ├── MODEL_ISSUES.md └── FEATURE_REQUESTS.md ``` ### Create Learning Files ```bash mkdir -p ~/.openclaw/workspace/.learnings ``` Then create the log files (or copy from `assets/`): - `LEARNINGS.md` — model selection, prompt optimization, cost efficiency, hallucination patterns - `MODEL_ISSUES.md` — inference failures, model regressions, RAG problems, multimodal errors - `FEATURE_REQUESTS.md` — AI tooling, evaluation frameworks, model management capabilities ### Promotion Targets When AI learnings prove broadly applicable, promote them: | Learning Type | Promote To | Example | |---------------|------------|---------| | Model behavior patterns | `SOUL.md` | "Claude 4 tends to over-qualify, use direct prompting" | | Model selection & routing | `AGENTS.md` | "Use fast model for triage, capable model for code gen" | | Model/tool configuration | `TOOLS.md` | "Set temperature 0.1 for code, 0.7 for creative" | | Model selection insights | Model selection matrix | "Sonnet for code gen, Opus for complex reasoning" | | Prompt patterns that work | Prompt library | "Chain-of-thought improves code quality by 35%" | | Fine-tuning lessons | Fine-tuning runbook | "Always include replay data to prevent forgetting" | | RAG improvements | RAG architecture doc | "Chunk by content type, not fixed token size" | | Inference optimizations | Performance tuning guide | "Cache system prompts, batch similar requests" | | Evaluation findings | Benchmark suite docs | "HumanEval + internal eval for code gen models" | | Guardrail tuning | Guardrail policy doc | "Lower toxicity threshold for customer-facing" | ### Optional: Enable Hook For automatic reminders at session start: ```bash cp -r hooks/openclaw ~/.openclaw/hooks/self-improving-ai openclaw hooks enable self-improving-ai ``` See `references/openclaw-integration.md` for complete details. --- ## Generic Setup (Other Agents) For Claude Code, Codex, Copilot, or other agents, create `.learnings/` in the project or workspace root: ```bash mkdir -p .learnings ``` Create the files inline using the headers shown above. ### Add reference to agent files Add to `AGENTS.md`, `CLAUDE.md`, or `.github/copilot-instructions.md`: #### Self-Improving AI Workflow When AI/model issues or patterns are discovered: 1. Log to `.learnings/MODEL_ISSUES.md`, `LEARNINGS.md`, or `FEATURE_REQUESTS.md` 2. Review and promote broadly applicable learnings to: - Model selection matrix — which model for which task - Prompt library — proven prompt patterns and templates - Fine-tuning runbook — training procedures and eval gates - RAG architecture doc — chunking, retrieval, and reranking strategies - Benchmark suite — evaluation methodology and baselines ## Logging Format ### Learning Entry [LRN-YYYYMMDD-XXX] Append to `.learnings/LEARNINGS.md`: ```markdown ## [LRN-YYYYMMDD-XXX] category **Logged**: ISO-8601 timestamp **Priority**: low | medium | high | critical **Status**: pending **Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails ### Summary One-line description of the AI/model insight ### Details Full context: what model behavior was observed, why it matters for quality/cost/latency, what the correct configuration or approach should be. Include metrics and benchmarks. ### Impact - **Quality**: eval score change (e.g., HumanEval pass@1 improvement) - **Cost**: per-request or daily cost delta - **Latency**: P50/P99 change ### Suggested Action Specific model swap, config change, prompt update, or pipeline improvement to adopt ### Metadata - Source: benchmark_evaluation | a_b_test | production_monitoring | incident | user_report - Model: model name and version (e.g., claude-4-sonnet, gpt-4o, llama-3.1-70b) - Provider: anthropic | openai | google | meta | mistral | local | custom - Modality: text | image | audio | video | multimodal - Temperature/Top-P/Top-K: generation parameters used - Token Usage: input/output tokens (e.g., 2400 input / 1800 output) - Latency: response time in ms (e.g., P50 1200ms, P99 3400ms) - Cost: estimated cost per request (e.g., $0.027/task) - Tags: tag1, tag2 - See Also: LRN-20250110-001 (if related to existing entry) - Pattern-Key: model_selection.code_generation (optional) - Recurrence-Count: 1 (optional) - First-Seen: 2025-01-15 (optional) - Last-Seen: 2025-01-15 (optional) --- ``` **Categories for learnings:** | Category | Use When | |----------|----------| | `model_selection` | A different model performs better for a task (quality, cost, or latency) | | `prompt_optimization` | A prompt change significantly improves output quality or efficiency | | `inference_latency` | Latency exceeds thresholds or optimization opportunity found | | `fine_tune_regression` | Fine-tuned model scores below baseline on eval benchmarks | | `context_management` | Context window overflow, lost-in-the-middle, or prompt structure issue | | `modality_gap` | Model fails on specific input type (image, audio, video, PDF) | | `hallucination_rate` | Model produces factually incorrect output on known-fact queries | | `cost_efficiency` | Token cost exceeds budget or cost optimization opportunity found | ### Model Issue Entry [MDL-YYYYMMDD-XXX] Append to `.learnings/MODEL_ISSUES.md`: ```markdown ## [MDL-YYYYMMDD-XXX] issue_name **Logged**: ISO-8601 timestamp **Priority**: high **Status**: pending **Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails ### Summary Brief description of the model/inference issue ### Details What happened: model regression, inference failure, RAG retrieval miss, embedding drift, multimodal pipeline error, or guardrail misfire. Include specific inputs and outputs (redacted). ### Root Cause What in the model, config, or pipeline caused this issue. ### Fix Steps taken or recommended to resolve: model swap, config change, preprocessing, reindex, etc. ### Metadata - Model: model name and version - Provider: anthropic | openai | google | meta | mistral | local | custom - Modality: text | image | audio | video | multimodal - Temperature/Top-P/Top-K: generation parameters used - Token Usage: input/output tokens - Latency: response time in ms - Cost: estimated cost per request - Environment: production | staging | development - Related Issues: MDL-20250110-001 (if recurring) --- ``` ### Feature Request Entry [FEAT-YYYYMMDD-XXX] Append to `.learnings/FEATURE_REQUESTS.md`: ```markdown ## [FEAT-YYYYMMDD-XXX] capability_name **Logged**: ISO-8601 timestamp **Priority**: medium **Status**: pending **Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails ### Requested Capability What AI tool, framework, or capability is needed ### User Context Why it's needed, what manual process it replaces, estimated time saved ### Complexity Estimate simple | medium | complex ### Suggested Implementation How this could be built: eval framework, model router, A/B test harness, monitoring dashboard ### Metadata - Frequency: first_time | recurring - Related Features: existing_tool_or_capability --- ``` ## ID Generation Format: `TYPE-YYYYMMDD-XXX` - TYPE: `LRN` (learning), `MDL` (model issue), `FEAT` (feature request) - YYYYMMDD: Current date - XXX: Sequential number or random 3 chars (e.g., `001`, `A7B`) Examples: `LRN-20250415-001`, `MDL-20250415-A3F`, `FEAT-20250415-002` ## Resolving Entries When an issue is resolved, update the entry: 1. Change `**Status**: pending` → `**Status**: resolved` 2. Add resolution block after Metadata: ```markdown ### Resolution - **Resolved**: 2025-01-16T09:00:00Z - **Fix Applied**: model swap / config change / prompt update / pipeline fix - **Notes**: Updated model selection matrix / added to prompt library / retrained model ``` Other status values: - `in_progress` — Actively being investigated or benchmarked - `wont_fix` — Accepted trade-off (add reason and cost/quality analysis in Resolution notes) - `promoted` — Elevated to model selection matrix, prompt library, fine-tuning runbook, or RAG architecture doc - `promoted_to_skill` — Extracted as a reusable skill ## Promoting to Project Memory When a learning is broadly applicable (not a one-off model quirk), promote it to permanent project knowledge. ### When to Promote - Model selection insight confirmed across 3+ task types - Prompt pattern improves quality consistently across models - Fine-tuning lesson applies beyond one specific training run - RAG improvement verified on multiple document types ### Promotion Targets | Target | What Belongs There | |--------|-------------------| | Model selection matrix | Which model for which task, with benchmarks | | Prompt library | Proven prompt patterns, system prompt templates | | Fine-tuning runbook | Training procedures, data mixing ratios, eval gates | | RAG architecture doc | Chunking strategy, embedding models, retrieval ranking | | Performance tuning guide | Inference caching, batching, quantization, provider routing | | Benchmark suite docs | Eval methodology, baseline scores, regression thresholds | | Guardrail policy doc | Content filtering rules, PII detection, output validation | | `AGENTS.md` | Model routing, multi-agent workflows | ### How to Promote 1. **Distill** the learning into an actionable guideline or configuration 2. **Add** to appropriate target (matrix entry, prompt template, runbook step) 3. **Update** original entry: - Change `**Status**: pending` → `**Status**: promoted` - Add `**Promoted**: model selection matrix` (or `prompt library`, `fine-tuning runbook`, etc.) ## Recurring Pattern Detection If logging something similar to an existing entry: 1. **Search first**: `grep -r "keyword" .learnings/` 2. **Link entries**: Add `**See Also**: MDL-20250110-001` in Metadata 3. **Bump priority** if issue keeps recurring 4. **Consider systemic fix**: Recurring AI issues often indicate: - Same model failing on same input type (→ model swap or preprocessing) - Same prompt pattern needed across tasks (→ add to system prompt template) - Same cost overrun on specific use case (→ model downgrade or caching) - Same hallucination on same topic (→ RAG grounding or guardrail) ## Simplify & Harden Feed Ingest recurring AI patterns from `simplify-and-harden` into model configs or prompt libraries. 1. For each candidate, use `pattern_key` as the dedupe key. 2. Search `.learnings/LEARNINGS.md` for existing entry: `grep -n "Pattern-Key: <key>" .learnings/LEARNINGS.md` 3. If found: increment `Recurrence-Count`, update `Last-Seen`, add `See Also` links. 4. If not found: create new `LRN-...` entry with `Source: simplify-and-harden`. **Promotion threshold**: `Recurrence-Count >= 3`, seen in 2+ task types, within 30-day window. Targets: model selection matrix, prompt library, fine-tuning runbook, `AGENTS.md` / `TOOLS.md`. ## Periodic Review Review `.learnings/` at natural breakpoints: ### When to Review - When model provider announces updates or deprecations - After switching or upgrading models - Monthly cost review (compare actual vs budget) - Quarterly eval benchmark re-run - When new modality support is added - After fine-tuning iterations ### Quick Status Check ```bash grep -h "Status\*\*: pending" .learnings/*.md | wc -l grep -B5 "Priority\*\*: critical\|Priority\*\*: high" .learnings/MODEL_ISSUES.md | grep "^## \[" grep -B2 "hallucination_rate\|fine_tune_regression" .learnings/LEARNINGS.md | grep "^## \[" ``` ### Review Actions - Resolve fixed model issues with config change notes - Promote recurring patterns to model selection matrix or prompt library - Update cost projections based on actual usage - Re-run eval benchmarks after model updates ## Detection Triggers Automatically log when you encounter: **Model Response Quality** (→ model issue or learning): - Model response includes "I don't have access to" or "As an AI" when it shouldn't - Model output contradicts known facts (hallucination) - Response quality noticeably worse than previous interactions with same model **Inference Performance** (→ model issue with latency/cost context): - Response latency >5s for simple queries - Token count approaching context limit (>80% utilization) - Cost per session exceeding threshold - Rate limit or quota exceeded - Model version deprecated warning **RAG Pipeline** (→ model issue with rag_pipeline area): - RAG context retrieved but answer ignores it - Retrieved chunks are irrelevant to the query - Embedding similarity scores below threshold **Fine-Tuning** (→ learning with fine_tune_regression): - Fine-tuned model scoring below baseline on eval - Training loss not converging or eval loss increasing - Catastrophic forgetting on non-target tasks **Multimodal** (→ model issue with multimodal area): - Image/audio/video input returns "I can't process this" - OCR quality drops on rotated/skewed/low-contrast scans - Cross-modal retrieval fails (text query → image result) **Guardrails** (→ learning with guardrails note): - Guardrail blocks legitimate request (false positive) - Harmful content passes through guardrail (false negative) - Structured output validation rejects valid response ## Priority Guidelines | Priority | When to Use | AI Examples | |----------|-------------|-------------| | `critical` | Model producing harmful/dangerous output, data leaking through model, fine-tuned model catastrophic regression, guardrail completely bypassed | Hallucination in medical/legal advice, PII in model output, 45% drop on coding benchmarks after fine-tune | | `high` | Significant quality drop, latency >10x baseline, cost overrun >50%, hallucination on critical facts, multimodal pipeline broken | Model fails all rotated PDF scans, 3x latency after provider update, daily cost doubled | | `medium` | Model selection could be better, prompt optimization opportunity, minor eval regression, cost optimization, embedding refresh needed | Sonnet 23% better than GPT-4o for code gen, chain-of-thought adds 35% quality, embedding dimension reduction saves 60% | | `low` | Documentation of model behavior, minor prompt tweak, config cleanup | Model prefers bullet lists over paragraphs, temperature 0.1 vs 0.0 negligible difference | ## Area Tags Use to filter learnings by AI domain: | Area | Scope | |------|-------| | `model_config` | Model selection, version pinning, parameter tuning (temperature, top-p, top-k, max tokens, stop sequences), provider configuration, fallback chains | | `prompt_engineering` | System prompts, few-shot examples, chain-of-thought, prompt templates, prompt compression, prompt caching | | `fine_tuning` | Training data curation, hyperparameters, eval sets, RLHF/DPO, LoRA/QLoRA, catastrophic forgetting, checkpoint management | | `rag_pipeline` | Chunking strategy, embedding models, vector stores, retrieval ranking, reranking, hybrid search, context assembly | | `inference` | Latency optimization, batching, streaming, caching, quantization, speculative decoding, KV cache, provider routing | | `embeddings` | Model selection, dimension reduction, index management, drift detection, similarity thresholds, multi-language | | `multimodal` | Vision (image/PDF), audio (speech/music), video, cross-modal retrieval, modality-specific prompting, output format handling | | `evaluation` | Benchmarks, human eval, automated eval (LLM-as-judge), A/B testing, regression testing, domain-specific metrics | | `guardrails` | Content filtering, PII detection, toxicity, factuality checks, output validation, structured output enforcement, jailbreak prevention | ## Model Lifecycle Management Track model deprecation schedules and plan migrations proactively. ### Deprecation Tracking Maintain awareness of provider deprecation timelines. OpenAI models typically deprecated 6-12 months after successor release. Anthropic version-pinned models maintained per published schedule. Gemini versions follow quarterly cadence. Open-weight models (Llama, Mistral) don't deprecate but lose community support. ### Migration Planning When a model version is deprecated: (1) log to `MODEL_ISSUES.md` with deprecation date and replacement, (2) run eval suite against replacement before cutover, (3) update all config files and fallback chains, (4) monitor quality for 7 days post-migration, (5) update model selection matrix. Pin exact versions in production. Keep eval results versioned alongside model configs. Document behavioral differences between generations. Maintain fallback chains: primary → secondary → tertiary. ## Best Practices 1. **Pin model versions** — never use "latest" in production 2. **Benchmark before switching models** — gut feeling is not evaluation 3. **Measure cost per task, not just per token** — task-level cost captures retries and multi-turn 4. **Cache aggressively** — same prompt + same input = same output 5. **Use structured output (JSON mode)** when downstream parsing is needed 6. **Separate system prompt from user context** for cleaner caching 7. **Test multimodal inputs with edge cases** — blurry images, noisy audio, long video 8. **Keep eval sets versioned alongside fine-tuned models** 9. **Log temperature/parameters with every issue** — reproducibility matters 10. **Set up fallback chains** — provider outages happen 11. **Monitor embedding drift** after source data changes 12. **Review guardrail logs monthly** for false positive trends ## Gitignore Options **Keep learnings local** (per-developer): ```gitignore .learnings/ ``` **Track learnings in repo** (team-wide): Don't add to .gitignore — learnings become shared knowledge. **Hybrid** (track templates, ignore entries): ```gitignore .learnings/*.md !.learnings/.gitkeep ``` ## Hook Integration Enable automatic reminders through agent hooks. This is **opt-in**. ### Quick Setup (Claude Code / Codex) Create `.claude/settings.json` in your project: ```json { "hooks": { "UserPromptSubmit": [{ "matcher": "", "hooks": [{ "type": "command", "command": "./skills/self-improving-ai/scripts/activator.sh" }] }] } } ``` This injects an AI-focused learning evaluation reminder after each prompt (~50-100 tokens overhead). ### Advanced Setup (With Error Detection) ```json { "hooks": { "UserPromptSubmit": [{ "matcher": "", "hooks": [{ "type": "command", "command": "./skills/self-improving-ai/scripts/activator.sh" }] }], "PostToolUse": [{ "matcher": "Bash", "hooks": [{ "type": "command", "command": "./skills/self-improving-ai/scripts/error-detector.sh" }] }] } } ``` Enable `PostToolUse` only if you want the hook to inspect command output for model API errors, rate limits, context overflows, and inference failures. ### Available Hook Scripts | Script | Hook Type | Purpose | |--------|-----------|---------| | `scripts/activator.sh` | UserPromptSubmit | Reminds to evaluate AI/model learnings after tasks | | `scripts/error-detector.sh` | PostToolUse (Bash) | Triggers on model API errors, rate limits, inference failures | See `references/hooks-setup.md` for detailed configuration and troubleshooting. ## Automatic Skill Extraction When an AI learning is valuable enough to become a reusable skill, extract it. ### Skill Extraction Criteria | Criterion | Description | |-----------|-------------| | **Recurring** | Same model issue or pattern in 2+ projects or task types | | **Verified** | Status is `resolved` with confirmed benchmark improvement | | **Non-obvious** | Required investigation, benchmarking, or A/B testing | | **Broadly applicable** | Not specific to one model version; useful across providers | | **User-flagged** | User says "save this as a skill" or similar | ### Extraction Workflow 1. **Identify candidate**: Learning meets extraction criteria 2. **Run helper** (or create manually): ```bash ./skills/self-improving-ai/scripts/extract-skill.sh skill-name --dry-run ./skills/self-improving-ai/scripts/extract-skill.sh skill-name ``` 3. **Customize SKILL.md**: Fill in template with AI-specific content, model compatibility matrix, and benchmark results 4. **Update learning**: Set status to `promoted_to_skill`, add `Skill-Path` 5. **Verify**: Read skill in fresh session to ensure it's self-contained ### Extraction Detection Triggers **In conversation**: "This model keeps failing on this", "Save this prompt pattern as a skill", "We always need this RAG config". **In entries**: Multiple `See Also` links, high priority + resolved, same `Pattern-Key` across projects, verified benchmark improvement. ## Multi-Agent Support | Agent | Activation | Detection | |-------|-----------|-----------| | Claude Code | Hooks (UserPromptSubmit, PostToolUse) | Automatic via error-detector.sh | | Codex CLI | Hooks (same pattern) | Automatic via hook scripts | | GitHub Copilot | Manual (`.github/copilot-instructions.md`) | Manual review | | OpenClaw | Workspace injection + inter-agent messaging | Via session tools | ### Model Routing for Multi-Agent Setups Different agents benefit from different models: - **Triage agent**: fast model (e.g., Claude Haiku, GPT-4o-mini) for classification and routing - **Coding agent**: capable model (e.g., Claude Sonnet, GPT-4o) for code generation and review - **Document processing agent**: multimodal model (e.g., Claude Opus, Gemini Pro) for image/PDF/video - **Evaluation agent**: strong reasoning model for LLM-as-judge tasks Log model routing decisions as `model_selection` learnings. Promote proven routing patterns to `AGENTS.md`. ## Stackability Contract (Standalone + Multi-Skill) This skill is standalone-compatible and stackable with other self-improving skills. ### Namespaced Logging (recommended for 2+ skills) - Namespace for this skill: `.learnings/ai/` - Keep current standalone behavior if you prefer flat files. - Optional shared index for all skills: `.learnings/INDEX.md` ### Required Metadata Every new entry must include: ```markdown **Skill**: ai ``` ### Hook Arbitration (when 2+ skills are enabled) - Use one dispatcher hook as the single entrypoint. - Dispatcher responsibilities: route by matcher, dedupe repeated events, and rate-limit reminders. - Suggested defaults: dedupe key = `event + matcher + file + 5m_window`; max 1 reminder per skill every 5 minutes. ### Narrow Matcher Scope (ai) Only trigger this skill automatically for AI/LLM signals such as: - `rate limit|context window|token|hallucination|model deprecation` - `rag|embedding|fine-tune|inference latency|guardrail` - explicit AI/model intent in user prompt ### Cross-Skill Precedence When guidance conflicts, apply: 1. `security` 2. `engineering` 3. `coding` 4. `ai` 5. user-explicit domain skill 6. `meta` as tie-breaker ### Ownership Rules - This skill writes only to `.learnings/ai/` in stackable mode. - It may read other skill folders for cross-linking, but should not rewrite their entries.

path: assets/FEATURE_REQUESTS.md
content: # Feature Requests

Capabilities needed for model selection, inference optimization, fine-tuning, RAG pipelines, multimodal processing, and evaluation.

---


path: assets/ERRORS.md
content: # Model Issues Log

Inference failures, model regressions, RAG retrieval problems, embedding drift, multimodal pipeline errors, and guardrail misfires.

---


path: assets/SKILL-TEMPLATE.md
content: # AI / LLM Skill Template

Template for creating skills extracted from AI/LLM learnings. Copy and customize.

---

## SKILL.md Template

```markdown
---
name: skill-name-here
description: "Concise description of the model behavior, prompt pattern, inference optimization, or RAG improvement this skill addresses. Include trigger conditions."
---

# Skill Name

Brief introduction: what AI/model problem this skill solves, which models and providers it applies to, and its origin.

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Model/inference trigger] | [Config change, prompt fix, or model swap] |
| [Related trigger] | [Alternative approach] |

## Background

Why this AI/LLM knowledge matters. What quality degradation, cost overrun, or latency issue it prevents.

## The Problem

### Current Behavior

```
// Model output, error message, or metric showing the issue
```

### Impact

How the problem manifests: quality drop, cost increase, latency spike, hallucination, or modality failure.

## Solution

### Fix

```
// Updated config, prompt, model selection, or pipeline change
```

### Step-by-Step

1. Identify the model/config issue
2. Benchmark current performance (latency, quality, cost)
3. Apply the fix
4. Re-evaluate on benchmark suite
5. Monitor in production for regression

## Model Compatibility Matrix

| Model | Provider | Version | Tested | Result |
|-------|----------|---------|--------|--------|
| [model-name] | [provider] | [version] | [date] | [pass/fail with notes] |

## Benchmark Results

| Metric | Before | After | Delta |
|--------|--------|-------|-------|
| Quality (eval score) | [score] | [score] | [+/-] |
| Latency (P50/P99) | [ms] | [ms] | [+/-] |
| Cost (per 1K requests) | [$] | [$] | [+/-] |
| Token usage (avg) | [tokens] | [tokens] | [+/-] |

## Configuration

### Model Parameters

```json
{
  "model": "model-name",
  "temperature": 0.1,
  "top_p": 0.95,
  "max_tokens": 4096,
  "stop_sequences": []
}
```

### Prompt Template

```
[System prompt or prompt pattern that implements the fix]
```

## Common Variations

- **Variation A**: Description and how to handle across different model families
- **Variation B**: Description for multimodal vs text-only scenarios

## Provider Compatibility

| Provider | Models Affected | Notes |
|----------|----------------|-------|
| Anthropic | Claude 4 Opus, Claude 4 Sonnet | [compatibility notes] |
| OpenAI | GPT-4o, GPT-4o-mini | [compatibility notes] |
| Google | Gemini 2.5 Pro, Gemini 2.5 Flash | [compatibility notes] |
| Meta | Llama 3.1 70B, Llama 3.1 8B | [compatibility notes] |
| Mistral | Mistral Large, Codestral | [compatibility notes] |

## Gotchas

- Warning or common mistake when applying the fix
- Edge case with specific model versions or providers
- Behavior differences between text and multimodal inputs

## Related

- Link to model provider documentation
- Link to related evaluation benchmark
- Link to related skill

## Source

Extracted from AI/LLM learning entry.
- **Learning ID**: LRN-YYYYMMDD-XXX or MDL-YYYYMMDD-XXX
- **Original Category**: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency
- **Area**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails
- **Extraction Date**: YYYY-MM-DD
```

---

## Minimal Template

For simple AI/LLM skills that don't need all sections:

```markdown
---
name: skill-name-here
description: "What model behavior or AI pattern this addresses and when to apply it."
---

# Skill Name

[Problem statement in one sentence]

## Problem

Description of the model behavior, inference issue, or prompt failure.

## Solution

Updated config, prompt change, or model swap that resolves the issue.

## Benchmark Results

| Metric | Before | After |
|--------|--------|-------|
| [key metric] | [value] | [value] |

## Source

- Learning ID: LRN-YYYYMMDD-XXX
```

---

## Naming Conventions

- **Skill name**: lowercase, hyphens for spaces
  - Good: `code-gen-model-selection`, `rag-chunk-sizing`, `lost-in-the-middle-fix`
  - Bad: `ModelSelection`, `rag_chunks`, `fix1`

- **Description**: Start with action verb, mention the model or AI component
  - Good: "Optimizes RAG chunk sizes by content type to improve retrieval relevance. Use when retrieval scores drop below 0.8."
  - Bad: "RAG stuff"

- **Files**:
  - `SKILL.md` — Required, main documentation
  - `scripts/` — Optional, automation (benchmark runner, config validator)
  - `references/` — Optional, detailed docs
  - `assets/` — Optional, templates

---

## Extraction Checklist

Before creating a skill from an AI/LLM learning:

- [ ] Fix is verified (benchmarked, tested with eval suite)
- [ ] Solution is broadly applicable (not one-off model quirk)
- [ ] Model compatibility matrix is filled in
- [ ] Benchmark results are documented (before/after)
- [ ] Configuration parameters are specified
- [ ] Provider compatibility is noted
- [ ] Name follows conventions
- [ ] Description is concise but informative

After creating:

- [ ] Update original learning with `promoted_to_skill` status
- [ ] Add `Skill-Path: skills/skill-name` to learning metadata
- [ ] Test skill by reading it in a fresh session
- [ ] Verify config changes work across target models
- [ ] Re-run eval suite to confirm no regression


path: assets/LEARNINGS.md
content: # AI / LLM Learnings

Model selection insights, prompt optimization patterns, inference tuning, fine-tuning lessons, RAG pipeline improvements, embedding management, multimodal processing, evaluation findings, and guardrail adjustments captured during AI/ML work.

**Categories**: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency
**Areas**: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails
**Statuses**: pending | in_progress | resolved | wont_fix | promoted | promoted_to_skill

## Status Definitions

| Status | Meaning |
|--------|---------|
| `pending` | Not yet addressed |
| `in_progress` | Actively being investigated or tested |
| `resolved` | Issue fixed, model swapped, or config tuned |
| `wont_fix` | Accepted trade-off (reason in Resolution) |
| `promoted` | Elevated to model selection matrix, prompt library, fine-tuning runbook, or RAG architecture doc |
| `promoted_to_skill` | Extracted as a reusable skill |

## Skill Extraction Fields

When a learning is promoted to a skill, add these fields:

```markdown
**Status**: promoted_to_skill
**Skill-Path**: skills/skill-name
```

Example:
```markdown
## [LRN-20250415-001] model_selection

**Logged**: 2025-04-15T10:00:00Z
**Priority**: high
**Status**: promoted_to_skill
**Skill-Path**: skills/code-gen-model-selection
**Area**: model_config

### Summary
Claude Sonnet outperforms GPT-4o on multi-file edit instructions by 23% on HumanEval
...
```

---


path: scripts/activator.sh
content: #!/bin/bash
# AI Self-Improvement Activator Hook
# Triggers on UserPromptSubmit to remind about AI/model learning capture
# Keep output minimal (~50-100 tokens) to minimize overhead

set -e

cat << 'EOF'
<ai-improvement-reminder>
After completing this task, evaluate if AI/model learnings emerged:
- Did the model struggle with a specific input type or modality?
- Was response quality lower than expected for the model used?
- Could a different model or configuration have performed better?
- Did latency or cost exceed expectations?
- Was context window management a problem?
- Did a prompt pattern significantly improve output?
- Were hallucinations detected in factual responses?
- Did RAG retrieval return relevant context?

If yes: Log to .learnings/ using the self-improving-ai format.
If model config change needed: Update model selection or prompt library.
</ai-improvement-reminder>
EOF


path: scripts/extract-skill.sh
content: #!/bin/bash
# AI Skill Extraction Helper
# Creates a new skill from an AI/LLM learning entry
# Usage: ./extract-skill.sh <skill-name> [--dry-run]

set -e

SKILLS_DIR="./skills"

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

usage() {
    cat << EOF
Usage: $(basename "$0") <skill-name> [options]

Create a new AI/LLM skill from a learning entry.

Arguments:
  skill-name     Name of the skill (lowercase, hyphens for spaces)

Options:
  --dry-run      Show what would be created without creating files
  --output-dir   Relative output directory under current path (default: ./skills)
  -h, --help     Show this help message

Examples:
  $(basename "$0") code-gen-model-selection
  $(basename "$0") rag-chunk-sizing --dry-run
  $(basename "$0") lost-in-the-middle-fix --output-dir ./skills/ai

The skill will be created in: \$SKILLS_DIR/<skill-name>/
EOF
}

log_info() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

log_warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

log_error() {
    echo -e "${RED}[ERROR]${NC} $1" >&2
}

SKILL_NAME=""
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case $1 in
        --dry-run)
            DRY_RUN=true
            shift
            ;;
        --output-dir)
            if [ -z "${2:-}" ] || [[ "${2:-}" == -* ]]; then
                log_error "--output-dir requires a relative path argument"
                usage
                exit 1
            fi
            SKILLS_DIR="$2"
            shift 2
            ;;
        -h|--help)
            usage
            exit 0
            ;;
        -*)
            log_error "Unknown option: $1"
            usage
            exit 1
            ;;
        *)
            if [ -z "$SKILL_NAME" ]; then
                SKILL_NAME="$1"
            else
                log_error "Unexpected argument: $1"
                usage
                exit 1
            fi
            shift
            ;;
    esac
done

if [ -z "$SKILL_NAME" ]; then
    log_error "Skill name is required"
    usage
    exit 1
fi

if ! [[ "$SKILL_NAME" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; then
    log_error "Invalid skill name format. Use lowercase letters, numbers, and hyphens only."
    log_error "Examples: 'code-gen-model-selection', 'rag-chunk-sizing', 'lost-in-the-middle-fix'"
    exit 1
fi

if [[ "$SKILLS_DIR" = /* ]]; then
    log_error "Output directory must be a relative path under the current directory."
    exit 1
fi

if [[ "$SKILLS_DIR" =~ (^|/)\.\.(/|$) ]]; then
    log_error "Output directory cannot include '..' path segments."
    exit 1
fi

SKILLS_DIR="${SKILLS_DIR#./}"
SKILLS_DIR="./$SKILLS_DIR"

SKILL_PATH="$SKILLS_DIR/$SKILL_NAME"

if [ -d "$SKILL_PATH" ] && [ "$DRY_RUN" = false ]; then
    log_error "Skill already exists: $SKILL_PATH"
    log_error "Use a different name or remove the existing skill first."
    exit 1
fi

if [ "$DRY_RUN" = true ]; then
    log_info "Dry run - would create:"
    echo "  $SKILL_PATH/"
    echo "  $SKILL_PATH/SKILL.md"
    echo ""
    echo "Template content would be:"
    echo "---"
    cat << TEMPLATE
name: $SKILL_NAME
description: "[TODO: Describe the model behavior, prompt pattern, or inference optimization this skill addresses]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction — what AI/model problem this solves]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Model/inference trigger] | [Config change, prompt fix, or model swap] |

## The Problem

[TODO: Describe the model behavior, quality issue, or pipeline failure]

## Solution

[TODO: Updated config, prompt pattern, model swap, or pipeline change]

## Model Compatibility Matrix

| Model | Provider | Version | Tested | Result |
|-------|----------|---------|--------|--------|
| [TODO] | [TODO] | [TODO] | [TODO] | [TODO] |

## Benchmark Results

| Metric | Before | After | Delta |
|--------|--------|-------|-------|
| Quality (eval score) | [TODO] | [TODO] | [TODO] |
| Latency (P50/P99) | [TODO] | [TODO] | [TODO] |
| Cost (per 1K requests) | [TODO] | [TODO] | [TODO] |

## Configuration

\`\`\`json
{
  "model": "[TODO: model name]",
  "temperature": 0.1,
  "top_p": 0.95,
  "max_tokens": 4096
}
\`\`\`

## Source

- Learning ID: [TODO: Add original learning ID]
- Category: [TODO: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency]
- Area: [TODO: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails]
- Original File: .learnings/LEARNINGS.md or .learnings/MODEL_ISSUES.md
TEMPLATE
    echo "---"
    exit 0
fi

log_info "Creating AI skill: $SKILL_NAME"

mkdir -p "$SKILL_PATH"

cat > "$SKILL_PATH/SKILL.md" << TEMPLATE
---
name: $SKILL_NAME
description: "[TODO: Describe the model behavior, prompt pattern, or inference optimization this skill addresses]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction — what AI/model problem this solves]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Model/inference trigger] | [Config change, prompt fix, or model swap] |

## The Problem

[TODO: Describe the model behavior, quality issue, or pipeline failure]

## Solution

[TODO: Updated config, prompt pattern, model swap, or pipeline change]

## Model Compatibility Matrix

| Model | Provider | Version | Tested | Result |
|-------|----------|---------|--------|--------|
| [TODO] | [TODO] | [TODO] | [TODO] | [TODO] |

## Benchmark Results

| Metric | Before | After | Delta |
|--------|--------|-------|-------|
| Quality (eval score) | [TODO] | [TODO] | [TODO] |
| Latency (P50/P99) | [TODO] | [TODO] | [TODO] |
| Cost (per 1K requests) | [TODO] | [TODO] | [TODO] |

## Configuration

\`\`\`json
{
  "model": "[TODO: model name]",
  "temperature": 0.1,
  "top_p": 0.95,
  "max_tokens": 4096
}
\`\`\`

## Source

- Learning ID: [TODO: Add original learning ID]
- Category: [TODO: model_selection | prompt_optimization | inference_latency | fine_tune_regression | context_management | modality_gap | hallucination_rate | cost_efficiency]
- Area: [TODO: model_config | prompt_engineering | fine_tuning | rag_pipeline | inference | embeddings | multimodal | evaluation | guardrails]
- Original File: .learnings/LEARNINGS.md or .learnings/MODEL_ISSUES.md
TEMPLATE

log_info "Created: $SKILL_PATH/SKILL.md"

echo ""
log_info "Skill scaffold created."
echo ""
echo "Next steps:"
echo "  1. Edit $SKILL_PATH/SKILL.md"
echo "  2. Fill in the TODO sections with model/config details from your learning"
echo "  3. Add model compatibility matrix (which models/providers were tested)"
echo "  4. Add benchmark results (before/after metrics)"
echo "  5. Add references/ folder if you have eval data or detailed docs"
echo "  6. Update the original learning entry with:"
echo "     **Status**: promoted_to_skill"
echo "     **Skill-Path**: skills/$SKILL_NAME"


path: scripts/error-detector.sh
content: #!/bin/bash
# AI Self-Improvement Error Detector Hook
# Triggers on PostToolUse for Bash to detect model API errors, inference failures,
# RAG issues, and other AI/LLM-related problems
# Reads CLAUDE_TOOL_OUTPUT environment variable

set -e

OUTPUT="${CLAUDE_TOOL_OUTPUT:-}"

ERROR_PATTERNS=(
    "rate limit"
    "quota exceeded"
    "context length"
    "token limit"
    "max_tokens"
    "model not found"
    "deprecated"
    "hallucination"
    "factual error"
    "embedding"
    "vector"
    "retrieval"
    "chunk"
    "fine-tune"
    "training loss"
    "eval loss"
    "CUDA"
    "GPU"
    "out of memory"
    "OOM"
    "inference"
    "latency"
    "timeout"
    "modality"
    "unsupported format"
    "content filter"
    "guardrail"
    "blocked"
    "safety"
    "refusal"
    "RateLimitError"
    "APIError"
    "AuthenticationError"
    "InvalidRequestError"
    "ContextWindowExceededError"
    "ModelNotFoundError"
    "ServiceUnavailableError"
    "overloaded"
    "capacity"
    "throttle"
    "429"
    "503"
)

contains_error=false
for pattern in "${ERROR_PATTERNS[@]}"; do
    if [[ "$OUTPUT" == *"$pattern"* ]]; then
        contains_error=true
        break
    fi
done

if [ "$contains_error" = true ]; then
    cat << 'EOF'
<ai-error-detected>
An AI/model error was detected in command output. Consider logging to .learnings/ if:
- Rate limit or quota exceeded → MODEL_ISSUES.md [MDL-YYYYMMDD-XXX] with provider details
- Context window overflow → LEARNINGS.md with context_management category
- Model not found or deprecated → MODEL_ISSUES.md with migration notes
- Inference timeout or latency spike → MODEL_ISSUES.md with latency measurement
- Fine-tuning loss anomaly → LEARNINGS.md with fine_tune_regression category
- GPU/CUDA out of memory → MODEL_ISSUES.md with model size and hardware specs
- Content filter or guardrail block → LEARNINGS.md with guardrails note
- Embedding or retrieval failure → MODEL_ISSUES.md with RAG pipeline details

Include model name, provider, parameters, token usage, and latency.
</ai-error-detected>
EOF
fi


path: hooks/openclaw/handler.js
content: /**
 * Self-Improving AI Hook for OpenClaw
 *
 * Injects an AI/LLM-specific reminder to evaluate learnings during agent bootstrap.
 * Fires on agent:bootstrap event before workspace files are injected.
 */

const REMINDER_CONTENT = `
## AI / LLM Self-Improvement Reminder

After completing tasks, evaluate if any AI/model learnings should be captured:

**Log model issues when:**
- Model response quality dropped after provider update → \`.learnings/MODEL_ISSUES.md\`
- Inference latency spiked or exceeded threshold → \`.learnings/MODEL_ISSUES.md\`
- RAG retrieval returned irrelevant or stale chunks → \`.learnings/MODEL_ISSUES.md\`
- Embedding quality degraded after reindex → \`.learnings/MODEL_ISSUES.md\`
- Multimodal input failed (image/audio/video/PDF) → \`.learnings/MODEL_ISSUES.md\`
- Guardrail misfire (false positive or bypass) → \`.learnings/MODEL_ISSUES.md\`

**Log learnings when:**
- Better model or config discovered for a task → \`.learnings/LEARNINGS.md\` (model_selection)
- Prompt tweak significantly improved output → \`.learnings/LEARNINGS.md\` (prompt_optimization)
- Hallucination detected in factual response → \`.learnings/LEARNINGS.md\` (hallucination_rate)
- Context window overflow caused truncation → \`.learnings/LEARNINGS.md\` (context_management)
- Fine-tuned model regressed on eval set → \`.learnings/LEARNINGS.md\` (fine_tune_regression)
- Token cost exceeded budget projections → \`.learnings/LEARNINGS.md\` (cost_efficiency)

**Log feature requests when:**
- Missing AI capability (model A/B testing, eval framework, etc.) → \`.learnings/FEATURE_REQUESTS.md\`

**Promote when pattern is proven:**
- Model behavior patterns → SOUL.md (e.g., "Claude 4 tends to over-qualify, use direct prompting")
- Model selection and workflow → AGENTS.md (e.g., "Use fast model for triage, capable model for code gen")
- Model/tool configuration → TOOLS.md (e.g., "Set temperature 0.1 for code, 0.7 for creative")

Log model name, provider, parameters (temperature, top-p), token usage, and latency with every entry.
`.trim();

const handler = async (event) => {
  if (!event || typeof event !== 'object') {
    return;
  }

  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  if (Array.isArray(event.context.bootstrapFiles)) {
    event.context.bootstrapFiles.push({
      path: 'AI_SELF_IMPROVEMENT_REMINDER.md',
      content: REMINDER_CONTENT,
      virtual: true,
    });
  }
};

module.exports = handler;
module.exports.default = handler;


path: hooks/openclaw/HOOK.md
content: ---
name: self-improving-ai
description: "Injects AI/LLM self-improvement reminder during agent bootstrap"
metadata: {"openclaw":{"emoji":"🤖","events":["agent:bootstrap"]}}
---

# AI Self-Improvement Hook

Injects a reminder to evaluate AI/model learnings during agent bootstrap.

## What It Does

- Fires on `agent:bootstrap` (before workspace files are injected)
- Adds an AI-specific reminder block to check `.learnings/` for relevant entries
- Prompts the agent to log model quality issues, inference failures, prompt optimization discoveries, hallucinations, context overflows, RAG retrieval problems, and multimodal pipeline errors
- Emphasizes **NEVER logging API keys, model access tokens, or customer data**

## Reminder Content

The hook injects reminders to log to the appropriate file based on what occurred:

| Trigger | Target File | Category |
|---------|-------------|----------|
| Model response quality issue | `MODEL_ISSUES.md` | model regression |
| Latency or cost anomaly | `MODEL_ISSUES.md` | inference/cost |
| Better model/config discovered | `LEARNINGS.md` | `model_selection` |
| Prompt pattern improved output | `LEARNINGS.md` | `prompt_optimization` |
| Hallucination detected | `LEARNINGS.md` | `hallucination_rate` |
| Context overflow | `LEARNINGS.md` | `context_management` |
| Multimodal input failed | `MODEL_ISSUES.md` | modality_gap |
| Missing AI capability | `FEATURE_REQUESTS.md` | feature request |

## Promotion Targets

| Learning Type | Promote To |
|---------------|------------|
| Model behavior patterns | `SOUL.md` |
| Model workflows and routing | `AGENTS.md` |
| Model/tool configuration | `TOOLS.md` |

## Configuration

No configuration needed. Enable with:

```bash
openclaw hooks enable self-improving-ai
```


path: hooks/openclaw/handler.ts
content: /**
 * Self-Improving AI Hook for OpenClaw
 *
 * Injects an AI/LLM-specific reminder to evaluate learnings during agent bootstrap.
 * Fires on agent:bootstrap event before workspace files are injected.
 */

import type { HookHandler } from 'openclaw/hooks';

const REMINDER_CONTENT = `## AI / LLM Self-Improvement Reminder

After completing tasks, evaluate if any AI/model learnings should be captured:

**Log model issues when:**
- Model response quality dropped after provider update → \`.learnings/MODEL_ISSUES.md\`
- Inference latency spiked or exceeded threshold → \`.learnings/MODEL_ISSUES.md\`
- RAG retrieval returned irrelevant or stale chunks → \`.learnings/MODEL_ISSUES.md\`
- Embedding quality degraded after reindex → \`.learnings/MODEL_ISSUES.md\`
- Multimodal input failed (image/audio/video/PDF) → \`.learnings/MODEL_ISSUES.md\`
- Guardrail misfire (false positive or bypass) → \`.learnings/MODEL_ISSUES.md\`

**Log learnings when:**
- Better model or config discovered for a task → \`.learnings/LEARNINGS.md\` (model_selection)
- Prompt tweak significantly improved output → \`.learnings/LEARNINGS.md\` (prompt_optimization)
- Hallucination detected in factual response → \`.learnings/LEARNINGS.md\` (hallucination_rate)
- Context window overflow caused truncation → \`.learnings/LEARNINGS.md\` (context_management)
- Fine-tuned model regressed on eval set → \`.learnings/LEARNINGS.md\` (fine_tune_regression)
- Token cost exceeded budget projections → \`.learnings/LEARNINGS.md\` (cost_efficiency)

**Log feature requests when:**
- Missing AI capability (model A/B testing, eval framework, etc.) → \`.learnings/FEATURE_REQUESTS.md\`

**Promote when pattern is proven:**
- Model behavior patterns → SOUL.md (e.g., "Claude 4 tends to over-qualify, use direct prompting")
- Model selection and workflow → AGENTS.md (e.g., "Use fast model for triage, capable model for code gen")
- Model/tool configuration → TOOLS.md (e.g., "Set temperature 0.1 for code, 0.7 for creative")

Log model name, provider, parameters (temperature, top-p), token usage, and latency with every entry.`;

const handler: HookHandler = async (event) => {
  if (!event || typeof event !== 'object') {
    return;
  }

  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  const sessionKey = event.sessionKey || '';
  if (sessionKey.includes(':subagent:')) {
    return;
  }

  if (Array.isArray(event.context.bootstrapFiles)) {
    event.context.bootstrapFiles.push({
      path: 'AI_SELF_IMPROVEMENT_REMINDER.md',
      content: REMINDER_CONTENT,
      virtual: true,
    });
  }
};

export default handler;


path: references/examples.md
content: # Entry Examples

Concrete examples of well-formatted AI/LLM entries with all fields.

## Learning: Model Selection (Code Generation)

```markdown
## [LRN-20250415-001] model_selection

**Logged**: 2025-04-15T10:30:00Z
**Priority**: high
**Status**: resolved
**Area**: model_config

### Summary
Switched from GPT-4o to Claude Sonnet for code generation; 23% improvement on HumanEval, 40% cost reduction

### Details
Evaluated Claude 4 Sonnet against GPT-4o on our internal code generation benchmark (500 multi-file
edit tasks). Sonnet scored 89.2% vs GPT-4o's 72.5% on correct first-attempt edits. Sonnet is
significantly better at following complex multi-file edit instructions where changes span 3+ files
with interdependencies. GPT-4o tended to miss downstream type changes in dependent files.

Cost comparison at 1M tokens/day: GPT-4o ~$45/day vs Sonnet ~$27/day (40% reduction).

### Impact
- **Quality**: 23% improvement on HumanEval pass@1 (72.5% → 89.2%)
- **Cost**: 40% reduction in daily API spend
- **Latency**: Comparable (Sonnet P50: 1.2s, GPT-4o P50: 1.1s)

### Suggested Action
Update model selection matrix: use Claude Sonnet as default for code generation tasks.
Keep GPT-4o as fallback and for tasks requiring function calling with complex schemas.

### Metadata
- Source: benchmark_evaluation
- Model: claude-4-sonnet vs gpt-4o
- Provider: anthropic vs openai
- Modality: text
- Temperature: 0.1
- Token Usage: avg 2,400 input / 1,800 output per task
- Latency: Sonnet P50 1,200ms / GPT-4o P50 1,100ms
- Cost: Sonnet $0.027/task vs GPT-4o $0.045/task
- Tags: model-selection, code-generation, benchmark, cost-optimization
- Pattern-Key: model_selection.code_generation

---
```

## Learning: Prompt Optimization (Chain-of-Thought)

```markdown
## [LRN-20250416-001] prompt_optimization

**Logged**: 2025-04-16T14:00:00Z
**Priority**: medium
**Status**: resolved
**Area**: prompt_engineering

### Summary
Adding "Think step by step before writing code" to system prompt reduced bug rate by 35% on internal eval

### Details
Tested chain-of-thought prompting on our internal eval suite (200 coding tasks with known correct
solutions). Adding "Think step by step before writing code. Identify edge cases before
implementing." to the system prompt reduced incorrect outputs from 31% to 20% (35% relative
improvement).

Effect varies by model:
- Claude 4 Sonnet: 28% → 16% error rate (43% improvement)
- GPT-4o: 34% → 24% error rate (29% improvement)
- Llama 3.1 70B: 45% → 42% error rate (7% improvement — minimal effect)
- Mistral Large: 38% → 30% error rate (21% improvement)

The improvement is most pronounced on tasks requiring multi-step reasoning. Smaller models
(<30B parameters) show minimal benefit, likely because they lack the capacity to execute
chain-of-thought effectively.

### Suggested Action
Add chain-of-thought instruction to system prompt for all code generation tasks using
models >30B parameters. Do not apply to small/fast models as it wastes tokens without benefit.

### Metadata
- Source: a_b_test
- Model: claude-4-sonnet, gpt-4o, llama-3.1-70b, mistral-large
- Provider: anthropic, openai, meta, mistral
- Modality: text
- Temperature: 0.1
- Token Usage: +150 tokens average overhead from CoT reasoning
- Tags: prompt-engineering, chain-of-thought, eval, code-generation
- Pattern-Key: prompt_optimization.chain_of_thought

---
```

## Learning: Context Management (Lost-in-the-Middle)

```markdown
## [LRN-20250417-001] context_management

**Logged**: 2025-04-17T09:15:00Z
**Priority**: high
**Status**: promoted
**Promoted**: RAG architecture doc
**Area**: rag_pipeline

### Summary
RAG context placed at beginning of prompt was ignored; moving chunks to just before user query improved relevance from 67% to 89%

### Details
Our RAG pipeline placed retrieved chunks at the top of the context window, followed by system
instructions, then the user query. Evaluation showed the model frequently ignored retrieved
context, especially when total context exceeded 8K tokens.

This is the "lost-in-the-middle" effect documented in Liu et al. (2023): LLMs attend most strongly
to the beginning and end of context, with a U-shaped attention curve. Information in the middle
of long contexts gets effectively ignored.

Restructured prompt layout:
1. System instructions (brief, ~200 tokens)
2. User query
3. Retrieved context chunks (placed right before the query, within the attention "end" zone)

Result: answer relevance jumped from 67% to 89% on our 500-question eval set.

### Suggested Action
Always place RAG-retrieved context immediately before the user's question, not at the
beginning of the prompt. Keep system instructions minimal and at the very start.

### Metadata
- Source: evaluation
- Model: claude-4-sonnet
- Provider: anthropic
- Modality: text
- Token Usage: same total tokens, different ordering
- Tags: rag, context-management, lost-in-the-middle, prompt-structure
- Pattern-Key: context_management.lost_in_the_middle

---
```

## Learning: Cost Efficiency (Embedding Dimensionality Reduction)

```markdown
## [LRN-20250418-001] cost_efficiency

**Logged**: 2025-04-18T11:00:00Z
**Priority**: medium
**Status**: resolved
**Area**: embeddings

### Summary
Switching from ada-002 (1536d) to a 512d model with Matryoshka dimensionality reduction saved 60% on vector storage with only 2% recall drop

### Details
Our vector store held 2.3M documents embedded with text-embedding-ada-002 at 1536 dimensions.
Storage cost was $840/month on Pinecone. Evaluated switching to text-embedding-3-small with
Matryoshka dimensionality reduction to 512 dimensions.

Benchmark results on our domain-specific retrieval eval (1,000 queries):
- ada-002 (1536d): Recall@10 = 94.2%, MRR = 0.87
- text-embedding-3-small (512d): Recall@10 = 92.1%, MRR = 0.85
- text-embedding-3-small (256d): Recall@10 = 86.4%, MRR = 0.79 (too much degradation)

512d was the sweet spot: 2.1% recall drop, 2.3% MRR drop, but 66% reduction in vector
dimensions → 60% storage cost reduction ($840 → $336/month).

### Suggested Action
Use text-embedding-3-small with 512 dimensions for general-purpose retrieval.
Reserve 1536d for high-precision retrieval tasks (legal, medical).
Never go below 512d without evaluating domain-specific recall.

### Metadata
- Source: benchmark_evaluation
- Model: text-embedding-ada-002 vs text-embedding-3-small
- Provider: openai
- Modality: text
- Cost: $840/month → $336/month (60% reduction)
- Tags: embeddings, dimensionality-reduction, cost, matryoshka, vector-store
- Pattern-Key: cost_efficiency.embedding_reduction

---
```

## Model Issue: Fine-Tune Regression (Catastrophic Forgetting)

```markdown
## [MDL-20250420-001] fine_tune_regression

**Logged**: 2025-04-20T16:00:00Z
**Priority**: critical
**Status**: resolved
**Area**: fine_tuning

### Summary
Fine-tuned Llama 3.1 70B on customer support data; improved tone but catastrophic forgetting on code tasks

### Details
Fine-tuned Llama 3.1 70B with QLoRA (rank 16, alpha 32) on 12K customer support conversation
pairs. Goal: improve empathetic tone and company-specific knowledge.

Results on eval suites:
- Customer support tone: 72% → 91% (target achieved)
- Company FAQ accuracy: 65% → 88% (target achieved)
- HumanEval (coding): 68% → 37% (catastrophic regression)
- MMLU (general knowledge): 79% → 71% (significant regression)

The fine-tuning data was 100% customer support with zero coding or general knowledge examples.
The model over-specialized, losing capabilities in unrelated domains.

### Root Cause
Pure domain-specific fine-tuning without replay data from general capabilities.
QLoRA rank 16 was high enough to significantly modify model weights.

### Fix
1. Mixed training data: 70% customer support + 20% general instruction + 10% coding
2. Reduced QLoRA rank from 16 to 8
3. Added eval gates: abort training if any benchmark drops >5% from baseline

Post-fix results:
- Customer support tone: 72% → 87% (slightly lower but acceptable)
- HumanEval: 68% → 65% (within 5% threshold)
- MMLU: 79% → 77% (within 5% threshold)

### Metadata
- Model: llama-3.1-70b (fine-tuned)
- Provider: meta (self-hosted)
- Modality: text
- Temperature: 0.3
- Training: QLoRA rank 16 → rank 8, 3 epochs, lr 2e-4
- Eval: HumanEval, MMLU, custom tone scorer, custom FAQ accuracy
- Tags: fine-tuning, catastrophic-forgetting, qlora, regression, eval
- Pattern-Key: fine_tune_regression.catastrophic_forgetting

### Resolution
- **Resolved**: 2025-04-22T10:00:00Z
- **Notes**: Mixed training data prevents catastrophic forgetting. Always include replay data.

---
```

## Model Issue: Multimodal Gap (Scanned PDF with Rotation)

```markdown
## [MDL-20250421-001] modality_gap

**Logged**: 2025-04-21T13:30:00Z
**Priority**: high
**Status**: resolved
**Area**: multimodal

### Summary
Claude vision fails to read handwritten text in scanned PDFs with >15 degree rotation

### Details
Document processing pipeline sends scanned PDFs to Claude 4 Opus vision for OCR and extraction.
Quality was high for clean scans but dropped severely for documents with rotation, skew, or
handwritten annotations.

Failure modes:
- Rotation >15°: model returns "I can't read the text clearly" or hallucinates content
- Handwritten annotations overlapping printed text: annotation ignored or misattributed
- Low-contrast scans (<100 DPI effective): complete extraction failure

Testing across 200 scanned documents:
- Clean scans (0° rotation, typed text): 96% accuracy
- Minor rotation (<15°): 89% accuracy
- Significant rotation (>15°): 42% accuracy
- Handwritten + rotation: 23% accuracy

### Fix
Added preprocessing pipeline before model ingestion:
1. Deskew with OpenCV `cv2.minAreaRect` + affine transform
2. Contrast normalization (CLAHE)
3. Upscale low-DPI images to minimum 300 DPI
4. Separate handwritten region detection with dedicated model

Post-fix: accuracy on rotated documents improved from 42% to 91%.

### Metadata
- Model: claude-4-opus
- Provider: anthropic
- Modality: multimodal (image/PDF)
- Token Usage: ~1,200 input tokens per page (image)
- Latency: 3,200ms per page
- Cost: $0.048 per page
- Tags: multimodal, vision, pdf, ocr, rotation, preprocessing
- Pattern-Key: modality_gap.rotated_pdf

### Resolution
- **Resolved**: 2025-04-22T15:00:00Z
- **Notes**: Always preprocess scanned documents before sending to vision models.

---
```

## Feature Request: Automated Model A/B Testing

```markdown
## [FEAT-20250419-001] model_ab_testing_framework

**Logged**: 2025-04-19T10:00:00Z
**Priority**: high
**Status**: pending
**Area**: evaluation

### Requested Capability
Automated model A/B testing framework that routes N% of requests to a candidate model
and compares outputs via LLM-as-judge evaluation.

### User Context
Currently comparing models requires manual benchmark runs on static eval sets.
We need to evaluate models on live production traffic to catch real-world differences
that synthetic benchmarks miss. Manual A/B testing is error-prone and takes 2-3 days
per comparison.

### Complexity Estimate
complex

### Suggested Implementation
1. Traffic router: split N% of requests to candidate model (configurable per endpoint)
2. Dual inference: both models process the same input, only primary response is served
3. LLM-as-judge: third model evaluates both outputs on quality, relevance, safety
4. Metrics dashboard: win rate, quality score distribution, latency comparison, cost delta
5. Auto-promote: if candidate wins by >5% over 1,000 samples, flag for promotion
6. Guardrails: abort test if candidate error rate exceeds 2x baseline

Architecture:
```
Request → Router → [Primary Model] → Response to user
                 → [Candidate Model] → Shadow response
                                     → LLM Judge → Metrics DB → Dashboard
```

### Metadata
- Frequency: recurring (every model evaluation cycle)
- Related Features: model selection matrix, eval pipeline, provider routing

---
```

## Learning: Promoted to Skill (RAG Chunk Sizing)

```markdown
## [LRN-20250412-001] context_management

**Logged**: 2025-04-12T15:00:00Z
**Priority**: high
**Status**: promoted_to_skill
**Skill-Path**: skills/rag-chunk-sizing
**Area**: rag_pipeline

### Summary
Optimal RAG chunk sizes vary significantly by content type; one-size-fits-all chunking degrades retrieval quality

### Details
After 3 months of RAG pipeline tuning across 4 projects, established optimal chunk sizes
by content type based on retrieval recall@10 benchmarks:

| Content Type | Optimal Chunk Size | Overlap | Boundary Strategy |
|--------------|-------------------|---------|-------------------|
| Source code | 100-200 lines | 20 lines | Function/class boundaries |
| Documentation (prose) | 500-800 tokens | 50 tokens | Semantic paragraph boundaries |
| Tables / structured data | Full table as single chunk | None | Preserve table integrity |
| API references | One endpoint per chunk | None | Endpoint boundary |
| Chat transcripts | One conversation turn | 1 turn | Speaker change |
| Legal documents | 300-500 tokens | 100 tokens | Section/clause boundaries |

Key findings:
- Code chunks that split mid-function lose 34% retrieval accuracy
- Tables split across chunks are effectively useless (retrieval recall drops to 12%)
- Prose chunked at arbitrary token boundaries vs semantic boundaries: 15% recall difference
- Overlap is critical for prose but harmful for structured data (causes duplicate retrieval)

### Suggested Action
Implement content-type-aware chunking in the RAG pipeline. Detect content type before
chunking and apply the appropriate strategy. Never use a single chunk size for all content.

### Metadata
- Source: benchmark_evaluation
- Model: text-embedding-3-small (512d)
- Provider: openai (embeddings)
- Tags: rag, chunking, retrieval, content-type, benchmark
- See Also: LRN-20250320-003, LRN-20250401-007, MDL-20250405-002
- Pattern-Key: context_management.chunk_sizing
- Recurrence-Count: 4
- First-Seen: 2025-01-20
- Last-Seen: 2025-04-12

---
```


path: references/openclaw-integration.md
content: # OpenClaw Integration

Complete setup and usage guide for integrating the self-improving-ai skill with OpenClaw.

## Overview

OpenClaw uses workspace-based prompt injection combined with event-driven hooks. Context is injected from workspace files at session start, and hooks can trigger on lifecycle events.

## Workspace Structure

```
~/.openclaw/
├── workspace/                   # Working directory
│   ├── AGENTS.md               # Multi-agent coordination, model routing
│   ├── SOUL.md                 # Model behavior guidelines, personality
│   ├── TOOLS.md                # Model/tool configuration, parameters
│   ├── MEMORY.md               # Long-term memory (main session only)
│   └── memory/                 # Daily memory files
│       └── YYYY-MM-DD.md
├── skills/                      # Installed skills
│   └── self-improving-ai/
│       └── SKILL.md
└── hooks/                       # Custom hooks
    └── self-improving-ai/
        ├── HOOK.md
        └── handler.ts
```

## Quick Setup

### 1. Install the Skill

```bash
clawdhub install self-improving-ai
```

Or copy manually:

```bash
cp -r self-improving-ai ~/.openclaw/skills/
```

### 2. Install the Hook (Optional)

```bash
cp -r hooks/openclaw ~/.openclaw/hooks/self-improving-ai
openclaw hooks enable self-improving-ai
```

### 3. Create Learning Files

```bash
mkdir -p ~/.openclaw/workspace/.learnings
```

## Promotion Targets (AI-Specific)

When AI/model learnings prove broadly applicable, promote them to workspace files:

| Learning Type | Promote To | Example |
|---------------|------------|---------|
| Model behavior patterns | `SOUL.md` | "Claude 4 tends to over-qualify, use direct prompting" |
| Model selection and routing | `AGENTS.md` | "Use fast model for triage, capable model for code gen" |
| Model/tool configuration | `TOOLS.md` | "Set temperature 0.1 for code, 0.7 for creative" |
| Model selection insights | Model selection matrix | "Sonnet for code gen, Opus for complex reasoning" |
| Prompt patterns | Prompt library | "Chain-of-thought improves code quality by 35%" |
| Fine-tuning lessons | Fine-tuning runbook | "Always include replay data to prevent forgetting" |
| RAG improvements | RAG architecture doc | "Chunk by content type, not fixed token size" |
| Inference optimizations | Performance tuning guide | "Cache system prompts, batch similar requests" |
| Evaluation findings | Benchmark suite docs | "HumanEval + internal eval for code gen models" |
| Guardrail tuning | Guardrail policy doc | "Lower toxicity threshold for customer-facing" |

### Promotion Decision Tree

```
Is it about which model to use?
├── Yes → Model selection matrix
└── No → Is it about how to prompt?
    ├── Yes → Prompt library
    └── No → Is it about model behavior/personality?
        ├── Yes → SOUL.md
        └── No → Is it about model workflow/routing?
            ├── Yes → AGENTS.md
            └── No → Is it about model configuration?
                ├── Yes → TOOLS.md
                └── No → Is it about fine-tuning?
                    ├── Yes → Fine-tuning runbook
                    └── No → Is it about RAG?
                        ├── Yes → RAG architecture doc
                        └── No → Is it about evaluation?
                            ├── Yes → Benchmark suite
                            └── No → Keep in .learnings/
```

## AI-Specific Detection Triggers

| Trigger | Action | Target |
|---------|--------|--------|
| Model quality dropped after update | Log model issue | MODEL_ISSUES.md |
| Inference latency spike | Log model issue | MODEL_ISSUES.md |
| RAG retrieval returned irrelevant chunks | Log model issue | MODEL_ISSUES.md |
| Embedding quality degraded | Log model issue | MODEL_ISSUES.md |
| Multimodal input failed | Log model issue | MODEL_ISSUES.md |
| Guardrail misfire | Log model issue | MODEL_ISSUES.md |
| Better model discovered for a task | Log learning | LEARNINGS.md (model_selection) |
| Prompt pattern improved output | Log learning | LEARNINGS.md (prompt_optimization) |
| Hallucination detected | Log learning | LEARNINGS.md (hallucination_rate) |
| Context window overflow | Log learning | LEARNINGS.md (context_management) |
| Fine-tuned model regressed on eval | Log learning | LEARNINGS.md (fine_tune_regression) |
| Token cost exceeded budget | Log learning | LEARNINGS.md (cost_efficiency) |

## Inter-Agent Communication

OpenClaw provides tools for cross-session communication. Use only when cross-session sharing is explicitly needed.

### sessions_send

Share an AI/model insight with another session:
```
sessions_send(sessionKey="session-id", message="Model insight: Claude Sonnet 23% better than GPT-4o on multi-file edits, 40% cheaper")
```

### sessions_spawn

Spawn a background agent to analyze model performance patterns:
```
sessions_spawn(task="Analyze .learnings/MODEL_ISSUES.md for recurring model failures and promotion candidates", label="model-review")
```

## Available Hook Events

| Event | When It Fires |
|-------|---------------|
| `agent:bootstrap` | Before workspace files inject |
| `command:new` | When `/new` command issued |
| `command:reset` | When `/reset` command issued |
| `command:stop` | When `/stop` command issued |
| `gateway:startup` | When gateway starts |

## Verification

```bash
openclaw hooks list        # Check hook is registered
openclaw status            # Check skill is loaded
```

## Troubleshooting

### Hook not firing
1. Ensure hooks enabled in config
2. Restart gateway after config changes
3. Check gateway logs for errors

### Learnings not persisting
1. Verify `.learnings/` directory exists
2. Check file permissions
3. Ensure workspace path is configured correctly

### Skill not loading
1. Check skill is in skills directory
2. Verify SKILL.md has correct frontmatter
3. Run `openclaw status` to see loaded skills


path: references/hooks-setup.md
content: # Hook Setup Guide

Configure automatic AI/LLM self-improvement triggers for AI coding agents.

## Overview

Hooks enable proactive AI-pattern capture by injecting reminders at key moments:
- **UserPromptSubmit**: Reminder after each prompt to evaluate AI/model learnings
- **PostToolUse (Bash)**: Error detection when model API calls, inference, or RAG operations fail

Note: For model API calls specifically, consider using a PostToolUse matcher to detect API errors from provider SDKs (Anthropic, OpenAI, etc.).

## Claude Code Setup

### Option 1: Project-Level Configuration

Create `.claude/settings.json` in your project root:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-ai/scripts/activator.sh"
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-ai/scripts/error-detector.sh"
          }
        ]
      }
    ]
  }
}
```

### Option 2: User-Level Configuration

Add to `~/.claude/settings.json` for global activation:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "~/.claude/skills/self-improving-ai/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

### Minimal Setup (Activator Only)

For lower overhead, use only the UserPromptSubmit hook:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-ai/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## Codex CLI Setup

Codex uses the same hook system. Create `.codex/settings.json`:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-ai/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## GitHub Copilot Setup

Copilot doesn't support hooks. Add guidance to `.github/copilot-instructions.md`:

```markdown
## AI Self-Improvement

After encountering AI/model issues, consider logging to `.learnings/`:
- Model quality drops, latency spikes, inference failures → MODEL_ISSUES.md
- Model selection insights, prompt patterns, hallucinations → LEARNINGS.md
- Missing AI capabilities (eval framework, A/B testing) → FEATURE_REQUESTS.md

Include model name, provider, temperature, token usage, and latency.
Promote recurring patterns to model selection matrix or prompt library.
```

## Verification

### Test Activator Hook

1. Enable the hook configuration
2. Start a new Claude Code session
3. Send any prompt
4. Verify you see `<ai-improvement-reminder>` in the context

### Test Error Detector Hook

1. Enable PostToolUse hook for Bash
2. Run a command that produces a model API error (rate limit, context overflow, etc.)
3. Verify you see `<ai-error-detected>` reminder

### Dry Run Extract Script

```bash
./skills/self-improving-ai/scripts/extract-skill.sh test-skill --dry-run
```

## Troubleshooting

### Hook Not Triggering

1. **Check script permissions**: `chmod +x scripts/*.sh`
2. **Verify path**: Use absolute paths or paths relative to project root
3. **Check settings location**: Project vs user-level settings
4. **Restart session**: Hooks are loaded at session start

### Permission Denied

```bash
chmod +x ./skills/self-improving-ai/scripts/activator.sh
chmod +x ./skills/self-improving-ai/scripts/error-detector.sh
chmod +x ./skills/self-improving-ai/scripts/extract-skill.sh
```

### Script Not Found

If using relative paths, ensure you're in the correct directory or use absolute paths:

```json
{
  "command": "/absolute/path/to/skills/self-improving-ai/scripts/activator.sh"
}
```

### Too Much Overhead

If the activator feels intrusive:

1. **Use minimal setup**: Only UserPromptSubmit, skip PostToolUse
2. **Add matcher filter**: Only trigger for AI-related prompts:

```json
{
  "matcher": "model|inference|prompt|rag|embed|hallucin|latency|fine-tune",
  "hooks": [...]
}
```

## Hook Output Budget

The activator is designed to be lightweight:
- **Target**: ~50-100 tokens per activation
- **Content**: Structured AI-specific reminder, not verbose instructions
- **Format**: XML tags for easy parsing

## Security Considerations

- Hook scripts run with the same permissions as the agent
- Scripts only output text; they don't modify files or run commands
- Error detector reads `CLAUDE_TOOL_OUTPUT` environment variable
- Treat `CLAUDE_TOOL_OUTPUT` as potentially sensitive; do not log verbatim
- Never log API keys, model access tokens, or customer data in learnings
- All scripts are opt-in
- Recommended default: enable `UserPromptSubmit` only; add `PostToolUse` when you want model API error detection

## Disabling Hooks

To temporarily disable without removing configuration:

1. Comment out in settings
2. Or delete the settings file — hooks won't run without configuration


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)