Skill: jose-compu/self-improving-science (version 1.1.0)

--- name: self-improving-science description: "Captures learnings, experiment issues, and methodology corrections for continuous improvement in scientific research and ML workflows. Use when: (1) Data leakage detected in train/test split, (2) Model fails to reproduce across seeds or environments, (3) Statistical test misapplied or p-value misinterpreted, (4) Hypothesis test fails or needs revision, (5) Feature distribution shift detected, (6) User corrects methodology or analysis approach, (7) Experiment design flaw discovered. Also review learnings before designing new experiments." --- # Self-Improving Science Skill Log learnings, experiment issues, and methodology corrections to markdown files for continuous improvement in scientific research, data science, and ML/AI experimentation. Important findings get promoted to experiment checklists, data governance docs, model cards, and methodology standards. ## First-Use Initialisation Before logging anything, ensure the `.learnings/` directory and files exist in the project or workspace root. If any are missing, create them: ```bash mkdir -p .learnings [ -f .learnings/LEARNINGS.md ] || printf "# Learnings\n\nMethodology insights, statistical corrections, and knowledge gaps captured during research.\n\n**Categories**: methodology_flaw | data_quality | reproducibility_issue | statistical_error | hypothesis_revision | experiment_design\n\n---\n" > .learnings/LEARNINGS.md [ -f .learnings/EXPERIMENT_ISSUES.md ] || printf "# Experiment Issues\n\nFailed experiments, data quality problems, and reproducibility failures.\n\n---\n" > .learnings/EXPERIMENT_ISSUES.md [ -f .learnings/FEATURE_REQUESTS.md ] || printf "# Feature Requests\n\nResearch tooling and ML pipeline capabilities requested by the user.\n\n---\n" > .learnings/FEATURE_REQUESTS.md ``` Never overwrite existing files. This is a no-op if `.learnings/` is already initialised. Do not log proprietary datasets, patient identifiers, API keys, or raw data samples unless the user explicitly asks. Prefer summary statistics and redacted excerpts over full data dumps. If you want automatic reminders or setup assistance, use the opt-in hook workflow described in [Hook Integration](#hook-integration). ## Quick Reference | Situation | Action | |-----------|--------| | Data leakage found in pipeline | Log to `.learnings/EXPERIMENT_ISSUES.md` with `data_quality` | | Model fails to reproduce | Log to `.learnings/EXPERIMENT_ISSUES.md` with `reproducibility_issue` | | Statistical test misapplied | Log to `.learnings/LEARNINGS.md` with `statistical_error` | | Hypothesis test fails | Log to `.learnings/LEARNINGS.md` with `hypothesis_revision` | | Methodology flaw discovered | Log to `.learnings/LEARNINGS.md` with `methodology_flaw` | | Experiment design improvement | Log to `.learnings/LEARNINGS.md` with `experiment_design` | | Feature distribution shift | Log to `.learnings/EXPERIMENT_ISSUES.md` with `data_quality` | | User wants missing ML tool | Log to `.learnings/FEATURE_REQUESTS.md` | | NaN loss or training divergence | Log to `.learnings/EXPERIMENT_ISSUES.md` | | Missing data pattern discovered | Log to `.learnings/LEARNINGS.md` with `data_quality` | | Similar to existing entry | Link with `**See Also**`, consider priority bump | | Broadly applicable finding | Promote to experiment checklist, model card, or methodology standard | | Data governance insight | Promote to data governance docs | | Model behavior documentation | Promote to model card | | Pipeline best practice | Promote to `AGENTS.md` (OpenClaw workspace) | ## OpenClaw Setup (Recommended) OpenClaw is the primary platform for this skill. It uses workspace-based prompt injection with automatic skill loading. ### Installation **Via ClawdHub (recommended):** ```bash clawdhub install self-improving-science ``` **Manual:** ```bash git clone https://github.com/jose-compu/self-improving-science.git ~/.openclaw/skills/self-improving-science ``` ### Workspace Structure OpenClaw injects these files into every session: ``` ~/.openclaw/workspace/ ├── AGENTS.md # Multi-agent workflows, experiment orchestration ├── SOUL.md # Research principles, scientific rigor guidelines ├── TOOLS.md # ML framework gotchas, data tool capabilities ├── MEMORY.md # Long-term memory (main session only) ├── memory/ # Daily memory files │ └── YYYY-MM-DD.md └── .learnings/ # This skill's log files ├── LEARNINGS.md ├── EXPERIMENT_ISSUES.md └── FEATURE_REQUESTS.md ``` ### Create Learning Files ```bash mkdir -p ~/.openclaw/workspace/.learnings ``` Then create the log files (or copy from `assets/`): - `LEARNINGS.md` — methodology corrections, statistical insights, experiment design lessons - `EXPERIMENT_ISSUES.md` — data quality failures, reproducibility problems, model drift events - `FEATURE_REQUESTS.md` — requested research tooling and pipeline capabilities ### Promotion Targets When learnings prove broadly applicable, promote them to research artifacts: | Learning Type | Promote To | Example | |---------------|------------|---------| | Experiment design patterns | Experiment Checklist | "Always check class balance before training" | | Data handling rules | Data Governance Docs | "PII must be hashed before feature extraction" | | Model documentation | Model Card | "Model degrades on inputs > 512 tokens" | | Pipeline best practices | `AGENTS.md` | "Run distribution check before retraining" | | ML framework gotchas | `TOOLS.md` | "PyTorch DataLoader workers leak memory on macOS" | | Research communication | `SOUL.md` | "Report confidence intervals, not just point estimates" | ### Inter-Session Communication OpenClaw provides tools to share learnings across sessions: - **sessions_list** — View active/recent sessions - **sessions_history** — Read another session's transcript - **sessions_send** — Send a learning to another session - **sessions_spawn** — Spawn a sub-agent for background work Use these only in trusted environments and only when the user explicitly wants cross-session sharing. Prefer sending summary statistics and methodology notes, not raw datasets or credentials. ### Optional: Enable Hook For automatic reminders at session start: ```bash cp -r hooks/openclaw ~/.openclaw/hooks/self-improving-science openclaw hooks enable self-improving-science ``` See `references/openclaw-integration.md` for complete details. --- ## Generic Setup (Other Agents) For Claude Code, Codex, Copilot, or other agents, create `.learnings/` in the project or workspace root: ```bash mkdir -p .learnings ``` Create the files inline using the headers shown above. Avoid reading templates from the current repo or workspace unless you explicitly trust that path. ### Add reference to agent files to remind yourself to log learnings #### Self-Improvement Workflow (Science) When experiment issues or methodology corrections occur: 1. Log to `.learnings/EXPERIMENT_ISSUES.md`, `LEARNINGS.md`, or `FEATURE_REQUESTS.md` 2. Review and promote broadly applicable learnings to: - Experiment checklists — pre-run validation steps - Model cards — known limitations, performance bounds - Data governance docs — handling rules, quality gates - `CLAUDE.md` or `AGENTS.md` — project-level conventions ## Logging Format ### Learning Entry Append to `.learnings/LEARNINGS.md`: ```markdown ## [LRN-YYYYMMDD-XXX] category **Logged**: ISO-8601 timestamp **Priority**: low | medium | high | critical **Status**: pending **Area**: data_collection | preprocessing | analysis | modeling | validation | publication ### Summary One-line description of what was learned ### Details Full context: what happened, what was wrong, what's correct. Include relevant metrics, sample sizes, or statistical values. ### Suggested Action Specific fix or improvement to make ### Metadata - Source: experiment | peer_review | user_feedback | analysis - Related Files: path/to/notebook.ipynb, path/to/data.csv - Tags: tag1, tag2 - See Also: LRN-20260101-001 (if related to existing entry) - Dataset: dataset_name (optional) - Model: model_name_or_version (optional) - Metric-Before: 0.85 (optional) - Metric-After: 0.91 (optional) - Pattern-Key: leakage.timestamp | stats.normality_assumption (optional) - Recurrence-Count: 1 (optional) - First-Seen: 2026-01-15 (optional) - Last-Seen: 2026-01-15 (optional) --- ``` ### Experiment Issue Entry Append to `.learnings/EXPERIMENT_ISSUES.md`: ```markdown ## [EXP-YYYYMMDD-XXX] category **Logged**: ISO-8601 timestamp **Priority**: high **Status**: pending **Area**: data_collection | preprocessing | analysis | modeling | validation | publication ### Summary Brief description of what failed or went wrong ### Error ``` Actual error message, unexpected metric, or reproducibility delta ``` ### Context - Experiment/notebook attempted - Dataset and split used - Model architecture and hyperparameters (if relevant) - Hardware/environment details - Summary of relevant output (avoid full data dumps) ### Root Cause If identifiable, what caused the issue ### Suggested Fix How to prevent or resolve this ### Metadata - Reproducible: yes | no | unknown - Related Files: path/to/notebook.ipynb - Seeds Tested: 42, 123, 7 (if reproducibility issue) - See Also: EXP-20260101-001 (if recurring) --- ``` ### Feature Request Entry Append to `.learnings/FEATURE_REQUESTS.md`: ```markdown ## [FEAT-YYYYMMDD-XXX] capability_name **Logged**: ISO-8601 timestamp **Priority**: medium **Status**: pending **Area**: data_collection | preprocessing | analysis | modeling | validation | publication ### Requested Capability What the user wanted to do ### Research Context Why they need it — what experiment, analysis, or pipeline step it supports ### Complexity Estimate simple | medium | complex ### Suggested Implementation How this could be built, what libraries or tools it might use ### Metadata - Frequency: first_time | recurring - Related Features: existing_pipeline_step --- ``` ## ID Generation Format: `TYPE-YYYYMMDD-XXX` - TYPE: `LRN` (learning), `EXP` (experiment issue), `FEAT` (feature request) - YYYYMMDD: Current date - XXX: Sequential number or random 3 chars (e.g., `001`, `A7B`) Examples: `LRN-20260412-001`, `EXP-20260412-A3F`, `FEAT-20260412-002` ## Resolving Entries When an issue is fixed, update the entry: 1. Change `**Status**: pending` → `**Status**: resolved` 2. Add resolution block after Metadata: ```markdown ### Resolution - **Resolved**: 2026-04-13T09:00:00Z - **Commit/PR**: abc123 or #42 - **Experiment-Run**: run_id_or_notebook_version - **Notes**: Brief description of what was done ``` Other status values: - `in_progress` — Actively being investigated - `wont_fix` — Decided not to address (add reason in Resolution notes) - `promoted` — Elevated to experiment checklist, model card, or methodology standard - `promoted_to_skill` — Extracted as a reusable skill ## Promoting to Research Artifacts When a learning is broadly applicable (not a one-off fix), promote it to permanent research memory. ### When to Promote - Learning applies across multiple experiments or datasets - Knowledge any researcher (human or AI) working on this project should know - Prevents recurring methodology mistakes - Documents data constraints or model limitations ### Promotion Targets | Target | What Belongs There | |--------|-------------------| | Experiment Checklist | Pre-run validation: data checks, split verification, seed logging | | Model Card | Known limitations, performance bounds, failure modes, training data description | | Data Governance Docs | PII handling, data quality gates, provenance requirements | | Methodology Standards | Statistical test selection, sample size requirements, reporting conventions | | `CLAUDE.md` / `AGENTS.md` | Project-level facts, pipeline conventions, automation rules | | `TOOLS.md` | ML framework gotchas, library version constraints (OpenClaw) | | `SOUL.md` | Research communication style, rigor principles (OpenClaw) | ### How to Promote 1. **Distill** the learning into a concise rule or checklist item 2. **Add** to appropriate section in target document (create if needed) 3. **Update** original entry: - Change `**Status**: pending` → `**Status**: promoted` - Add `**Promoted**: experiment-checklist.md` (or target doc) ### Promotion Examples **Learning** (verbose): > Used t-test on highly skewed revenue data. User pointed out normality > assumption was violated. Switched to Mann-Whitney U test. P-value changed > from 0.03 to 0.12 — original conclusion was invalid. **In Methodology Standards** (concise): ```markdown ## Statistical Test Selection - Check normality (Shapiro-Wilk) before parametric tests - Skewed data → use non-parametric alternatives (Mann-Whitney U, Kruskal-Wallis) - Report both test choice rationale and assumption checks ``` **Learning** (verbose): > Timestamp feature in training data was leaking the target. Model had 0.99 > AUC in validation but 0.52 in production. The timestamp encoded when the > label was assigned, not when the event occurred. **In Experiment Checklist** (actionable): ```markdown ## Pre-Training Checks - [ ] Verify no temporal leakage: features must predate the label event - [ ] Check feature-target correlation for suspiciously high values (>0.95) - [ ] Validate that train/test split respects time ordering if data is temporal ``` ## Recurring Pattern Detection If logging something similar to an existing entry: 1. **Search first**: `grep -r "keyword" .learnings/` 2. **Link entries**: Add `**See Also**: EXP-20260101-001` in Metadata 3. **Bump priority** if issue keeps recurring 4. **Consider systemic fix**: Recurring issues often indicate: - Missing experiment checklist item (→ promote to checklist) - Missing data validation step (→ add to pipeline) - Architectural problem (→ create tech debt ticket) ## Detection Triggers Automatically log when you notice: **Data Quality Issues** (→ experiment issue with `data_quality`): - Data leakage between train and test sets - Feature-target correlation > 0.95 - Missing data patterns (MNAR, MAR, MCAR) not accounted for - Distribution shift between training and serving data - Class imbalance not addressed - Duplicate records in dataset **Statistical Errors** (→ learning with `statistical_error`): - P-value reported without effect size - Multiple comparisons without correction (Bonferroni, FDR) - Parametric test on non-normal data - Confidence interval misinterpreted - Sample size too small for chosen test - P-hacking signals: many tested hypotheses, only significant ones reported **Methodology Flaws** (→ learning with `methodology_flaw`): - No holdout set for hyperparameter tuning - Validation set used for both tuning and evaluation - Feature engineering on full dataset before splitting - Cross-validation leaking preprocessing steps - Evaluation metric misaligned with business objective **Reproducibility Issues** (→ experiment issue with `reproducibility_issue`): - Different results across random seeds - Results differ between local and cloud environments - Notebook cells executed out of order - Missing dependency versions in requirements - GPU non-determinism not documented **Hypothesis Revisions** (→ learning with `hypothesis_revision`): - Hypothesis test fails to reject null - Effect size smaller than expected - Confounding variable discovered - Causal assumption violated **Model/Training Errors** (→ experiment issue): - NaN loss during training - Gradient explosion or vanishing - CUDA out of memory - Shape mismatch in tensors - Model accuracy degradation after data update - Convergence failure **Feature Requests** (→ feature request): - "Can you add automated leakage detection?" - "I need a data drift monitoring pipeline" - "Is there a way to version datasets?" - "Why can't we do A/B test analysis here?" ## Priority Guidelines | Priority | When to Use | Example | |----------|-------------|---------| | `critical` | Data leakage in production model, results published with error | Target leakage shipped to production scoring | | `high` | Irreproducible published result, major statistical error | T-test on non-normal data changing conclusion | | `medium` | Methodology improvement, better experiment design | Adding stratified splitting to pipeline | | `low` | Documentation of approach, minor analysis note | Noting which random seed was used | ## Area Tags Use to filter learnings by research phase: | Area | Scope | |------|-------| | `data_collection` | Surveys, scraping, APIs, sensor data, database queries | | `preprocessing` | Cleaning, imputation, encoding, normalization, feature engineering | | `analysis` | EDA, statistical tests, hypothesis testing, visualization | | `modeling` | Model selection, training, hyperparameter tuning, architecture | | `validation` | Cross-validation, holdout testing, A/B tests, model evaluation | | `publication` | Reports, papers, model cards, dashboards, presentations | ## Best Practices 1. **Log immediately** — context and metric values are freshest right after the issue 2. **Include metrics** — always note before/after values, sample sizes, p-values 3. **Record seeds and versions** — library versions, random seeds, GPU type 4. **Link notebooks** — reference the exact notebook and cell where the issue occurred 5. **Suggest concrete fixes** — not just "investigate further" 6. **Use consistent categories** — enables filtering by issue type 7. **Promote aggressively** — if a mistake could recur, add to experiment checklist 8. **Review before experiments** — check past learnings for the dataset/method you're about to use ## Gitignore Options **Keep learnings local** (per-researcher): ```gitignore .learnings/ ``` This repo uses that default to avoid committing sensitive data or noisy local logs. **Track learnings in repo** (team-wide): Don't add to .gitignore — learnings become shared research knowledge. **Hybrid** (track templates, ignore entries): ```gitignore .learnings/*.md !.learnings/.gitkeep ``` ## Hook Integration Enable automatic reminders through agent hooks. This is **opt-in** — you must explicitly configure hooks. ### Quick Setup (Claude Code / Codex) Create `.claude/settings.json` in your project: ```json { "hooks": { "UserPromptSubmit": [{ "matcher": "", "hooks": [{ "type": "command", "command": "./skills/self-improving-science/scripts/activator.sh" }] }] } } ``` This injects a science-specific learning evaluation reminder after each prompt (~60-120 tokens overhead). ### Advanced Setup (With Error Detection) ```json { "hooks": { "UserPromptSubmit": [{ "matcher": "", "hooks": [{ "type": "command", "command": "./skills/self-improving-science/scripts/activator.sh" }] }], "PostToolUse": [{ "matcher": "Bash", "hooks": [{ "type": "command", "command": "./skills/self-improving-science/scripts/error-detector.sh" }] }] } } ``` Enable `PostToolUse` only if you want error-pattern reminders from ML training output and data pipeline commands. ### Available Hook Scripts | Script | Hook Type | Purpose | |--------|-----------|---------| | `scripts/activator.sh` | UserPromptSubmit | Reminds to evaluate experiment learnings | | `scripts/error-detector.sh` | PostToolUse (Bash) | Triggers on ML/data errors | See `references/hooks-setup.md` for detailed configuration and troubleshooting. ## Automatic Skill Extraction When a learning is valuable enough to become a reusable skill, extract it: ```bash ./skills/self-improving-science/scripts/extract-skill.sh skill-name --dry-run ./skills/self-improving-science/scripts/extract-skill.sh skill-name ``` **Extraction criteria** — any of: recurring (2+ See Also links), verified (resolved status), non-obvious (required investigation), broadly applicable, or user-flagged. After extraction: set status to `promoted_to_skill`, add `Skill-Path`, verify in fresh session. ## Periodic Review Review `.learnings/` before new experiments, after training runs, and before publication. ```bash grep -h "Status\*\*: pending" .learnings/*.md | wc -l grep -B5 "Priority\*\*: high" .learnings/*.md | grep "^## \[" ``` ## Multi-Agent Support | Agent | Activation | Setup | |-------|-----------|-------| | Claude Code / Codex | Hooks (UserPromptSubmit, PostToolUse) | `.claude/settings.json` | | GitHub Copilot | Manual | `.github/copilot-instructions.md` | | OpenClaw | Workspace injection | See OpenClaw Setup above | Apply self-improvement when you: discover data leakage, get irreproducible results, misapply a statistical test, find methodology flaws, hit training errors, or learn dataset quirks. ## Stackability Contract (Standalone + Multi-Skill) This skill is standalone-compatible and stackable with other self-improving skills. ### Namespaced Logging (recommended for 2+ skills) - Namespace for this skill: `.learnings/science/` - Keep current standalone behavior if you prefer flat files. - Optional shared index for all skills: `.learnings/INDEX.md` ### Required Metadata Every new entry must include: ```markdown **Skill**: science ``` ### Hook Arbitration (when 2+ skills are enabled) - Use one dispatcher hook as the single entrypoint. - Dispatcher responsibilities: route by matcher, dedupe repeated events, and rate-limit reminders. - Suggested defaults: dedupe key = `event + matcher + file + 5m_window`; max 1 reminder per skill every 5 minutes. ### Narrow Matcher Scope (science) Only trigger this skill automatically for science signals such as: - `experiment|hypothesis|p-value|confidence interval|reproducibility` - `dataset shift|data leakage|methodology flaw|benchmark drift` - explicit science intent in user prompt ### Cross-Skill Precedence When guidance conflicts, apply: 1. `security` 2. `engineering` 3. `coding` 4. `ai` 5. user-explicit domain skill 6. `meta` as tie-breaker ### Ownership Rules - This skill writes only to `.learnings/science/` in stackable mode. - It may read other skill folders for cross-linking, but should not rewrite their entries.

path: assets/FEATURE_REQUESTS.md
content: # Feature Requests

Research tooling, ML pipeline capabilities, and analysis tools requested by the user.

---


path: assets/ERRORS.md
content: # Experiment Issues Log

Failed experiments, data quality problems, reproducibility failures, and training errors.

---


path: assets/SKILL-TEMPLATE.md
content: # Skill Template (Science)

Template for creating skills extracted from research learnings. Copy and customize.

---

## SKILL.md Template

```markdown
---
name: skill-name-here
description: "Concise description of when and why to use this skill. Include trigger conditions."
---

# Skill Name

Brief introduction explaining the research problem this skill solves and its origin.

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Trigger 1] | [Action 1] |
| [Trigger 2] | [Action 2] |

## Background

Why this knowledge matters. What methodology mistakes it prevents. Context from the original learning or experiment.

## Solution

### Step-by-Step

1. First step with code or command
2. Second step
3. Verification step (include expected metric ranges)

### Code Example

\`\`\`python
# Example code demonstrating the solution
# Include library versions in comments
import pandas as pd  # >= 2.0
import sklearn  # >= 1.3
\`\`\`

## Common Variations

- **Variation A**: Different dataset type or scale
- **Variation B**: Different ML framework or statistical test

## Gotchas

- Warning or common mistake #1 (include metric impact if known)
- Warning or common mistake #2

## Related

- Link to related methodology documentation
- Link to related skill or experiment checklist

## Source

Extracted from learning entry.
- **Learning ID**: LRN-YYYYMMDD-XXX
- **Original Category**: methodology_flaw | data_quality | statistical_error | etc.
- **Dataset**: dataset_name (if applicable)
- **Extraction Date**: YYYY-MM-DD
```

---

## Minimal Template

For simple skills that don't need all sections:

```markdown
---
name: skill-name-here
description: "What this skill does and when to use it."
---

# Skill Name

[Problem statement in one sentence]

## Solution

[Direct solution with code/commands, including library versions]

## Source

- Learning ID: LRN-YYYYMMDD-XXX
```

---

## Template with Scripts

For skills that include executable helpers (data validation, leakage detection, etc.):

```markdown
---
name: skill-name-here
description: "What this skill does and when to use it."
---

# Skill Name

[Introduction]

## Quick Reference

| Command | Purpose |
|---------|---------|
| `./scripts/validate-data.sh` | [What it checks] |
| `./scripts/check-leakage.py` | [What it detects] |

## Usage

### Automated (Recommended)

\`\`\`bash
./skills/skill-name/scripts/validate-data.sh path/to/dataset.csv
\`\`\`

### Manual Steps

1. Step one
2. Step two

## Scripts

| Script | Description |
|--------|-------------|
| `scripts/validate-data.sh` | Data quality checks |
| `scripts/check-leakage.py` | Feature-target leakage detection |

## Source

- Learning ID: LRN-YYYYMMDD-XXX
```

---

## Naming Conventions

- **Skill name**: lowercase, hyphens for spaces
  - Good: `leakage-detection`, `normality-checks`, `model-card-template`
  - Bad: `Leakage_Detection`, `NormalityChecks`

- **Description**: Start with action verb, mention trigger
  - Good: "Detects data leakage in train/test splits. Use when feature-target correlation is suspiciously high."
  - Bad: "Data stuff"

- **Files**:
  - `SKILL.md` — Required, main documentation
  - `scripts/` — Optional, executable code (validation, detection)
  - `references/` — Optional, detailed methodology docs
  - `assets/` — Optional, templates and data schemas

---

## Extraction Checklist

Before creating a skill from a learning:

- [ ] Learning is verified (status: resolved, fix validated with data)
- [ ] Solution is broadly applicable (not dataset-specific)
- [ ] Content is complete (has all needed context and metrics)
- [ ] Statistical claims include sample sizes and effect sizes
- [ ] Name follows conventions
- [ ] Description is concise but informative
- [ ] Quick Reference table is actionable
- [ ] Code examples include library version requirements

After creating:

- [ ] Update original learning with `promoted_to_skill` status
- [ ] Add `Skill-Path: skills/skill-name` to learning metadata
- [ ] Test skill by reading it in a fresh session


path: assets/LEARNINGS.md
content: # Learnings

Methodology insights, statistical corrections, and knowledge gaps captured during research.

**Categories**: methodology_flaw | data_quality | reproducibility_issue | statistical_error | hypothesis_revision | experiment_design
**Areas**: data_collection | preprocessing | analysis | modeling | validation | publication
**Statuses**: pending | in_progress | resolved | wont_fix | promoted | promoted_to_skill

## Status Definitions

| Status | Meaning |
|--------|---------|
| `pending` | Not yet addressed |
| `in_progress` | Actively being investigated |
| `resolved` | Issue fixed or knowledge integrated |
| `wont_fix` | Decided not to address (reason in Resolution) |
| `promoted` | Elevated to experiment checklist, model card, or methodology standard |
| `promoted_to_skill` | Extracted as a reusable skill |

## Skill Extraction Fields

When a learning is promoted to a skill, add these fields:

```markdown
**Status**: promoted_to_skill
**Skill-Path**: skills/skill-name
```

Example:
```markdown
## [LRN-20260412-001] methodology_flaw

**Logged**: 2026-04-12T10:00:00Z
**Priority**: high
**Status**: promoted_to_skill
**Skill-Path**: skills/leakage-detection
**Area**: preprocessing

### Summary
Timestamp features leak target variable when timestamps encode label assignment time
...
```

---



path: scripts/activator.sh
content: #!/bin/bash
# Self-Improving Science Activator Hook
# Triggers on UserPromptSubmit to remind about experiment learning capture
# Keep output minimal (~60-120 tokens) to minimize overhead

set -e

cat << 'EOF'
<science-improvement-reminder>
After completing this task, evaluate if extractable research knowledge emerged:
- Data quality issue or leakage discovered?
- Statistical test assumption violated?
- Methodology flaw found in experiment design?
- Model failed to reproduce across seeds or environments?
- Hypothesis needs revision based on results?
- Training error (NaN loss, OOM, convergence failure)?

If yes: Log to .learnings/ using the self-improving-science format.
  - Data/reproducibility issues → EXPERIMENT_ISSUES.md
  - Methodology/statistical insights → LEARNINGS.md
  - Missing ML tooling → FEATURE_REQUESTS.md

If high-value (recurring, broadly applicable): Promote to experiment checklist, model card, or methodology standard.
</science-improvement-reminder>
EOF


path: scripts/extract-skill.sh
content: #!/bin/bash
# Science Skill Extraction Helper
# Creates a new skill from a research learning entry
# Usage: ./extract-skill.sh <skill-name> [--dry-run]

set -e

SKILLS_DIR="./skills"

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

usage() {
    cat << EOF
Usage: $(basename "$0") <skill-name> [options]

Create a new skill from a research learning entry.

Arguments:
  skill-name     Name of the skill (lowercase, hyphens for spaces)

Options:
  --dry-run      Show what would be created without creating files
  --output-dir   Relative output directory under current path (default: ./skills)
  -h, --help     Show this help message

Examples:
  $(basename "$0") leakage-detection
  $(basename "$0") normality-checks --dry-run
  $(basename "$0") model-card-template --output-dir ./skills/custom

The skill will be created in: \$SKILLS_DIR/<skill-name>/
EOF
}

log_info() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

log_warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

log_error() {
    echo -e "${RED}[ERROR]${NC} $1" >&2
}

SKILL_NAME=""
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case $1 in
        --dry-run)
            DRY_RUN=true
            shift
            ;;
        --output-dir)
            if [ -z "${2:-}" ] || [[ "${2:-}" == -* ]]; then
                log_error "--output-dir requires a relative path argument"
                usage
                exit 1
            fi
            SKILLS_DIR="$2"
            shift 2
            ;;
        -h|--help)
            usage
            exit 0
            ;;
        -*)
            log_error "Unknown option: $1"
            usage
            exit 1
            ;;
        *)
            if [ -z "$SKILL_NAME" ]; then
                SKILL_NAME="$1"
            else
                log_error "Unexpected argument: $1"
                usage
                exit 1
            fi
            shift
            ;;
    esac
done

if [ -z "$SKILL_NAME" ]; then
    log_error "Skill name is required"
    usage
    exit 1
fi

if ! [[ "$SKILL_NAME" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; then
    log_error "Invalid skill name format. Use lowercase letters, numbers, and hyphens only."
    log_error "Examples: 'leakage-detection', 'normality-checks', 'model-card-template'"
    exit 1
fi

if [[ "$SKILLS_DIR" = /* ]]; then
    log_error "Output directory must be a relative path under the current directory."
    exit 1
fi

if [[ "$SKILLS_DIR" =~ (^|/)\.\.(/|$) ]]; then
    log_error "Output directory cannot include '..' path segments."
    exit 1
fi

SKILLS_DIR="${SKILLS_DIR#./}"
SKILLS_DIR="./$SKILLS_DIR"

SKILL_PATH="$SKILLS_DIR/$SKILL_NAME"

if [ -d "$SKILL_PATH" ] && [ "$DRY_RUN" = false ]; then
    log_error "Skill already exists: $SKILL_PATH"
    log_error "Use a different name or remove the existing skill first."
    exit 1
fi

if [ "$DRY_RUN" = true ]; then
    log_info "Dry run - would create:"
    echo "  $SKILL_PATH/"
    echo "  $SKILL_PATH/SKILL.md"
    echo ""
    echo "Template content would be:"
    echo "---"
    cat << TEMPLATE
name: $SKILL_NAME
description: "[TODO: Add a concise description of what this skill does and when to use it]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction explaining the research problem this skill solves]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Data/methodology trigger] | [What to do] |

## Background

[TODO: Why this matters — what methodology mistake or data issue it prevents]

## Solution

### Step-by-Step

1. First step with code or command
2. Second step (include expected metric ranges)
3. Verification step

### Code Example

\`\`\`python
# Include library versions
# import sklearn  # >= 1.3
\`\`\`

## Gotchas

- [TODO: Common mistakes when applying this solution]

## Source Learning

This skill was extracted from a research learning entry.
- Learning ID: [TODO: Add original learning ID]
- Original Category: [TODO: methodology_flaw | data_quality | statistical_error | etc.]
- Original File: .learnings/LEARNINGS.md
- Dataset: [TODO: dataset name if applicable]
TEMPLATE
    echo "---"
    exit 0
fi

log_info "Creating skill: $SKILL_NAME"

mkdir -p "$SKILL_PATH"

cat > "$SKILL_PATH/SKILL.md" << TEMPLATE
---
name: $SKILL_NAME
description: "[TODO: Add a concise description of what this skill does and when to use it]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction explaining the research problem this skill solves]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Data/methodology trigger] | [What to do] |

## Background

[TODO: Why this matters — what methodology mistake or data issue it prevents]

## Solution

### Step-by-Step

1. First step with code or command
2. Second step (include expected metric ranges)
3. Verification step

### Code Example

\`\`\`python
# Include library versions
# import sklearn  # >= 1.3
\`\`\`

## Gotchas

- [TODO: Common mistakes when applying this solution]

## Source Learning

This skill was extracted from a research learning entry.
- Learning ID: [TODO: Add original learning ID]
- Original Category: [TODO: methodology_flaw | data_quality | statistical_error | etc.]
- Original File: .learnings/LEARNINGS.md
- Dataset: [TODO: dataset name if applicable]
TEMPLATE

log_info "Created: $SKILL_PATH/SKILL.md"

echo ""
log_info "Skill scaffold created."
echo ""
echo "Next steps:"
echo "  1. Edit $SKILL_PATH/SKILL.md"
echo "  2. Fill in the TODO sections with content from your learning"
echo "  3. Add references/ folder for methodology documentation"
echo "  4. Add scripts/ folder for data validation or detection code"
echo "  5. Update the original learning entry with:"
echo "     **Status**: promoted_to_skill"
echo "     **Skill-Path**: skills/$SKILL_NAME"


path: scripts/error-detector.sh
content: #!/bin/bash
# Self-Improving Science Error Detector Hook
# Triggers on PostToolUse for Bash to detect ML/data pipeline failures
# Reads CLAUDE_TOOL_OUTPUT environment variable

set -e

OUTPUT="${CLAUDE_TOOL_OUTPUT:-}"

ERROR_PATTERNS=(
    "data leakage"
    "overfitting"
    "underfitting"
    "NaN loss"
    "nan loss"
    "NaN"
    "convergence"
    "not converge"
    "diverge"
    "p-value"
    "p_value"
    "not reproducible"
    "irreproducible"
    "CUDA error"
    "CUDA out of memory"
    "out of memory"
    "OOM"
    "shape mismatch"
    "RuntimeError"
    "ValueError"
    "dimension mismatch"
    "Traceback"
    "ModuleNotFoundError"
    "ImportError"
    "FileNotFoundError"
    "KeyError"
    "IndexError"
    "TypeError"
    "ZeroDivisionError"
    "MemoryError"
    "gradient explosion"
    "vanishing gradient"
    "loss is nan"
    "loss: nan"
    "accuracy: 0.0"
    "f1: 0.0"
    "inf loss"
    "class imbalance"
    "data drift"
    "distribution shift"
    "feature mismatch"
    "column not found"
    "missing values"
    "null values"
    "SettingWithCopyWarning"
    "FutureWarning"
    "DeprecationWarning"
    "failed"
    "FAILED"
    "Error:"
    "error:"
    "ERROR:"
    "fatal:"
    "Permission denied"
    "No such file"
    "command not found"
    "exit code"
    "non-zero"
)

contains_error=false
for pattern in "${ERROR_PATTERNS[@]}"; do
    if [[ "$OUTPUT" == *"$pattern"* ]]; then
        contains_error=true
        break
    fi
done

if [ "$contains_error" = true ]; then
    cat << 'EOF'
<science-error-detected>
A data science / ML error was detected. Consider logging to .learnings/ if:
- Data leakage, distribution shift, or quality issue found
- Model training failed (NaN loss, OOM, convergence)
- Statistical assumption violated or test misapplied
- Results not reproducible across runs
- Shape mismatch, missing data, or pipeline failure

Log experiment issues to EXPERIMENT_ISSUES.md: [EXP-YYYYMMDD-XXX]
Log methodology learnings to LEARNINGS.md: [LRN-YYYYMMDD-XXX]
Include: metrics, seeds, library versions, dataset name.
</science-error-detected>
EOF
fi


path: hooks/openclaw/handler.js
content: /**
 * Self-Improving Science Hook for OpenClaw
 *
 * Injects a reminder to evaluate experiment and methodology learnings
 * during agent bootstrap. Fires on agent:bootstrap event before
 * workspace files are injected.
 */

const REMINDER_CONTENT = `
## Self-Improving Science Reminder

After completing tasks, evaluate if any research learnings should be captured:

**Log when:**
- Data quality issue found → \`.learnings/EXPERIMENT_ISSUES.md\`
- Methodology flaw discovered → \`.learnings/LEARNINGS.md\` with \`methodology_flaw\`
- Statistical error detected → \`.learnings/LEARNINGS.md\` with \`statistical_error\`
- Model fails to reproduce → \`.learnings/EXPERIMENT_ISSUES.md\` with \`reproducibility_issue\`
- Hypothesis needs revision → \`.learnings/LEARNINGS.md\` with \`hypothesis_revision\`
- User wants missing ML tool → \`.learnings/FEATURE_REQUESTS.md\`

**Promote when pattern is proven:**
- Experiment design patterns → Experiment Checklist
- Data handling rules → Data Governance Docs
- Model limitations → Model Card
- Pipeline best practices → \`AGENTS.md\`
- ML framework gotchas → \`TOOLS.md\`

Keep entries specific: include metrics, sample sizes, seeds, and library versions.
`.trim();

const handler = async (event) => {
  if (!event || typeof event !== 'object') {
    return;
  }

  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  if (Array.isArray(event.context.bootstrapFiles)) {
    event.context.bootstrapFiles.push({
      path: 'SELF_IMPROVING_SCIENCE_REMINDER.md',
      content: REMINDER_CONTENT,
      virtual: true,
    });
  }
};

module.exports = handler;
module.exports.default = handler;


path: hooks/openclaw/HOOK.md
content: ---
name: self-improving-science
description: "Injects science-specific self-improvement reminder during agent bootstrap"
metadata: {"openclaw":{"emoji":"🔬","events":["agent:bootstrap"]}}
---

# Self-Improving Science Hook

Injects a reminder to evaluate experiment learnings during agent bootstrap.

## What It Does

- Fires on `agent:bootstrap` (before workspace files are injected)
- Adds a reminder block to check `.learnings/` for relevant experiment entries
- Prompts the agent to log methodology flaws, data quality issues, statistical errors, and reproducibility failures

## Configuration

No configuration needed. Enable with:

```bash
openclaw hooks enable self-improving-science
```


path: hooks/openclaw/handler.ts
content: /**
 * Self-Improving Science Hook for OpenClaw
 *
 * Injects a reminder to evaluate experiment and methodology learnings
 * during agent bootstrap. Fires on agent:bootstrap event before
 * workspace files are injected.
 */

import type { HookHandler } from 'openclaw/hooks';

const REMINDER_CONTENT = `## Self-Improving Science Reminder

After completing tasks, evaluate if any research learnings should be captured:

**Log when:**
- Data quality issue found → \`.learnings/EXPERIMENT_ISSUES.md\`
- Methodology flaw discovered → \`.learnings/LEARNINGS.md\` with \`methodology_flaw\`
- Statistical error detected → \`.learnings/LEARNINGS.md\` with \`statistical_error\`
- Model fails to reproduce → \`.learnings/EXPERIMENT_ISSUES.md\` with \`reproducibility_issue\`
- Hypothesis needs revision → \`.learnings/LEARNINGS.md\` with \`hypothesis_revision\`
- User wants missing ML tool → \`.learnings/FEATURE_REQUESTS.md\`

**Promote when pattern is proven:**
- Experiment design patterns → Experiment Checklist
- Data handling rules → Data Governance Docs
- Model limitations → Model Card
- Pipeline best practices → \`AGENTS.md\`
- ML framework gotchas → \`TOOLS.md\`

Keep entries specific: include metrics, sample sizes, seeds, and library versions.`;

const handler: HookHandler = async (event) => {
  if (!event || typeof event !== 'object') {
    return;
  }

  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  const sessionKey = event.sessionKey || '';
  if (sessionKey.includes(':subagent:')) {
    return;
  }

  if (Array.isArray(event.context.bootstrapFiles)) {
    event.context.bootstrapFiles.push({
      path: 'SELF_IMPROVING_SCIENCE_REMINDER.md',
      content: REMINDER_CONTENT,
      virtual: true,
    });
  }
};

export default handler;


path: references/examples.md
content: # Entry Examples

Concrete examples of well-formatted entries with all fields for scientific research and ML workflows.

## Learning: Data Quality (Target Leakage)

```markdown
## [LRN-20260410-001] data_quality

**Logged**: 2026-04-10T14:30:00Z
**Priority**: critical
**Status**: resolved
**Area**: preprocessing

### Summary
Target leakage through timestamp feature — `event_timestamp` encoded label assignment time

### Details
The `event_timestamp` feature had near-perfect correlation (0.98) with the
target variable `churned`. Investigation revealed the timestamp recorded when
the churn label was applied by the CRM system, not when the user event
occurred. Model achieved 0.99 AUC in validation but 0.52 AUC on live traffic.

### Suggested Action
Audit all timestamp-derived features against the label generation process.
Features must strictly predate the label event. Add a temporal leakage check
to the preprocessing pipeline.

### Metadata
- Source: experiment
- Related Files: notebooks/churn_model_v3.ipynb, src/features/temporal.py
- Tags: leakage, timestamp, churn, target-leakage
- Dataset: customer_events_2025
- Model: XGBoost v1.7.6
- Metric-Before: 0.99 AUC (validation), 0.52 AUC (production)
- Metric-After: 0.81 AUC (validation), 0.79 AUC (production)
- Pattern-Key: leakage.timestamp_target

### Resolution
- **Resolved**: 2026-04-11T09:00:00Z
- **Commit/PR**: #287
- **Notes**: Removed event_timestamp, added temporal_leakage_check() to pipeline

---
```

## Learning: Statistical Error (T-Test on Non-Normal Data)

```markdown
## [LRN-20260408-002] statistical_error

**Logged**: 2026-04-08T11:15:00Z
**Priority**: high
**Status**: promoted
**Promoted**: methodology-standards.md
**Area**: analysis

### Summary
Applied t-test to highly skewed revenue data — normality assumption violated

### Details
Used independent samples t-test to compare revenue between control and
treatment groups in A/B test. Revenue data had skewness of 4.7 and Shapiro-Wilk
p < 0.001. T-test yielded p = 0.03 (significant). After switching to
Mann-Whitney U test, p = 0.12 (not significant). Original conclusion of
"treatment increases revenue" was invalid.

### Suggested Action
Always run Shapiro-Wilk test before parametric tests. For skewed financial
data, default to non-parametric tests or log-transform with justification.
Report both the test choice rationale and assumption check results.

### Metadata
- Source: peer_review
- Related Files: notebooks/ab_test_revenue_q1.ipynb
- Tags: t-test, normality, mann-whitney, skewness, a-b-test
- Dataset: ab_test_revenue_2026q1
- Pattern-Key: stats.normality_assumption

---
```

## Learning: Methodology Flaw (No Holdout for Tuning)

```markdown
## [LRN-20260405-003] methodology_flaw

**Logged**: 2026-04-05T16:45:00Z
**Priority**: high
**Status**: resolved
**Area**: validation

### Summary
Hyperparameter tuning used validation set that was also final evaluation set

### Details
GridSearchCV was run with 5-fold CV on the full dataset minus test set. The
"best" hyperparameters were then evaluated on the same test set used to select
among model architectures. This means the test set influenced both architecture
selection and hyperparameter tuning, inflating reported performance by ~3-5
percentage points. Proper approach: split into train/validation/test, use
validation for tuning, test only once at the end.

### Suggested Action
Implement three-way split: train (60%), validation (20%), test (20%). Use
validation exclusively for hyperparameter search. Touch test set only for
final reported metrics. Document the split strategy in experiment metadata.

### Metadata
- Source: user_feedback
- Related Files: notebooks/model_selection.ipynb, src/training/tune.py
- Tags: hyperparameter-tuning, data-splitting, overfitting, validation
- Model: RandomForest, XGBoost, LightGBM (all affected)
- Metric-Before: 0.93 accuracy (inflated)
- Metric-After: 0.88 accuracy (honest estimate)

### Resolution
- **Resolved**: 2026-04-06T10:00:00Z
- **Commit/PR**: #274
- **Notes**: Refactored pipeline to enforce train/val/test split with StratifiedShuffleSplit

---
```

## Experiment Issue: Reproducibility (Different Results Across Seeds)

```markdown
## [EXP-20260403-001] reproducibility_issue

**Logged**: 2026-04-03T09:30:00Z
**Priority**: high
**Status**: resolved
**Area**: modeling

### Summary
Neural network accuracy varies by 8 percentage points across random seeds

### Error
```
Seed 42:  accuracy=0.87, F1=0.83
Seed 123: accuracy=0.79, F1=0.74
Seed 7:   accuracy=0.85, F1=0.81
Seed 0:   accuracy=0.82, F1=0.78
Seed 999: accuracy=0.84, F1=0.80
```

### Context
- Experiment: Sentiment classification on product reviews
- Model: 2-layer LSTM with attention, PyTorch 2.2.0
- Dataset: 50k reviews, 80/10/10 split
- Hardware: NVIDIA A100, CUDA 12.1
- Batch size: 64, learning rate: 1e-3, epochs: 20

### Root Cause
Small dataset combined with random weight initialization and dropout creates
high variance. GPU non-determinism (cuDNN) also contributes. No seed was set
for `torch.backends.cudnn`.

### Suggested Fix
1. Set all seeds: `torch.manual_seed`, `numpy.random.seed`, `random.seed`,
   `torch.backends.cudnn.deterministic = True`
2. Report mean and std across 5+ seeds instead of single-run results
3. Use `torch.use_deterministic_algorithms(True)` for full reproducibility
4. Consider larger dataset or pre-trained embeddings to reduce variance

### Metadata
- Reproducible: yes (variance is the issue)
- Related Files: notebooks/sentiment_lstm.ipynb, src/models/lstm.py
- Seeds Tested: 42, 123, 7, 0, 999

### Resolution
- **Resolved**: 2026-04-04T14:00:00Z
- **Commit/PR**: #268
- **Notes**: Added seed_everything() utility, now report mean±std across 5 seeds

---
```

## Experiment Issue: Model Drift (Accuracy Degradation After Data Update)

```markdown
## [EXP-20260401-002] data_quality

**Logged**: 2026-04-01T13:00:00Z
**Priority**: critical
**Status**: resolved
**Area**: validation

### Summary
Fraud detection model accuracy dropped from 0.94 to 0.71 after monthly data refresh

### Error
```
March model (trained on Jan-Feb data): precision=0.94, recall=0.89, F1=0.91
April evaluation (on March data):      precision=0.71, recall=0.63, F1=0.67
```

### Context
- Model: Gradient Boosted Trees (LightGBM 4.3.0)
- Training data: Jan-Feb 2026 transactions (1.2M rows)
- Evaluation data: March 2026 transactions (450k rows)
- Features: 47 transaction features + 12 engineered features
- Production serving via batch scoring pipeline

### Root Cause
Distribution shift in two key features:
1. `transaction_amount` mean shifted from $127 to $89 (seasonal pattern)
2. `merchant_category` had 3 new categories not seen during training

Feature drift detection (PSI) was not in place. The model had no mechanism
to flag out-of-distribution inputs.

### Suggested Fix
1. Add Population Stability Index (PSI) monitoring for all features
2. Set PSI > 0.2 as alert threshold, PSI > 0.5 as retrain trigger
3. Handle unseen categorical values with an "unknown" bucket
4. Implement monthly retraining with sliding window

### Metadata
- Reproducible: yes
- Related Files: src/models/fraud_detector.py, pipelines/monthly_retrain.py
- See Also: EXP-20260115-004 (similar drift in Q4 2025)

### Resolution
- **Resolved**: 2026-04-02T16:00:00Z
- **Commit/PR**: #281
- **Notes**: Added PSI monitoring, unknown category handling, and sliding window retraining

---
```

## Feature Request: Automated Data Leakage Detection

```markdown
## [FEAT-20260409-001] automated_leakage_detection

**Logged**: 2026-04-09T10:00:00Z
**Priority**: high
**Status**: pending
**Area**: preprocessing

### Requested Capability
Automated detection of data leakage in feature engineering pipelines

### Research Context
After two incidents of target leakage (LRN-20260410-001 and EXP-20260215-003),
user wants a reusable tool that scans feature-target relationships for
leakage signals before model training begins.

### Complexity Estimate
medium

### Suggested Implementation
1. Compute feature-target mutual information; flag features with MI > threshold
2. Check temporal ordering: all feature timestamps must predate label timestamp
3. Detect features derived from target (e.g., aggregations that include label)
4. Run on every pipeline execution as a pre-training gate
5. Libraries: `sklearn.feature_selection.mutual_info_classif`, custom temporal checks

### Metadata
- Frequency: recurring (2 incidents in 60 days)
- Related Features: preprocessing pipeline, feature store validation

---
```

## Learning: Promoted to Experiment Checklist

```markdown
## [LRN-20260320-004] experiment_design

**Logged**: 2026-03-20T08:30:00Z
**Priority**: high
**Status**: promoted
**Promoted**: experiment-checklist.md
**Area**: validation

### Summary
Cross-validation must nest preprocessing inside folds to prevent data leakage

### Details
StandardScaler was fit on the full training set before cross-validation.
This means fold validation sets were scaled using statistics computed on
data that included those same validation samples. Mean AUC dropped from
0.89 to 0.86 after fixing with Pipeline + CV nesting. The 3-point
difference is the "optimism" from leaking scaling parameters.

### Suggested Action
Always use sklearn Pipeline to encapsulate preprocessing + model. Pass
the Pipeline (not just the model) to cross_val_score or GridSearchCV.

### Metadata
- Source: experiment
- Related Files: notebooks/cv_pipeline_fix.ipynb
- Tags: cross-validation, preprocessing, leakage, pipeline
- Pattern-Key: leakage.cv_preprocessing
- Recurrence-Count: 3
- First-Seen: 2026-01-15
- Last-Seen: 2026-03-20

---
```

**In experiment-checklist.md:**
```markdown
## Pre-Training Checks
- [ ] Preprocessing is inside CV folds (use sklearn Pipeline)
- [ ] No feature statistics computed on full dataset before splitting
- [ ] StandardScaler/MinMaxScaler fit only on training fold
```

## Learning: Promoted to Skill (Model Card Template)

```markdown
## [LRN-20260315-005] experiment_design

**Logged**: 2026-03-15T14:00:00Z
**Priority**: medium
**Status**: promoted_to_skill
**Skill-Path**: skills/model-card-template
**Area**: publication

### Summary
Standardized model card format needed for documenting model limitations and performance bounds

### Details
Team had no consistent format for documenting model behavior, failure modes,
and ethical considerations. After three incidents where model limitations were
not communicated to downstream consumers, created a reusable model card
template based on Mitchell et al. (2019) framework.

### Suggested Action
Use the extracted skill template for every model shipped to production or
shared with stakeholders.

### Metadata
- Source: user_feedback
- Related Files: docs/model-card-template.md
- Tags: model-card, documentation, model-governance, responsible-ai

---
```


path: references/openclaw-integration.md
content: # OpenClaw Integration (Science)

Complete setup and usage guide for integrating the self-improving science skill with OpenClaw.

## Overview

OpenClaw uses workspace-based prompt injection combined with event-driven hooks. Context is injected from workspace files at session start, and hooks can trigger on lifecycle events.

## Workspace Structure

```
~/.openclaw/
├── workspace/                   # Working directory
│   ├── AGENTS.md               # Experiment orchestration, pipeline patterns
│   ├── SOUL.md                 # Research rigor principles, communication style
│   ├── TOOLS.md                # ML framework gotchas, data tool capabilities
│   ├── MEMORY.md               # Long-term memory (main session only)
│   └── memory/                 # Daily memory files
│       └── YYYY-MM-DD.md
├── skills/                      # Installed skills
│   └── self-improving-science/
│       └── SKILL.md
└── hooks/                       # Custom hooks
    └── self-improving-science/
        ├── HOOK.md
        └── handler.ts
```

## Quick Setup

### 1. Install the Skill

```bash
clawdhub install self-improving-science
```

Or copy manually:

```bash
cp -r self-improving-science ~/.openclaw/skills/
```

### 2. Install the Hook (Optional)

Copy the hook to OpenClaw's hooks directory:

```bash
cp -r hooks/openclaw ~/.openclaw/hooks/self-improving-science
```

Enable the hook:

```bash
openclaw hooks enable self-improving-science
```

### 3. Create Learning Files

Create the `.learnings/` directory in your workspace:

```bash
mkdir -p ~/.openclaw/workspace/.learnings
```

## Injected Prompt Files

### AGENTS.md (Science-Oriented)

Purpose: Experiment orchestration and ML pipeline patterns.

```markdown
# Experiment Orchestration

## Pipeline Rules
- Run data validation before any training step
- Log all hyperparameters and random seeds
- Use experiment tracking (MLflow, W&B) for all runs

## Delegation Rules
- Use explore agent for dataset investigation
- Spawn sub-agents for hyperparameter sweeps
- Use sessions_send for sharing experiment results across sessions
```

### SOUL.md (Science-Oriented)

Purpose: Research rigor and communication guidelines.

```markdown
# Research Principles

## Scientific Rigor
- Report confidence intervals, not just point estimates
- Distinguish correlation from causation
- Acknowledge limitations explicitly

## Communication Style
- Use precise statistical language
- Cite methodology choices with references
- Flag uncertainty honestly
```

### TOOLS.md (Science-Oriented)

Purpose: ML framework gotchas and data tool capabilities.

```markdown
# Tool Knowledge

## PyTorch
- DataLoader workers leak memory on macOS — use num_workers=0 for debugging
- torch.use_deterministic_algorithms(True) for reproducibility

## scikit-learn
- Pipeline must wrap preprocessing for proper CV
- StratifiedKFold for imbalanced classes

## pandas
- .copy() after slicing to avoid SettingWithCopyWarning
- Use .astype("category") for memory efficiency on categoricals
```

## Learning Workflow

### Capturing Learnings

1. **In-session**: Log to `.learnings/` as usual
2. **Cross-session**: Promote to workspace files or research artifacts

### Promotion Decision Tree

```
Is the learning project-specific?
├── Yes → Keep in .learnings/
└── No → Is it a methodology/statistics pattern?
    ├── Yes → Promote to Methodology Standards
    └── No → Is it about data handling?
        ├── Yes → Promote to Data Governance Docs
        └── No → Is it about model behavior?
            ├── Yes → Promote to Model Card
            └── No → Is it an ML tool gotcha?
                ├── Yes → Promote to TOOLS.md
                └── No → Promote to AGENTS.md (pipeline pattern)
```

### Promotion Format Examples

**From learning:**
> PyTorch DataLoader with num_workers > 0 causes memory leak on macOS during long training runs

**To TOOLS.md:**
```markdown
## PyTorch
- DataLoader num_workers > 0 leaks memory on macOS — set to 0 for debugging, or use persistent_workers=True
```

**From learning:**
> Model silently predicts majority class when class ratio exceeds 20:1

**To Model Card:**
```markdown
## Known Limitations
- Performance degrades when minority class ratio < 5% — use class weights or SMOTE
```

## Inter-Agent Communication

OpenClaw provides tools for cross-session communication. Use only when cross-session sharing is explicitly needed.

### sessions_list

View active and recent sessions:
```
sessions_list(activeMinutes=30, messageLimit=3)
```

### sessions_history

Read transcript from another session:
```
sessions_history(sessionKey="session-id", limit=50)
```

### sessions_send

Send experiment result to another session:
```
sessions_send(sessionKey="session-id", message="Learning: Model requires temporal split, random split leaks")
```

Prefer sending concise findings with metric summaries rather than raw data or full notebooks.

### sessions_spawn

Spawn a background sub-agent:
```
sessions_spawn(task="Run hyperparameter sweep on config_v2.yaml", label="hp-sweep")
```

## Available Hook Events

| Event | When It Fires |
|-------|---------------|
| `agent:bootstrap` | Before workspace files inject |
| `command:new` | When `/new` command issued |
| `command:reset` | When `/reset` command issued |
| `command:stop` | When `/stop` command issued |
| `gateway:startup` | When gateway starts |

## Science-Specific Detection Triggers

| Trigger | Action |
|---------|--------|
| Data quality issue found | Log to `.learnings/EXPERIMENT_ISSUES.md` with `data_quality` |
| Methodology flaw | Log to `.learnings/LEARNINGS.md` with `methodology_flaw` |
| Statistical test misapplied | Log to `.learnings/LEARNINGS.md` with `statistical_error` |
| Model reproducibility failure | Log to `.learnings/EXPERIMENT_ISSUES.md` with `reproducibility_issue` |
| Hypothesis revision needed | Log to `.learnings/LEARNINGS.md` with `hypothesis_revision` |
| ML framework issue | Log to `TOOLS.md` with framework name |
| Pipeline pattern discovered | Log to `AGENTS.md` with pipeline step |

## Verification

Check hook is registered:

```bash
openclaw hooks list
```

Check skill is loaded:

```bash
openclaw status
```

## Troubleshooting

### Hook not firing

1. Ensure hooks enabled in config
2. Restart gateway after config changes
3. Check gateway logs for errors

### Learnings not persisting

1. Verify `.learnings/` directory exists
2. Check file permissions
3. Ensure workspace path is configured correctly

### Skill not loading

1. Check skill is in skills directory
2. Verify SKILL.md has correct frontmatter
3. Run `openclaw status` to see loaded skills


path: references/hooks-setup.md
content: # Hook Setup Guide (Science)

Configure automatic self-improvement triggers for scientific research and ML workflows.

## Overview

Hooks enable proactive learning capture by injecting reminders at key moments:
- **UserPromptSubmit**: Reminder after each prompt to evaluate experiment learnings
- **PostToolUse (Bash)**: ML/data error detection when commands fail

## Claude Code Setup

### Option 1: Project-Level Configuration

Create `.claude/settings.json` in your project root:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-science/scripts/activator.sh"
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-science/scripts/error-detector.sh"
          }
        ]
      }
    ]
  }
}
```

### Option 2: User-Level Configuration

Add to `~/.claude/settings.json` for global activation:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "~/.claude/skills/self-improving-science/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

### Minimal Setup (Activator Only)

For lower overhead, use only the UserPromptSubmit hook:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-science/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## Codex CLI Setup

Codex uses the same hook system. Create `.codex/settings.json`:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improving-science/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## GitHub Copilot Setup

Copilot doesn't support hooks directly. Add guidance to `.github/copilot-instructions.md`:

```markdown
## Self-Improvement (Science)

After completing tasks that involved:
- Debugging data quality issues or leakage
- Fixing statistical test misapplication
- Resolving model training failures
- Discovering reproducibility problems
- Revising hypotheses based on results

Consider logging the learning to `.learnings/` using the self-improving-science format.

For high-value learnings, promote to experiment checklists, model cards, or methodology standards.
```

## Verification

### Test Activator Hook

1. Enable the hook configuration
2. Start a new Claude Code session
3. Send any prompt
4. Verify you see `<science-improvement-reminder>` in the context

### Test Error Detector Hook

1. Enable PostToolUse hook for Bash
2. Run a command that produces ML errors: `python -c "import torch; torch.tensor([1.0]) / torch.tensor([0.0])"`
3. Verify you see `<science-error-detected>` reminder

### Dry Run Extract Script

```bash
./skills/self-improving-science/scripts/extract-skill.sh test-skill --dry-run
```

## Troubleshooting

### Hook Not Triggering

1. **Check script permissions**: `chmod +x scripts/*.sh`
2. **Verify path**: Use absolute paths or paths relative to project root
3. **Check settings location**: Project vs user-level settings
4. **Restart session**: Hooks are loaded at session start

### Permission Denied

```bash
chmod +x ./skills/self-improving-science/scripts/activator.sh
chmod +x ./skills/self-improving-science/scripts/error-detector.sh
chmod +x ./skills/self-improving-science/scripts/extract-skill.sh
```

### Script Not Found

If using relative paths, ensure you're in the correct directory or use absolute paths:

```json
{
  "command": "/absolute/path/to/skills/self-improving-science/scripts/activator.sh"
}
```

### Too Much Overhead

If the activator feels intrusive:

1. **Use minimal setup**: Only UserPromptSubmit, skip PostToolUse
2. **Add matcher filter**: Only trigger for experiment-related prompts:

```json
{
  "matcher": "train|model|experiment|data|analysis|test",
  "hooks": [...]
}
```

## Hook Output Budget

The activator is designed to be lightweight:
- **Target**: ~60-120 tokens per activation
- **Content**: Structured reminder with science-specific triggers
- **Format**: XML tags for easy parsing

## Security Considerations

- Hook scripts run with the same permissions as Claude Code
- Scripts only output text; they don't modify files or run commands
- Error detector reads `CLAUDE_TOOL_OUTPUT` environment variable
- Treat `CLAUDE_TOOL_OUTPUT` as potentially sensitive; do not log or forward it verbatim
- All scripts are opt-in (you must configure them explicitly)
- Recommended default: enable `UserPromptSubmit` only

## Disabling Hooks

To temporarily disable without removing configuration:

1. **Comment out in settings**:
```json
{
  "hooks": {
    // "UserPromptSubmit": [...]
  }
}
```

2. **Or delete the settings file**: Hooks won't run without configuration


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)