Skill: brianhearn/expertpack-export (version 1.1.0)

--- name: expertpack-export description: Export an OpenClaw instance's accumulated knowledge into a structured ExpertPack composite. Use when backing up an agent's identity, exporting for migration, or creating a portable knowledge snapshot. Handles auto-discovery (scanning workspace state to identify constituent packs), distillation (compressing raw state into structured EP files), and packaging (writing EP-compliant packs + composite manifest). Output is Obsidian-compatible — includes YAML frontmatter on all content files and can be opened as an Obsidian vault. NOT for importing/hydrating from an existing EP. metadata: openclaw: homepage: https://expertpack.ai requires: bins: - python3 --- # ExpertPack Export Part of the [ExpertPack](https://expertpack.ai) framework — a structured, portable knowledge format for AI agents. Export an OpenClaw instance into a composite ExpertPack — an agent pack (subtype: agent) as the voice, plus person/product/process packs as knowledge constituents. **Learn more:** [expertpack.ai](https://expertpack.ai) · [GitHub](https://github.com/brianhearn/ExpertPack) · [Schema docs](https://expertpack.ai/#schemas) ## Prerequisites - Read `references/schemas-summary.md` for the EP schema rules this export must follow. - The export writes to a target directory (default: `{workspace}/export/`). It does NOT modify the agent's live workspace files. ## Export Flow ### 1. Scan Run `scripts/scan.py` to inventory the workspace. It outputs a JSON manifest of discovered files, their categories, and proposed pack assignments. ```bash python3 {skill_dir}/scripts/scan.py --workspace /root/.openclaw/workspace --output /tmp/ep-scan.json ``` Review the scan output. It proposes: - Which files map to which pack type (agent, person, product, process) - Which knowledge domains were detected - Confidence scores for ambiguous classifications ### 2. Propose Present the proposed composite to the user: - List each proposed pack with type, slug, and key content sources - Flag ambiguous classifications for user decision - Note any gaps (e.g., "No process packs detected — skip or create stubs?") Wait for user confirmation before proceeding. ### 3. Distill Run `scripts/distill.py` for each proposed pack. It reads source files, extracts knowledge, deduplicates, and writes EP-compliant output. ```bash python3 {skill_dir}/scripts/distill.py \ --scan /tmp/ep-scan.json \ --pack agent:easybot \ --output /root/.openclaw/workspace/export/packs/easybot/ ``` Repeat for each pack. The script: - Reads source files listed in the scan manifest - Extracts and classifies knowledge assertions - Deduplicates (prefers newest for conflicts) - Writes structured .md files with proper headers and frontmatter - Writes manifest.yaml per pack - Strips secrets (API keys, tokens, passwords) automatically ### 4. Compose Run `scripts/compose.py` to generate the composite manifest and overview. ```bash python3 {skill_dir}/scripts/compose.py \ --scan /tmp/ep-scan.json \ --export-dir /root/.openclaw/workspace/export/ ``` ### 5. Validate Run `scripts/validate.py` to check the export against schema rules. ```bash python3 {skill_dir}/scripts/validate.py --export-dir /root/.openclaw/workspace/export/ ``` It checks: - All required files exist per schema - manifest.yaml fields are valid - No secrets leaked (scans for API key patterns) - File sizes within guidelines - Cross-references resolve ### 6. Review & Ship Present the validation report and a summary of what was exported. The user decides whether to commit/push or adjust. ## Important Rules - **Never include secrets.** The scan and distill scripts strip known patterns, but always review `operational/tools.md` and `operational/infrastructure.md` manually. - **Distill, don't copy.** Raw journal entries and session states should be compressed into structured knowledge. The export should be 10-20% the volume of raw state. - **Respect privacy.** Flag personal information about the user for access tier review. Default user-specific content to `private` access. - **Preserve provenance.** Each distilled file should note its source files in frontmatter. - **Don't modify the live workspace.** All output goes to the export directory.

path: references/schemas-summary.md
content: # ExpertPack Schema Summary for Export

Condensed reference for the export scripts. Full schemas at `schemas/` in the ExpertPack repo.

## Pack Types

| Type | Subtype | Purpose |
|------|---------|---------|
| person | (none) | Human knowledge — stories, mind, relationships, presentation |
| person | agent | AI agent identity — operational config, prescriptive mind, tools, safety |
| product | — | Product knowledge — concepts, workflows, interfaces, troubleshooting |
| process | — | Process knowledge — phases, decisions, checklists, gotchas |
| composite | — | Orchestration layer — wires packs together with roles and conflict rules |

## Agent Pack Structure (person, subtype: agent)

```
packs/{agent-slug}/
├── manifest.yaml          # type: person, subtype: agent
├── overview.md            # Identity, personality, vibe
├── MIGRATION.md           # How to hydrate a new instance
├── operational/
│   ├── tools.md           # Tool inventory (shape, not secrets)
│   ├── infrastructure.md  # Hosts, services, topology
│   ├── integrations.md    # Messaging channels, APIs, accounts
│   ├── routines.md        # Heartbeats, backups, cron schedules
│   └── safety.md          # Behavioral contracts, guardrails
├── mind/
│   ├── values.md          # Prescriptive operational principles
│   ├── skills.md          # Capabilities list
│   ├── relational.md      # Interaction rules
│   ├── preferences.md     # Learned formatting/behavior preferences
│   ├── reasoning.md       # Decision patterns
│   └── tensions.md        # Known limitations, failure modes
├── relationships/
│   └── people.md          # Primary user, contacts, peer agents
├── facts/
│   ├── personal.md        # Agent identity facts
│   └── timeline.md        # Significant events
├── presentation/
│   ├── speech_patterns.md # Communication style
│   └── modes.md           # Context-dependent voices
├── summaries/
│   └── lessons.md         # Patterns learned, failure post-mortems
├── meta/
│   └── privacy.md         # Export access tier rules
└── verbatim/              # Optional: significant decisions/conversations
```

## Person Pack Structure (standard)

```
packs/{person-slug}/
├── manifest.yaml          # type: person
├── overview.md
├── facts/                 # personal.md, career.md, education.md, timeline.md
├── relationships/people.md
├── mind/                  # 9 core files + optional reasoning, influences
├── preferences/           # (captured in mind/preferences.md)
├── presentation/          # speech_patterns.md, modes.md
└── meta/privacy.md
```

## Product Pack Structure

```
packs/{product-slug}/
├── manifest.yaml          # type: product
├── overview.md
├── concepts/              # Core concepts, mental model
├── workflows/             # Step-by-step procedures
├── interfaces/            # UI/API documentation
├── commercial/            # Pricing, deployment, security
├── troubleshooting/       # Errors, diagnostics, common mistakes
├── faq/
└── glossary.md
```

## Process Pack Structure

```
packs/{process-slug}/
├── manifest.yaml          # type: process
├── overview.md
├── phases/                # Sequential steps
├── decisions/             # Decision points, criteria
├── checklists/            # Verification steps
└── gotchas/               # Common mistakes, edge cases
```

## Composite Manifest

```yaml
name: "Composite Name"
slug: "composite-slug"
type: "composite"
version: "1.0.0"
schema_version: "1.0"
entry_point: "overview.md"

packs:
  - path: "../packs/agent-slug"
    role: voice              # At most one voice pack
  - path: "../packs/person-slug"
    role: knowledge          # Multiple knowledge packs OK
  - path: "../packs/product-slug"
    role: knowledge

conflicts:
  priority: [agent-slug, person-slug, product-slug]
  strategy: "flag"
```

## Key Rules

1. **Markdown canonical** — all knowledge in .md files
2. **1-3KB per file** — one topic per file
3. **## headers** at natural breaks for RAG chunking
4. **kebab-case** filenames and slugs
5. **No secrets** — never include API keys, tokens, passwords
6. **manifest.yaml required** for every pack
7. **overview.md required** for every pack
8. **Distill, don't copy** — compress raw state into structured knowledge


path: scripts/compose.py
content: #!/usr/bin/env python3
"""
ExpertPack Export — Composite Generator

Reads the scan manifest and generates a composite manifest + overview
that wires all exported packs together.

Usage:
    python3 compose.py --scan /tmp/ep-scan.json --export-dir /path/to/export/
"""

import argparse
import json
import sys
from pathlib import Path
from datetime import datetime


def compose(scan_path: str, export_dir: str):
    """Generate composite manifest and overview."""
    with open(scan_path) as f:
        scan = json.load(f)

    export = Path(export_dir)
    packs = scan.get("packs", {})

    if not packs:
        print("Error: No packs found in scan manifest", file=sys.stderr)
        sys.exit(1)

    # Determine composite slug from agent pack or workspace
    agent_packs = [s for s, p in packs.items() if p.get("subtype") == "agent"]
    agent_slug = agent_packs[0] if agent_packs else "agent"
    composite_slug = f"{agent_slug}-full"

    composite_dir = export / "composites" / composite_slug
    composite_dir.mkdir(parents=True, exist_ok=True)

    # Build pack entries
    pack_entries = []
    priority_order = []

    # Agent pack first (voice role)
    for slug, pack in packs.items():
        if pack.get("subtype") == "agent":
            pack_entries.append({
                "path": f"../../packs/{slug}",
                "role": "voice",
            })
            priority_order.append(slug)

    # Then person packs, then product, then process (knowledge role)
    type_order = ["person", "product", "process"]
    for pack_type in type_order:
        for slug, pack in packs.items():
            if pack["type"] == pack_type and pack.get("subtype") != "agent":
                pack_entries.append({
                    "path": f"../../packs/{slug}",
                    "role": "knowledge",
                })
                priority_order.append(slug)

    # Write composite manifest
    manifest_lines = [
        f'name: "{agent_slug.replace("-", " ").title()} — Full Instance Export"',
        f'slug: "{composite_slug}"',
        'type: "composite"',
        'version: "1.0.0"',
        'schema_version: "1.1"',
        f'description: "Complete knowledge export of the {agent_slug.replace("-", " ").title()} instance"',
        'entry_point: "overview.md"',
        '',
        'packs:',
    ]

    for entry in pack_entries:
        manifest_lines.append(f'  - path: "{entry["path"]}"')
        manifest_lines.append(f'    role: "{entry["role"]}"')

    manifest_lines.extend([
        '',
        'conflicts:',
        f'  priority: [{", ".join(priority_order)}]',
        '  strategy: "flag"',
    ])

    (composite_dir / "manifest.yaml").write_text("\n".join(manifest_lines) + "\n")

    # Write composite overview
    overview_lines = [
        f"# {agent_slug.replace('-', ' ').title()} — Full Export",
        "",
        f"Composite ExpertPack exported from an OpenClaw instance on {datetime.utcnow().strftime('%Y-%m-%d')}.",
        "",
        "## Constituent Packs",
        "",
    ]

    for entry in pack_entries:
        slug = entry["path"].split("/")[-1]
        pack_info = packs.get(slug, {})
        pack_type = pack_info.get("type", "unknown")
        subtype = f" ({pack_info['subtype']})" if pack_info.get("subtype") else ""
        overview_lines.append(f"- **{slug}** — {pack_type}{subtype}, role: {entry['role']}")
        if pack_info.get("description"):
            overview_lines.append(f"  - {pack_info['description']}")

    overview_lines.extend([
        "",
        "## Import",
        "",
        "See each pack's MIGRATION.md (agent pack) or overview.md for hydration instructions.",
        "Configure credentials separately — no secrets are included in this export.",
    ])

    (composite_dir / "overview.md").write_text("\n".join(overview_lines) + "\n")

    print(f"Composite '{composite_slug}' generated at {composite_dir}")
    print(f"  {len(pack_entries)} packs wired:")
    for e in pack_entries:
        print(f"    [{e['role']}] {e['path']}")


def main():
    parser = argparse.ArgumentParser(description="Generate EP composite from scan")
    parser.add_argument("--scan", required=True, help="Path to scan manifest JSON")
    parser.add_argument("--export-dir", required=True, help="Export root directory")
    args = parser.parse_args()

    compose(args.scan, args.export_dir)


if __name__ == "__main__":
    main()


path: scripts/distill.py
content: #!/usr/bin/env python3
"""
ExpertPack Export — Knowledge Distiller

Reads source files from the scan manifest and produces structured EP-compliant
output files for a single pack.

This script handles the STRUCTURAL work: creating directories, writing manifests,
and scaffolding files. The actual CONTENT distillation (compressing journals into
lessons, extracting operational knowledge from memory files) is done by the AI agent
that calls this script — the agent reads source files, distills the knowledge, and
writes the content to the scaffolded files.

Usage:
    python3 distill.py --scan /tmp/ep-scan.json --pack agent:slug --output /path/to/pack/
    python3 distill.py --scan /tmp/ep-scan.json --pack person:slug --output /path/to/pack/
    python3 distill.py --scan /tmp/ep-scan.json --pack product:slug --output /path/to/pack/
    python3 distill.py --scan /tmp/ep-scan.json --pack process:slug --output /path/to/pack/
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path
from datetime import datetime


# Secret patterns to strip from content
SECRET_PATTERNS = [
    (re.compile(r'((?:api[_-]?key|token|secret|password|bearer)\s*[:=]\s*)\S+', re.IGNORECASE),
     r'\1<REDACTED>'),
    (re.compile(r'(sk-)[a-zA-Z0-9]{20,}'), r'\1<REDACTED>'),
    (re.compile(r'(ghp_)[a-zA-Z0-9]{36,}'), r'\1<REDACTED>'),
    (re.compile(r'(xoxb-)[a-zA-Z0-9\-]+'), r'\1<REDACTED>'),
]


def strip_secrets(text: str) -> str:
    """Replace known secret patterns with <REDACTED>."""
    for pattern, replacement in SECRET_PATTERNS:
        text = pattern.sub(replacement, text)
    return text


def scaffold_agent_pack(output_dir: Path, slug: str, scan: dict):
    """Create directory structure and manifest for an agent pack."""
    dirs = [
        "operational",
        "mind",
        "relationships",
        "facts",
        "presentation",
        "summaries",
        "meta",
    ]
    for d in dirs:
        (output_dir / d).mkdir(parents=True, exist_ok=True)

    # Write manifest
    manifest = {
        "name": slug.replace("-", " ").title(),
        "slug": slug,
        "type": "person",
        "subtype": "agent",
        "version": "1.0.0",
        "schema_version": "1.6",
        "description": f"Agent identity and operational knowledge for {slug}",
        "entry_point": "overview.md",
        "subject": {
            "name": slug.replace("-", " ").title(),
            "platform": "OpenClaw",
            "created": datetime.utcnow().strftime("%Y-%m-%d"),
        },
        "context": {
            "always": [
                "overview.md",
                "presentation/speech_patterns.md",
                "operational/safety.md",
                "mind/values.md",
            ],
            "searchable": [
                "mind/",
                "relationships/",
                "facts/",
                "operational/",
                "summaries/",
                "presentation/modes.md",
            ],
            "on_demand": [
                "meta/",
            ],
        },
        "sources": [
            {
                "type": "platform-export",
                "description": "Auto-generated from OpenClaw instance state",
                "date": datetime.utcnow().strftime("%Y-%m-%d"),
            }
        ],
    }

    _write_yaml(output_dir / "manifest.yaml", manifest)

    # Write stub files that the agent will fill with distilled content
    stubs = {
        "overview.md": "# {name}\n\n> Agent identity and personality. To be distilled from SOUL.md + IDENTITY.md.\n",
        "MIGRATION.md": "# Migration Guide\n\n> Hydration instructions. To be filled after distillation.\n",
        "operational/tools.md": "# Tools\n\n> Distilled from TOOLS.md. No secrets.\n",
        "operational/infrastructure.md": "# Infrastructure\n\n> Distilled from TOOLS.md + memory files.\n",
        "operational/integrations.md": "# Integrations\n\n> Messaging channels, APIs, external services.\n",
        "operational/routines.md": "# Routines\n\n> Distilled from HEARTBEAT.md + cron jobs.\n",
        "operational/safety.md": "# Safety Contracts\n\n> Distilled from AGENTS.md safety rules.\n",
        "mind/values.md": "# Values & Operational Principles\n\n> Prescriptive rules from AGENTS.md.\n",
        "mind/skills.md": "# Capabilities\n\n> What this agent can do.\n",
        "mind/relational.md": "# Interaction Rules\n\n> How this agent communicates in different contexts.\n",
        "mind/preferences.md": "# Learned Preferences\n\n> Formatting, behavior, platform-specific rules.\n",
        "mind/reasoning.md": "# Reasoning Patterns\n\n> How this agent approaches decisions.\n",
        "mind/tensions.md": "# Known Limitations\n\n> Failure modes, weaknesses, things to improve.\n",
        "relationships/people.md": "# Relationships\n\n> Primary user, contacts, peer agents.\n",
        "facts/personal.md": "# Agent Identity\n\n> Name, creation date, platform, avatar.\n",
        "facts/timeline.md": "# Timeline\n\n> Significant events in this agent's history.\n",
        "presentation/speech_patterns.md": "# Communication Style\n\n> Tone, humor, formality, emoji usage.\n",
        "presentation/modes.md": "# Modes\n\n> Context-dependent voices.\n",
        "summaries/lessons.md": "# Lessons Learned\n\n> Patterns, post-mortems, accumulated wisdom.\n",
        "meta/privacy.md": "# Privacy\n\n## Public\n- Identity, capabilities, communication style\n\n## Private\n- User details, infrastructure specifics, relationship details\n",
    }

    for relpath, template in stubs.items():
        fpath = output_dir / relpath
        content = template.replace("{name}", slug.replace("-", " ").title())
        fpath.write_text(content)

    return manifest


def scaffold_person_pack(output_dir: Path, slug: str, scan: dict):
    """Create directory structure and manifest for a person pack."""
    dirs = ["facts", "relationships", "mind", "presentation", "meta"]
    for d in dirs:
        (output_dir / d).mkdir(parents=True, exist_ok=True)

    manifest = {
        "name": slug.replace("-", " ").title(),
        "slug": slug,
        "type": "person",
        "version": "1.0.0",
        "schema_version": "1.5",
        "description": f"Knowledge about {slug.replace('-', ' ').title()}",
        "entry_point": "overview.md",
        "subject": {
            "full_name": slug.replace("-", " ").title(),
            "alive": True,
        },
        "context": {
            "always": ["overview.md", "facts/personal.md"],
            "searchable": ["facts/", "relationships/", "mind/"],
            "on_demand": ["meta/"],
        },
        "sources": [
            {
                "type": "platform-export",
                "description": "Distilled from OpenClaw workspace knowledge",
                "date": datetime.utcnow().strftime("%Y-%m-%d"),
            }
        ],
    }

    _write_yaml(output_dir / "manifest.yaml", manifest)

    stubs = {
        "overview.md": "# {name}\n\n> Person overview. To be distilled from USER.md + MEMORY.md.\n",
        "facts/personal.md": "# Personal Facts\n\n> Bio, timezone, role, background.\n",
        "relationships/people.md": "# Relationships\n\n> Key people in this person's life/work.\n",
        "mind/preferences.md": "# Preferences\n\n> Communication style, work patterns, interests.\n",
        "meta/privacy.md": "# Privacy\n\n## Public\n- Name, role, professional info\n\n## Private\n- Personal details, family, beliefs\n",
    }

    for relpath, template in stubs.items():
        fpath = output_dir / relpath
        content = template.replace("{name}", slug.replace("-", " ").title())
        fpath.write_text(content)

    return manifest


def scaffold_product_pack(output_dir: Path, slug: str, scan: dict):
    """Create directory structure and manifest for a product pack."""
    dirs = ["concepts", "workflows", "commercial", "troubleshooting", "faq"]
    for d in dirs:
        (output_dir / d).mkdir(parents=True, exist_ok=True)

    manifest = {
        "name": slug.replace("-", " ").title(),
        "slug": slug,
        "type": "product",
        "version": "1.0.0",
        "schema_version": "1.4",
        "description": f"Product knowledge for {slug.replace('-', ' ').title()}",
        "entry_point": "overview.md",
        "context": {
            "always": ["overview.md"],
            "searchable": ["concepts/", "workflows/", "commercial/", "faq/"],
            "on_demand": ["troubleshooting/"],
        },
        "sources": [
            {
                "type": "platform-export",
                "description": "Distilled from OpenClaw workspace knowledge",
                "date": datetime.utcnow().strftime("%Y-%m-%d"),
            }
        ],
    }

    _write_yaml(output_dir / "manifest.yaml", manifest)

    stubs = {
        "overview.md": f"# {slug.replace('-', ' ').title()}\n\n> Product overview. To be distilled.\n",
    }

    for relpath, content in stubs.items():
        (output_dir / relpath).write_text(content)

    return manifest


def scaffold_process_pack(output_dir: Path, slug: str, scan: dict):
    """Create directory structure and manifest for a process pack."""
    dirs = ["phases", "decisions", "checklists", "gotchas"]
    for d in dirs:
        (output_dir / d).mkdir(parents=True, exist_ok=True)

    manifest = {
        "name": slug.replace("-", " ").title(),
        "slug": slug,
        "type": "process",
        "version": "1.0.0",
        "schema_version": "1.2",
        "description": f"Process documentation for {slug.replace('-', ' ').title()}",
        "entry_point": "overview.md",
        "context": {
            "always": ["overview.md"],
            "searchable": ["phases/", "decisions/", "checklists/"],
            "on_demand": ["gotchas/"],
        },
        "sources": [
            {
                "type": "platform-export",
                "description": "Distilled from OpenClaw workspace knowledge",
                "date": datetime.utcnow().strftime("%Y-%m-%d"),
            }
        ],
    }

    _write_yaml(output_dir / "manifest.yaml", manifest)

    stubs = {
        "overview.md": f"# {slug.replace('-', ' ').title()}\n\n> Process overview. To be distilled.\n",
    }

    for relpath, content in stubs.items():
        (output_dir / relpath).write_text(content)

    return manifest


def _write_yaml(path: Path, data: dict):
    """Write a dict as YAML (simple serializer — avoids PyYAML dependency)."""
    def _serialize(obj, indent=0):
        lines = []
        prefix = "  " * indent
        if isinstance(obj, dict):
            for k, v in obj.items():
                if isinstance(v, (dict, list)):
                    lines.append(f"{prefix}{k}:")
                    lines.extend(_serialize(v, indent + 1))
                elif isinstance(v, bool):
                    lines.append(f"{prefix}{k}: {'true' if v else 'false'}")
                elif v is None:
                    lines.append(f"{prefix}{k}: null")
                else:
                    lines.append(f'{prefix}{k}: "{v}"')
        elif isinstance(obj, list):
            for item in obj:
                if isinstance(item, dict):
                    first = True
                    for k, v in item.items():
                        if first:
                            lines.append(f"{prefix}- {k}: \"{v}\"" if isinstance(v, str) else f"{prefix}- {k}: {v}")
                            first = False
                        else:
                            lines.append(f"{prefix}  {k}: \"{v}\"" if isinstance(v, str) else f"{prefix}  {k}: {v}")
                elif isinstance(item, str):
                    lines.append(f'{prefix}- "{item}"')
                else:
                    lines.append(f"{prefix}- {item}")
        return lines

    content = "\n".join(_serialize(data)) + "\n"
    path.write_text(content)


def main():
    parser = argparse.ArgumentParser(description="Scaffold and distill an EP pack")
    parser.add_argument("--scan", required=True, help="Path to scan manifest JSON")
    parser.add_argument("--pack", required=True, help="Pack to distill: type:slug (e.g., agent:easybot)")
    parser.add_argument("--output", "-o", required=True, help="Output directory for the pack")
    args = parser.parse_args()

    with open(args.scan) as f:
        scan = json.load(f)

    pack_type, pack_slug = args.pack.split(":", 1)
    output_dir = Path(args.output)
    output_dir.mkdir(parents=True, exist_ok=True)

    scaffolders = {
        "agent": scaffold_agent_pack,
        "person": scaffold_person_pack,
        "product": scaffold_product_pack,
        "process": scaffold_process_pack,
    }

    if pack_type not in scaffolders:
        print(f"Error: unknown pack type '{pack_type}'. Use: {', '.join(scaffolders.keys())}", file=sys.stderr)
        sys.exit(1)

    manifest = scaffolders[pack_type](output_dir, pack_slug, scan)
    print(f"Scaffolded {pack_type} pack '{pack_slug}' at {output_dir}")

    # List source files for this pack from the scan
    source_files = [f for f in scan.get("files", []) if f.get("pack") == pack_slug]
    print(f"Source files to distill: {len(source_files)}")
    for sf in source_files:
        flag = " ⚠️ SECRETS" if sf.get("has_secret_patterns") else ""
        flag += " 🔍 NEEDS ANALYSIS" if sf.get("needs_content_analysis") else ""
        print(f"  [{sf['confidence']:.1f}] {sf['path']} → {sf['section']}{flag}")

    print(f"\nNext: Agent reads source files and writes distilled content to {output_dir}/")


if __name__ == "__main__":
    main()


path: scripts/scan.py
content: #!/usr/bin/env python3
"""
ExpertPack Export — Workspace Scanner

Scans an OpenClaw workspace and produces a JSON manifest describing:
- All discovered files and their categories
- Proposed pack assignments (agent, person, product, process)
- Knowledge domains detected
- Confidence scores for ambiguous items

Usage:
    python3 scan.py --workspace /path/to/workspace --output /tmp/ep-scan.json
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path
from datetime import datetime


# --- File classification rules ---

# Files that define the agent itself
AGENT_FILES = {
    "SOUL.md": {"section": "overview", "confidence": 1.0},
    "IDENTITY.md": {"section": "facts/personal", "confidence": 1.0},
    "AGENTS.md": {"section": "mind/values+safety", "confidence": 1.0},
    "TOOLS.md": {"section": "operational/tools+infrastructure", "confidence": 1.0},
    "HEARTBEAT.md": {"section": "operational/routines", "confidence": 1.0},
}

# Files that describe the user (person pack)
USER_FILES = {
    "USER.md": {"section": "overview+facts", "confidence": 1.0},
}

# Memory files — need content analysis to classify
MEMORY_PATTERNS = {
    "MEMORY.md": {"default_pack": "agent", "section": "mixed", "confidence": 0.7},
    "memory/session-state.md": {"default_pack": "agent", "section": "operational/routines", "confidence": 0.6},
    "memory/archive.md": {"default_pack": "mixed", "section": "mixed", "confidence": 0.5},
    "memory/droplet-architecture.md": {"default_pack": "agent", "section": "operational/infrastructure", "confidence": 0.9},
}

# Secret patterns to flag
SECRET_PATTERNS = [
    re.compile(r'(?:api[_-]?key|token|secret|password|bearer)\s*[:=]\s*\S+', re.IGNORECASE),
    re.compile(r'sk-[a-zA-Z0-9]{20,}'),
    re.compile(r'ghp_[a-zA-Z0-9]{36,}'),
    re.compile(r'xoxb-[a-zA-Z0-9\-]+'),
]


def scan_workspace(workspace: str) -> dict:
    """Scan workspace and produce classification manifest."""
    ws = Path(workspace)
    if not ws.exists():
        print(f"Error: workspace {workspace} does not exist", file=sys.stderr)
        sys.exit(1)

    manifest = {
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "workspace": str(ws.resolve()),
        "packs": {},
        "files": [],
        "warnings": [],
    }

    # Track proposed packs
    packs = {}

    # --- Pass 1: Scan root-level OC files ---
    agent_slug = None
    user_slug = None

    # Detect agent identity
    identity_file = ws / "IDENTITY.md"
    if identity_file.exists():
        content = identity_file.read_text(errors="replace")
        name_match = re.search(r'\*\*Name:\*\*\s*(.+)', content)
        if name_match:
            agent_slug = name_match.group(1).strip().lower().replace(" ", "-")
    if not agent_slug:
        agent_slug = "agent"

    # Detect user identity
    user_file = ws / "USER.md"
    if user_file.exists():
        content = user_file.read_text(errors="replace")
        name_match = re.search(r'\*\*Name:\*\*\s*(.+)', content)
        if name_match:
            raw_name = name_match.group(1).strip()
            user_slug = raw_name.lower().replace(" ", "-")
    if not user_slug:
        user_slug = "user"

    # Register agent pack
    packs[agent_slug] = {
        "type": "person",
        "subtype": "agent",
        "slug": agent_slug,
        "sources": [],
        "description": f"AI agent identity and operational knowledge",
    }

    # Register person pack for user
    packs[user_slug] = {
        "type": "person",
        "subtype": None,
        "slug": user_slug,
        "sources": [],
        "description": f"Knowledge about the primary user",
    }

    # --- Pass 2: Classify root workspace files ---
    for fname, meta in AGENT_FILES.items():
        fpath = ws / fname
        if fpath.exists():
            entry = _make_file_entry(fpath, ws, agent_slug, meta["section"], meta["confidence"])
            _check_secrets(fpath, entry, manifest)
            manifest["files"].append(entry)
            packs[agent_slug]["sources"].append(str(fpath.relative_to(ws)))

    for fname, meta in USER_FILES.items():
        fpath = ws / fname
        if fpath.exists():
            entry = _make_file_entry(fpath, ws, user_slug, meta["section"], meta["confidence"])
            _check_secrets(fpath, entry, manifest)
            manifest["files"].append(entry)
            packs[user_slug]["sources"].append(str(fpath.relative_to(ws)))

    # --- Pass 3: Memory files ---
    for pattern, meta in MEMORY_PATTERNS.items():
        fpath = ws / pattern
        if fpath.exists():
            pack_assignment = agent_slug if meta["default_pack"] != "mixed" else agent_slug
            entry = _make_file_entry(fpath, ws, pack_assignment, meta["section"], meta["confidence"])
            if meta["default_pack"] == "mixed":
                entry["needs_content_analysis"] = True
            _check_secrets(fpath, entry, manifest)
            manifest["files"].append(entry)
            packs[agent_slug]["sources"].append(str(fpath.relative_to(ws)))

    # Daily journals
    memory_dir = ws / "memory"
    if memory_dir.exists():
        for f in sorted(memory_dir.glob("*.md")):
            rel = str(f.relative_to(ws))
            if rel in [v for v in MEMORY_PATTERNS.keys()]:
                continue  # already handled
            if re.match(r'memory/\d{4}-\d{2}-\d{2}\.md', rel):
                entry = _make_file_entry(f, ws, agent_slug, "summaries/lessons", 0.6)
                entry["content_type"] = "daily_journal"
                entry["needs_content_analysis"] = True
                manifest["files"].append(entry)
                packs[agent_slug]["sources"].append(rel)
            elif re.match(r'memory/\d{4}-\d{2}-summary\.md', rel):
                entry = _make_file_entry(f, ws, agent_slug, "summaries/lessons", 0.7)
                entry["content_type"] = "monthly_summary"
                manifest["files"].append(entry)
                packs[agent_slug]["sources"].append(rel)
            else:
                # Other memory files — classify by name
                entry = _make_file_entry(f, ws, agent_slug, "mixed", 0.5)
                entry["needs_content_analysis"] = True
                manifest["files"].append(entry)
                packs[agent_slug]["sources"].append(rel)

    # --- Pass 4: Detect product/process knowledge ---
    # Look for project directories with STATUS.md or known structures
    skip_dirs = {"memory", "export", ".git", "node_modules", "logs", "scripts",
                 ".secrets", "bot-profiles", "memory/old"}

    for item in ws.iterdir():
        if not item.is_dir():
            continue
        if item.name in skip_dirs or item.name.startswith("."):
            continue

        status_file = item / "STATUS.md"
        manifest_file = item / "manifest.yaml"
        readme_file = item / "README.md"

        # Check if this looks like an ExpertPack already
        if manifest_file.exists():
            # It's already an EP — note it but don't re-export
            manifest["warnings"].append(
                f"Directory {item.name}/ appears to be an existing ExpertPack (has manifest.yaml). Skipping."
            )
            continue

        # Check for product/process indicators
        has_status = status_file.exists()
        has_concepts = (item / "concepts").exists()
        has_workflows = (item / "workflows").exists()
        has_phases = (item / "phases").exists()

        if has_concepts or has_workflows or has_status:
            # Likely a product knowledge directory
            slug = item.name.lower().replace(" ", "-")
            if slug not in packs:
                packs[slug] = {
                    "type": "product",
                    "subtype": None,
                    "slug": slug,
                    "sources": [],
                    "description": f"Product knowledge from {item.name}/",
                }
            # Scan .md files in the directory
            for md_file in item.rglob("*.md"):
                entry = _make_file_entry(md_file, ws, slug, _guess_section(md_file, item), 0.7)
                manifest["files"].append(entry)
                packs[slug]["sources"].append(str(md_file.relative_to(ws)))

        elif has_phases:
            # Likely a process directory
            slug = item.name.lower().replace(" ", "-")
            if slug not in packs:
                packs[slug] = {
                    "type": "process",
                    "subtype": None,
                    "slug": slug,
                    "sources": [],
                    "description": f"Process knowledge from {item.name}/",
                }
            for md_file in item.rglob("*.md"):
                entry = _make_file_entry(md_file, ws, slug, "phases", 0.6)
                manifest["files"].append(entry)
                packs[slug]["sources"].append(str(md_file.relative_to(ws)))

    # --- Pass 5: Scan logs for operational knowledge ---
    logs_dir = ws / "logs"
    if logs_dir.exists():
        for md_file in logs_dir.rglob("*.md"):
            entry = _make_file_entry(md_file, ws, agent_slug, "operational/routines", 0.5)
            entry["content_type"] = "log"
            entry["needs_content_analysis"] = True
            manifest["files"].append(entry)
            packs[agent_slug]["sources"].append(str(md_file.relative_to(ws)))

    # --- Pass 6: Scripts ---
    scripts_dir = ws / "scripts"
    if scripts_dir.exists():
        for script in scripts_dir.iterdir():
            if script.is_file():
                entry = _make_file_entry(script, ws, agent_slug, "operational/tools", 0.6)
                entry["content_type"] = "script"
                manifest["files"].append(entry)
                packs[agent_slug]["sources"].append(str(script.relative_to(ws)))

    manifest["packs"] = packs

    # --- Summary stats ---
    manifest["summary"] = {
        "total_files": len(manifest["files"]),
        "total_packs": len(packs),
        "pack_types": {slug: p["type"] + (f"/{p['subtype']}" if p.get("subtype") else "")
                       for slug, p in packs.items()},
        "files_needing_analysis": sum(1 for f in manifest["files"] if f.get("needs_content_analysis")),
        "secret_warnings": sum(1 for f in manifest["files"] if f.get("has_secret_patterns")),
    }

    return manifest


def _make_file_entry(fpath: Path, workspace: Path, pack_slug: str,
                     section: str, confidence: float) -> dict:
    """Create a file entry for the manifest."""
    stat = fpath.stat()
    return {
        "path": str(fpath.relative_to(workspace)),
        "pack": pack_slug,
        "section": section,
        "confidence": confidence,
        "size_bytes": stat.st_size,
        "modified": datetime.fromtimestamp(stat.st_mtime).isoformat(),
    }


def _check_secrets(fpath: Path, entry: dict, manifest: dict):
    """Check file for potential secret patterns."""
    try:
        content = fpath.read_text(errors="replace")
        for pattern in SECRET_PATTERNS:
            if pattern.search(content):
                entry["has_secret_patterns"] = True
                manifest["warnings"].append(
                    f"Potential secret found in {entry['path']} — review before export"
                )
                return
    except Exception:
        pass


def _guess_section(fpath: Path, project_root: Path) -> str:
    """Guess the EP section for a file within a project directory."""
    rel = fpath.relative_to(project_root)
    parts = rel.parts

    if len(parts) > 1:
        parent = parts[0].lower()
        section_map = {
            "concepts": "concepts",
            "workflows": "workflows",
            "interfaces": "interfaces",
            "commercial": "commercial",
            "troubleshooting": "troubleshooting",
            "faq": "faq",
            "phases": "phases",
            "decisions": "decisions",
            "checklists": "checklists",
        }
        if parent in section_map:
            return section_map[parent]

    name = fpath.stem.lower()
    if "status" in name:
        return "overview"
    if "readme" in name:
        return "overview"
    if "glossary" in name:
        return "glossary"

    return "concepts"  # default for product files


def main():
    parser = argparse.ArgumentParser(description="Scan OpenClaw workspace for EP export")
    parser.add_argument("--workspace", "-w", required=True, help="Path to workspace")
    parser.add_argument("--output", "-o", required=True, help="Output JSON path")
    args = parser.parse_args()

    manifest = scan_workspace(args.workspace)

    with open(args.output, "w") as f:
        json.dump(manifest, f, indent=2)

    print(f"Scan complete: {manifest['summary']['total_files']} files, "
          f"{manifest['summary']['total_packs']} packs proposed")
    print(f"Output: {args.output}")

    if manifest["warnings"]:
        print(f"\n⚠️  {len(manifest['warnings'])} warnings:")
        for w in manifest["warnings"][:10]:
            print(f"  - {w}")
        if len(manifest["warnings"]) > 10:
            print(f"  ... and {len(manifest['warnings']) - 10} more")


if __name__ == "__main__":
    main()


path: scripts/validate.py
content: #!/usr/bin/env python3
"""
ExpertPack Export — Validator

Checks an exported EP against schema rules:
- Required files exist
- Manifest fields are valid
- No secrets leaked
- File sizes within guidelines
- Cross-references resolve

Usage:
    python3 validate.py --export-dir /path/to/export/
"""

import argparse
import os
import re
import sys
from pathlib import Path


SECRET_PATTERNS = [
    re.compile(r'(?:api[_-]?key|token|secret|password|bearer)\s*[:=]\s*(?!<REDACTED>)\S{10,}', re.IGNORECASE),
    re.compile(r'sk-[a-zA-Z0-9]{20,}'),
    re.compile(r'ghp_[a-zA-Z0-9]{36,}'),
    re.compile(r'xoxb-[a-zA-Z0-9\-]+'),
]

REQUIRED_PACK_FILES = ["manifest.yaml", "overview.md"]

AGENT_REQUIRED_DIRS = ["operational", "mind", "relationships", "facts", "presentation"]
AGENT_REQUIRED_FILES = [
    "operational/tools.md",
    "operational/safety.md",
    "mind/values.md",
    "relationships/people.md",
    "presentation/speech_patterns.md",
]

MAX_FILE_SIZE_KB = 5  # Soft limit — warn above this
MAX_FILE_SIZE_HARD_KB = 20  # Hard limit — error above this


def validate(export_dir: str) -> dict:
    """Validate an EP export directory."""
    export = Path(export_dir)
    results = {
        "passed": True,
        "errors": [],
        "warnings": [],
        "stats": {
            "packs": 0,
            "files": 0,
            "total_size_kb": 0,
        },
    }

    if not export.exists():
        results["errors"].append(f"Export directory does not exist: {export_dir}")
        results["passed"] = False
        return results

    # Find all packs
    packs_dir = export / "packs"
    composites_dir = export / "composites"

    pack_dirs = []
    if packs_dir.exists():
        pack_dirs = [d for d in packs_dir.iterdir() if d.is_dir()]
    if composites_dir.exists():
        pack_dirs += [d for d in composites_dir.iterdir() if d.is_dir()]

    if not pack_dirs:
        results["errors"].append("No packs or composites found in export directory")
        results["passed"] = False
        return results

    results["stats"]["packs"] = len(pack_dirs)

    for pack_dir in pack_dirs:
        _validate_pack(pack_dir, results)

    # Summary
    if results["errors"]:
        results["passed"] = False

    return results


def _validate_pack(pack_dir: Path, results: dict):
    """Validate a single pack directory."""
    pack_name = pack_dir.name

    # Check required files
    for req in REQUIRED_PACK_FILES:
        if not (pack_dir / req).exists():
            results["errors"].append(f"[{pack_name}] Missing required file: {req}")

    # Parse manifest
    manifest_path = pack_dir / "manifest.yaml"
    manifest = {}
    if manifest_path.exists():
        manifest = _parse_simple_yaml(manifest_path)

    pack_type = manifest.get("type", "unknown")
    subtype = manifest.get("subtype", "")

    # Validate manifest fields
    required_manifest = ["name", "slug", "type", "version", "entry_point"]
    for field in required_manifest:
        if field not in manifest:
            results["errors"].append(f"[{pack_name}] manifest.yaml missing field: {field}")

    # Agent-specific checks
    if subtype == "agent":
        for req_dir in AGENT_REQUIRED_DIRS:
            if not (pack_dir / req_dir).exists():
                results["warnings"].append(f"[{pack_name}] Agent pack missing directory: {req_dir}/")
        for req_file in AGENT_REQUIRED_FILES:
            fpath = pack_dir / req_file
            if not fpath.exists():
                results["warnings"].append(f"[{pack_name}] Agent pack missing file: {req_file}")
            elif fpath.stat().st_size < 50:
                results["warnings"].append(f"[{pack_name}] {req_file} appears to be a stub ({fpath.stat().st_size} bytes)")

    # Check all files
    for fpath in pack_dir.rglob("*"):
        if not fpath.is_file():
            continue

        results["stats"]["files"] += 1
        size_kb = fpath.stat().st_size / 1024
        results["stats"]["total_size_kb"] += size_kb

        # File size checks (only for .md files)
        if fpath.suffix == ".md":
            if size_kb > MAX_FILE_SIZE_HARD_KB:
                results["errors"].append(
                    f"[{pack_name}] {fpath.relative_to(pack_dir)} is {size_kb:.1f}KB — exceeds hard limit ({MAX_FILE_SIZE_HARD_KB}KB)")
            elif size_kb > MAX_FILE_SIZE_KB:
                results["warnings"].append(
                    f"[{pack_name}] {fpath.relative_to(pack_dir)} is {size_kb:.1f}KB — consider splitting")

        # Secret scan
        if fpath.suffix in (".md", ".yaml", ".yml", ".json", ".txt"):
            try:
                content = fpath.read_text(errors="replace")
                for pattern in SECRET_PATTERNS:
                    match = pattern.search(content)
                    if match:
                        results["errors"].append(
                            f"[{pack_name}] Potential secret in {fpath.relative_to(pack_dir)}: ...{match.group()[:30]}...")
                        break
            except Exception:
                pass

        # Naming convention check
        if fpath.suffix == ".md" and fpath.name != fpath.name.lower() and fpath.name not in (
            "MIGRATION.md", "LEGACY.md", "README.md", "SCHEMA.md", "STATUS.md"
        ):
            # Allow uppercase for conventional filenames
            if not fpath.name.isupper() and fpath.name not in ("MIGRATION.md",):
                pass  # kebab-case not enforced on well-known names


def _parse_simple_yaml(path: Path) -> dict:
    """Very simple YAML parser — handles flat key: value pairs and quoted strings."""
    result = {}
    try:
        for line in path.read_text().splitlines():
            line = line.strip()
            if not line or line.startswith("#") or line.startswith("-"):
                continue
            if ":" in line:
                key, _, value = line.partition(":")
                key = key.strip()
                value = value.strip().strip('"').strip("'")
                if value.lower() == "true":
                    value = True
                elif value.lower() == "false":
                    value = False
                elif value.lower() == "null":
                    value = None
                result[key] = value
    except Exception:
        pass
    return result


def main():
    parser = argparse.ArgumentParser(description="Validate EP export")
    parser.add_argument("--export-dir", required=True, help="Export root directory")
    args = parser.parse_args()

    results = validate(args.export_dir)

    print(f"{'✅ PASSED' if results['passed'] else '❌ FAILED'}")
    print(f"\nStats: {results['stats']['packs']} packs, {results['stats']['files']} files, "
          f"{results['stats']['total_size_kb']:.1f}KB total")

    if results["errors"]:
        print(f"\n❌ Errors ({len(results['errors'])}):")
        for e in results["errors"]:
            print(f"  {e}")

    if results["warnings"]:
        print(f"\n⚠️  Warnings ({len(results['warnings'])}):")
        for w in results["warnings"]:
            print(f"  {w}")

    sys.exit(0 if results["passed"] else 1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)