Skill: theshadowrose/crucible-forge (version 1.0.3)

--- name: "Forge AI Workspace Architect" description: "Systematic workspace reorganization for AI agent users. Scans workspace, builds safety-first reorganization plan, executes with zero data loss, and verifies everything works afterward." author: "@TheShadowRose" version: "1.0.2" tags: ["workspace", "reorganization", "architecture", "cleanup", "organization", "safety"] license: "MIT" --- # Forge AI Workspace Architect Systematic workspace reorganization for AI agent users. Scans workspace, builds safety-first reorganization plan, executes with zero data loss, and verifies everything works afterward. --- **Systematic workspace reorganization for AI agent users.** Forge scans your AI agent's workspace, builds a safety-first reorganization plan, executes it with zero data loss, and verifies everything works afterward. Built from a battle-tested methodology used on production AI workspaces. --- ## The Problem AI agent workspaces accumulate chaos fast. Files pile up at root. Memory files reference moved documents. Cron jobs point to old paths. Config files drift. One bad reorganization can brick your agent's memory, break running processes, or corrupt critical state files. Manual cleanup is dangerous. Automated cleanup without safeguards is worse. ## What Forge Does ### 1. **Scan** (`forge_scan.py`) - Maps your entire workspace: files, directories, sizes, types - Discovers all cross-references (file A mentions file B) - Finds broken references (file A mentions file B, but B doesn't exist) - Identifies running processes and their file dependencies - Detects hardcoded paths in scripts and configs - Catalogs cron jobs, scheduled tasks, and their path dependencies - Outputs a complete workspace report ### 2. **Plan** (`forge_plan.py`) - Generates a reorganization plan based on your configured directory template - Creates hardline safety rules before touching anything - Produces an ordered execution sequence with dependency tracking - Identifies protected files that must never be modified - Plans reference patches (what needs updating after moves) - Generates rollback procedures for every step - Outputs a human-reviewable plan — nothing executes automatically ### 3. **Audit** (`forge_audit.py`) - **Pre-move audit:** Verifies backup exists, creates file manifest, checks all protected files - **Post-move audit:** Compares file counts, verifies no files lost, checks all references resolve, validates protected files unchanged - **Content audit:** Scans for misplaced content, secrets in wrong locations, orphaned files - **Reference audit:** Verifies every cross-reference in every file still resolves --- ## Quick Start ```bash # 1. Configure your workspace cp config_example.py forge_config.py # Edit forge_config.py with your workspace path, protected files, directory template # 2. Scan your workspace python3 forge_scan.py --config forge_config.py # 3. Generate a reorganization plan python3 forge_plan.py --config forge_config.py --scan-report scan_report.json # 4. Review the plan (ALWAYS review before executing) cat reorg_plan.json # 5. Run pre-move audit python3 forge_audit.py --config forge_config.py --phase pre # 6. Execute the plan (you do this manually or with your own script) # Forge generates the plan. YOU execute it. This is deliberate. # 7. Run post-move audit python3 forge_audit.py --config forge_config.py --phase post --manifest pre_manifest.txt ``` ## Design Philosophy ### Safety First, Always - **Never deletes files.** Moves only. If something looks like garbage, it goes to archive. - **Never modifies protected files.** You define what's protected; Forge won't touch them. - **Backup before changes.** Forge refuses to generate a plan without a verified backup. - **Human reviews the plan.** Forge doesn't auto-execute. You read the plan, you decide. - **Rollback for every step.** Every move in the plan includes how to undo it. ### Zero-Deletion Policy No file is ever deleted during reorganization. Files are moved, never removed. Even files that appear to be duplicates are preserved until you explicitly confirm deletion. ### Atomic Reference Updates When a file moves, every reference to that file must update in the same logical step. Forge tracks these as paired operations: move file + update references = one unit. If the reference update fails, the move should be rolled back. ### Framework, Not Opinions Forge doesn't dictate your directory structure. You configure the template. Forge handles the mechanics of getting there safely. --- ## Use Cases - **New AI agent setup:** Start with a clean, structured workspace from day one - **Workspace recovery:** Untangle months of accumulated file chaos - **Platform migration:** Moving from one AI agent platform to another - **Team onboarding:** Standardize workspace structure across multiple agents - **Pre-deployment cleanup:** Verify workspace integrity before going live - **Post-incident audit:** Check what broke and what's still intact after a failure ## What's Included | File | Purpose | |------|---------| | `forge_scan.py` | Workspace scanner and reference mapper | | `forge_plan.py` | Reorganization planner with safety rules | | `forge_audit.py` | Pre/post audit verification | | `config_example.py` | Configuration template | | `LIMITATIONS.md` | What Forge doesn't do | | `LICENSE` | MIT License | ## Requirements - Python 3.8+ - No external dependencies (stdlib only) - Works on any OS (Linux, macOS, Windows) - Platform-agnostic (OpenClaw, AutoGPT, CrewAI, or any file-based workspace) ## Configuration See `config_example.py` for the full configuration reference. Key settings: - **`WORKSPACE_ROOT`** — Path to your workspace - **`PROTECTED_FILES`** — Files that must never be modified or moved - **`PROTECTED_DIRS`** — Directories that must never be touched - **`DIRECTORY_TEMPLATE`** — Your desired directory structure - **`BACKUP_DIR`** — Where backups are stored - **`ARCHIVE_DIR`** — Where "deleted" files actually go - **`README_TEMPLATE`** — Content template for per-directory `_README.md` files --- ## quality-verified ## License MIT — See `LICENSE` file. --- ## ⚠️ Security Note — Config File Execution This skill loads configuration from a user-provided Python file using `importlib.exec_module`. **This executes the config file as Python code.** - Only run config files you have written or fully reviewed - A malicious or unreviewed config file can execute arbitrary code on your system - Config path is validated for existence, `.py` extension, and size (1MB cap) before execution - Do not run configs provided by untrusted sources without inspection `PROCESS_CHECK_COMMANDS` in your config file runs subprocess calls. Only include trusted, pre-validated commands. --- ## ⚠️ Disclaimer This software is provided "AS IS", without warranty of any kind, express or implied. **USE AT YOUR OWN RISK.** - The author(s) are NOT liable for any damages, losses, or consequences arising from the use or misuse of this software — including but not limited to financial loss, data loss, security breaches, business interruption, or any indirect/consequential damages. - This software does NOT constitute financial, legal, trading, or professional advice. - Users are solely responsible for evaluating whether this software is suitable for their use case, environment, and risk tolerance. - No guarantee is made regarding accuracy, reliability, completeness, or fitness for any particular purpose. - The author(s) are not responsible for how third parties use, modify, or distribute this software after purchase. By downloading, installing, or using this software, you acknowledge that you have read this disclaimer and agree to use the software entirely at your own risk. **DATA DISCLAIMER:** This software processes and stores data locally on your system. The author(s) are not responsible for data loss, corruption, or unauthorized access resulting from software bugs, system failures, or user error. Always maintain independent backups of important data. This software does not transmit data externally unless explicitly configured by the user. --- ## Support & Links | | | |---|---| | 🐛 **Bug Reports** | [REDACTED_SECRET] | | ☕ **Ko-fi** | [ko-fi.com/theshadowrose](https://ko-fi.com/theshadowrose) | | 🛒 **Gumroad** | [shadowyrose.gumroad.com](https://shadowyrose.gumroad.com) | | 🐦 **Twitter** | [@TheShadowyRose](https://twitter.com/TheShadowyRose) | | 🐙 **GitHub** | [github.com/TheShadowRose](https://github.com/TheShadowRose) | | 🧠 **PromptBase** | [promptbase.com/profile/shadowrose](https://promptbase.com/profile/shadowrose) | *Built with [OpenClaw](https://github.com/openclaw/openclaw) — thank you for making this possible.* --- 🛠️ **Need something custom?** Custom OpenClaw agents & skills starting at $500. If you can describe it, I can build it. → [Hire me on Fiverr](https://www.fiverr.com/s/jjmlZ0v)

path: config_example.py
content: """
Forge Configuration — AI Workspace Architect
=============================================

Copy this file to forge_config.py and customize for your workspace.
Every setting is documented. Read before editing.
"""

import os

# =============================================================================
# WORKSPACE ROOT
# =============================================================================
# Absolute path to your AI agent's workspace directory.
# All scanning, planning, and auditing operates within this directory.
WORKSPACE_ROOT = os.path.expanduser("~/workspace")

# =============================================================================
# BACKUP SETTINGS
# =============================================================================
# Directory for backups. Forge refuses to plan without a verified backup.
BACKUP_DIR = os.path.join(WORKSPACE_ROOT, "backups")

# Create a backup before generating any plan
REQUIRE_BACKUP_BEFORE_PLAN = True

# Maximum age of backup (in hours) to consider "fresh enough"
MAX_BACKUP_AGE_HOURS = 24

# Files/dirs to exclude from backups (glob patterns)
BACKUP_EXCLUDES = [
    "backups/*.tar.gz",
    ".git",
    "__pycache__",
    "*.pyc",
    "node_modules",
    "venv",
    ".env",
]

# =============================================================================
# PROTECTED FILES — NEVER MODIFY OR MOVE
# =============================================================================
# These files are sacred. Forge will not include them in any move plan.
# Add your agent's critical files here.
PROTECTED_FILES = [
    # Identity / personality files
    # "SOUL.md",
    # "IDENTITY.md",
    # "AGENTS.md",

    # Memory files
    # "MEMORY.md",
    # "memory/*.md",

    # State files (active processes depend on these)
    # "drift_state.json",

    # Configuration
    # "USER.md",
    # "TOOLS.md",
]

# =============================================================================
# PROTECTED DIRECTORIES — NEVER TOUCH CONTENTS
# =============================================================================
# Entire directories that Forge must not scan for moves.
# Use for live systems, running bots, etc.
PROTECTED_DIRS = [
    # "bots",           # Live bot directory
    # "skills",         # Installed agent skills
    # ".secrets",       # Credentials
]

# =============================================================================
# DIRECTORY TEMPLATE — YOUR DESIRED STRUCTURE
# =============================================================================
# Define the target directory structure. Forge will plan moves to match this.
# Each entry is: directory_path -> description (used in _README.md generation)
#
# Customize this entirely for your use case.
DIRECTORY_TEMPLATE = {
    "archive": "Retired files. Nothing is deleted — it comes here instead.",
    "backups": "System backups. Automated and manual snapshots.",
    "docs": "Documentation, guides, and reference material.",
    "inbox": "Quick-capture landing zone. New files go here first, then get sorted.",
    "memory": "Agent memory files. Daily notes, long-term memory.",
    "memory/archive": "Archived monthly memory rollups.",
    "projects": "Active projects and their working files.",
    "scripts": "Utility scripts, automation, and tools.",
    "config": "Configuration files for various systems.",
    "templates": "Reusable templates and scaffolds.",
}

# =============================================================================
# FILE CLASSIFICATION RULES
# =============================================================================
# Rules for automatically classifying files during scan.
# Format: (glob_pattern, target_directory)
# Files matching a pattern get suggested for the target directory in the plan.
# These are SUGGESTIONS — the plan is always human-reviewed.
FILE_CLASSIFICATION_RULES = [
    # ("*.log", "logs"),
    # ("DRAFT-*.md", "inbox"),
    # ("*.tar.gz", "backups"),
    # ("*.bak", "archive"),
    # ("memory/????-??-??.md", "memory"),  # Daily memory files stay in memory/
]

# =============================================================================
# REFERENCE SCANNING
# =============================================================================
# File extensions to scan for cross-references
REFERENCE_SCAN_EXTENSIONS = [
    ".md", ".txt", ".py", ".sh", ".js", ".ts",
    ".json", ".yaml", ".yml", ".toml", ".cfg", ".ini",
]

# Patterns to detect file references (regex)
# These are used to find mentions of files inside other files.
REFERENCE_PATTERNS = [
    r'(?:^|[\s"`\'/\\(])([A-Za-z0-9_\-./]+\.(?:md|txt|py|sh|js|json|yaml|yml|toml))',
    r'(?:path|file|source|include|import)\s*[:=]\s*["\']?([^\s"\']+)',
]

# =============================================================================
# PROCESS DETECTION
# =============================================================================
# Forge checks for running processes that depend on workspace files.
# If a process is found using a file, that file becomes protected.
DETECT_RUNNING_PROCESSES = True

# Commands to check for running processes (platform-specific)
PROCESS_CHECK_COMMANDS = [
    "ps aux",           # Unix/Linux/macOS
    # "tasklist /v",    # Windows
]

# =============================================================================
# AUDIT SETTINGS
# =============================================================================
# Directory where audit reports and manifests are stored
AUDIT_DIR = os.path.join(WORKSPACE_ROOT, "backups")

# Secrets patterns — files containing these patterns trigger warnings
SECRETS_PATTERNS = [
    r"(?i)api[_\-]?key",
    r"(?i)password\s*[:=]",
    r"(?i)secret\s*[:=]",
    r"(?i)token\s*[:=]",
    r"sk-[a-zA-Z0-9]{20,}",
    r"(?i)BEGIN\s+(RSA|DSA|EC|OPENSSH)\s+PRIVATE\s+KEY",
]

# Maximum file size to scan for references (skip huge files)
MAX_SCAN_FILE_SIZE_MB = 10

# =============================================================================
# OUTPUT SETTINGS
# =============================================================================
# Where Forge writes its reports and plans
OUTPUT_DIR = os.path.join(WORKSPACE_ROOT, "forge-output")

# Generate _README.md files for new directories
GENERATE_README_FILES = True

# README template for new directories
README_TEMPLATE = """# {dir_name}

{description}

## What belongs here
{belongs_here}

## What does NOT belong here
Files that don't match this directory's purpose should go in `inbox/` for triage.
"""

# =============================================================================
# ARCHIVE SETTINGS
# =============================================================================
ARCHIVE_DIR = os.path.join(WORKSPACE_ROOT, "archive")

# Use year subdirectories in archive (archive/2026/, archive/2025/)
ARCHIVE_YEAR_SUBDIRS = True

# =============================================================================
# SAFETY SETTINGS
# =============================================================================
# These are the hardline rules. Override at your own risk.

# Never delete any file (moves to archive instead)
ZERO_DELETION_POLICY = True

# Require file count verification before and after moves
REQUIRE_FILE_COUNT_CHECK = True

# Maximum number of files to move in a single plan (safety limit)
MAX_MOVES_PER_PLAN = 500

# Abort if more than this percentage of files would be moved
MAX_MOVE_PERCENTAGE = 80  # If >80% of files need moving, something is wrong

# Require manifest creation before any plan
REQUIRE_PRE_MANIFEST = True


path: forge_audit.py
content: #!/usr/bin/env python3
"""
Forge Auditor — Pre/Post Verification
======================================

Runs integrity checks before and after workspace reorganization:

Pre-move audit:
  - Verifies backup exists and is recent
  - Creates file manifest (snapshot of current state)
  - Validates all protected files exist and are intact
  - Checks for running processes
  - Scans for exposed secrets

Post-move audit:
  - Compares file count (no files lost)
  - Verifies all protected files unchanged
  - Checks all cross-references resolve
  - Validates directory structure matches template
  - Reports any orphaned files

Usage:
    python3 forge_audit.py --config forge_config.py --phase pre
    python3 forge_audit.py --config forge_config.py --phase post --manifest pre_manifest.txt
    python3 forge_audit.py --config forge_config.py --phase content
"""

import argparse
import hashlib
import importlib.util
import json
import os
import re
import subprocess
import sys
import time
from datetime import datetime
from pathlib import Path


def load_config(config_path: str) -> object:
    # Security: validate path before executing config file
    if not os.path.isfile(config_path):
        print(f"ERROR: Config not found: {config_path}")
        sys.exit(1)
    if not config_path.endswith(".py"):
        print(f"ERROR: Config must be a .py file: {config_path}")
        sys.exit(1)
    if os.path.getsize(config_path) > 1_000_000:
        print(f"ERROR: Config file too large (>1MB): {config_path}")
        sys.exit(1)
    spec = importlib.util.spec_from_file_location("config", config_path)
    if spec is None or spec.loader is None:
        print(f"ERROR: Cannot load config from {config_path}")
        sys.exit(1)
    config = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(config)
    return config


class AuditResult:
    """Collects audit findings."""

    def __init__(self, phase: str):
        self.phase = phase
        self.timestamp = datetime.now().isoformat()
        self.checks = []
        self.passed = 0
        self.warnings = 0
        self.failures = 0

    def check_pass(self, name: str, detail: str = ""):
        self.checks.append({"name": name, "result": "PASS", "detail": detail})
        self.passed += 1

    def check_warn(self, name: str, detail: str):
        self.checks.append({"name": name, "result": "WARNING", "detail": detail})
        self.warnings += 1

    def check_fail(self, name: str, detail: str):
        self.checks.append({"name": name, "result": "FAIL", "detail": detail})
        self.failures += 1

    @property
    def overall(self) -> str:
        if self.failures > 0:
            return "FAIL"
        if self.warnings > 0:
            return "WARNING"
        return "PASS"

    def to_dict(self) -> dict:
        return {
            "phase": self.phase,
            "timestamp": self.timestamp,
            "overall": self.overall,
            "passed": self.passed,
            "warnings": self.warnings,
            "failures": self.failures,
            "checks": self.checks,
        }

    def print_report(self):
        symbols = {"PASS": "✅", "WARNING": "⚠️", "FAIL": "❌"}
        print(f"\n{'=' * 60}")
        print(f"FORGE {self.phase.upper()} AUDIT REPORT")
        print(f"{'=' * 60}")
        print(f"Time:     {self.timestamp}")
        print(f"Overall:  {symbols.get(self.overall, '?')} {self.overall}")
        print(f"Passed:   {self.passed}")
        print(f"Warnings: {self.warnings}")
        print(f"Failures: {self.failures}")
        print(f"\n--- CHECKS ---")
        for c in self.checks:
            sym = symbols.get(c["result"], "?")
            detail = f" — {c['detail']}" if c["detail"] else ""
            print(f"  {sym} {c['name']}{detail}")
        print(f"{'=' * 60}")


def get_file_hash(filepath: str) -> str:
    """SHA-256 hash of a file."""
    h = hashlib.sha256()
    try:
        with open(filepath, "rb") as f:
            for chunk in iter(lambda: f.read(8192), b""):
                h.update(chunk)
        return h.hexdigest()
    except OSError:
        return ""


def create_manifest(workspace_root: str) -> list:
    """Create a file manifest with paths and hashes."""
    manifest = []
    workspace = Path(workspace_root).resolve()
    for root, _, filenames in os.walk(workspace):
        for fn in filenames:
            fp = Path(root) / fn
            try:
                rel = str(fp.relative_to(workspace))
                stat = fp.stat()
                manifest.append({
                    "path": rel,
                    "size": stat.st_size,
                    "hash": get_file_hash(str(fp)),
                    "modified": stat.st_mtime,
                })
            except OSError:
                pass
    return manifest


def run_pre_audit(config) -> AuditResult:
    """Pre-move audit: verify workspace is safe to reorganize."""
    result = AuditResult("pre-move")
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")
    workspace = Path(workspace_root).resolve()

    # Check 1: Workspace exists
    if workspace.is_dir():
        result.check_pass("Workspace exists", str(workspace))
    else:
        result.check_fail("Workspace exists", f"{workspace} not found")
        return result

    # Check 2: Backup exists and is recent
    backup_dir = getattr(config, "BACKUP_DIR", os.path.join(workspace_root, "backups"))
    max_age = getattr(config, "MAX_BACKUP_AGE_HOURS", 24) * 3600

    if os.path.isdir(backup_dir):
        backups = []
        for f in os.listdir(backup_dir):
            fp = os.path.join(backup_dir, f)
            if f.endswith((".tar.gz", ".tgz", ".zip")):
                backups.append((fp, os.path.getmtime(fp)))

        if backups:
            newest = max(backups, key=lambda x: x[1])
            age = time.time() - newest[1]
            if age < max_age:
                result.check_pass("Backup exists and recent", f"{newest[0]} ({age/3600:.1f}h old)")
            else:
                result.check_warn("Backup exists but stale", f"{newest[0]} ({age/3600:.1f}h old)")
        else:
            result.check_fail("No backup archives found", f"Checked {backup_dir}")
    else:
        result.check_fail("Backup directory missing", backup_dir)

    # Check 3: Protected files exist and are readable
    protected_files = getattr(config, "PROTECTED_FILES", [])
    for pf in protected_files:
        import glob as glob_mod
        matches = glob_mod.glob(os.path.join(workspace_root, pf))
        if matches:
            for m in matches:
                if os.path.isfile(m) and os.path.getsize(m) > 0:
                    result.check_pass(f"Protected file: {pf}", f"{os.path.getsize(m)} bytes")
                else:
                    result.check_fail(f"Protected file: {pf}", "Empty or not a file")
        else:
            result.check_warn(f"Protected file: {pf}", "Not found (check glob pattern)")

    # Check 4: Protected directories exist
    protected_dirs = getattr(config, "PROTECTED_DIRS", [])
    for pd in protected_dirs:
        full = os.path.join(workspace_root, pd)
        if os.path.isdir(full):
            count = sum(1 for _, _, files in os.walk(full) for _ in files)
            result.check_pass(f"Protected dir: {pd}", f"{count} files")
        else:
            result.check_warn(f"Protected dir: {pd}", "Not found")

    # Check 5: Running processes
    try:
        ps = subprocess.run(["ps", "aux"], capture_output=True, text=True, timeout=10)
        ws_procs = [l for l in ps.stdout.splitlines() if workspace_root in l]
        if ws_procs:
            result.check_warn(
                "Running processes using workspace",
                f"{len(ws_procs)} process(es) found — their files will be auto-protected"
            )
        else:
            result.check_pass("No running processes using workspace")
    except Exception:
        result.check_warn("Process check", "Could not run ps aux")

    # Check 6: Create manifest
    print("  Creating file manifest...")
    manifest = create_manifest(workspace_root)

    manifest_dir = getattr(config, "AUDIT_DIR", os.path.join(workspace_root, "backups"))
    os.makedirs(manifest_dir, exist_ok=True)
    manifest_path = os.path.join(manifest_dir, f"pre_manifest_{int(time.time())}.json")

    with open(manifest_path, "w") as f:
        json.dump(manifest, f, indent=2)

    result.check_pass("Manifest created", f"{len(manifest)} files → {manifest_path}")

    # Check 7: File count baseline
    total = len(manifest)
    result.check_pass("File count baseline", f"{total} files")

    # Check 8: Secrets scan
    secrets_patterns = getattr(config, "SECRETS_PATTERNS", [])
    if secrets_patterns:
        warnings = 0
        compiled = []
        for p in secrets_patterns:
            try:
                compiled.append(re.compile(p))
            except re.error:
                pass

        for entry in manifest:
            fp = os.path.join(workspace_root, entry["path"])
            if entry["size"] > 10 * 1024 * 1024:
                continue
            try:
                content = Path(fp).read_text(encoding="utf-8", errors="replace")
                for pat in compiled:
                    if pat.search(content):
                        warnings += 1
                        break
            except OSError:
                pass

        if warnings:
            result.check_warn("Secrets scan", f"{warnings} file(s) contain potential secrets")
        else:
            result.check_pass("Secrets scan", "No exposed secrets detected")

    return result


def run_post_audit(config, manifest_path: str = None) -> AuditResult:
    """Post-move audit: verify nothing broke."""
    result = AuditResult("post-move")
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")
    workspace = Path(workspace_root).resolve()

    # Check 1: Workspace still exists
    if workspace.is_dir():
        result.check_pass("Workspace exists")
    else:
        result.check_fail("Workspace exists", "MISSING")
        return result

    # Check 2: File count comparison
    current_manifest = create_manifest(workspace_root)
    current_count = len(current_manifest)

    if manifest_path and os.path.exists(manifest_path):
        with open(manifest_path) as f:
            pre_manifest = json.load(f)
        pre_count = len(pre_manifest)

        if current_count >= pre_count:
            result.check_pass("File count", f"Pre: {pre_count}, Post: {current_count} (no files lost)")
        else:
            lost = pre_count - current_count
            result.check_fail("File count", f"Pre: {pre_count}, Post: {current_count} — {lost} files MISSING")

            # Find missing files
            pre_paths = {e["path"] for e in pre_manifest}
            current_paths = {e["path"] for e in current_manifest}
            missing = pre_paths - current_paths
            if missing:
                for m in sorted(missing)[:20]:
                    result.check_fail(f"Missing file", m)
    else:
        result.check_warn("File count", f"No pre-manifest to compare. Current: {current_count}")

    # Check 3: Protected files unchanged
    protected_files = getattr(config, "PROTECTED_FILES", [])
    if manifest_path and os.path.exists(manifest_path):
        with open(manifest_path) as f:
            pre_manifest = json.load(f)
        pre_hashes = {e["path"]: e["hash"] for e in pre_manifest}

        import glob as glob_mod
        for pf in protected_files:
            matches = glob_mod.glob(os.path.join(workspace_root, pf))
            for m in matches:
                rel = str(Path(m).relative_to(workspace))
                current_hash = get_file_hash(m)
                pre_hash = pre_hashes.get(rel, "")
                if pre_hash and current_hash == pre_hash:
                    result.check_pass(f"Protected unchanged: {rel}")
                elif pre_hash:
                    result.check_fail(f"Protected CHANGED: {rel}", "Hash mismatch — file was modified!")
                else:
                    result.check_warn(f"Protected file: {rel}", "Not in pre-manifest")

    # Check 4: Directory structure matches template
    template = getattr(config, "DIRECTORY_TEMPLATE", {})
    for dir_path in template:
        full = os.path.join(workspace_root, dir_path)
        if os.path.isdir(full):
            result.check_pass(f"Directory exists: {dir_path}")
        else:
            result.check_warn(f"Directory missing: {dir_path}")

    # Check 5: _README.md files exist
    if getattr(config, "GENERATE_README_FILES", True):
        for dir_path in template:
            readme = os.path.join(workspace_root, dir_path, "_README.md")
            if os.path.isfile(readme):
                result.check_pass(f"README exists: {dir_path}/_README.md")
            else:
                result.check_warn(f"README missing: {dir_path}/_README.md")

    # Check 6: Broken references scan
    print("  Scanning for broken references...")
    extensions = set(getattr(config, "REFERENCE_SCAN_EXTENSIONS", [".md", ".txt", ".py", ".sh"]))
    current_paths = {e["path"] for e in current_manifest}
    current_basenames = {os.path.basename(e["path"]) for e in current_manifest}
    broken_count = 0

    for entry in current_manifest:
        if Path(entry["path"]).suffix.lower() not in extensions:
            continue
        if entry["size"] > 10 * 1024 * 1024:
            continue
        fp = os.path.join(workspace_root, entry["path"])
        try:
            content = Path(fp).read_text(encoding="utf-8", errors="replace")
        except OSError:
            continue

        # Look for file references that don't resolve
        file_refs = re.findall(r'[\w\-./]+\.(?:md|txt|py|sh|json|yaml|yml)', content)
        for ref in file_refs:
            ref = ref.strip("./")
            basename = os.path.basename(ref)
            if (ref not in current_paths and
                    basename not in current_basenames and
                    len(basename) > 4):
                broken_count += 1

    if broken_count == 0:
        result.check_pass("Reference integrity", "No broken references detected")
    else:
        result.check_warn("Reference integrity", f"{broken_count} potential broken references")

    # Check 7: Running processes still alive
    protected_dirs = getattr(config, "PROTECTED_DIRS", [])
    try:
        ps = subprocess.run(["ps", "aux"], capture_output=True, text=True, timeout=10)
        ws_procs = [l for l in ps.stdout.splitlines() if workspace_root in l]
        if ws_procs:
            result.check_pass("Running processes survived", f"{len(ws_procs)} still active")
    except Exception:
        pass

    # Save post-manifest
    audit_dir = getattr(config, "AUDIT_DIR", os.path.join(workspace_root, "backups"))
    os.makedirs(audit_dir, exist_ok=True)
    post_manifest_path = os.path.join(audit_dir, f"post_manifest_{int(time.time())}.json")
    with open(post_manifest_path, "w") as f:
        json.dump(current_manifest, f, indent=2)
    result.check_pass("Post-manifest created", post_manifest_path)

    return result


def run_content_audit(config) -> AuditResult:
    """Content audit: scan for misplaced content, orphans, issues."""
    result = AuditResult("content")
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")
    workspace = Path(workspace_root).resolve()
    template = getattr(config, "DIRECTORY_TEMPLATE", {})

    # Check 1: Root-level files that don't belong
    protected_files = set(getattr(config, "PROTECTED_FILES", []))
    root_items = os.listdir(workspace_root)
    root_files = [f for f in root_items if os.path.isfile(os.path.join(workspace_root, f))]
    unexpected_root = [f for f in root_files if f not in protected_files and not f.startswith(".")]

    if unexpected_root:
        result.check_warn(
            "Root-level files",
            f"{len(unexpected_root)} files at root: {', '.join(unexpected_root[:10])}"
        )
    else:
        result.check_pass("Root-level files", "All root files are protected or expected")

    # Check 2: Empty directories
    for root, dirs, files in os.walk(workspace_root):
        if not dirs and not files:
            rel = str(Path(root).relative_to(workspace))
            result.check_warn(f"Empty directory: {rel}")

    # Check 3: Duplicate filenames (potential confusion)
    from collections import Counter
    all_basenames = []
    for root, _, filenames in os.walk(workspace_root):
        all_basenames.extend(filenames)
    dupes = {name: count for name, count in Counter(all_basenames).items() if count > 1 and not name.startswith(".")}
    if dupes:
        top_dupes = sorted(dupes.items(), key=lambda x: -x[1])[:10]
        result.check_warn(
            "Duplicate filenames",
            "; ".join(f"{name} ({count}x)" for name, count in top_dupes)
        )
    else:
        result.check_pass("No duplicate filenames")

    # Check 4: Secrets in non-protected locations
    secrets_patterns = getattr(config, "SECRETS_PATTERNS", [])
    protected_dirs = set(getattr(config, "PROTECTED_DIRS", []))
    compiled = []
    for p in secrets_patterns:
        try:
            compiled.append(re.compile(p))
        except re.error:
            pass

    exposed = []
    for root, _, filenames in os.walk(workspace_root):
        rel_root = str(Path(root).relative_to(workspace))
        in_protected = any(rel_root.startswith(pd) for pd in protected_dirs)
        if in_protected:
            continue
        for fn in filenames:
            fp = Path(root) / fn
            if fp.stat().st_size > 10 * 1024 * 1024:
                continue
            try:
                content = fp.read_text(encoding="utf-8", errors="replace")
                for pat in compiled:
                    if pat.search(content):
                        exposed.append(str(fp.relative_to(workspace)))
                        break
            except OSError:
                pass

    if exposed:
        result.check_warn("Secrets in non-protected files", f"{len(exposed)} files: {', '.join(exposed[:5])}")
    else:
        result.check_pass("No secrets in non-protected files")

    return result


def main():
    parser = argparse.ArgumentParser(description="Forge Auditor — Pre/Post Verification")
    parser.add_argument("--config", required=True, help="Path to forge_config.py")
    parser.add_argument("--phase", required=True, choices=["pre", "post", "content"],
                        help="Audit phase: pre (before moves), post (after moves), content (deep scan)")
    parser.add_argument("--manifest", default=None, help="Pre-manifest path (for post-move comparison)")
    parser.add_argument("--output", default=None, help="Output report JSON path")
    args = parser.parse_args()

    config = load_config(args.config)

    if args.phase == "pre":
        result = run_pre_audit(config)
    elif args.phase == "post":
        result = run_post_audit(config, args.manifest)
    elif args.phase == "content":
        result = run_content_audit(config)
    else:
        print(f"Unknown phase: {args.phase}")
        sys.exit(1)

    result.print_report()

    # Save report
    output_dir = getattr(config, "OUTPUT_DIR", os.path.join(config.WORKSPACE_ROOT, "forge-output"))
    os.makedirs(output_dir, exist_ok=True)
    output_path = args.output or os.path.join(output_dir, f"audit_{args.phase}_{int(time.time())}.json")

    with open(output_path, "w") as f:
        json.dump(result.to_dict(), f, indent=2)

    print(f"\nAudit report written to: {output_path}")

    # Exit code reflects result
    sys.exit(1 if result.failures > 0 else 0)


if __name__ == "__main__":
    main()


path: forge_plan.py
content: #!/usr/bin/env python3
"""
Forge Planner — Reorganization Plan Generator
==============================================

Takes a scan report and configuration, produces:
- Hardline safety rules (generated before any moves)
- Ordered execution plan with dependency tracking
- Reference patch list (what updates after each move)
- Per-directory _README.md content
- Rollback procedures for every step

Usage:
    python3 forge_plan.py --config forge_config.py --scan-report scan_report.json
    python3 forge_plan.py --config forge_config.py --scan-report scan_report.json --output plan.json
"""

import argparse
import importlib.util
import json
import os
import re
import sys
import time
from collections import defaultdict
from datetime import datetime
from pathlib import Path


def load_config(config_path: str) -> object:
    # Security: validate path before executing config file
    if not os.path.isfile(config_path):
        print(f"ERROR: Config not found: {config_path}")
        sys.exit(1)
    if not config_path.endswith(".py"):
        print(f"ERROR: Config must be a .py file: {config_path}")
        sys.exit(1)
    if os.path.getsize(config_path) > 1_000_000:
        print(f"ERROR: Config file too large (>1MB): {config_path}")
        sys.exit(1)
    spec = importlib.util.spec_from_file_location("config", config_path)
    if spec is None or spec.loader is None:
        print(f"ERROR: Cannot load config from {config_path}")
        sys.exit(1)
    config = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(config)
    return config


def load_scan_report(path: str) -> dict:
    try:
        with open(path) as f:
            return json.load(f)
    except (OSError, json.JSONDecodeError) as e:
        print(f"ERROR: Cannot load scan report: {e}")
        sys.exit(1)


def generate_safety_rules(config, scan_report: dict) -> dict:
    """Generate hardline safety rules based on workspace state."""
    protected_files = set(getattr(config, "PROTECTED_FILES", []))
    protected_dirs = set(getattr(config, "PROTECTED_DIRS", []))

    # Auto-protect files referenced by running processes
    auto_protected = set()
    for proc in scan_report.get("running_processes", []):
        if proc.get("workspace_reference"):
            # Extract file paths from process command line
            line = proc.get("line", "")
            for f in scan_report.get("files", []):
                if os.path.basename(f["path"]) in line:
                    auto_protected.add(f["path"])

    # Auto-protect files with secrets warnings
    secrets_files = set()
    for w in scan_report.get("secrets_warnings", []):
        secrets_files.add(w["file"])

    rules = {
        "generated_at": datetime.now().isoformat(),
        "zero_deletion_policy": True,
        "require_backup": getattr(config, "REQUIRE_BACKUP_BEFORE_PLAN", True),
        "require_file_count_check": getattr(config, "REQUIRE_FILE_COUNT_CHECK", True),
        "max_moves_per_plan": getattr(config, "MAX_MOVES_PER_PLAN", 500),
        "max_move_percentage": getattr(config, "MAX_MOVE_PERCENTAGE", 80),
        "protected_files": sorted(protected_files),
        "protected_dirs": sorted(protected_dirs),
        "auto_protected_by_processes": sorted(auto_protected),
        "secrets_warning_files": sorted(secrets_files),
        "all_protected": sorted(protected_files | auto_protected),
        "rules": [
            "NEVER delete any file. Move to archive only.",
            "NEVER modify protected files. Read-only access only.",
            "ALWAYS create backup before executing any move.",
            "ALWAYS verify file count before and after moves.",
            "ALWAYS update references atomically with moves.",
            "NEVER move files referenced by running processes.",
            "ALWAYS review plan before execution. No auto-execute.",
            "If a file's purpose is unclear, move to inbox/ for triage.",
            "If a file contains secrets, flag but do not move.",
            "Rollback procedure must exist for every planned move.",
        ],
    }

    return rules


def classify_file(filepath: str, config) -> str:
    """Determine target directory for a file based on classification rules."""
    rules = getattr(config, "FILE_CLASSIFICATION_RULES", [])

    for pattern, target_dir in rules:
        # Simple glob matching
        if _glob_match(filepath, pattern):
            return target_dir

    return ""  # No classification — stays where it is


def _glob_match(filepath: str, pattern: str) -> bool:
    """Simple glob pattern matching."""
    import fnmatch
    basename = os.path.basename(filepath)
    return fnmatch.fnmatch(basename, pattern) or fnmatch.fnmatch(filepath, pattern)


def generate_moves(config, scan_report: dict, safety_rules: dict) -> list:
    """Generate the list of file moves needed."""
    template = getattr(config, "DIRECTORY_TEMPLATE", {})
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")
    all_protected = set(safety_rules["all_protected"])
    protected_dirs = set(safety_rules["protected_dirs"])

    moves = []
    references = scan_report.get("references", {})

    for f in scan_report.get("files", []):
        path = f["path"]

        # Skip protected
        if path in all_protected or f.get("protected"):
            continue

        # Skip files already in a template directory
        top_dir = path.split("/")[0] if "/" in path else ""
        if top_dir in template:
            continue

        # Skip files in protected directories
        in_protected = False
        for pd in protected_dirs:
            if path.startswith(pd + "/") or path == pd:
                in_protected = True
                break
        if in_protected:
            continue

        # Try classification rules
        target = classify_file(path, config)
        if target:
            new_path = os.path.join(target, os.path.basename(path))
            # Find references that need updating
            ref_updates = []
            for source, refs in references.items():
                if path in refs or os.path.basename(path) in refs:
                    ref_updates.append({
                        "file": source,
                        "old_ref": path,
                        "new_ref": new_path,
                    })

            moves.append({
                "source": path,
                "destination": new_path,
                "reason": f"Classification rule: {target}",
                "reference_updates": ref_updates,
                "rollback": f"mv '{new_path}' '{path}'",
            })

    return moves


def generate_directory_creation(config) -> list:
    """Generate list of directories to create."""
    template = getattr(config, "DIRECTORY_TEMPLATE", {})
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")

    dirs_to_create = []
    for dir_path, description in template.items():
        full_path = os.path.join(workspace_root, dir_path)
        if not os.path.exists(full_path):
            dirs_to_create.append({
                "path": dir_path,
                "description": description,
                "create_readme": getattr(config, "GENERATE_README_FILES", True),
            })

    return dirs_to_create


def generate_readmes(config) -> list:
    """Generate _README.md content for each template directory."""
    template = getattr(config, "DIRECTORY_TEMPLATE", {})
    readme_template = getattr(config, "README_TEMPLATE", "# {dir_name}\n\n{description}\n")

    readmes = []
    for dir_path, description in template.items():
        dir_name = os.path.basename(dir_path) or dir_path
        content = readme_template.format(
            dir_name=dir_name.replace("-", " ").title(),
            description=description,
            belongs_here=description,
        )
        readmes.append({
            "path": os.path.join(dir_path, "_README.md"),
            "content": content,
        })

    return readmes


def build_execution_order(moves: list, safety_rules: dict) -> list:
    """Order moves by dependency — files referenced by other moves go first."""
    # Build dependency graph
    move_sources = {m["source"] for m in moves}
    move_destinations = {m["destination"] for m in moves}

    # Separate moves into phases
    phases = []

    # Phase 0: Create directories
    phases.append({
        "phase": 0,
        "name": "Create directories",
        "description": "Create target directories before any file moves.",
    })

    # Phase 1: Create backup and manifest
    phases.append({
        "phase": 1,
        "name": "Backup and manifest",
        "description": "Create fresh backup and file manifest for rollback.",
    })

    # Phase 2: Files with no reference dependencies
    independent = []
    dependent = []
    for m in moves:
        has_deps = bool(m.get("reference_updates"))
        if has_deps:
            dependent.append(m)
        else:
            independent.append(m)

    phases.append({
        "phase": 2,
        "name": "Independent moves",
        "description": "Move files with no cross-references (safe, parallel-capable).",
        "moves": independent,
        "move_count": len(independent),
    })

    # Phase 3: Files with reference dependencies (move + patch atomically)
    phases.append({
        "phase": 3,
        "name": "Dependent moves with reference patches",
        "description": "Move files and update all references atomically.",
        "moves": dependent,
        "move_count": len(dependent),
    })

    # Phase 4: Post-move verification
    phases.append({
        "phase": 4,
        "name": "Post-move verification",
        "description": "Run forge_audit.py --phase post to verify everything.",
    })

    return phases


def validate_plan(moves: list, safety_rules: dict, scan_report: dict) -> list:
    """Validate the plan against safety rules. Return list of issues."""
    issues = []
    total_files = scan_report["summary"]["total_files"]
    max_moves = safety_rules["max_moves_per_plan"]
    max_pct = safety_rules["max_move_percentage"]

    if len(moves) > max_moves:
        issues.append({
            "severity": "ERROR",
            "message": f"Plan has {len(moves)} moves, exceeding limit of {max_moves}",
        })

    if total_files > 0:
        pct = (len(moves) / total_files) * 100
        if pct > max_pct:
            issues.append({
                "severity": "ERROR",
                "message": f"Plan moves {pct:.1f}% of files, exceeding {max_pct}% safety limit",
            })

    # Check for destination conflicts
    destinations = defaultdict(list)
    for m in moves:
        destinations[m["destination"]].append(m["source"])

    for dest, sources in destinations.items():
        if len(sources) > 1:
            issues.append({
                "severity": "ERROR",
                "message": f"Multiple files targeting same destination: {dest} ← {sources}",
            })

    # Check for moves of protected files (shouldn't happen, but verify)
    all_protected = set(safety_rules["all_protected"])
    for m in moves:
        if m["source"] in all_protected:
            issues.append({
                "severity": "CRITICAL",
                "message": f"Plan includes move of protected file: {m['source']}",
            })

    return issues


def generate_plan(config, scan_report: dict) -> dict:
    """Generate complete reorganization plan."""
    print("Generating safety rules...")
    safety_rules = generate_safety_rules(config, scan_report)

    print("Planning directory creation...")
    dirs = generate_directory_creation(config)

    print("Generating _README.md content...")
    readmes = generate_readmes(config)

    print("Planning file moves...")
    moves = generate_moves(config, scan_report, safety_rules)

    print("Building execution order...")
    execution_order = build_execution_order(moves, safety_rules)

    print("Validating plan...")
    issues = validate_plan(moves, safety_rules, scan_report)

    plan = {
        "generated_at": datetime.now().isoformat(),
        "workspace_root": getattr(config, "WORKSPACE_ROOT", "."),
        "safety_rules": safety_rules,
        "directories_to_create": dirs,
        "readmes_to_generate": readmes,
        "moves": moves,
        "execution_order": execution_order,
        "validation_issues": issues,
        "summary": {
            "directories_to_create": len(dirs),
            "readmes_to_generate": len(readmes),
            "files_to_move": len(moves),
            "reference_patches": sum(len(m.get("reference_updates", [])) for m in moves),
            "validation_errors": len([i for i in issues if i["severity"] in ("ERROR", "CRITICAL")]),
            "validation_warnings": len([i for i in issues if i["severity"] == "WARNING"]),
        },
        "rollback_command": (
            "# To rollback, restore from the pre-move backup:\n"
            f"# tar -xzf backups/forge-pre-move-*.tar.gz -C {getattr(config, 'WORKSPACE_ROOT', '.')}\n"
            "# Then verify with: python3 forge_audit.py --config forge_config.py --phase post"
        ),
    }

    return plan


def print_plan_summary(plan: dict):
    """Print human-readable plan summary."""
    s = plan["summary"]
    print("\n" + "=" * 60)
    print("FORGE REORGANIZATION PLAN")
    print("=" * 60)
    print(f"Workspace:             {plan['workspace_root']}")
    print(f"Directories to create: {s['directories_to_create']}")
    print(f"READMEs to generate:   {s['readmes_to_generate']}")
    print(f"Files to move:         {s['files_to_move']}")
    print(f"Reference patches:     {s['reference_patches']}")
    print(f"Validation errors:     {s['validation_errors']}")
    print(f"Validation warnings:   {s['validation_warnings']}")

    if plan["validation_issues"]:
        print("\n--- VALIDATION ISSUES ---")
        for issue in plan["validation_issues"]:
            print(f"  [{issue['severity']}] {issue['message']}")

    if s["validation_errors"] > 0:
        print("\n⛔ PLAN HAS ERRORS — Do not execute until resolved.")
    else:
        print("\n✅ Plan validated. Review the full plan before executing.")

    print("\n--- SAFETY RULES ---")
    for rule in plan["safety_rules"]["rules"]:
        print(f"  • {rule}")

    if plan["moves"]:
        print(f"\n--- PLANNED MOVES (showing first 20 of {len(plan['moves'])}) ---")
        for m in plan["moves"][:20]:
            patches = len(m.get("reference_updates", []))
            patch_note = f" (+{patches} ref patches)" if patches else ""
            print(f"  {m['source']} → {m['destination']}{patch_note}")
        if len(plan["moves"]) > 20:
            print(f"  ... and {len(plan['moves']) - 20} more")

    print("\n" + "=" * 60)


def main():
    parser = argparse.ArgumentParser(description="Forge Planner — Reorganization Plan Generator")
    parser.add_argument("--config", required=True, help="Path to forge_config.py")
    parser.add_argument("--scan-report", required=True, help="Path to scan_report.json")
    parser.add_argument("--output", default=None, help="Output plan JSON path")
    args = parser.parse_args()

    config = load_config(args.config)
    scan_report = load_scan_report(args.scan_report)

    plan = generate_plan(config, scan_report)
    print_plan_summary(plan)

    output_dir = getattr(config, "OUTPUT_DIR", os.path.join(config.WORKSPACE_ROOT, "forge-output"))
    os.makedirs(output_dir, exist_ok=True)
    output_path = args.output or os.path.join(output_dir, "reorg_plan.json")

    with open(output_path, "w") as f:
        json.dump(plan, f, indent=2, default=str)

    print(f"\nFull plan written to: {output_path}")
    print("Review the plan carefully before executing any moves.")


if __name__ == "__main__":
    main()


path: forge_scan.py
content: #!/usr/bin/env python3
"""
Forge Scanner — Workspace Analysis Tool
========================================

Scans an AI agent workspace and produces a comprehensive report:
- File inventory (paths, sizes, types, modification times)
- Directory structure map
- Cross-reference graph (which files mention which)
- Broken references (mentioned files that don't exist)
- Running process dependencies
- Hardcoded path detection
- Secrets exposure warnings

Usage:
    python3 forge_scan.py --config forge_config.py
    python3 forge_scan.py --config forge_config.py --output scan_report.json
"""

import argparse
import importlib.util
import json
import os
import re
import shlex
import subprocess
import sys
import time
from collections import defaultdict
from datetime import datetime
from pathlib import Path


def load_config(config_path: str) -> object:
    """Load configuration from a Python file."""
    # Security: validate path before executing config file
    if not os.path.isfile(config_path):
        print(f"ERROR: Config not found: {config_path}")
        sys.exit(1)
    if not config_path.endswith(".py"):
        print(f"ERROR: Config must be a .py file: {config_path}")
        sys.exit(1)
    if os.path.getsize(config_path) > 1_000_000:
        print(f"ERROR: Config file too large (>1MB): {config_path}")
        sys.exit(1)
    spec = importlib.util.spec_from_file_location("config", config_path)
    if spec is None or spec.loader is None:
        print(f"ERROR: Cannot load config from {config_path}")
        sys.exit(1)
    config = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(config)
    return config


def scan_files(workspace_root: str, protected_dirs: list) -> list:
    """Walk workspace and collect file metadata."""
    files = []
    workspace = Path(workspace_root).resolve()

    protected_resolved = set()
    for pd in protected_dirs:
        p = (workspace / pd).resolve()
        protected_resolved.add(p)

    for root, dirs, filenames in os.walk(workspace):
        root_path = Path(root).resolve()

        # Skip protected directories
        skip = False
        for pd in protected_resolved:
            if root_path == pd or pd in root_path.parents:
                skip = True
                break
        if skip:
            # Still record the directory exists, but mark as protected
            for fn in filenames:
                fp = root_path / fn
                try:
                    stat = fp.stat()
                    files.append({
                        "path": str(fp.relative_to(workspace)),
                        "size": stat.st_size,
                        "modified": stat.st_mtime,
                        "extension": fp.suffix.lower(),
                        "protected": True,
                    })
                except OSError:
                    pass
            continue

        for fn in filenames:
            fp = root_path / fn
            try:
                stat = fp.stat()
                files.append({
                    "path": str(fp.relative_to(workspace)),
                    "size": stat.st_size,
                    "modified": stat.st_mtime,
                    "extension": fp.suffix.lower(),
                    "protected": False,
                })
            except OSError:
                pass

    return files


def scan_directories(workspace_root: str) -> list:
    """Map directory structure."""
    dirs = []
    workspace = Path(workspace_root).resolve()

    for root, dirnames, filenames in os.walk(workspace):
        root_path = Path(root).resolve()
        rel = str(root_path.relative_to(workspace))
        if rel == ".":
            rel = ""
        dirs.append({
            "path": rel,
            "file_count": len(filenames),
            "subdir_count": len(dirnames),
            "has_readme": "_README.md" in filenames or "README.md" in filenames,
        })

    return dirs


def find_references(workspace_root: str, files: list, config) -> dict:
    """Scan files for cross-references to other files."""
    workspace = Path(workspace_root).resolve()
    extensions = set(getattr(config, "REFERENCE_SCAN_EXTENSIONS", [".md", ".txt", ".py", ".sh", ".json"]))
    patterns = getattr(config, "REFERENCE_PATTERNS", [])
    max_size = getattr(config, "MAX_SCAN_FILE_SIZE_MB", 10) * 1024 * 1024

    # Build a set of known file basenames and paths for matching
    known_files = set()
    known_basenames = defaultdict(list)
    for f in files:
        known_files.add(f["path"])
        basename = os.path.basename(f["path"])
        known_basenames[basename].append(f["path"])

    compiled_patterns = []
    for p in patterns:
        try:
            compiled_patterns.append(re.compile(p))
        except re.error as e:
            print(f"WARNING: Invalid regex pattern '{p}': {e}")

    references = defaultdict(set)  # source_file -> set of referenced files
    broken_refs = defaultdict(set)  # source_file -> set of unresolved references

    for f in files:
        if f["protected"]:
            continue
        if f["extension"] not in extensions:
            continue
        if f["size"] > max_size:
            continue

        filepath = workspace / f["path"]
        try:
            content = filepath.read_text(encoding="utf-8", errors="replace")
        except OSError:
            continue

        found_refs = set()

        # Pattern-based reference detection
        for pat in compiled_patterns:
            for match in pat.finditer(content):
                ref = match.group(1) if match.lastindex else match.group(0)
                ref = ref.strip("\"'`()[] ")
                if ref:
                    found_refs.add(ref)

        # Simple basename detection — look for known filenames in content
        for basename, paths in known_basenames.items():
            if len(basename) > 4 and basename in content:  # Skip very short names
                for p in paths:
                    if p != f["path"]:
                        found_refs.add(basename)

        # Resolve references
        source_dir = os.path.dirname(f["path"])
        for ref in found_refs:
            resolved = False

            # Try as-is (relative to workspace root)
            if ref in known_files:
                references[f["path"]].add(ref)
                resolved = True

            # Try relative to source file's directory
            if not resolved:
                rel_path = os.path.normpath(os.path.join(source_dir, ref))
                if rel_path in known_files:
                    references[f["path"]].add(rel_path)
                    resolved = True

            # Try basename match
            if not resolved:
                basename = os.path.basename(ref)
                if basename in known_basenames:
                    for p in known_basenames[basename]:
                        references[f["path"]].add(p)
                    resolved = True

            # If it looks like a file path but doesn't resolve, it's broken
            if not resolved and re.search(r'\.\w{1,5}$', ref):
                broken_refs[f["path"]].add(ref)

    # Convert sets to lists for JSON serialization
    return {
        "references": {k: sorted(v) for k, v in references.items()},
        "broken_references": {k: sorted(v) for k, v in broken_refs.items()},
    }


def detect_secrets(workspace_root: str, files: list, config) -> list:
    """Scan for potential secrets exposure."""
    workspace = Path(workspace_root).resolve()
    patterns = getattr(config, "SECRETS_PATTERNS", [])
    max_size = getattr(config, "MAX_SCAN_FILE_SIZE_MB", 10) * 1024 * 1024

    compiled = []
    for p in patterns:
        try:
            compiled.append((p, re.compile(p)))
        except re.error:
            pass

    warnings = []
    for f in files:
        if f["size"] > max_size:
            continue
        filepath = workspace / f["path"]
        try:
            content = filepath.read_text(encoding="utf-8", errors="replace")
        except OSError:
            continue

        for pattern_str, pat in compiled:
            matches = pat.findall(content)
            if matches:
                warnings.append({
                    "file": f["path"],
                    "pattern": pattern_str,
                    "match_count": len(matches),
                    "protected": f["protected"],
                })
                break  # One warning per file is enough

    return warnings


def detect_processes(config) -> list:
    """Check for running processes that might depend on workspace files."""
    if not getattr(config, "DETECT_RUNNING_PROCESSES", True):
        return []

    commands = getattr(config, "PROCESS_CHECK_COMMANDS", ["ps aux"])
    workspace_root = getattr(config, "WORKSPACE_ROOT", "")

    processes = []
    for cmd in commands:
        try:
            cmd_list = shlex.split(cmd) if isinstance(cmd, str) else list(cmd)
            result = subprocess.run(
                cmd_list, capture_output=True, text=True, timeout=10
            )
            for line in result.stdout.splitlines():
                if workspace_root and workspace_root in line:
                    processes.append({
                        "command": cmd,
                        "line": line.strip(),
                        "workspace_reference": True,
                    })
        except (subprocess.TimeoutExpired, OSError) as e:
            processes.append({
                "command": cmd,
                "error": str(e),
            })

    return processes


def generate_report(config) -> dict:
    """Generate complete workspace scan report."""
    workspace_root = getattr(config, "WORKSPACE_ROOT", ".")
    protected_dirs = getattr(config, "PROTECTED_DIRS", [])

    print(f"Scanning workspace: {workspace_root}")

    print("  Collecting files...")
    files = scan_files(workspace_root, protected_dirs)

    print("  Mapping directories...")
    directories = scan_directories(workspace_root)

    print("  Finding cross-references...")
    ref_data = find_references(workspace_root, files, config)

    print("  Checking for secrets...")
    secrets = detect_secrets(workspace_root, files, config)

    print("  Detecting running processes...")
    processes = detect_processes(config)

    # Summary statistics
    total_files = len(files)
    protected_count = sum(1 for f in files if f["protected"])
    total_size = sum(f["size"] for f in files)
    extensions = defaultdict(int)
    for f in files:
        extensions[f["extension"]] += 1

    root_files = [f for f in files if os.sep not in f["path"] and "/" not in f["path"]]

    report = {
        "generated_at": datetime.now().isoformat(),
        "workspace_root": workspace_root,
        "summary": {
            "total_files": total_files,
            "protected_files": protected_count,
            "movable_files": total_files - protected_count,
            "total_size_bytes": total_size,
            "total_size_mb": round(total_size / (1024 * 1024), 2),
            "total_directories": len(directories),
            "root_level_files": len(root_files),
            "file_types": dict(sorted(extensions.items(), key=lambda x: -x[1])),
            "broken_reference_count": sum(len(v) for v in ref_data["broken_references"].values()),
            "secrets_warnings": len(secrets),
            "active_processes": len([p for p in processes if p.get("workspace_reference")]),
        },
        "files": files,
        "directories": directories,
        "references": ref_data["references"],
        "broken_references": ref_data["broken_references"],
        "secrets_warnings": secrets,
        "running_processes": processes,
        "root_files": [f["path"] for f in root_files],
    }

    return report


def print_summary(report: dict):
    """Print human-readable summary to stdout."""
    s = report["summary"]
    print("\n" + "=" * 60)
    print("FORGE SCAN REPORT")
    print("=" * 60)
    print(f"Workspace:          {report['workspace_root']}")
    print(f"Total files:        {s['total_files']}")
    print(f"Protected files:    {s['protected_files']}")
    print(f"Movable files:      {s['movable_files']}")
    print(f"Total size:         {s['total_size_mb']} MB")
    print(f"Directories:        {s['total_directories']}")
    print(f"Root-level files:   {s['root_level_files']}")
    print(f"Broken references:  {s['broken_reference_count']}")
    print(f"Secrets warnings:   {s['secrets_warnings']}")
    print(f"Active processes:   {s['active_processes']}")

    if report["broken_references"]:
        print("\n--- BROKEN REFERENCES ---")
        for source, refs in report["broken_references"].items():
            for ref in refs:
                print(f"  {source} → {ref} (NOT FOUND)")

    if report["secrets_warnings"]:
        print("\n--- SECRETS WARNINGS ---")
        for w in report["secrets_warnings"]:
            prot = " [PROTECTED]" if w["protected"] else ""
            print(f"  {w['file']}: {w['pattern']} ({w['match_count']} matches){prot}")

    if report["running_processes"]:
        procs = [p for p in report["running_processes"] if p.get("workspace_reference")]
        if procs:
            print("\n--- ACTIVE PROCESSES USING WORKSPACE ---")
            for p in procs:
                print(f"  {p['line']}")

    print("\n" + "=" * 60)


def main():
    parser = argparse.ArgumentParser(description="Forge Scanner — Workspace Analysis")
    parser.add_argument("--config", required=True, help="Path to forge_config.py")
    parser.add_argument("--output", default=None, help="Output JSON report path (default: forge-output/scan_report.json)")
    args = parser.parse_args()

    config = load_config(args.config)
    report = generate_report(config)
    print_summary(report)

    # Write JSON report
    output_dir = getattr(config, "OUTPUT_DIR", os.path.join(config.WORKSPACE_ROOT, "forge-output"))
    os.makedirs(output_dir, exist_ok=True)
    output_path = args.output or os.path.join(output_dir, "scan_report.json")

    with open(output_path, "w") as f:
        json.dump(report, f, indent=2, default=str)

    print(f"\nFull report written to: {output_path}")


if __name__ == "__main__":
    main()


path: LICENSE.md
content: MIT License

Copyright (c) 2026 Forge Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


path: LIMITATIONS.md
content: # Forge — Limitations

## What Forge Does NOT Do

### No Auto-Execution
Forge generates plans. It does not execute them. You read the plan, you move the files. This is a deliberate safety choice, not a missing feature.

### No Platform-Specific Integration
Forge works with files and directories. It does not integrate with:
- Cron job management (you update cron jobs manually based on the plan)
- Git operations (commit/push after reorg is your responsibility)
- Cloud storage sync
- Database state files
- Container or VM management

### No Content Understanding
Forge classifies files by name, extension, and location — not by reading and understanding their contents. It won't know that a file named `notes.md` contains trading strategies that belong in a `trading/` directory unless you configure a classification rule for it.

### No Conflict Resolution
If two files have the same name and both need to move to the same directory, Forge flags the conflict. It does not resolve it. You decide which file wins.

### No Incremental Mode
Each scan is a full workspace scan. There is no "only scan files changed since last run." For large workspaces (10,000+ files), scans may take several minutes.

### No Undo Button
Forge creates manifests and rollback commands, but there is no single "undo" command. Rolling back requires restoring from the pre-move backup. Always verify your backup before executing a plan.

## When NOT to Use Forge

- **During active incidents.** If your agent is mid-task or a live system is unstable, don't reorganize.
- **Without a backup.** Forge enforces this, but if you override it, you're on your own.
- **On workspaces you don't own.** Forge moves files. Make sure you have permission.
- **As a substitute for understanding your workspace.** Forge is a tool, not a consultant. If you don't know what your files do, scanning them won't tell you. Read them first.

## Known Edge Cases

1. **Symlinks:** Forge follows symlinks during scanning but does not create or preserve them during moves. If your workspace relies on symlinks, verify them manually after reorganization.

2. **Binary files:** Reference scanning only works on text files. Binary files are inventoried but not scanned for cross-references.

3. **Very long paths:** Some operating systems have path length limits. Forge does not check for this. Deep directory nesting may cause issues on Windows.

4. **Concurrent access:** If another process modifies files during a scan, the scan results may be inconsistent. Run scans during quiet periods.

5. **Unicode filenames:** Forge handles UTF-8 filenames. Other encodings may cause issues on some platforms.

## Assumptions

- Python 3.8+ is available
- The workspace is on a local filesystem (not a network mount with high latency)
- You have read/write permissions to the workspace
- Files are small enough to read into memory for reference scanning (configurable limit, default 10MB)


path: README.md
content: # Forge — AI Workspace Architect

**Systematic workspace reorganization for AI agent users.**

Forge scans your AI agent's workspace, builds a safety-first reorganization plan, executes it with zero data loss, and verifies everything works afterward. Built from a battle-tested methodology used on production AI workspaces.

---

## The Problem

AI agent workspaces accumulate chaos fast. Files pile up at root. Memory files reference moved documents. Cron jobs point to old paths. Config files drift. One bad reorganization can brick your agent's memory, break running processes, or corrupt critical state files.

Manual cleanup is dangerous. Automated cleanup without safeguards is worse.

## What Forge Does

### 1. **Scan** (`forge_scan.py`)
- Maps your entire workspace: files, directories, sizes, types
- Discovers all cross-references (file A mentions file B)
- Finds broken references (file A mentions file B, but B doesn't exist)
- Identifies running processes and their file dependencies
- Detects hardcoded paths in scripts and configs
- Catalogs cron jobs, scheduled tasks, and their path dependencies
- Outputs a complete workspace report

### 2. **Plan** (`forge_plan.py`)
- Generates a reorganization plan based on your configured directory template
- Creates hardline safety rules before touching anything
- Produces an ordered execution sequence with dependency tracking
- Identifies protected files that must never be modified
- Plans reference patches (what needs updating after moves)
- Generates rollback procedures for every step
- Outputs a human-reviewable plan — nothing executes automatically

### 3. **Audit** (`forge_audit.py`)
- **Pre-move audit:** Verifies backup exists, creates file manifest, checks all protected files
- **Post-move audit:** Compares file counts, verifies no files lost, checks all references resolve, validates protected files unchanged
- **Content audit:** Scans for misplaced content, secrets in wrong locations, orphaned files
- **Reference audit:** Verifies every cross-reference in every file still resolves

---

## Quick Start

```bash
# 1. Configure your workspace
cp config_example.py forge_config.py
# Edit forge_config.py with your workspace path, protected files, directory template

# 2. Scan your workspace
python3 forge_scan.py --config forge_config.py

# 3. Generate a reorganization plan
python3 forge_plan.py --config forge_config.py --scan-report scan_report.json

# 4. Review the plan (ALWAYS review before executing)
cat reorg_plan.json

# 5. Run pre-move audit
python3 forge_audit.py --config forge_config.py --phase pre

# 6. Execute the plan (you do this manually or with your own script)
# Forge generates the plan. YOU execute it. This is deliberate.

# 7. Run post-move audit
python3 forge_audit.py --config forge_config.py --phase post --manifest pre_manifest.txt
```

## Design Philosophy

### Safety First, Always
- **Never deletes files.** Moves only. If something looks like garbage, it goes to archive.
- **Never modifies protected files.** You define what's protected; Forge won't touch them.
- **Backup before changes.** Forge refuses to generate a plan without a verified backup.
- **Human reviews the plan.** Forge doesn't auto-execute. You read the plan, you decide.
- **Rollback for every step.** Every move in the plan includes how to undo it.

### Zero-Deletion Policy
No file is ever deleted during reorganization. Files are moved, never removed. Even files that appear to be duplicates are preserved until you explicitly confirm deletion.

### Atomic Reference Updates
When a file moves, every reference to that file must update in the same logical step. Forge tracks these as paired operations: move file + update references = one unit. If the reference update fails, the move should be rolled back.

### Framework, Not Opinions
Forge doesn't dictate your directory structure. You configure the template. Forge handles the mechanics of getting there safely.

---

## Use Cases

- **New AI agent setup:** Start with a clean, structured workspace from day one
- **Workspace recovery:** Untangle months of accumulated file chaos
- **Platform migration:** Moving from one AI agent platform to another
- **Team onboarding:** Standardize workspace structure across multiple agents
- **Pre-deployment cleanup:** Verify workspace integrity before going live
- **Post-incident audit:** Check what broke and what's still intact after a failure

## What's Included

| File | Purpose |
|------|---------|
| `forge_scan.py` | Workspace scanner and reference mapper |
| `forge_plan.py` | Reorganization planner with safety rules |
| `forge_audit.py` | Pre/post audit verification |
| `config_example.py` | Configuration template |
| `LIMITATIONS.md` | What Forge doesn't do |
| `LICENSE` | MIT License |

## Requirements

- Python 3.8+
- No external dependencies (stdlib only)
- Works on any OS (Linux, macOS, Windows)
- Platform-agnostic (OpenClaw, AutoGPT, CrewAI, or any file-based workspace)

## Configuration

See `config_example.py` for the full configuration reference. Key settings:

- **`WORKSPACE_ROOT`** — Path to your workspace
- **`PROTECTED_FILES`** — Files that must never be modified or moved
- **`PROTECTED_DIRS`** — Directories that must never be touched
- **`DIRECTORY_TEMPLATE`** — Your desired directory structure
- **`BACKUP_DIR`** — Where backups are stored
- **`ARCHIVE_DIR`** — Where "deleted" files actually go
- **`README_TEMPLATE`** — Content template for per-directory `_README.md` files

---

## quality-verified


## License

MIT — See `LICENSE` file.


---

## ⚠️ Disclaimer

This software is provided "AS IS", without warranty of any kind, express or implied.

**USE AT YOUR OWN RISK.**

- The author(s) are NOT liable for any damages, losses, or consequences arising from 
  the use or misuse of this software — including but not limited to financial loss, 
  data loss, security breaches, business interruption, or any indirect/consequential damages.
- This software does NOT constitute financial, legal, trading, or professional advice.
- Users are solely responsible for evaluating whether this software is suitable for 
  their use case, environment, and risk tolerance.
- No guarantee is made regarding accuracy, reliability, completeness, or fitness 
  for any particular purpose.
- The author(s) are not responsible for how third parties use, modify, or distribute 
  this software after purchase.

By downloading, installing, or using this software, you acknowledge that you have read 
this disclaimer and agree to use the software entirely at your own risk.


**DATA DISCLAIMER:** This software processes and stores data locally on your system. 
The author(s) are not responsible for data loss, corruption, or unauthorized access 
resulting from software bugs, system failures, or user error. Always maintain 
independent backups of important data. This software does not transmit data externally 
unless explicitly configured by the user.

---

## Support & Links

| | |
|---|---|
| 🐛 **Bug Reports** | [REDACTED_SECRET] |
| ☕ **Ko-fi** | [ko-fi.com/theshadowrose](https://ko-fi.com/theshadowrose) |
| 🛒 **Gumroad** | [shadowyrose.gumroad.com](https://shadowyrose.gumroad.com) |
| 🐦 **Twitter** | [@TheShadowyRose](https://twitter.com/TheShadowyRose) |
| 🐙 **GitHub** | [github.com/TheShadowRose](https://github.com/TheShadowRose) |
| 🧠 **PromptBase** | [promptbase.com/profile/shadowrose](https://promptbase.com/profile/shadowrose) |

*Built with [OpenClaw](https://github.com/openclaw/openclaw) — thank you for making this possible.*

---

🛠️ **Need something custom?** Custom OpenClaw agents & skills starting at $500. If you can describe it, I can build it. → [Hire me on Fiverr](https://www.fiverr.com/s/jjmlZ0v)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)