Skill: billwanttobetop/ai-project-scaffold (version 1.0.1)

--- name: ai-project-scaffold description: "为 AI 模型项目创建标准化目录结构：覆盖数据→分词→模型→实验全链路，支持多方案并行、来源追踪、质量记录。内置健康检查、追踪链、实验对比、checkpoint管理。严格文档规范防止 AI 生成冗余文档。" --- # AI 项目目录结构管理 v2.0 适用于涉及数据清洗、分词/Tokenization、模型训练、后训练、强化学习等完整 AI 管线的项目。 ## 🔴 文档铁律（AI 助手必须遵守） > ⚠️ 这是本 skill 最重要的部分。违反任何一条都应被视为严重错误。 ### 原则 1. **表 > 文** — 能用表格记录的，不写段落。表格可扫描、可对比、不冗余。 2. **记录决策，不写教程** — 文档回答"做了什么选择、为什么"，不写"怎么用、是什么"。 3. **一处一真理** — 同一信息绝不出现在两个文件里。信息有且仅有一个权威位置。 4. **日志即文档** — `experiment_log.md` 是唯一持续增长的文档。其他 README 只在结构变化时更新。 5. **写完删一半** — 写完任何文档后，删掉至少一半字数。留下的才是精华。 ### 禁止清单 | ❌ 绝对禁止 | ✅ 应该做 | |-------------|----------| | 写超过 200 字的段落说明 | 用表格或列表，每行 ≤ 一行 | | 在 README 里粘贴代码 | 代码放在 scripts/，README 只写脚本名+用途 | | 创建 `GUIDE.md`、`TUTORIAL.md`、`NOTES.md` 等随意命名的文档 | 只使用约定文件名 | | 重复其他文件已有的信息 | 用链接引用（如 `详见 01_data/README.md`） | | 自动生成"项目概述""背景介绍"等水字数的章节 | 空着也比写废话强 | | 写 `detailed_explanation_v2_final.md` | 一个主题一个文件，版本在文件名里（如 `token_eval_v2.md`） | | 用 `final`、`new`、`latest` 等模糊后缀 | 用数字版本号 v1, v2, v3 | ### 文档命名规范 ``` ✅ 好命名: architecture_decision_001_transformer.md # ADR 风格 qc_report_wikipedia_zh_2024-03-15.md # 数据集_日期 eval_benchmark_mmlu_v2.md # 主题_版本 ❌ 坏命名: notes.md # 太模糊 final_report.md # 哪个 final？ test.md # 测什么？ new_doc_v3_final_revised.md # 💀 ``` ### 允许的文档类型及字数上限 | 文档 | 最大字数 | 更新频率 | 说明 | |------|---------|---------|------| | 各层 `README.md` | 300 字 | 结构变化时 | 表格为主，不写段落 | | `experiment_log.md` | 不限（按条目追加） | 每次实验 | 唯一允许持续增长的文件 | | `metadata.yaml` | 不限 | 数据源变更时 | 结构化数据，不是散文 | | `eval.md`（分词方案下） | 500 字 | 评估完成后 | 表格 + 关键数字 | | `architecture_decision_*.md` | 500 字 | 做决策时 | 背景+选项+决策+后果 | | `qc_report_*.md` | 300 字 | 每次 QC | 数据集+问题+处理+结果 | | `meeting_*.md` | 200 字 | 每次会议 | 日期+议题+结论，不要流水账 | 超过字数上限的文档应拆分为多个文件，或用表格代替段落。 --- ## 设计理念 1. **分层管理** — 数据层 / 分词层 / 模型层 各自独立，互不污染 2. **多方案并行** — 每层支持多种技术方案/尝试并存，各自有独立空间 3. **来源可追溯** — 数据从哪来、经过什么处理、用到了哪个模型，全程可追踪 4. **质量可见** — 数据质量、分词效果、模型评估都有独立的记录空间 5. **文档极简** — 遵循上方🔴铁律，拒绝 AI 生成冗余文档 --- ## 目录结构 ``` <project_name>/ ├── README.md # 项目总览（≤300字） ├── _project_config.yaml # 项目元信息 ├── .gitignore # 自动生成，忽略 checkpoints/logs/数据/.env ├── scripts/ # 管理脚本（自动复制） │ ├── check_project.py # 健康检查 │ ├── trace.py # 追踪链生成 │ ├── compare_runs.py # 实验对比 │ └── checkpoint_mgmt.py # Checkpoint 管理 │ ├── 01_data/ # 📊 数据层 │ ├── README.md # 数据来源登记表（表格，≤300字） │ ├── 01_raw/ # 原始数据（按来源分目录） │ │ └── source_<name>/ │ │ ├── metadata.yaml # 结构化元信息 │ │ └── data/ │ ├── 02_qc/ # 质量控制 │ │ ├── scripts/ # QC 脚本 │ │ ├── reports/ # QC 报告（命名: qc_report_<dataset>_YYYY-MM-DD.md） │ │ └── cleaned/ # 清洗后数据 │ ├── 03_processed/ # 处理后 │ │ ├── text/ │ │ ├── image/ │ │ ├── audio/ │ │ └── multimodal/ # 多模态细化 │ │ ├── image_text/ │ │ ├── video_text/ │ │ ├── audio_text/ │ │ └── interleaved/ │ └── 04_quality_log.md # 数据质量追踪日志（表格追加） │ ├── 02_tokenization/ # 🔤 分词层 │ ├── README.md # 分词策略总览（表格，≤300字） │ ├── 01_existing/ # 现成 tokenizer │ │ └── <tokenizer_name>/ │ │ ├── config.yaml │ │ └── notes.md # 选用理由（≤150字） │ ├── 02_custom/ # 自制 │ │ └── <approach_name>/ │ │ ├── README.md # 方案说明（≤200字） │ │ ├── scripts/ │ │ ├── configs/ │ │ ├── vocab_output/ │ │ └── eval.md # 评估结果（表格，≤500字） │ ├── 03_training/ # token 相关训练 │ ├── 04_evaluation/ # 内在 + 下游评估 │ │ ├── intrinsic/ │ │ └── extrinsic/ │ └── 05_generated/ # 分词产生的新数据记录 │ ├── 03_models/ # 🧠 模型层 │ ├── README.md # 架构选型决策（表格，≤300字） │ ├── 01_pretraining/ │ │ └── <arch_name>/<run_name>/ │ │ ├── configs/ # 训练配置 │ │ ├── scripts/ # 训练脚本 │ │ ├── checkpoints/ # gitignored │ │ ├── logs/ # gitignored │ │ └── eval/ │ ├── 02_post_training/ │ │ ├── sft/ # 监督微调 │ │ ├── dpo/ # DPO │ │ ├── orpo/ # ORPO │ │ ├── kto/ # KTO │ │ ├── simpo/ # SimPO │ │ └── rlhf/ # RLHF 流程 │ ├── 03_rl/ │ │ ├── ppo/ │ │ ├── grpo/ │ │ └── reward_models/ │ ├── 04_evaluation/ │ │ ├── benchmarks/ │ │ └── reports/ │ └── 05_deployment/ │ ├── 04_experiments/ # 🔬 实验管理 │ ├── experiment_log.md # 时间线日志（唯一可无限增长的文件） │ └── runs/ │ ├── 05_docs/ # 📚 文档（严格遵守铁律） │ ├── design/ # 架构决策记录（ADR 风格） │ ├── meetings/ # 会议记录（命名: meeting_YYYY-MM-DD_主题.md） │ └── references/ # 外部参考文献 │ └── 06_shared/ # 🔧 共享资源 ├── scripts/ # 通用脚本 └── configs/ ├── training/ # DeepSpeed/LoRA/QLoRA/WandB 模板 ├── data/ # 数据处理配置模板 └── deployment/ # 部署配置模板 ``` --- ## 🛠 管理工具 ### 新建项目 ```bash python3 init_ai_project.py --name "MyLLM" [--output /path/to/parent] ``` 自动生成：完整目录 + 各层 README 模板 + `.gitignore` + 5 个配置模板 + 管理脚本副本 ### 添加组件 ```bash # 数据来源（自动生成 metadata.yaml） --add-source --source <名称> # 分词方案（自动创建 scripts/configs/vocab/eval） --add-tokenizer --approach <方案名> # 预训练运行 --add-training --arch <架构> --run <运行名> # 后训练运行 --add-posttrain --method <sft/dpo/orpo/kto/simpo/rlhf> --run <运行名> # RL 运行 --add-rl --method <ppo/grpo/...> --run <运行名> ``` ### 升级已有项目 ```bash python3 init_ai_project.py --name "MyLLM" --upgrade # 补充缺失目录 + .gitignore + 配置模板 + 管理脚本 ``` ### 健康检查 ```bash python3 check_project.py --name "MyLLM" # 扫描: 目录完整度 / metadata 字段 / eval 是否填写 / 日志是否更新 # 输出: 健康评分 0-100 + 具体问题清单 ``` ### 追踪链 ```bash python3 trace.py --name "MyLLM" # 全局追踪 python3 trace.py --name "MyLLM" --layer models # 只看模型层 python3 trace.py --name "MyLLM" --full # 含文件详情 ``` ### 实验对比 ```bash python3 compare_runs.py --name "MyLLM" # 全部对比 python3 compare_runs.py --name "MyLLM" --arch llama3 # 指定架构 python3 compare_runs.py --name "MyLLM" --stage post_training # 只看后训练 ``` ### Checkpoint 管理 ```bash python3 checkpoint_mgmt.py --name "MyLLM" list # 列出所有 python3 checkpoint_mgmt.py --name "MyLLM" clean --keep 3 --dry-run # 预览清理 python3 checkpoint_mgmt.py --name "MyLLM" clean --keep 3 # 每运行保留3个最新 python3 checkpoint_mgmt.py --name "MyLLM" archive --to /backup/ # 归档 python3 checkpoint_mgmt.py --name "MyLLM" stats # 磁盘统计 ``` --- ## 命名规范 | 层级 | 目录命名 | 示例 | |------|---------|------| | 数据来源 | `source_<描述>` | `source_wikipedia_zh`, `source_internal_docs` | | QC 报告 | `qc_report_<dataset>_YYYY-MM-DD.md` | `qc_report_wikipedia_2024-03-15.md` | | 分词方案 | `<方法>_v<版本>` | `bpe_v1`, `unigram_v2`, `sentencepiece_v1` | | 模型架构 | `<架构名>` | `llama3`, `qwen2`, `custom_moe` | | 训练运行 | `<描述>_v<版本>` | `baseline_v1`, `lr_3e4_v2`, `wd_0.1_v1` | | 后训练方法 | `<缩写>` | `sft`, `dpo`, `ppo`, `grpo` | | 会议记录 | `meeting_YYYY-MM-DD_<主题>.md` | `meeting_2024-03-15_arch_review.md` | | 设计决策 | `architecture_decision_<序号>_<主题>.md` | `architecture_decision_001_transformer.md` | --- ## 追踪链 每个阶段记录输入/输出依赖，形成完整链路： ``` source_wikipedia_zh (01_data/01_raw/) → qc_report_wikipedia_2024-03-15.md (01_data/02_qc/reports/) → bpe_v1 (02_tokenization/02_custom/) → eval.md 评估通过 → llama3/baseline_v1 (03_models/01_pretraining/) → sft/chat_v1 (03_models/02_post_training/) → ppo/reward_v2 (03_models/03_rl/) ``` > 💡 运行 `python3 scripts/trace.py --name <项目>` 自动生成追踪链。 --- ## AI 助手行为准则 使用本 skill 时，AI 助手必须: 1. **初始化项目时** — 只生成模板骨架，不填充假数据。空表格比假数据好。 2. **写 README 时** — 表格 > 列表 > 段落。默认用表格。 3. **记录实验时** — 追加到 `experiment_log.md`，用模板格式，不要新建文件。 4. **写评估报告时** — 数字和表格是主角，文字只是表头。 5. **不要做的事**: - ❌ 不要创建 `05_docs/` 下的随意命名的 `.md` 文件 - ❌ 不要在多个 README 里重复同一段话 - ❌ 不要写"本项目旨在..."这种水字数开头 - ❌ 不要生成超过 300 字的 README - ❌ 不要用 `final`、`new`、`latest`、`updated` 命名文件

path: scripts/check_project.py
content: #!/usr/bin/env python3
"""
AI 项目健康检查脚本
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
扫描项目目录，检查：
  - 数据来源是否都有 metadata.yaml
  - 分词方案是否都有 eval.md
  - 训练运行是否有日志
  - 实验日志是否更新
  - metadata.yaml 字段完整度
  - 目录结构完整性

用法:
  python3 check_project.py --name MyProject [--output .]
"""

import argparse
import os
import sys
from datetime import datetime, timedelta

# ═══════════════════════════════════════════════════════════════
# 检查规则
# ═══════════════════════════════════════════════════════════════

REQUIRED_DIRS = [
    "01_data", "02_tokenization", "03_models",
    "04_experiments", "05_docs", "06_shared",
]

REQUIRED_README = [
    "README.md", "_project_config.yaml", ".gitignore",
    "01_data/README.md", "01_data/04_quality_log.md",
    "02_tokenization/README.md",
    "03_models/README.md",
    "04_experiments/experiment_log.md",
]

METADATA_REQUIRED_FIELDS = [
    "source_name", "source_type", "url", "access_date",
    "license", "language", "description",
]

EVAL_REQUIRED_SECTIONS = [
    "内在评估", "下游评估",
]


def check_color(status: str) -> str:
    """简单的终端着色"""
    colors = {"OK": "✅", "WARN": "⚠️", "ERROR": "❌", "INFO": "ℹ️"}
    return colors.get(status, "  ")


class HealthChecker:
    def __init__(self, base: str, project_name: str):
        self.base = base
        self.name = project_name
        self.warnings = []
        self.errors = []
        self.passes = []

    def run_all(self):
        print(f"\n{'='*60}")
        print(f"🏥 AI 项目健康检查: {self.name}")
        print(f"📁 {os.path.abspath(self.base)}")
        print(f"📅 {datetime.now().strftime('%Y-%m-%d %H:%M')}")
        print(f"{'='*60}\n")

        self._check_dirs()
        self._check_readmes()
        self._check_gitignore()
        self._check_data_sources()
        self._check_quality_log()
        self._check_tokenizers()
        self._check_models()
        self._check_experiment_log()
        self._check_config_templates()

        self._print_summary()

    def _check_dirs(self):
        print("📂 目录结构 ...")
        for d in REQUIRED_DIRS:
            path = os.path.join(self.base, d)
            if os.path.isdir(path):
                self.passes.append(f"  目录存在: {d}/")
            else:
                self.errors.append(f"  缺少目录: {d}/")

    def _check_readmes(self):
        print("📝 README 文件 ...")
        for f in REQUIRED_README:
            path = os.path.join(self.base, f)
            if os.path.isfile(path):
                size = os.path.getsize(path)
                if size < 50:
                    self.warnings.append(f"  {f} 内容过少 ({size} bytes)，可能未填写")
                else:
                    self.passes.append(f"  README 正常: {f}")
            else:
                self.errors.append(f"  缺少文件: {f}")

    def _check_gitignore(self):
        gi_path = os.path.join(self.base, ".gitignore")
        if not os.path.isfile(gi_path):
            self.errors.append("  缺少 .gitignore！建议立即创建")
            return
        with open(gi_path) as f:
            content = f.read()
        checks = [
            ("checkpoints", "**/checkpoints/"),
            ("*.pth", "*.pth"),
            ("*.safetensors", "*.safetensors"),
            (".env", ".env"),
            ("logs", "*.log"),
        ]
        for name, pattern in checks:
            if pattern not in content:
                self.warnings.append(f"  .gitignore 可能缺少 {name} 的忽略规则")

    def _check_data_sources(self):
        print("📊 数据来源 ...")
        raw_dir = os.path.join(self.base, "01_data/01_raw")
        if not os.path.isdir(raw_dir):
            self.warnings.append("  01_data/01_raw/ 目录不存在")
            return

        sources = [d for d in os.listdir(raw_dir)
                   if os.path.isdir(os.path.join(raw_dir, d)) and d.startswith("source_")]
        if not sources:
            self.warnings.append("  尚未添加任何数据来源（01_data/01_raw/ 为空）")
            return

        self.passes.append(f"  发现 {len(sources)} 个数据来源")
        for src in sorted(sources):
            meta_path = os.path.join(raw_dir, src, "metadata.yaml")
            if not os.path.isfile(meta_path):
                self.errors.append(f"  {src}/ 缺少 metadata.yaml")
                continue

            with open(meta_path) as f:
                meta = f.read()

            missing_fields = []
            for field in METADATA_REQUIRED_FIELDS:
                if f"{field}:" not in meta:
                    missing_fields.append(field)
            # 检查字段是否为空值
            for line in meta.split("\n"):
                if ":" in line:
                    key, val = line.split(":", 1)
                    key = key.strip()
                    val = val.strip()
                    if key in METADATA_REQUIRED_FIELDS and (not val or val == ""):
                        if key not in missing_fields:
                            missing_fields.append(f"{key}(空值)")

            if missing_fields:
                self.warnings.append(
                    f"  {src}/metadata.yaml 缺少或为空: {', '.join(missing_fields)}")
            else:
                self.passes.append(f"  ✅ source_{src} metadata 完整")

    def _check_quality_log(self):
        ql_path = os.path.join(self.base, "01_data/04_quality_log.md")
        if not os.path.isfile(ql_path):
            self.warnings.append("  缺少 01_data/04_quality_log.md")
            return
        with open(ql_path) as f:
            content = f.read()
        # 检查是否有实际的质量记录（至少一行表格数据）
        table_lines = [l for l in content.split("\n") if l.startswith("|") and "---" not in l and l.count("|") >= 3]
        data_lines = [l for l in table_lines if any(c.isalpha() for c in l)]
        if len(data_lines) <= 2:  # 只有表头
            self.warnings.append("  04_quality_log.md 尚无质量记录")

    def _check_tokenizers(self):
        print("🔤 分词方案 ...")
        custom_dir = os.path.join(self.base, "02_tokenization/02_custom")
        if not os.path.isdir(custom_dir):
            return

        approaches = [d for d in os.listdir(custom_dir)
                      if os.path.isdir(os.path.join(custom_dir, d))]
        if not approaches:
            self.warnings.append("  02_tokenization/02_custom/ 中尚无分词方案")
            return

        self.passes.append(f"  发现 {len(approaches)} 个分词方案")
        for app in sorted(approaches):
            app_dir = os.path.join(custom_dir, app)
            eval_path = os.path.join(app_dir, "eval.md")

            if not os.path.isfile(eval_path):
                self.warnings.append(f"  {app}/ 缺少 eval.md")
                continue

            with open(eval_path) as f:
                eval_content = f.read()

            if len(eval_content.strip()) < 100:
                self.warnings.append(f"  {app}/eval.md 内容过少，可能未填写评估结果")
            else:
                self.passes.append(f"  ✅ {app} eval 已填写")

    def _check_models(self):
        print("🧠 模型训练 ...")
        pt_dir = os.path.join(self.base, "03_models/01_pretraining")
        if not os.path.isdir(pt_dir):
            return

        archs = [d for d in os.listdir(pt_dir)
                 if os.path.isdir(os.path.join(pt_dir, d))]
        total_runs = 0
        for arch in sorted(archs):
            arch_dir = os.path.join(pt_dir, arch)
            runs = [d for d in os.listdir(arch_dir)
                    if os.path.isdir(os.path.join(arch_dir, d))]
            total_runs += len(runs)
            for run in sorted(runs):
                run_dir = os.path.join(arch_dir, run)
                # 检查是否有日志
                logs_dir = os.path.join(run_dir, "logs")
                if os.path.isdir(logs_dir) and os.listdir(logs_dir):
                    self.passes.append(f"  ✅ {arch}/{run} 有训练日志")
                else:
                    self.warnings.append(f"  {arch}/{run} logs/ 为空，训练可能未启动或无日志")
                # 检查 eval
                eval_dir = os.path.join(run_dir, "eval")
                if os.path.isdir(eval_dir) and os.listdir(eval_dir):
                    self.passes.append(f"  ✅ {arch}/{run} 有评估结果")
                else:
                    self.warnings.append(f"  {arch}/{run} eval/ 为空")

        if total_runs > 0:
            self.passes.append(f"  共 {total_runs} 个训练运行（{len(archs)} 个架构）")
        else:
            self.info = "  尚未开始模型训练"

        # 检查后训练
        post_dir = os.path.join(self.base, "03_models/02_post_training")
        if os.path.isdir(post_dir):
            post_methods = [d for d in os.listdir(post_dir)
                            if os.path.isdir(os.path.join(post_dir, d))]
            post_runs = 0
            for m in post_methods:
                m_dir = os.path.join(post_dir, m)
                post_runs += len([d for d in os.listdir(m_dir)
                                  if os.path.isdir(os.path.join(m_dir, d))])
            if post_runs > 0:
                self.passes.append(f"  后训练: {post_runs} 个运行")

    def _check_experiment_log(self):
        log_path = os.path.join(self.base, "04_experiments/experiment_log.md")
        if not os.path.isfile(log_path):
            self.errors.append("  缺少 04_experiments/experiment_log.md")
            return
        with open(log_path) as f:
            content = f.read()

        # 检查是否有实际记录（查找日期格式）
        import re
        dates = re.findall(r'\d{4}-\d{2}-\d{2}', content)
        if len(dates) <= 1:  # 只有模板中的日期
            self.warnings.append("  experiment_log.md 尚无实验记录（无日期条目）")
        else:
            self.passes.append(f"  experiment_log.md 有 {len(set(dates))} 条日期记录")

        # 检查最近更新
        mtime = os.path.getmtime(log_path)
        days_ago = (datetime.now() - datetime.fromtimestamp(mtime)).days
        if days_ago > 7:
            self.warnings.append(f"  experiment_log.md 已 {days_ago} 天未更新")

    def _check_config_templates(self):
        config_dir = os.path.join(self.base, "06_shared/configs/training")
        if os.path.isdir(config_dir):
            templates = os.listdir(config_dir)
            if templates:
                self.passes.append(f"  配置模板: {len(templates)} 个")
            else:
                self.warnings.append("  06_shared/configs/training/ 无配置模板")

    def _print_summary(self):
        print(f"\n{'='*60}")
        print(f"📊 检查总结")
        print(f"{'='*60}")
        print(f"  ✅ 通过: {len(self.passes)}")
        print(f"  ⚠️  警告: {len(self.warnings)}")
        print(f"  ❌ 错误: {len(self.errors)}")

        if self.errors:
            print(f"\n❌ 错误（必须修复）:")
            for e in self.errors:
                print(f"  {e}")

        if self.warnings:
            print(f"\n⚠️  警告（建议修复）:")
            for w in self.warnings:
                print(f"  {w}")

        if self.passes and (self.errors or self.warnings):
            print(f"\n✅ 通过项 (共 {len(self.passes)} 项，用 --verbose 查看)")

        score = max(0, 100 - len(self.errors) * 15 - len(self.warnings) * 5)
        grade = "🏆" if score >= 90 else ("👍" if score >= 70 else ("⚠️" if score >= 50 else "🔴"))
        print(f"\n📈 健康评分: {score}/100 {grade}")


def main():
    parser = argparse.ArgumentParser(description="AI 项目健康检查工具")
    parser.add_argument("--name", type=str, required=True, help="项目名称")
    parser.add_argument("--output", type=str, default=".", help="项目父目录")
    parser.add_argument("--verbose", "-v", action="store_true", help="显示全部通过项")
    args = parser.parse_args()

    base = os.path.join(args.output, args.name)
    if not os.path.isdir(base):
        print(f"❌ 项目不存在: {base}")
        sys.exit(1)

    checker = HealthChecker(base, args.name)
    checker.run_all()

    if args.verbose and checker.passes:
        print(f"\n✅ 全部通过项:")
        for p in checker.passes:
            print(f"  {p}")

    sys.exit(0 if not checker.errors else 1)


if __name__ == "__main__":
    main()


path: scripts/checkpoint_mgmt.py
content: #!/usr/bin/env python3
"""
Checkpoint 管理脚本
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
功能:
  - 列出所有 checkpoint 及大小
  - 清理旧 checkpoint（保留 best + 每 N 步一个）
  - 归档 checkpoint 到指定位置
  - 磁盘用量统计

用法:
  python3 checkpoint_mgmt.py --name MyProject list            # 列出所有 checkpoint
  python3 checkpoint_mgmt.py --name MyProject clean --keep 3  # 每个运行只保留最近3个
  python3 checkpoint_mgmt.py --name MyProject archive --to /archive/path
  python3 checkpoint_mgmt.py --name MyProject stats           # 磁盘用量统计
  python3 checkpoint_mgmt.py --name MyProject clean --dry-run # 预览清理
"""

import argparse
import os
import sys
import shutil
from datetime import datetime


def human_size(size_bytes: int) -> str:
    """字节转可读大小"""
    for unit in ['B', 'KB', 'MB', 'GB', 'TB']:
        if size_bytes < 1024:
            return f"{size_bytes:.1f} {unit}"
        size_bytes /= 1024
    return f"{size_bytes:.1f} PB"


def find_all_checkpoints(base: str) -> list:
    """扫描项目中所有 checkpoint"""
    checkpoints = []
    for root, dirs, files in os.walk(base):
        if 'checkpoints' in root:
            for f in files:
                if f.startswith('.') or f == '.gitkeep':
                    continue
                fpath = os.path.join(root, f)
                size = os.path.getsize(fpath)
                mtime = os.path.getmtime(fpath)
                rel = os.path.relpath(fpath, base)
                checkpoints.append({
                    "path": fpath,
                    "rel_path": rel,
                    "size": size,
                    "mtime": mtime,
                    "run_dir": os.path.dirname(os.path.dirname(fpath)),
                })
    return sorted(checkpoints, key=lambda x: x["size"], reverse=True)


def cmd_list(base: str, project_name: str):
    """列出所有 checkpoint"""
    print(f"\n{'='*60}")
    print(f"📦 Checkpoint 清单: {project_name}")
    print(f"{'='*60}\n")

    ckpts = find_all_checkpoints(base)
    if not ckpts:
        print("  🕐 未找到任何 checkpoint")
        return

    total_size = sum(c["size"] for c in ckpts)
    print(f"  📊 共 {len(ckpts)} 个 checkpoint，总大小 {human_size(total_size)}\n")
    print(f"  {'大小':<12} {'修改时间':<20} {'路径'}")
    print(f"  {'─'*12} {'─'*20} {'─'*40}")

    # 按运行分组显示
    run_groups = {}
    for c in ckpts:
        run_key = c.get("run_dir", "unknown")
        if run_key not in run_groups:
            run_groups[run_key] = []
        run_groups[run_key].append(c)

    for run_dir, items in sorted(run_groups.items()):
        run_name = os.path.relpath(run_dir, base)
        run_size = sum(i["size"] for i in items)
        print(f"\n  📁 {run_name} ({len(items)} 个, {human_size(run_size)})")
        for item in items[:10]:  # 每个运行最多显示10个
            mtime = datetime.fromtimestamp(item["mtime"]).strftime("%Y-%m-%d %H:%M")
            print(f"     {human_size(item['size']):<12} {mtime:<20} {os.path.basename(item['rel_path'])}")
        if len(items) > 10:
            print(f"     ... 还有 {len(items) - 10} 个文件")


def cmd_clean(base: str, project_name: str, keep: int = 3, dry_run: bool = False):
    """清理旧 checkpoint，每个运行只保留最近 N 个"""
    print(f"\n{'='*60}")
    print(f"🧹 Checkpoint 清理: {project_name}")
    if dry_run:
        print(f"   🔍 预览模式（不会实际删除）")
    print(f"   📌 每个运行保留最近 {keep} 个 checkpoint")
    print(f"{'='*60}\n")

    ckpts = find_all_checkpoints(base)
    if not ckpts:
        print("  🕐 未找到任何 checkpoint")
        return

    # 按运行分组，每组保留 keep 个最新的
    run_groups = {}
    for c in ckpts:
        run_key = c.get("run_dir", "unknown")
        if run_key not in run_groups:
            run_groups[run_key] = []
        run_groups[run_key].append(c)

    total_deleted = 0
    total_freed = 0

    for run_dir, items in sorted(run_groups.items()):
        run_name = os.path.relpath(run_dir, base)
        # 按修改时间排序，最新的在前
        items_sorted = sorted(items, key=lambda x: x["mtime"], reverse=True)
        to_delete = items_sorted[keep:]
        to_keep = items_sorted[:keep]

        if not to_delete:
            print(f"  ✅ {run_name}: {len(items)} 个 (全部保留)")
            continue

        freed = sum(d["size"] for d in to_delete)
        total_deleted += len(to_delete)
        total_freed += freed

        print(f"  🗑 {run_name}: 保留 {len(to_keep)} 个, 删除 {len(to_delete)} 个 "
              f"(释放 {human_size(freed)})")

        if not dry_run:
            for d in to_delete:
                try:
                    os.remove(d["path"])
                    print(f"     ❌ 删除: {os.path.basename(d['rel_path'])}")
                except OSError as e:
                    print(f"     ⚠️ 删除失败: {os.path.basename(d['rel_path'])} - {e}")

    if total_deleted > 0:
        action = "将删除" if dry_run else "已删除"
        print(f"\n  📊 {action} {total_deleted} 个 checkpoint, 释放 {human_size(total_freed)}")
    else:
        print(f"\n  ✅ 无需清理")


def cmd_archive(base: str, project_name: str, archive_to: str):
    """归档所有 checkpoint 到指定位置"""
    print(f"\n{'='*60}")
    print(f"📦 Checkpoint 归档: {project_name}")
    print(f"   目标: {archive_to}")
    print(f"{'='*60}\n")

    ckpts = find_all_checkpoints(base)
    if not ckpts:
        print("  🕐 未找到任何 checkpoint")
        return

    os.makedirs(archive_to, exist_ok=True)
    today = datetime.now().strftime("%Y%m%d")
    archive_dir = os.path.join(archive_to, f"{project_name}_ckpt_{today}")
    os.makedirs(archive_dir, exist_ok=True)

    total_size = sum(c["size"] for c in ckpts)
    print(f"  📊 共 {len(ckpts)} 个 checkpoint, {human_size(total_size)}")
    print(f"  📁 归档到: {archive_dir}\n")

    copied = 0
    for c in ckpts:
        # 保持目录结构
        rel = os.path.relpath(c["path"], base)
        dest = os.path.join(archive_dir, rel)
        os.makedirs(os.path.dirname(dest), exist_ok=True)
        try:
            shutil.copy2(c["path"], dest)
            copied += 1
            if copied % 10 == 0:
                print(f"  ✅ 已复制 {copied}/{len(ckpts)} ...")
        except OSError as e:
            print(f"  ⚠️ 复制失败: {rel} - {e}")

    print(f"\n  ✅ 归档完成: {copied} 个文件 → {archive_dir}")


def cmd_stats(base: str, project_name: str):
    """磁盘用量统计"""
    print(f"\n{'='*60}")
    print(f"💾 磁盘用量统计: {project_name}")
    print(f"{'='*60}\n")

    # 按模块统计
    modules = {
        "01_data": "数据层",
        "02_tokenization": "分词层",
        "03_models": "模型层",
        "04_experiments": "实验管理",
        "05_docs": "文档",
        "06_shared": "共享资源",
    }

    total = 0
    for mod_dir, mod_name in modules.items():
        mod_path = os.path.join(base, mod_dir)
        if not os.path.isdir(mod_path):
            continue
        mod_size = 0
        for dirpath, dirnames, filenames in os.walk(mod_path):
            for f in filenames:
                fp = os.path.join(dirpath, f)
                if os.path.isfile(fp):
                    mod_size += os.path.getsize(fp)
        total += mod_size
        print(f"  {mod_name:<10} {human_size(mod_size):>12}")

    print(f"  {'─'*25}")
    print(f"  {'总计':<10} {human_size(total):>12}")

    # Checkpoint 专项统计
    ckpts = find_all_checkpoints(base)
    if ckpts:
        ckpt_size = sum(c["size"] for c in ckpts)
        print(f"\n  📦 Checkpoint: {len(ckpts)} 个, {human_size(ckpt_size)} "
              f"({ckpt_size/total*100:.1f}% 总空间)")


def main():
    parser = argparse.ArgumentParser(
        description="AI 项目 Checkpoint 管理工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  checkpoint_mgmt.py --name MyLLM list                        # 列出所有 checkpoint
  checkpoint_mgmt.py --name MyLLM clean --keep 3               # 每运行保留3个最新
  checkpoint_mgmt.py --name MyLLM clean --keep 3 --dry-run     # 预览清理
  checkpoint_mgmt.py --name MyLLM archive --to /backup/ckpts   # 归档
  checkpoint_mgmt.py --name MyLLM stats                        # 磁盘统计
        """)

    parser.add_argument("--name", type=str, required=True, help="项目名称")
    parser.add_argument("--output", type=str, default=".", help="项目父目录")
    parser.add_argument("command", choices=["list", "clean", "archive", "stats"],
                        help="操作命令")
    parser.add_argument("--keep", type=int, default=3,
                        help="每个运行保留的 checkpoint 数量 (用于 clean, 默认3)")
    parser.add_argument("--to", type=str, help="归档目标路径 (用于 archive)")
    parser.add_argument("--dry-run", action="store_true", help="预览模式，不实际删除")
    args = parser.parse_args()

    base = os.path.join(args.output, args.name)
    if not os.path.isdir(base):
        print(f"❌ 项目不存在: {base}")
        sys.exit(1)

    if args.command == "list":
        cmd_list(base, args.name)
    elif args.command == "clean":
        cmd_clean(base, args.name, keep=args.keep, dry_run=args.dry_run)
    elif args.command == "archive":
        if not args.to:
            parser.error("archive 命令需要 --to <目标路径>")
        cmd_archive(base, args.name, args.to)
    elif args.command == "stats":
        cmd_stats(base, args.name)


if __name__ == "__main__":
    main()


path: scripts/compare_runs.py
content: #!/usr/bin/env python3
"""
AI 项目实验对比脚本
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
自动扫描训练运行，汇总关键指标对比表。

支持:
  - 同一架构下多个运行的对比
  - 跨架构对比
  - 预训练 / 后训练 / RL 各阶段对比

用法:
  python3 compare_runs.py --name MyProject                     # 全部对比
  python3 compare_runs.py --name MyProject --arch llama3        # 指定架构
  python3 compare_runs.py --name MyProject --stage post_training # 只看后训练
"""

import argparse
import os
import sys
import json
from datetime import datetime


class RunComparator:
    def __init__(self, base: str, project_name: str):
        self.base = base
        self.name = project_name

    def run(self, arch: str = None, stage: str = None):
        print(f"\n{'='*60}")
        print(f"📊 AI 项目实验对比: {self.name}")
        print(f"📅 {datetime.now().strftime('%Y-%m-%d %H:%M')}")
        print(f"{'='*60}")

        if stage is None or stage == "pretraining":
            self._compare_pretraining(arch)
        if stage is None or stage == "post_training":
            self._compare_post_training()
        if stage is None or stage == "rl":
            self._compare_rl()
        if stage is None or stage == "tokenization":
            self._compare_tokenization()

        print()

    def _scan_run_metrics(self, run_dir: str) -> dict:
        """扫描单个运行的目录，提取关键指标"""
        metrics = {}

        # 检查 configs 目录
        configs_dir = os.path.join(run_dir, "configs")
        if os.path.isdir(configs_dir):
            for f in os.listdir(configs_dir):
                if f.endswith(('.yaml', '.yml', '.json')):
                    metrics["has_config"] = True
                    break

        # 检查 logs 目录
        logs_dir = os.path.join(run_dir, "logs")
        if os.path.isdir(logs_dir):
            log_files = [f for f in os.listdir(logs_dir) if f.endswith(('.log', '.txt', '.out'))]
            metrics["log_files"] = len(log_files)
            # 尝试读取训练日志找 loss
            for lf in log_files:
                # 简单采样，不读全文件
                lf_path = os.path.join(logs_dir, lf)
                try:
                    with open(lf_path, errors='ignore') as f:
                        for line in f:
                            line = line.strip()
                            if 'loss' in line.lower() and ('=' in line or ':' in line):
                                metrics["has_loss_in_log"] = True
                                break
                except Exception:
                    pass

        # 检查 checkpoints
        ckpt_dir = os.path.join(run_dir, "checkpoints")
        if os.path.isdir(ckpt_dir):
            ckpt_files = [f for f in os.listdir(ckpt_dir) if not f.startswith('.')]
            metrics["checkpoint_count"] = len(ckpt_files)
            # 估算大小
            total_size = 0
            for f in ckpt_files:
                fpath = os.path.join(ckpt_dir, f)
                if os.path.isfile(fpath):
                    total_size += os.path.getsize(fpath)
            metrics["checkpoint_size_mb"] = round(total_size / (1024 * 1024), 1)

        # 检查 eval
        eval_dir = os.path.join(run_dir, "eval")
        if os.path.isdir(eval_dir):
            eval_files = [f for f in os.listdir(eval_dir) if not f.startswith('.')]
            metrics["eval_files"] = len(eval_files)

        return metrics

    def _compare_pretraining(self, arch_filter: str = None):
        print(f"\n🧠 ── 预训练对比 ──")

        pt_dir = os.path.join(self.base, "03_models/01_pretraining")
        if not os.path.isdir(pt_dir):
            print("  (尚无预训练运行)")
            return

        all_runs = []
        for arch in sorted(os.listdir(pt_dir)):
            arch_dir = os.path.join(pt_dir, arch)
            if not os.path.isdir(arch_dir):
                continue
            if arch_filter and arch != arch_filter:
                continue

            for run in sorted(os.listdir(arch_dir)):
                run_dir = os.path.join(arch_dir, run)
                if not os.path.isdir(run_dir):
                    continue
                metrics = self._scan_run_metrics(run_dir)
                all_runs.append({
                    "arch": arch,
                    "run": run,
                    "path": run_dir,
                    "metrics": metrics,
                })

        if not all_runs:
            print("  (无匹配的运行)")
            return

        # 打印对比表
        print(f"\n  {'架构':<15} {'运行':<20} {'配置':<8} {'日志':<8} {'Checkpoint':<12} {'评估':<8}")
        print(f"  {'─'*15} {'─'*20} {'─'*8} {'─'*8} {'─'*12} {'─'*8}")
        for r in all_runs:
            m = r["metrics"]
            has_config = "✅" if m.get("has_config") else "❌"
            log_info = f"{m.get('log_files', 0)}文件"
            ckpt_info = f"{m.get('checkpoint_count', 0)}个/{m.get('checkpoint_size_mb', 0)}MB"
            eval_info = f"{m.get('eval_files', 0)}文件"
            print(f"  {r['arch']:<15} {r['run']:<20} {has_config:<8} {log_info:<8} {ckpt_info:<12} {eval_info:<8}")

        print(f"\n  共 {len(all_runs)} 个预训练运行")

    def _compare_post_training(self):
        print(f"\n🔧 ── 后训练对比 ──")

        post_dir = os.path.join(self.base, "03_models/02_post_training")
        if not os.path.isdir(post_dir):
            print("  (无后训练运行)")
            return

        all_runs = []
        for method in sorted(os.listdir(post_dir)):
            method_dir = os.path.join(post_dir, method)
            if not os.path.isdir(method_dir):
                continue
            for run in sorted(os.listdir(method_dir)):
                run_dir = os.path.join(method_dir, run)
                if not os.path.isdir(run_dir):
                    continue
                metrics = self._scan_run_metrics(run_dir)
                all_runs.append({
                    "method": method,
                    "run": run,
                    "metrics": metrics,
                })

        if not all_runs:
            print("  (无匹配的运行)")
            return

        print(f"\n  {'方法':<12} {'运行':<20} {'配置':<8} {'日志':<8} {'Checkpoint':<12} {'评估':<8}")
        print(f"  {'─'*12} {'─'*20} {'─'*8} {'─'*8} {'─'*12} {'─'*8}")
        for r in all_runs:
            m = r["metrics"]
            has_config = "✅" if m.get("has_config") else "❌"
            log_info = f"{m.get('log_files', 0)}文件"
            ckpt_info = f"{m.get('checkpoint_count', 0)}个"
            eval_info = f"{m.get('eval_files', 0)}文件"
            print(f"  {r['method']:<12} {r['run']:<20} {has_config:<8} {log_info:<8} {ckpt_info:<12} {eval_info:<8}")

        print(f"\n  共 {len(all_runs)} 个后训练运行")

    def _compare_rl(self):
        print(f"\n🎮 ── RL 对比 ──")

        rl_dir = os.path.join(self.base, "03_models/03_rl")
        if not os.path.isdir(rl_dir):
            print("  (无 RL 运行)")
            return

        all_runs = []
        for method in sorted(os.listdir(rl_dir)):
            method_dir = os.path.join(rl_dir, method)
            if not os.path.isdir(method_dir):
                continue
            for run in sorted(os.listdir(method_dir)):
                run_dir = os.path.join(method_dir, run)
                if not os.path.isdir(run_dir):
                    continue
                metrics = self._scan_run_metrics(run_dir)
                all_runs.append({
                    "method": method,
                    "run": run,
                    "metrics": metrics,
                })

        if not all_runs:
            print("  (无匹配的运行)")
            return

        print(f"\n  {'方法':<16} {'运行':<20} {'配置':<8} {'日志':<8} {'Checkpoint':<12} {'评估':<8}")
        print(f"  {'─'*16} {'─'*20} {'─'*8} {'─'*8} {'─'*12} {'─'*8}")
        for r in all_runs:
            m = r["metrics"]
            has_config = "✅" if m.get("has_config") else "❌"
            log_info = f"{m.get('log_files', 0)}文件"
            ckpt_info = f"{m.get('checkpoint_count', 0)}个"
            eval_info = f"{m.get('eval_files', 0)}文件"
            print(f"  {r['method']:<16} {r['run']:<20} {has_config:<8} {log_info:<8} {ckpt_info:<12} {eval_info:<8}")

        print(f"\n  共 {len(all_runs)} 个 RL 运行")

    def _compare_tokenization(self):
        print(f"\n🔤 ── 分词对比 ──")

        custom_dir = os.path.join(self.base, "02_tokenization/02_custom")
        if not os.path.isdir(custom_dir):
            print("  (无分词方案)")
            return

        approaches = sorted([d for d in os.listdir(custom_dir)
                             if os.path.isdir(os.path.join(custom_dir, d))])
        if not approaches:
            print("  (无分词方案)")
            return

        print(f"\n  {'方案':<20} {'脚本':<8} {'词表':<8} {'评估':<8} {'README':<8}")
        print(f"  {'─'*20} {'─'*8} {'─'*8} {'─'*8} {'─'*8}")
        for app in approaches:
            app_dir = os.path.join(custom_dir, app)
            has_scripts = bool(os.path.isdir(os.path.join(app_dir, "scripts")) and
                               [f for f in os.listdir(os.path.join(app_dir, "scripts"))
                                if f.endswith(('.py', '.sh'))])
            has_vocab = bool(os.path.isdir(os.path.join(app_dir, "vocab_output")) and
                             os.listdir(os.path.join(app_dir, "vocab_output")))
            eval_path = os.path.join(app_dir, "eval.md")
            has_eval = os.path.isfile(eval_path) and os.path.getsize(eval_path) > 100
            readme_path = os.path.join(app_dir, "README.md")
            has_readme = os.path.isfile(readme_path) and os.path.getsize(readme_path) > 50

            print(f"  {app:<20} {'✅' if has_scripts else '❌':<8} "
                  f"{'✅' if has_vocab else '❌':<8} "
                  f"{'✅' if has_eval else '❌':<8} "
                  f"{'✅' if has_readme else '❌':<8}")

        print(f"\n  共 {len(approaches)} 个分词方案")


def main():
    parser = argparse.ArgumentParser(description="AI 项目实验对比工具")
    parser.add_argument("--name", type=str, required=True, help="项目名称")
    parser.add_argument("--output", type=str, default=".", help="项目父目录")
    parser.add_argument("--arch", type=str, help="只对比指定架构")
    parser.add_argument("--stage", type=str,
                        choices=["pretraining", "post_training", "rl", "tokenization"],
                        help="只对比指定阶段")
    args = parser.parse_args()

    base = os.path.join(args.output, args.name)
    if not os.path.isdir(base):
        print(f"❌ 项目不存在: {base}")
        sys.exit(1)

    comparator = RunComparator(base, args.name)
    comparator.run(arch=args.arch, stage=args.stage)


if __name__ == "__main__":
    main()


path: scripts/doc_lint.py
content: #!/usr/bin/env python3
"""
文档规范检查器 —— 铁律的代码化 enforcement
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
扫描项目中所有 .md/.yaml 文档，自动检测：
  - 重复内容（跨文件相似度 > 70%）
  - 超字数文件
  - 模糊命名（final/new/latest/draft/temp）
  - 空模板（生成后未填写）
  - 非约定文件名（随意命名的野文档）

用法:
  python3 doc_lint.py --name MyProject [--output .]
  python3 doc_lint.py --name MyProject --strict       # 严格模式，重复内容报错
  python3 doc_lint.py --name MyProject --fix           # 自动标记可疑文件
"""

import argparse
import os
import re
import sys
from collections import defaultdict
from datetime import datetime

# ═══════════════════════════════════════════════════════════════
# 规则定义
# ═══════════════════════════════════════════════════════════════

# 允许的文件名（白名单）
ALLOWED_FILES = {
    # 根目录
    "README.md", ".gitignore", "_project_config.yaml",
    # 01_data
    "01_data/README.md", "01_data/04_quality_log.md",
    # 02_tokenization
    "02_tokenization/README.md", "02_tokenization/05_generated/README.md",
    # 03_models
    "03_models/README.md",
    # 04_experiments
    "04_experiments/experiment_log.md",
    # 06_shared
    "06_shared/README.md", "06_shared/configs/README.md",
    # configs
    "06_shared/configs/training/deepspeed_zero2.yaml",
    "06_shared/configs/training/deepspeed_zero3.yaml",
    "06_shared/configs/training/lora_config.yaml",
    "06_shared/configs/training/qlora_config.yaml",
    "06_shared/configs/training/wandb_config.yaml",
}

# 允许的命名模式（正则）
ALLOWED_PATTERNS = [
    r"01_data/01_raw/source_.+/metadata\.yaml$",
    r"01_data/02_qc/reports/qc_report_.+\.md$",
    r"01_data/02_qc/scripts/.+\.(py|sh)$",
    r"02_tokenization/01_existing/.+/notes\.md$",
    r"02_tokenization/01_existing/.+/config\.yaml$",
    r"02_tokenization/02_custom/.+/README\.md$",
    r"02_tokenization/02_custom/.+/eval\.md$",
    r"02_tokenization/02_custom/.+/scripts/.+\.(py|sh)$",
    r"02_tokenization/02_custom/.+/configs/.+\.(yaml|json)$",
    r"03_models/.*/configs/.+\.(yaml|json|py|sh)$",
    r"03_models/.*/scripts/.+\.(py|sh)$",
    r"05_docs/design/architecture_decision_\d{3}_.+\.md$",
    r"05_docs/meetings/meeting_\d{4}-\d{2}-\d{2}_.+\.md$",
    r"05_docs/references/.+\.(md|pdf|bib)$",
    r"06_shared/scripts/.+\.(py|sh)$",
    r"06_shared/configs/.*\.(yaml|json)$",
    r"scripts/.+\.py$",
]

# 模糊命名关键词（禁止出现）
VAGUE_NAMES = [
    "final", "new", "latest", "old", "draft", "temp", "tmp",
    "test", "backup", "copy", "revised", "updated", "version",
    "v2_final", "v3_revised", "use_this_one",
]

# 各文件的字数上限
WORD_LIMITS = {
    "README.md": 350,
    "01_data/README.md": 400,
    "02_tokenization/README.md": 400,
    "03_models/README.md": 400,
    "06_shared/README.md": 300,
    "01_data/04_quality_log.md": None,  # 不限
    "04_experiments/experiment_log.md": None,  # 不限
}

# 重复内容阈值（0-1，越高越严格）
SIMILARITY_THRESHOLD = 0.6


def find_all_docs(base: str) -> list:
    """找到所有 .md 和 .yaml 文件"""
    docs = []
    for root, dirs, filenames in os.walk(base):
        # 跳过不需要检查的目录
        dirs[:] = [d for d in dirs if d not in [
            '.git', '__pycache__', 'checkpoints', 'logs',
            'vocab_output', 'data', 'cleaned', 'checkpoints',
            'runs', 'benchmarks', '.vscode', '.idea',
        ]]
        for f in filenames:
            if f.endswith(('.md', '.yaml', '.yml')):
                fpath = os.path.join(root, f)
                rel = os.path.relpath(fpath, base)
                docs.append({
                    "path": fpath,
                    "rel": rel,
                    "name": f,
                    "size": os.path.getsize(fpath),
                })
    return docs


def is_allowed_file(rel_path: str) -> bool:
    """检查文件是否在白名单或匹配允许模式"""
    if rel_path in ALLOWED_FILES:
        return True
    for pattern in ALLOWED_PATTERNS:
        if re.search(pattern, rel_path):
            return True
    return False


def check_vague_naming(name: str) -> list:
    """检查文件名是否包含模糊关键词"""
    issues = []
    name_lower = name.lower()
    for vn in VAGUE_NAMES:
        if vn in name_lower:
            issues.append(f"包含模糊词 '{vn}'")
    return issues


def check_word_count(path: str, rel: str) -> tuple:
    """检查字数是否超限"""
    # yaml/json 配置文件不检查字数（它们是数据，不是文档）
    if rel.endswith(('.yaml', '.yml', '.json')):
        return (True, 0, None)
    for pattern, limit in WORD_LIMITS.items():
        if rel.endswith(pattern) or rel == pattern:
            if limit is None:
                return (True, None, None)
            try:
                with open(path, errors='ignore') as f:
                    # 统计中文字数 + 英文单词数
                    content = f.read()
                    # 去掉 markdown 格式符号
                    clean = re.sub(r'[#*`\-|>\[\]()]', '', content)
                    clean = re.sub(r'\s+', ' ', clean).strip()
                    words = len(clean)
            except Exception:
                return (True, 0, limit)
            if words > limit:
                return (False, words, limit)
            return (True, words, limit)
    # 没有匹配的字数限制规则，用默认 500
    try:
        with open(path, errors='ignore') as f:
            content = f.read()
            clean = re.sub(r'[#*`\-|>\[\]()]', '', content)
            clean = re.sub(r'\s+', ' ', clean).strip()
            words = len(clean)
    except Exception:
        return (True, 0, 500)
    if words > 500:
        return (False, words, 500)
    return (True, words, 500)


def check_empty_template(path: str, size: int) -> bool:
    """检查模板是否为空（生成后未填写）"""
    if size < 50:
        return False
    try:
        with open(path, errors='ignore') as f:
            content = f.read()

        # 检查是否完全是模板占位符
        template_indicators = [
            "<!-- 简要描述项目目标 -->",
            "简要描述项目目标",
            "<实验名称>",
            "<描述>",
            "YYYY-MM-DD: <实验名称>",
        ]
        has_real_content = False
        for line in content.split("\n"):
            stripped = line.strip()
            if stripped and not stripped.startswith('<!--') and not stripped.startswith('#'):
                is_template = False
                for ti in template_indicators:
                    if ti in stripped:
                        is_template = True
                        break
                if not is_template:
                    has_real_content = True
                    break

        if not has_real_content:
            return False
    except Exception:
        pass
    return True


def extract_text_blocks(content: str, min_len: int = 30) -> list:
    """提取文本段落（>30字符的连续文本）"""
    # 去掉代码块
    content = re.sub(r'```.*?```', '', content, flags=re.DOTALL)
    # 去掉表格
    content = re.sub(r'\|.*\|', '', content)
    # 去掉标题
    content = re.sub(r'^#+\s.*$', '', content, flags=re.MULTILINE)
    # 去掉链接和图片
    content = re.sub(r'\[([^\]]*)\]\([^)]*\)', '', content)
    content = re.sub(r'!\[.*?\]\(.*?\)', '', content)

    # 分词（中英文混合）
    blocks = []
    lines = content.split('\n')
    for line in lines:
        line = line.strip()
        if len(line) >= min_len:
            blocks.append(line)
    return blocks


def simple_similarity(text1: str, text2: str) -> float:
    """简单 Jaccard 相似度（基于字符 trigram）"""
    def trigrams(s):
        return set(s[i:i+3] for i in range(len(s) - 2))

    t1 = trigrams(text1)
    t2 = trigrams(text2)
    if not t1 or not t2:
        return 0.0
    intersection = t1 & t2
    union = t1 | t2
    return len(intersection) / len(union)


def check_duplicates(docs: list) -> list:
    """检查跨文件重复内容"""
    issues = []
    # 只检查 .md 文件，且只检查组间（不同目录的文件）
    md_docs = [(d["path"], d["rel"]) for d in docs
               if d["name"].endswith('.md') and d["size"] > 100]

    if len(md_docs) < 2:
        return issues

    checked = set()
    for i, (path1, rel1) in enumerate(md_docs):
        try:
            with open(path1, errors='ignore') as f:
                content1 = f.read()
            # 只提取长段落
            blocks1 = extract_text_blocks(content1)
            text1 = " ".join(blocks1)
            if len(text1) < 100:
                continue
        except Exception:
            continue

        for j, (path2, rel2) in enumerate(md_docs):
            if i >= j:
                continue
            pair = (rel1, rel2)
            if pair in checked:
                continue
            checked.add(pair)

            # 跳过同一目录下的文件（允许内部引用）
            if os.path.dirname(rel1) == os.path.dirname(rel2):
                continue

            try:
                with open(path2, errors='ignore') as f:
                    content2 = f.read()
                blocks2 = extract_text_blocks(content2)
                text2 = " ".join(blocks2)
                if len(text2) < 100:
                    continue
            except Exception:
                continue

            sim = simple_similarity(text1, text2)
            if sim > SIMILARITY_THRESHOLD:
                issues.append({
                    "file1": rel1,
                    "file2": rel2,
                    "similarity": sim,
                })
    return issues


def run_lint(base: str, project_name: str, strict: bool = False, fix: bool = False):
    """主检查逻辑"""
    print(f"\n{'='*60}")
    print(f"📋 文档规范检查: {project_name}")
    print(f"📁 {os.path.abspath(base)}")
    print(f"📅 {datetime.now().strftime('%Y-%m-%d %H:%M')}")
    if strict:
        print(f"🔒 严格模式（重复内容 = 错误）")
    if fix:
        print(f"🔧 修复模式（自动标记可疑文件）")
    print(f"{'='*60}\n")

    docs = find_all_docs(base)
    if not docs:
        print("  🕐 未找到任何文档文件")
        return

    wild_files = []       # 野文档（不在白名单）
    vague_names = []      # 模糊命名
    overlimit = []        # 超字数
    empty_templates = []  # 空模板
    duplicate_issues = [] # 重复内容
    passes = []

    print(f"📄 扫描 {len(docs)} 个文档文件...\n")

    for doc in docs:
        rel = doc["rel"]
        name = doc["name"]
        size = doc["size"]
        path = doc["path"]

        # 1. 检查是否允许的文件
        if not is_allowed_file(rel):
            wild_files.append(rel)
            if fix:
                _mark_suspicious(path, "非约定文件名，请确认是否需要")

        # 2. 检查模糊命名
        name_issues = check_vague_naming(name)
        if name_issues:
            for ni in name_issues:
                vague_names.append(f"  {rel}: {ni}")
            if fix:
                _mark_suspicious(path, f"模糊命名: {', '.join(name_issues)}")

        # 3. 检查字数
        ok, words, limit = check_word_count(path, rel)
        if not ok and limit:
            overlimit.append(f"  {rel}: {words} 字 (上限 {limit})")

        # 4. 检查空模板
        if name.endswith('.md') and not check_empty_template(path, size):
            empty_templates.append(f"  {rel}: 模板未填写 ({size} bytes)")

        # 5. 对过大的文件标记
        if size > 10000:  # 超过 10KB
            overlimit.append(f"  {rel}: 文件过大 ({size:,} bytes)，建议拆分")

    # 6. 检查跨文件重复
    print("🔍 检查跨文件重复内容...")
    duplicate_issues = check_duplicates(docs)

    # ── 输出结果 ──

    has_issues = False

    if wild_files:
        has_issues = True
        print(f"\n🔴 野文档（不在约定位置，共 {len(wild_files)} 个）:")
        for f in wild_files:
            print(f"  ❌ {f}")
        print(f"\n  💡 这些文件不符合目录规范。如果是必要的，")
        print(f"     请移到约定目录或用约定命名（如 architecture_decision_NNN_xxx.md）")

    if vague_names:
        has_issues = True
        print(f"\n⚠️  模糊命名（共 {len(vague_names)} 个）:")
        for vn in vague_names:
            print(f"  {vn}")

    if overlimit:
        has_issues = True
        print(f"\n⚠️  超字数/过大文件（共 {len(overlimit)} 个）:")
        for ol in overlimit:
            print(f"  {ol}")

    if empty_templates:
        print(f"\n🟡 空模板（共 {len(empty_templates)} 个）:")
        for et in empty_templates:
            print(f"  {et}")
        print(f"\n  💡 这些是脚手架生成的模板，正常。填完内容后会自动通过。")

    if duplicate_issues:
        is_error = strict
        prefix = "🔴" if is_error else "⚠️"
        label = "错误" if is_error else "警告"
        print(f"\n{prefix} 跨文件重复内容（{label}，共 {len(duplicate_issues)} 处）:")
        for di in duplicate_issues:
            print(f"  📎 {di['file1']} ↔ {di['file2']} (相似度: {di['similarity']:.0%})")
        if not strict:
            print(f"\n  💡 用 --strict 将重复内容视为错误")
        if is_error:
            has_issues = True

    # ── 统计 ──
    ok_count = len(docs) - len(wild_files) - len(vague_names)
    print(f"\n{'='*60}")
    print(f"📊 检查统计")
    print(f"{'='*60}")
    print(f"  总文件: {len(docs)}")
    print(f"  ✅ 命名合规: {ok_count}")
    print(f"  🔴 野文档: {len(wild_files)}")
    print(f"  ⚠️  模糊命名: {len(vague_names)}")
    print(f"  ⚠️  超字数: {len(overlimit)}")
    print(f"  ⚠️  重复内容: {len(duplicate_issues)}")
    print(f"  🟡 空模板: {len(empty_templates)}")

    if not has_issues and not wild_files and not vague_names:
        print(f"\n  🎉 文档规范检查全部通过！")
    else:
        exit_code = 1 if (strict and duplicate_issues) or wild_files else 0
        print(f"\n  💡 修复建议:")
        if wild_files:
            print(f"     1. 移除或移走野文档到约定目录")
        if vague_names:
            print(f"     2. 重命名模糊文件，去掉 final/new/latest/draft")
        if overlimit:
            print(f"     3. 拆分/精简超字数文件")
        if duplicate_issues:
            print(f"     4. 删除重复内容，在一个位置维护唯一真理")

    if fix:
        print(f"\n  🔧 已在可疑文件头部添加 HTML 注释标记")

    return 0 if not has_issues else (1 if strict and duplicate_issues else 0)


def _mark_suspicious(path: str, reason: str):
    """在文件头部添加审核标记"""
    try:
        with open(path) as f:
            content = f.read()
        marker = f"<!-- 🔴 DOC-LINT: {reason} -->\n"
        if not content.startswith("<!-- 🔴 DOC-LINT:"):
            with open(path, "w") as f:
                f.write(marker + content)
    except Exception:
        pass


def main():
    parser = argparse.ArgumentParser(
        description="AI 项目文档规范检查器",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  doc_lint.py --name MyLLM                    # 常规检查
  doc_lint.py --name MyLLM --strict           # 严格模式
  doc_lint.py --name MyLLM --fix              # 自动标记可疑文件
        """)
    parser.add_argument("--name", type=str, required=True, help="项目名称")
    parser.add_argument("--output", type=str, default=".", help="项目父目录")
    parser.add_argument("--strict", action="store_true", help="严格模式：重复内容视为错误")
    parser.add_argument("--fix", action="store_true", help="自动在可疑文件头部添加标记")
    args = parser.parse_args()

    base = os.path.join(args.output, args.name)
    if not os.path.isdir(base):
        print(f"❌ 项目不存在: {base}")
        sys.exit(1)

    exit_code = run_lint(base, args.name, strict=args.strict, fix=args.fix)
    sys.exit(exit_code)


if __name__ == "__main__":
    main()


path: scripts/init_ai_project.py
content: #!/usr/bin/env python3
"""
AI 项目目录结构初始化脚本 v2.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
功能:
  - 新建项目（完整目录结构 + README 模板 + .gitignore + 配置模板）
  - 添加数据来源 / 分词方案 / 训练运行
  - 添加后训练 / RL 实验

用法:
  python3 init_ai_project.py --name MyProject
  python3 init_ai_project.py --name MyProject --add-source --source wikipedia_zh
  python3 init_ai_project.py --name MyProject --add-tokenizer --approach bpe_v2
  python3 init_ai_project.py --name MyProject --add-training --arch llama3 --run baseline_v1
  python3 init_ai_project.py --name MyProject --add-posttrain --method sft --run chat_v1
  python3 init_ai_project.py --name MyProject --add-rl --method ppo --run reward_v1
"""

import argparse
import os
import sys
from datetime import datetime

# ═══════════════════════════════════════════════════════════════
# 目录结构定义
# ═══════════════════════════════════════════════════════════════

DIRS = [
    # ── 01_data ──
    "01_data/01_raw",
    "01_data/02_qc/scripts",
    "01_data/02_qc/reports",
    "01_data/02_qc/cleaned",
    "01_data/03_processed/text",
    "01_data/03_processed/image",
    "01_data/03_processed/audio",
    # 多模态细化
    "01_data/03_processed/multimodal/image_text",
    "01_data/03_processed/multimodal/video_text",
    "01_data/03_processed/multimodal/audio_text",
    "01_data/03_processed/multimodal/interleaved",
    # ── 02_tokenization ──
    "02_tokenization/01_existing",
    "02_tokenization/02_custom",
    "02_tokenization/03_training",
    "02_tokenization/04_evaluation/intrinsic",
    "02_tokenization/04_evaluation/extrinsic",
    "02_tokenization/05_generated",
    # ── 03_models ──
    "03_models/01_pretraining",
    "03_models/02_post_training/sft",
    "03_models/02_post_training/dpo",
    "03_models/02_post_training/orpo",
    "03_models/02_post_training/kto",
    "03_models/02_post_training/simpo",
    "03_models/02_post_training/rlhf",
    "03_models/03_rl/ppo",
    "03_models/03_rl/grpo",
    "03_models/03_rl/reward_models",
    "03_models/04_evaluation/benchmarks",
    "03_models/04_evaluation/reports",
    "03_models/05_deployment",
    # ── 04_experiments ──
    "04_experiments/runs",
    # ── 05_docs ──
    "05_docs/design",
    "05_docs/meetings",
    "05_docs/references",
    # ── 06_shared ──
    "06_shared/scripts",
    "06_shared/configs/training",
    "06_shared/configs/data",
    "06_shared/configs/deployment",
]

# ═══════════════════════════════════════════════════════════════
# .gitignore 模板
# ═══════════════════════════════════════════════════════════════

GITIGNORE = """# ═══════════════════════════════════════════════
# AI 项目 .gitignore（自动生成 by ai-project-scaffold）
# ═══════════════════════════════════════════════

# ── Python ──
__pycache__/
*.py[cod]
*.egg-info/
dist/
build/
*.egg

# ── 模型文件（太大不进 git） ──
*.pth
*.pt
*.safetensors
*.bin
*.ckpt
*.h5
*.pb
*.onnx
**/checkpoints/
!**/checkpoints/.gitkeep

# ── 原始数据（太大 + 可能有版权问题） ──
01_data/01_raw/
!01_data/01_raw/**/metadata.yaml

# ── 日志 ──
*.log
**/logs/
!**/logs/.gitkeep

# ── 环境 & 密钥 ──
.env
.env.*
*.key
*.pem
secrets.yaml
credentials.json

# ── IDE ──
.vscode/
.idea/
*.swp
*.swo
*~

# ── OS ──
.DS_Store
Thumbs.db

# ── 临时文件 ──
tmp/
temp/
*.tmp
*.temp
.cache/

# ── 输出产物 ──
wandb/
outputs/
lightning_logs/
mlruns/
"""

# ═══════════════════════════════════════════════════════════════
# README 模板
# ═══════════════════════════════════════════════════════════════

README_ROOT = """# {project_name}

> 📅 创建时间: {date}
> 🛠 自动生成 by ai-project-scaffold v2.0

## 项目目标

<!-- 简要描述项目目标 -->

## 目录导航

| 目录 | 用途 |
|------|------|
| [01_data/](01_data/) | 📊 数据层：原始数据、质控、处理 |
| [02_tokenization/](02_tokenization/) | 🔤 分词层：tokenizer 选型与制作 |
| [03_models/](03_models/) | 🧠 模型层：预训练、后训练、RL |
| [04_experiments/](04_experiments/) | 🔬 实验管理 |
| [05_docs/](05_docs/) | 📚 文档与参考文献 |
| [06_shared/](06_shared/) | 🔧 共享脚本与配置 |

## 快速开始

```bash
# 检查项目健康状态
python3 scripts/check_project.py --name {project_name}

# 查看追踪链
python3 scripts/trace.py --name {project_name}

# 对比实验
python3 scripts/compare_runs.py --name {project_name}
```
"""

README_DATA = """# 📊 数据层

## 数据来源登记表

| 来源 | 类型 | 规模 | 获取方式 | 获取日期 | License | 质量评级 | 备注 |
|------|------|------|----------|----------|---------|----------|------|
|      |      |      |          |          |         |          |      |

## 数据流向

记录数据从原始 → QC → 处理 → 使用的完整链路。

> 💡 运行 `python3 scripts/trace.py --name <项目>` 自动生成追踪链。
"""

README_TOKEN = """# 🔤 分词层

## 分词策略决策

| 方案 | 类型 | 词表大小 | 训练语料 | 压缩率 | 下游任务 | 选用理由 |
|------|------|----------|----------|--------|----------|----------|
|      |      |          |          |        |          |          |

## 分词 → 模型 追踪

记录每个分词方案被哪些模型使用。

> 💡 运行 `python3 scripts/trace.py --name <项目> --layer tokenization` 查看分词追踪。
"""

README_MODEL = """# 🧠 模型层

## 架构选型决策

| 架构 | 参数量 | 选型理由 | 对比方案 | 决策日期 |
|------|--------|----------|----------|----------|
|      |        |          |          |          |

## 训练历史

| 运行 | 架构 | 数据 | Token | 状态 | Loss | 备注 |
|------|------|------|-------|------|------|------|
|      |      |      |       |      |      |      |

> 💡 运行 `python3 scripts/compare_runs.py --name <项目>` 自动对比实验。
"""

README_EXPERIMENT = """# 🔬 实验日志

## 实验时间线

按时间顺序记录每次实验：目的、配置、结果、结论。

---

### 模板

### YYYY-MM-DD: <实验名称>

- **目的:** 
- **涉及模块:** (数据/分词/预训练/后训练/RL)
- **关键配置:**
  - 
- **结果:**
  - 
- **结论/下一步:**
  - 

---

> 💡 运行 `python3 scripts/check_project.py --name <项目>` 检查是否有遗漏的实验记录。
"""

README_GENERATED_DATA = """# 🔄 分词过程中产生的新数据

记录分词/训练过程中自动生成的数据：
- 数据增强产物
- 重新采样数据
- 合成数据
- token 训练衍生物

| 来源实验 | 数据类型 | 规模 | 存放位置 | 用途 |
|----------|----------|------|----------|------|
|          |          |      |          |      |
"""

README_SHARED = """# 🔧 共享资源

跨模块通用的脚本和配置。

## scripts/
- 数据处理通用工具
- 评估指标计算
- 可视化工具

## configs/
- **training/** — DeepSpeed、LoRA、QLoRA 等训练配置模板
- **data/** — 数据加载、预处理配置模板
- **deployment/** — 模型部署配置模板

> 💡 初始化项目时会自动预置常用配置模板。
"""

QUALITY_LOG = """# 📋 数据质量追踪日志

> 持续更新，记录所有数据集的质量评估结果。

---

## 质量评级标准

| 评级 | 说明 |
|------|------|
| A | 高质量，可直接用于训练 |
| B | 需要轻度清洗 |
| C | 需要大量清洗，可能有噪音 |
| D | 质量差，仅作参考 |
| F | 不可用 |

## 质量记录

| 日期 | 数据集 | 评级 | 问题描述 | 处理方式 | 负责人 | QC 脚本 |
|------|--------|------|----------|----------|--------|---------|
|      |        |      |          |          |        |         |
"""

METADATA_TEMPLATE = """# 数据来源元信息

source_name: {source_name}
source_type:  # web_crawl / internal_db / public_dataset / api / manual
url: 
access_date: {date}
license: 
language: 
size:
  files: 
  tokens_estimate: 
format:  # jsonl / parquet / csv / txt / binary
description: >
  
quality_notes: >
  
contact: 
"""

PROJECT_CONFIG = """# ══════════════════════════════════
# 项目配置
# ══════════════════════════════════

project:
  name: {project_name}
  created: {date}
  description: >
    
  members: []
  repo: 

data:
  primary_language: 
  modalities: []  # text, image, audio, multimodal
  total_tokens_estimate: 

model:
  target_size: 
  architecture_candidates: []
  
training:
  framework:  # pytorch / jax / mindspore
  parallel_strategy:  # deepspeed_zero2 / zero3 / fsdp / ddp
  hardware:  # GPU型号和数量
  
goals:
  - 
"""

# ═══════════════════════════════════════════════════════════════
# 配置模板
# ═══════════════════════════════════════════════════════════════

DEEPSPEED_ZERO2 = """# DeepSpeed ZeRO-2 配置模板
# 适用: 中等规模模型，显存优化 + 梯度分片

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "none"
    },
    "allgather_partitions": true,
    "allgather_bucket_size": 2e8,
    "reduce_scatter": true,
    "reduce_bucket_size": 2e8,
    "overlap_comm": true,
    "contiguous_gradients": true
  },
  "fp16": {
    "enabled": false
  },
  "bf16": {
    "enabled": true
  },
  "gradient_clipping": 1.0,
  "wall_clock_breakdown": false
}
"""

DEEPSPEED_ZERO3 = """# DeepSpeed ZeRO-3 配置模板
# 适用: 超大模型，参数 + 梯度 + 优化器全分片

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "bf16": {
    "enabled": true
  },
  "gradient_clipping": 1.0,
  "wall_clock_breakdown": false
}
"""

LORA_CONFIG = """# LoRA (Low-Rank Adaptation) 配置模板
# 适用: 参数高效微调，冻结原模型，只训练低秩适配器

{
  "peft_type": "LORA",
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
  ],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}
"""

QLORA_CONFIG = """# QLoRA (Quantized LoRA) 配置模板
# 适用: 4-bit 量化 + LoRA，极低显存微调大模型

{
  "load_in_4bit": true,
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "peft_type": "LORA",
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
  ],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}
"""

WANDB_CONFIG = """# Weights & Biases 配置模板

wandb:
  project: "{project_name}"
  entity: ""
  mode: "online"  # online / offline / disabled
  tags: []
  notes: ""
  log_model: false
  settings:
    _disable_stats: false
"""

CONFIG_README = """# 配置模板

此目录包含常用训练/数据/部署配置模板。

## 使用方法

1. 复制需要的模板到对应实验目录
2. 修改其中的占位符（如 `{project_name}`）
3. 在启动脚本中引用

## 模板清单

### training/
| 模板 | 适用场景 |
|------|----------|
| `deepspeed_zero2.yaml` | 中等模型，ZeRO-2 优化 |
| `deepspeed_zero3.yaml` | 大模型，ZeRO-3 全分片 |
| `lora_config.yaml` | 参数高效微调 |
| `qlora_config.yaml` | 4-bit 量化微调 |
| `wandb_config.yaml` | 实验追踪配置 |
"""

# ═══════════════════════════════════════════════════════════════
# 工具函数
# ═══════════════════════════════════════════════════════════════

def create_dirs(base_path: str):
    for d in DIRS:
        os.makedirs(os.path.join(base_path, d), exist_ok=True)
    return base_path


def write_file(path: str, content: str):
    if os.path.exists(path):
        print(f"  ⏭  跳过（已存在）: {path}")
        return
    with open(path, "w", encoding="utf-8") as f:
        f.write(content)
    print(f"  ✅ 创建: {path}")


def force_write(path: str, content: str):
    """覆盖写入（用于配置模板等需要更新的文件）"""
    with open(path, "w", encoding="utf-8") as f:
        f.write(content)


def init_project(project_name: str, output_dir: str = "."):
    """初始化完整的 AI 项目目录结构"""
    base = os.path.join(output_dir, project_name)
    today = datetime.now().strftime("%Y-%m-%d")

    print(f"\n{'='*60}")
    print(f"🚀 初始化 AI 项目: {project_name}")
    print(f"📁 目标路径: {os.path.abspath(base)}")
    print(f"{'='*60}\n")

    # 1. 创建目录
    print("📂 创建目录结构...")
    create_dirs(base)

    # 2. README 模板
    print("\n📝 生成 README 模板...")
    readmes = {
        "README.md": README_ROOT.format(project_name=project_name, date=today),
        "01_data/README.md": README_DATA,
        "02_tokenization/README.md": README_TOKEN,
        "03_models/README.md": README_MODEL,
        "04_experiments/experiment_log.md": README_EXPERIMENT,
        "02_tokenization/05_generated/README.md": README_GENERATED_DATA,
        "06_shared/README.md": README_SHARED,
        "01_data/04_quality_log.md": QUALITY_LOG,
        "_project_config.yaml": PROJECT_CONFIG.format(project_name=project_name, date=today),
    }
    for rel_path, content in readmes.items():
        write_file(os.path.join(base, rel_path), content)

    # 3. .gitignore
    print("\n🔒 生成 .gitignore...")
    write_file(os.path.join(base, ".gitignore"), GITIGNORE)

    # 4. 配置模板
    print("\n⚙️  预置配置模板...")
    config_dir = os.path.join(base, "06_shared/configs")
    configs = {
        "README.md": CONFIG_README,
        "training/deepspeed_zero2.yaml": DEEPSPEED_ZERO2,
        "training/deepspeed_zero3.yaml": DEEPSPEED_ZERO3,
        "training/lora_config.yaml": LORA_CONFIG,
        "training/qlora_config.yaml": QLORA_CONFIG,
        "training/wandb_config.yaml": WANDB_CONFIG.format(project_name=project_name),
    }
    for rel_path, content in configs.items():
        force_write(os.path.join(config_dir, rel_path), content)
    print(f"  ✅ 预置 5 个配置模板到 06_shared/configs/")

    # 5. 复制管理脚本到项目
    print("\n🔧 复制管理脚本...")
    skill_scripts = os.path.dirname(os.path.abspath(__file__))
    mgmt_scripts = [
        "check_project.py",
        "trace.py",
        "compare_runs.py",
        "checkpoint_mgmt.py",
        "doc_lint.py",
    ]
    target_scripts_dir = os.path.join(base, "scripts")
    os.makedirs(target_scripts_dir, exist_ok=True)
    for script in mgmt_scripts:
        src = os.path.join(skill_scripts, script)
        if os.path.exists(src):
            import shutil
            shutil.copy2(src, os.path.join(target_scripts_dir, script))
            print(f"  ✅ 复制: scripts/{script}")
    if not mgmt_scripts or not any(os.path.exists(os.path.join(skill_scripts, s)) for s in mgmt_scripts):
        print(f"  ⚠️  管理脚本尚未就绪，稍后可用 --upgrade 补充")

    print(f"\n{'='*60}")
    print(f"✅ 项目初始化完成！")
    print(f"📁 {os.path.abspath(base)}")
    print(f"\n📋 已生成:")
    print(f"   • 30+ 目录      完整的三层管线结构")
    print(f"   • 9 个 README   各层级入口文档")
    print(f"   • .gitignore    自动忽略大文件/密钥/日志")
    print(f"   • 5 个配置模板   DeepSpeed/LoRA/QLoRA/WandB")
    print(f"\n💡 下一步:")
    print(f"   cd {os.path.abspath(base)}")
    print(f"   python3 scripts/check_project.py --name {project_name}  # 健康检查")
    print(f"   python3 scripts/trace.py --name {project_name}         # 追踪链")
    print(f"   编辑 01_data/README.md 登记数据来源")


def add_source(project_name: str, source_name: str, output_dir: str = "."):
    base = os.path.join(output_dir, project_name)
    source_dir = os.path.join(base, "01_data/01_raw", f"source_{source_name}")
    os.makedirs(os.path.join(source_dir, "data"), exist_ok=True)
    today = datetime.now().strftime("%Y-%m-%d")
    meta = METADATA_TEMPLATE.format(source_name=source_name, date=today)
    write_file(os.path.join(source_dir, "metadata.yaml"), meta)
    print(f"✅ 数据来源已添加: source_{source_name}")


def add_tokenizer(project_name: str, approach: str, output_dir: str = "."):
    base = os.path.join(output_dir, project_name)
    approach_dir = os.path.join(base, "02_tokenization/02_custom", approach)
    for sd in ["scripts", "configs", "vocab_output"]:
        os.makedirs(os.path.join(approach_dir, sd), exist_ok=True)
    eval_md = f"# {approach} 评估\n\n## 内在评估\n\n| 指标 | 值 |\n|------|----|\n| 压缩率 |  |\n| 覆盖率 |  |\n| 词表大小 |  |\n\n## 下游评估\n\n| 下游任务 | 指标 | 值 |\n|----------|------|----|\n|          |      |    |\n"
    write_file(os.path.join(approach_dir, "eval.md"), eval_md)
    write_file(os.path.join(approach_dir, "README.md"),
               f"# {approach}\n\n## 方案说明\n\n## 技术细节\n\n## 决策记录\n")
    print(f"✅ 分词方案已添加: {approach}")


def add_training(project_name: str, arch: str, run_name: str, output_dir: str = "."):
    base = os.path.join(output_dir, project_name)
    run_dir = os.path.join(base, "03_models/01_pretraining", arch, run_name)
    for sd in ["configs", "scripts", "checkpoints", "logs", "eval"]:
        os.makedirs(os.path.join(run_dir, sd), exist_ok=True)
    print(f"✅ 预训练运行已添加: {arch}/{run_name}")


def add_posttrain(project_name: str, method: str, run_name: str, output_dir: str = "."):
    base = os.path.join(output_dir, project_name)
    run_dir = os.path.join(base, "03_models/02_post_training", method, run_name)
    for sd in ["data", "configs", "scripts", "checkpoints", "eval"]:
        os.makedirs(os.path.join(run_dir, sd), exist_ok=True)
    print(f"✅ 后训练运行已添加: {method}/{run_name}")


def add_rl(project_name: str, method: str, run_name: str, output_dir: str = "."):
    base = os.path.join(output_dir, project_name)
    run_dir = os.path.join(base, "03_models/03_rl", method, run_name)
    for sd in ["configs", "scripts", "checkpoints", "logs", "eval"]:
        os.makedirs(os.path.join(run_dir, sd), exist_ok=True)
    print(f"✅ RL 运行已添加: {method}/{run_name}")


def upgrade_project(project_name: str, output_dir: str = "."):
    """升级已有项目：补充 .gitignore、配置模板、管理脚本"""
    base = os.path.join(output_dir, project_name)
    if not os.path.isdir(base):
        print(f"❌ 项目不存在: {base}")
        sys.exit(1)

    today = datetime.now().strftime("%Y-%m-%d")
    print(f"\n🔧 升级项目: {project_name}")

    # 补充缺失目录
    print("📂 补充目录...")
    create_dirs(base)

    # .gitignore
    gi_path = os.path.join(base, ".gitignore")
    if not os.path.exists(gi_path):
        print("🔒 生成 .gitignore...")
        write_file(gi_path, GITIGNORE)

    # 配置模板
    config_dir = os.path.join(base, "06_shared/configs")
    configs = {
        "README.md": CONFIG_README,
        "training/deepspeed_zero2.yaml": DEEPSPEED_ZERO2,
        "training/deepspeed_zero3.yaml": DEEPSPEED_ZERO3,
        "training/lora_config.yaml": LORA_CONFIG,
        "training/qlora_config.yaml": QLORA_CONFIG,
        "training/wandb_config.yaml": WANDB_CONFIG.format(project_name=project_name),
    }
    for rel_path, content in configs.items():
        write_file(os.path.join(config_dir, rel_path), content)

    # 管理脚本
    skill_scripts = os.path.dirname(os.path.abspath(__file__))
    mgmt_scripts = ["check_project.py", "trace.py", "compare_runs.py", "checkpoint_mgmt.py", "doc_lint.py"]
    target_scripts_dir = os.path.join(base, "scripts")
    os.makedirs(target_scripts_dir, exist_ok=True)
    for script in mgmt_scripts:
        src = os.path.join(skill_scripts, script)
        if os.path.exists(src):
            import shutil
            shutil.copy2(src, os.path.join(target_scripts_dir, script))
            print(f"  ✅ 更新: scripts/{script}")

    print(f"✅ 项目升级完成！")


def main():
    parser = argparse.ArgumentParser(
        description="AI 项目目录结构管理工具 v2.0",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  init_ai_project.py --name MyLLM                     # 新建项目
  init_ai_project.py --name MyLLM --upgrade            # 升级已有项目
  init_ai_project.py --name MyLLM --add-source --source wiki
  init_ai_project.py --name MyLLM --add-tokenizer --approach bpe_v2
  init_ai_project.py --name MyLLM --add-training --arch llama3 --run v1
  init_ai_project.py --name MyLLM --add-posttrain --method sft --run chat_v1
  init_ai_project.py --name MyLLM --add-rl --method ppo --run reward_v1
        """)

    parser.add_argument("--name", type=str, help="项目名称")
    parser.add_argument("--output", type=str, default=".", help="输出父目录（默认当前目录）")
    parser.add_argument("--upgrade", action="store_true", help="升级已有项目（补充配置和管理脚本）")

    # 子命令
    parser.add_argument("--add-source", action="store_true", help="添加数据来源")
    parser.add_argument("--source", type=str, help="来源名称")
    parser.add_argument("--add-tokenizer", action="store_true", help="添加分词方案")
    parser.add_argument("--approach", type=str, help="分词方案名")
    parser.add_argument("--add-training", action="store_true", help="添加预训练运行")
    parser.add_argument("--arch", type=str, help="模型架构")
    parser.add_argument("--run", type=str, help="运行名称")
    parser.add_argument("--add-posttrain", action="store_true", help="添加后训练运行")
    parser.add_argument("--method", type=str, help="后训练方法 (sft/dpo/orpo/kto/simpo/rlhf)")
    parser.add_argument("--add-rl", action="store_true", help="添加 RL 运行")
    # --method 和 --run 共用

    args = parser.parse_args()

    if not args.name:
        parser.error("必须指定 --name")

    if args.upgrade:
        upgrade_project(args.name, args.output)
    elif args.add_source:
        if not args.source:
            parser.error("--add-source 需要 --source <名称>")
        add_source(args.name, args.source, args.output)
    elif args.add_tokenizer:
        if not args.approach:
            parser.error("--add-tokenizer 需要 --approach <方案名>")
        add_tokenizer(args.name, args.approach, args.output)
    elif args.add_training:
        if not args.arch or not args.run:
            parser.error("--add-training 需要 --arch <架构> --run <运行名>")
        add_training(args.name, args.arch, args.run, args.output)
    elif args.add_posttrain:
        if not args.method or not args.run:
            parser.error("--add-posttrain 需要 --method <方法> --run <运行名>")
        add_posttrain(args.name, args.method, args.run, args.output)
    elif args.add_rl:
        if not args.method or not args.run:
            parser.error("--add-rl 需要 --method <方法> --run <运行名>")
        add_rl(args.name, args.method, args.run, args.output)
    else:
        init_project(args.name, args.output)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)