Skill: deusyu/translate-book (version 1.0.0)

--- name: translate-book description: Translate books (PDF/DOCX/EPUB) into any language using parallel sub-agents. Converts input -> Markdown chunks -> translated chunks -> HTML/DOCX/EPUB/PDF. allowed-tools: Read, Write, Edit, Bash, Glob, Grep, Agent, AskUserQuestion metadata: {"openclaw":{"requires":{"bins":["python3","pandoc","ebook-convert"],"anyBins":["calibre","ebook-convert"]},"homepage":"https://github.com/deusyu/translate-book"}} --- # Book Translation Skill You are a book translation assistant. You translate entire books from one language to another by orchestrating a multi-step pipeline. ## Workflow ### 1. Collect Parameters Determine the following from the user's message: - **file_path**: Path to the input file (PDF, DOCX, or EPUB) — REQUIRED - **target_lang**: Target language code (default: `zh`) — e.g. zh, en, ja, ko, fr, de, es - **concurrency**: Number of parallel sub-agents per batch (default: `8`) - **custom_instructions**: Any additional translation instructions from the user (optional) If the file path is not provided, ask the user. ### 2. Preprocess — Convert to Markdown Chunks Run the conversion script to produce chunks: ```bash python3 {baseDir}/scripts/convert.py "<file_path>" --olang "<target_lang>" ``` This creates a `{filename}_temp/` directory containing: - `input.html`, `input.md` — intermediate files - `chunk0001.md`, `chunk0002.md`, ... — source chunks for translation - `manifest.json` — chunk manifest for tracking and validation - `config.txt` — pipeline configuration with metadata ### 3. Discover Chunks Use Glob to find all source chunks and determine which still need translation: ``` Glob: {filename}_temp/chunk*.md Glob: {filename}_temp/output_chunk*.md ``` Calculate the set of chunks that have a source file but no corresponding `output_` file. These are the chunks to translate. If all chunks already have translations, skip to step 5. ### 3.5. Build Glossary (term consistency) A separate sub-agent translates each chunk with a fresh context. Without shared state, the same proper noun can drift across multiple translations. The glossary makes every sub-agent see the same canonical translation for the terms that appear in its chunk. If `<temp_dir>/glossary.json` already exists, skip the rebuild — re-running the skill must not overwrite a hand-edited glossary. To force a rebuild, delete the file. Otherwise: 1. **Sample chunks**: read `chunk0001.md`, the last chunk, and 3 evenly-spaced middle chunks. If `chunk_count < 5`, sample all of them. 2. **Extract terms**: from the samples, identify proper nouns and recurring domain terms that need consistent translation across the book — typically people, places, organizations, technical concepts. Translate each into the target language. Skip generic vocabulary that any translator would render the same way. 3. **Write `glossary.json`** in the temp dir, matching this v2 schema: ```json { "version": 2, "terms": [ {"id": "Manhattan", "source": "Manhattan", "target": "曼哈顿", "category": "place", "aliases": [], "gender": "unknown", "confidence": "medium", "frequency": 0, "evidence_refs": [], "notes": ""} ], "high_frequency_top_n": 20, "applied_meta_hashes": {} } ``` Existing v1 `glossary.json` files are auto-upgraded to v2 on first load. v2 forbids the same surface form (source or alias) appearing in two different terms; if a v1 file has polysemous duplicate sources, the upgrade aborts with a disambiguation message. 4. **Count frequencies** by running: ```bash python3 {baseDir}/scripts/glossary.py count-frequencies "<temp_dir>" ``` This scans every `chunk*.md` (excluding `output_chunk*.md`), updates each term's `frequency` field, and writes back atomically. The glossary is hand-editable. If the user edits a `target` field after a partial run, that's fine for this commit — affected chunks won't auto-re-translate yet (commit 3 adds precise re-translation). ### 4. Parallel Translation with Sub-Agents **Each chunk gets its own independent sub-agent** (1 chunk = 1 sub-agent = 1 fresh context). This prevents context accumulation and output truncation. Launch chunks in batches to respect API rate limits: - Each batch: up to `concurrency` sub-agents in parallel (default: 8) - Wait for the current batch to complete before launching the next **Spawn each sub-agent with the following task.** Use whatever sub-agent/background-agent mechanism your runtime provides (e.g. the Agent tool, sessions_spawn, or equivalent). The output file is `output_` prefixed to the source filename: `chunk0001.md` → `output_chunk0001.md`. > Translate the file `<temp_dir>/chunk<NNNN>.md` to {TARGET_LANGUAGE} and write the result to `<temp_dir>/output_chunk<NNNN>.md`. Follow the translation rules below. Output only the translated content — no commentary. Each sub-agent receives: - The single chunk file it is responsible for - The temp directory path - The target language - The translation prompt (see below) - A per-chunk term table (see "Term table assembly" below) - Any custom instructions **Term table assembly** — before spawning a sub-agent, run: ```bash python3 {baseDir}/scripts/glossary.py print-terms-for-chunk "<temp_dir>" "chunk<NNNN>.md" ``` Capture stdout. The CLI emits a 3-column markdown table (`原文 | 别名 | 译文`) of every term that either appears in this chunk (by source OR any alias) OR is in the top-N most-frequent terms book-wide. Inject the table as `{TERM_TABLE}` in rule #13 of the translation prompt. **If stdout is empty (no glossary, or no relevant terms), omit rule #13 from this chunk's prompt entirely** — do not leave a dangling `{TERM_TABLE}` placeholder. **Each sub-agent's task**: 1. Read the source chunk file (e.g. `chunk0001.md`) 2. Translate the content following the translation rules below 3. Write the translated content to `output_chunk0001.md` 4. Write observations to `output_chunk0001.meta.json` matching the schema below. **Non-blocking** — leave fields empty if unsure; do not invent entities. Always emit the file (even if all arrays are empty), because its presence + content hash is how the main agent tracks whether feedback was already merged. **Sub-agent meta schema** (`output_chunk<NNNN>.meta.json`): ```json { "schema_version": 1, "new_entities": [ {"source": "Taig", "target_proposal": "泰格", "category": "person", "evidence": "<≤200-char quote from the chunk>"} ], "alias_hypotheses": [ {"variant": "Taig", "may_be_alias_of_source": "Tai", "evidence": "<≤200-char quote>"} ], "attribute_hypotheses": [ {"entity_source": "Tai", "attribute": "gender", "value": "male", "confidence": "high", "evidence": "<≤200-char quote>"} ], "used_term_sources": ["Tai", "Manhattan"], "conflicts": [ {"entity_source": "Tai", "field": "target", "injected": "泰", "observed_better": "太一", "evidence": "<≤200-char quote>"} ] } ``` **Do NOT include a `chunk_id` field** — chunk identity is derived from the filename. Putting it in the payload creates a hallucination hole and validation will reject the file. The meta file is read by the main agent later and merged into `glossary.json` (see `merge_meta.py`). Sub-agents should fill the schema honestly: cite real quotes from the chunk, never invent entities to "look productive". An empty meta is a perfectly valid output. **IMPORTANT**: Each sub-agent translates exactly ONE chunk and writes the result directly to the output file. No START/END markers needed. #### Translation Prompt for Sub-Agents Include this translation prompt in each sub-agent's instructions (replace `{TARGET_LANGUAGE}` with the actual language name, e.g. "Chinese"): --- 请翻译markdown文件为 {TARGET_LANGUAGE}. IMPORTANT REQUIREMENTS: 1. 严格保持 Markdown 格式不变，包括标题、链接、图片引用等 2. 仅翻译文字内容，保留所有 Markdown 语法和文件名 3. 删除空链接、不必要的字符和如: 行末的'\\'。页码已由 convert.py 上游处理，不要再删除独立的数字行（可能是年份 1984、章节编号、引用编号等正文内容）。 4. 保证格式和语义准确翻译内容自然流畅 5. 只输出翻译后的正文内容，不要有任何说明、提示、注释或对话内容。 6. 表达清晰简洁，不要使用复杂的句式。请严格按顺序翻译，不要跳过任何内容。 7. 必须保留所有图片引用，包括： - 所有 ![alt](path) 格式的图片引用必须完整保留 - 图片文件名和路径不要修改（如 media/image-001.png） - 图片alt文本可以翻译，但必须保留图片引用结构 - 不要删除、过滤或忽略任何图片相关内容 - 图片引用示例：![Figure 1: Data Flow](media/image-001.png) -> ![图1：数据流](media/image-001.png) - **原始 HTML 标签（如 `<img alt="..." />`、`<a title="...">`）必须保持合法**：翻译 `alt`、`title` 等属性值内部文本时，下列字符会破坏 HTML 结构，必须替换为安全形式（仅适用于**原始 HTML 标签的属性值内部**；普通 Markdown 正文、代码块、URL 不要主动转义）： | 字符 | 在属性值内的危险 | 替换为 | |------|---------------|--------| | `"` | 闭合 `attr="..."` | 目标语言合适的弯引号（如中文 `“` `”`）或 `&quot;` | | `'` | 闭合 `attr='...'` | 目标语言合适的弯引号（如中文 `‘` `’`）或 `&#39;` | | `<` | 被解析为新标签 | `&lt;` | | `>` | 被解析为标签结束 | `&gt;` | | `&` | 被解析为实体起始（除非已是 `&xxx;`） | `&amp;` | 不要修改 `src`、`href` 等结构性属性的值，只翻译可见文本属性（`alt`、`title`）。 - 错误示例：`alt="爱丽丝拿着标着"喝我"的瓶子"` ← 内层英文 `"` 把外层 alt 撑断了 - 正确示例：`alt="爱丽丝拿着标着“喝我”的瓶子"` 或 `alt="爱丽丝拿着标着&quot;喝我&quot;的瓶子"` 8. 智能识别和处理多级标题，按照以下规则添加markdown标记： - 主标题（书名、章节名等）使用 # 标记 - 一级标题（大节标题）使用 ## 标记 - 二级标题（小节标题）使用 ### 标记 - 三级标题（子标题）使用 #### 标记 - 四级及以下标题使用 ##### 标记 9. 标题识别规则： - 独立成行的较短文本（通常少于50字符） - 具有总结性或概括性的语句 - 在文档结构中起到分隔和组织作用的文本 - 字体大小明显不同或有特殊格式的文本 - 数字编号开头的章节文本（如 "1.1 概述"、"第三章"等） 10. 标题层级判断： - 根据上下文和内容重要性判断标题层级 - 章节类标题通常为高层级（# 或 ##） - 小节、子节标题依次降级（### #### #####） - 保持同一文档内标题层级的一致性 11. 注意事项： - 不要过度添加标题标记，只对真正的标题文本添加 - 正文段落不要添加标题标记 - 如果原文已有markdown标题标记，保持其层级结构 12. {CUSTOM_INSTRUCTIONS if provided} 13. 术语一致性：以下术语必须严格使用指定译法，不要自行变换。表格中"原文"列**或"别名"列**任一形式出现在正文中时，都必须翻译为"译文"列对应的形式。 {TERM_TABLE} markdown文件正文: --- ### 4.5. Merge Sub-Agent Meta Into Glossary (after each batch) Each sub-agent emitted an `output_chunk<NNNN>.meta.json` alongside its translated chunk. After every batch completes, the main agent merges these observations into the canonical glossary so subsequent batches see an enriched glossary. 1. Run prepare-merge: ```bash python3 {baseDir}/scripts/merge_meta.py prepare-merge "<temp_dir>" ``` Capture stdout JSON. It contains four arrays: - `auto_apply` — new entities with no glossary collision and unanimous (target, category) across all proposing chunks. - `decisions_needed` — items requiring main-agent judgment. Each has `id`, `kind`, an `options` array, and the data needed to pick. Kinds: - `alias` — `{variant, candidate_source, evidence}`. Choices: `yes_alias` / `no_separate_entity` / `skip`. - `conflict` — `{entity_source, field, current, proposed, evidence}`. Choices: `keep_current` / `accept_proposed` / `record_in_notes`. - `new_entity_existing_alias` — sub-agents propose `proposed_source` as a new entity, but it's already someone's alias. `{proposed_source, currently_alias_of, promoted_variants: [{target_proposal, category, evidence, evidence_chunks}, ...]}`. Choices: one `use_variant_N` per distinct (target, category) promotion variant (promote `proposed_source` to standalone with that target+category, removing it from the host's aliases) / `keep_as_alias` / `skip`. - `existing_entity_conflict` — sub-agents proposed a (target, category) for `entity_source` that differs from the canonical. Multiple distinct differing proposals all get exposed. `{entity_source, current_target, current_category, proposed_variants: [{target_proposal, category, evidence, evidence_chunks}, ...]}`. Choices: `keep_current` / one `use_variant_N` per competing proposal (overwrites both target AND category, stamps the prior values into notes) / `record_in_notes` (canonical unchanged; every proposed variant gets logged to notes). - `alias_or_new_entity` — `variant` has multiple competing options that can't all coexist under v2's surface-form uniqueness rule. Triggered when (a) `variant` was proposed both as a new standalone entity AND as an alias of one or more candidates, OR (b) `variant` was proposed as an alias of two or more different candidates with no standalone competitor. `{variant, alias_candidates: [{candidate_source, evidence, evidence_chunks}, ...], standalone_variants: [{target_proposal, category, evidence, evidence_chunks}, ...]}`. Choices: one `use_alias_N` per candidate (attach as alias of that candidate), one `use_standalone_N` per competing standalone proposal (add as standalone with that target+category), or `skip`. - [REDACTED_SECRET] — `{source, variants: [{target_proposal, category, evidence, evidence_chunks}, ...]}`. Choices: `use_variant_0`, `use_variant_1`, ..., `skip`. - `consumed_chunk_ids` — every meta file scanned this round (regardless of whether it produced a finding). These hashes get recorded in `applied_meta_hashes` on apply. - `malformed_meta_chunk_ids` — meta files that failed validation. Quarantined: not consumed, not crashing the run. Surface them in your batch progress. 2. **If `consumed_chunk_ids` is empty** → nothing was scanned; skip to Step 5. 3. **If `consumed_chunk_ids` is non-empty but both `auto_apply` and `decisions_needed` are empty** → still pipe `{"auto_apply": [], "decisions": [], "consumed_chunk_ids": [...]}` into `apply-merge` so the hashes get recorded. **Skipping this is the bug** — no-op metas would re-scan forever otherwise. 4. **Otherwise, resolve each decision**: - Read its evidence quotes inline. - Pick one option from its `options` array. - Build a `decisions` entry that round-trips the original decision plus your choice. The entry MUST include the original `kind` and (for [REDACTED_SECRET]) the `variants` array, so apply-merge can validate and act: ```json {"id": "d1", "kind": "alias", "variant": "Taig", "candidate_source": "Tai", "choice": "yes_alias"} ``` 5. Pipe the decisions JSON into apply-merge: ```bash echo '{"auto_apply": [...], "decisions": [...], "consumed_chunk_ids": [...]}' \ | python3 {baseDir}/scripts/merge_meta.py apply-merge "<temp_dir>" ``` Surface the summary JSON (`auto_applied`, `decisions_resolved`, `consumed_chunks`, `errors`) in your batch progress message. **apply-merge is transactional.** If any decision is malformed (wrong choice for kind, missing fields, references a non-existent entity), the entire batch aborts with a non-zero exit and stderr details — no glossary mutation, no hashes recorded. On non-zero exit, fix the offending decision and re-pipe; `prepare-merge` will surface the same proposals because nothing was consumed. **Decision order in the input list is not significant.** `apply-merge` internally dispatches entity-creating decisions before alias-attaching ones, so `yes_alias` decisions whose candidate is created by another decision in the same batch (a `use_standalone_N`, `use_variant_N`, or `promote_to_separate_entity`) succeed regardless of the order you pass them in. Alias chains (e.g. `Taighi → Taig` where `Taig → Tai` is also a pending alias decision) resolve via a fixed-point loop within the alias-attacher pass — you don't need to topo-sort or sequence chained aliases manually. On a fresh run after a previous interrupted batch, `prepare-merge` will pick up any meta files left behind. Don't manually delete them. ### 5. Verify Completeness and Retry After all batches complete, use Glob to check that every source chunk has a corresponding output file. If any are missing, retry them — each missing chunk as its own sub-agent. Maximum 2 attempts per chunk (initial + 1 retry). Also read `manifest.json` and verify: - Every chunk id has a corresponding output file - No output file is empty (0 bytes) Then run the meta-merge observability snapshot: ```bash python3 {baseDir}/scripts/merge_meta.py status "<temp_dir>" ``` Surface a one-line summary in the verification report: > Translated chunks: 50 • Meta files: 48 found / 47 consumed • Malformed: 1 (chunk0099 — see stderr) • Chunks missing meta: chunk0017, chunk0042 Severity rules (none of these fail the run — meta is non-blocking): - `unmerged_meta_files > 0` after Step 4.5 ran → bug, flag prominently. Resume should have caught this. - `malformed_meta_files > 0` → sub-agent emitted invalid meta; print chunk_ids and a "fix the file by hand and re-run if you want this chunk's feedback merged" note. - `meta_files_found < translated_chunks` → sub-agent-compliance issue (some chunks didn't emit meta at all). Print missing chunk_ids. Report any chunks that failed translation after retry. ### 6. Translate Book Title Read `config.txt` from the temp directory to get the `original_title` field. Translate the title to the target language. For Chinese, wrap in 书名号: `《translated_title》`. ### 7. Post-process — Merge and Build Run the build script with the translated title: ```bash python3 {baseDir}/scripts/merge_and_build.py --temp-dir "<temp_dir>" --title "<translated_title>" --cleanup ``` The `--cleanup` flag removes intermediate files (chunks, input.html, etc.) after a fully successful build. If the user asked to keep intermediates, omit `--cleanup`. The script reads `output_lang` from `config.txt` automatically. Optional overrides: `--lang`, `--author`. This produces in the temp directory: - `output.md` — merged translated markdown - `book.html` — web version with floating TOC - `book_doc.html` — ebook version - `book.docx`, `book.epub`, `book.pdf` — format conversions (requires Calibre) ### 8. Report Results Tell the user: - Where the output files are located - How many chunks were translated - The translated title - List generated output files with sizes - Any format generation failures

path: AGENTS.md
content: # AGENTS.md

## Project

translate-book is a Codex Skill that translates books (PDF/DOCX/EPUB) into any language using parallel subagents. Published on ClawHub as `translate-book` and on GitHub as `deusyu/translate-book`.

## Structure

- `SKILL.md` — Skill definition, the orchestration logic that Codex / OpenClaw follows
- `scripts/convert.py` — PDF/DOCX/EPUB → Markdown chunks (via Calibre HTMLZ)
- `scripts/manifest.py` — SHA-256 chunk tracking and merge validation
- `scripts/glossary.py` — Term-consistency glossary; per-chunk term tables injected into sub-agent prompts
- `scripts/meta.py` — Per-chunk sub-agent observation file schema
- `scripts/merge_meta.py` — Batch-boundary merge of sub-agent observations into the canonical glossary
- `scripts/merge_and_build.py` — Merge translated chunks → HTML/DOCX/EPUB/PDF
- `scripts/calibre_html_publish.py` — Calibre format conversion wrapper
- `scripts/template.html`, `scripts/template_ebook.html` — HTML templates

## Testing changes

Use a small file for quick checks, or the checked-in baseline book for the repository's full-pipeline test.

Quick smoke test:

```bash
python3 scripts/convert.py /path/to/small.pdf --olang zh
# then run translation via the skill
python3 scripts/merge_and_build.py --temp-dir <name>_temp --title "test"
```

Full baseline test:

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "test"
```

Verify: all output_chunk*.md files exist, manifest validation passes, output formats generate.

## Conventions

- Only `chunk*.md` naming — no `page*` legacy support
- SKILL.md frontmatter must stay single-line per field (OpenClaw parser requirement)
- Script paths in SKILL.md use `{baseDir}` not hardcoded paths
- Subagent instructions in SKILL.md must be platform-neutral (work on Codex, OpenClaw, Codex)
- Checked-in baseline inputs live under `tests/baselines/<book-id>/`; generated full-pipeline outputs live under `tests/.artifacts/`
- README changes must be synced to both README.md and README.zh-CN.md
- Releases follow `.claude/commands/release.md` — three commands in order: `git push origin main`, `git tag vX.Y.Z && git push --tags`, `clawhub publish ./ --version X.Y.Z`. Do not skip the git tag; it's the only version anchor in the repo

## Do not

- Do not reintroduce `page*` file support — it was intentionally removed
- Do not hardcode `~/.Codex/skills/` paths in SKILL.md — use `{baseDir}`
- Do not put platform-specific tool names (Agent, sessions_spawn) in `allowed-tools` as the only option — keep the whitelist cross-platform
- Do not add mtime-based incremental rebuild for HTML/format generation — the current skip logic is intentionally simple (existence check). Metadata/template changes require manual cleanup. This is documented in the README.


path: CLAUDE.md
content: # CLAUDE.md

## Project

translate-book is a Claude Code Skill that translates books (PDF/DOCX/EPUB) into any language using parallel subagents. Published on ClawHub as `translate-book` and on GitHub as `deusyu/translate-book`.

## Structure

- `SKILL.md` — Skill definition, the orchestration logic that Claude Code / OpenClaw follows
- `scripts/convert.py` — PDF/DOCX/EPUB → Markdown chunks (via Calibre HTMLZ)
- `scripts/manifest.py` — SHA-256 chunk tracking and merge validation
- `scripts/glossary.py` — Term-consistency glossary; per-chunk term tables injected into sub-agent prompts
- `scripts/meta.py` — Per-chunk sub-agent observation file schema
- `scripts/merge_meta.py` — Batch-boundary merge of sub-agent observations into the canonical glossary
- `scripts/merge_and_build.py` — Merge translated chunks → HTML/DOCX/EPUB/PDF
- `scripts/calibre_html_publish.py` — Calibre format conversion wrapper
- `scripts/template.html`, `scripts/template_ebook.html` — HTML templates

## Testing changes

Test with a small PDF to verify the full pipeline:

```bash
python3 scripts/convert.py /path/to/small.pdf --olang zh
# then run translation via the skill
python3 scripts/merge_and_build.py --temp-dir <name>_temp --title "test"
```

Verify: all output_chunk*.md files exist, manifest validation passes, output formats generate.

## Conventions

- Only `chunk*.md` naming — no `page*` legacy support
- Pipeline output artifacts use the canonical names `book.html`, `book_doc.html`, `book.docx`, `book.epub`, `book.pdf`. Internal scripts and skip/cache logic depend on these names; if title-based filenames are added later they must be optional aliases/copies, not silent replacements
- SKILL.md frontmatter must stay single-line per field (OpenClaw parser requirement)
- Script paths in SKILL.md use `{baseDir}` not hardcoded paths
- Subagent instructions in SKILL.md must be platform-neutral (work on Claude Code, OpenClaw, Codex)
- README changes must be synced to both README.md and README.zh-CN.md
- Releases follow `.claude/commands/release.md` — three commands in order: `git push origin main`, `git tag vX.Y.Z && git push --tags`, `clawhub publish ./ --version X.Y.Z`. Do not skip the git tag; it's the only version anchor in the repo

## Do not

- Do not reintroduce `page*` file support — it was intentionally removed
- Do not hardcode `~/.claude/skills/` paths in SKILL.md — use `{baseDir}`
- Do not put platform-specific tool names (Agent, sessions_spawn) in `allowed-tools` as the only option — keep the whitelist cross-platform
- Do not add mtime-based incremental rebuild for HTML/format generation — the current skip logic is intentionally simple (existence check). Metadata/template changes require manual cleanup. This is documented in the README.


path: README.md
content: # Rainman Translate Book

English | [中文](README.zh-CN.md)

Claude Code skill that translates entire books (PDF/DOCX/EPUB) into any language using parallel subagents.

> Inspired by [claude_translater](https://github.com/wizlijun/claude_translater). The original project uses shell scripts as its entry point, coordinating the Claude CLI with multiple step scripts to perform chunked translation. This project restructures the workflow as a Claude Code Skill, using subagents to translate chunks in parallel, with manifest-driven integrity checks, resumable runs, and multi-format output unified into a single pipeline. As the project structure and implementation differ significantly from the original, this is an independent project rather than a fork.

---

## How It Works

```
Input (PDF/DOCX/EPUB)
  │
  ▼
Calibre ebook-convert → HTMLZ → HTML → Markdown
  │
  ▼
Split into chunks (chunk0001.md, chunk0002.md, ...)
  │  manifest.json tracks chunk hashes
  ▼
Parallel subagents (8 concurrent by default)
  │  each subagent: read 1 chunk → translate → write output_chunk*.md
  │  batched to respect API rate limits
  ▼
Validate (manifest hash check, 1:1 source↔output match)
  │
  ▼
Merge → Pandoc → HTML (with TOC) → Calibre → DOCX / EPUB / PDF
```

Each chunk gets its own independent subagent with a fresh context window. This prevents context accumulation and output truncation that happen when translating a full book in a single session.

## Features

- **Parallel subagents** — 8 concurrent translators per batch, each with isolated context
- **Resumable** — chunk-level resume; already-translated chunks are skipped on re-run (for metadata/template changes, use a fresh run)
- **Manifest validation** — SHA-256 hash tracking prevents stale or corrupt outputs from being merged
- **Multi-format output** — HTML (with floating TOC), DOCX, EPUB, PDF
- **Multi-language** — zh, en, ja, ko, fr, de, es (extensible)
- **PDF/DOCX/EPUB input** — Calibre handles the conversion heavy lifting

## Prerequisites

- **Claude Code CLI** — installed and authenticated
- **Calibre** — `ebook-convert` command must be available ([download](https://calibre-ebook.com/))
- **Pandoc** — for HTML↔Markdown conversion ([download](https://pandoc.org/))
- **Python 3** with:
  - `pypandoc` — required (`pip install pypandoc`)
  - `beautifulsoup4` — optional, for better TOC generation (`pip install beautifulsoup4`)

## Quick Start

### 1. Install the skill

**Option A: npx (recommended)**

```bash
npx skills add deusyu/translate-book -a claude-code -g
```

**Option B: ClawHub**

```bash
clawhub install translate-book
```

**Option C: Git clone**

```bash
git clone https://github.com/deusyu/translate-book.git ~/.claude/skills/translate-book
```


### 2. Translate a book

In Claude Code, say:

```
translate /path/to/book.pdf to Chinese
```

Or use the slash command:

```
/translate-book translate /path/to/book.pdf to Japanese
```

The skill handles the full pipeline automatically — convert, chunk, translate in parallel, validate, merge, and build all output formats.

### 3. Find your outputs

All files are in `{book_name}_temp/`:

| File | Description |
|------|-------------|
| `output.md` | Merged translated Markdown |
| `book.html` | Web version with floating TOC |
| `book.docx` | Word document |
| `book.epub` | E-book |
| `book.pdf` | Print-ready PDF |

## Repository Test Assets

- Checked-in baseline inputs live under `tests/baselines/<book-id>/`.
- Generated full-pipeline outputs live under `tests/.artifacts/` and should not be committed.
- Because `scripts/convert.py` writes `{book_name}_temp/` under the current working directory, run repository baseline tests from inside `tests/.artifacts/` to keep generated files out of the repo root.

### Full-Pipeline Baseline Example

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "test"
```

## Pipeline Details

### Step 1: Convert

```bash
python3 scripts/convert.py /path/to/book.pdf --olang zh
```

Calibre converts the input to HTMLZ, which is extracted and converted to Markdown, then split into chunks (~6000 chars each). A `manifest.json` records the SHA-256 hash of each source chunk for later validation.

### Step 1.5: Glossary (term consistency across chunks)

Each chunk is translated by a fresh-context sub-agent, which means the same proper noun can drift across multiple translations on a 100-chunk book. To fix this, the skill builds a glossary before translation:

1. Sample 5 chunks (first, last, 3 evenly-spaced middle).
2. Extract proper nouns and recurring domain terms; pick canonical translations.
3. Write `<temp_dir>/glossary.json` (hand-editable schema below).
4. Run `python3 scripts/glossary.py count-frequencies <temp_dir>` to populate per-term frequencies (ASCII terms use word-boundary regex so `cat` doesn't match `category`; CJK terms use substring; single-CJK-char terms are rejected; aliases count toward the term they belong to).
5. For each chunk, the orchestrator calls `python3 scripts/glossary.py print-terms-for-chunk <temp_dir> chunkNNNN.md` and injects the resulting 3-column (`原文 | 别名 | 译文`) markdown table into that chunk's prompt as a hard constraint. Term selection = (terms whose source OR any alias appears in this chunk) ∪ (top-N most-frequent book-wide).

```json
{
  "version": 2,
  "terms": [
    {"id": "Manhattan", "source": "Manhattan", "target": "曼哈顿",
     "category": "place", "aliases": [], "gender": "unknown",
     "confidence": "medium", "frequency": 12,
     "evidence_refs": [], "notes": ""}
  ],
  "high_frequency_top_n": 20,
  "applied_meta_hashes": {}
}
```

Existing v1 `glossary.json` files are auto-upgraded to v2 on first load. v2 forbids the same surface form (source or alias) appearing in two different terms; if a v1 file has polysemous duplicate sources, the upgrade aborts with a disambiguation message — fix the file by hand and reload.

Edit `glossary.json` between runs to fix translations; existing `glossary.json` is never overwritten — delete it to rebuild from scratch.

> **Note on partial reruns**: in the current release, editing `glossary.json` after some chunks have been translated does NOT auto-invalidate those chunks — they keep their old translations. Precise glossary-driven re-translation is planned for the next commit. For now, delete the affected `output_chunk*.md` files (or the whole temp dir) to apply glossary edits.

### Step 2: Translate (parallel subagents)

The skill launches subagents in batches (default: 8 concurrent). Each subagent:

1. Reads one source chunk (e.g. `chunk0042.md`)
2. Translates to the target language
3. Writes the result to `output_chunk0042.md`

If a run is interrupted, re-running skips chunks that already have valid output files. Failed chunks are retried once automatically.

### Step 3: Merge & Build

```bash
python3 scripts/merge_and_build.py --temp-dir book_temp --title "《translated title》"
```

Before merging, the script validates:
- Every source chunk has a corresponding output file (1:1 match)
- Source chunk hashes match the manifest (no stale outputs)
- No output files are empty

Then: merge → Pandoc HTML → inject TOC → Calibre generates DOCX, EPUB, PDF.

**Note:** `{book_name}_temp/` is a working directory for a single translation run. If you change the title, author, output language, template, or image assets, either use a fresh temp directory or delete the existing final artifacts (`output.md`, `book*.html`, `book.docx`, `book.epub`, `book.pdf`) before re-running.

## Project Structure

| File | Purpose |
|------|---------|
| `SKILL.md` | Claude Code skill definition — orchestrates the full pipeline |
| `scripts/convert.py` | PDF/DOCX/EPUB → Markdown chunks via Calibre HTMLZ |
| `scripts/manifest.py` | Chunk manifest: SHA-256 tracking and merge validation |
| `scripts/glossary.py` | Glossary management: per-chunk term tables for consistent terminology |
| `scripts/meta.py` | Per-chunk sub-agent observation file schema (`output_chunkNNNN.meta.json`) |
| `scripts/merge_meta.py` | Batch-boundary merge: sub-agent observations → canonical glossary |
| `scripts/merge_and_build.py` | Merge chunks → HTML → DOCX/EPUB/PDF |
| `scripts/calibre_html_publish.py` | Calibre wrapper for format conversion |
| `scripts/template.html` | Web HTML template with floating TOC |
| `scripts/template_ebook.html` | Ebook HTML template |
| `tests/baselines/` | Checked-in baseline book inputs for full-pipeline testing |
| `tests/.artifacts/` | Ignored full-pipeline test outputs |

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `Calibre ebook-convert not found` | Install Calibre and ensure `ebook-convert` is in PATH |
| `Manifest validation failed` | Source chunks changed since splitting — re-run `convert.py` |
| `Missing source chunk` | Source file deleted — re-run `convert.py` to regenerate |
| Incomplete translation | Re-run the skill — it resumes from where it stopped |
| Changed title/template/assets but output didn't update | Delete existing `output.md`, `book*.html`, `book.docx`, `book.epub`, `book.pdf` from the temp dir, then re-run `merge_and_build.py` |
| Want page-number footers stripped from PDF output | By default, monotonic page-number sequences (e.g. `1, 2, 3, ...`) are auto-detected and dropped while outliers like years (`1984`), chapter numbers, and citation indices stay preserved. If detection misses your case, pass `--strip-page-numbers` to `convert.py` to aggressively delete every standalone-digit line. The flag aborts if a cached `input.md` or `chunk*.md` already exists — delete them first so the flag actually takes effect. |
| `output.md exists but manifest invalid` | Stale output — the script auto-deletes and re-merges |
| `Glossary upgrade rejected: duplicate source` | v2 disallows two terms sharing a source/alias surface form. Edit `glossary.json` to disambiguate (e.g., rename one source from `Apple` to `Apple (Inc.)`) and reload. |
| PDF generation fails | Ensure Calibre is installed with PDF output support |

## Roadmap

Tracking [issue #7](https://github.com/deusyu/translate-book/issues/7) — name/term inconsistency and pronoun/gender errors across chunks. Today's glossary covers high-frequency main entities; secondary characters, spelling variants, and pronoun resolution are not yet addressed. The plan is four independently shippable phases.

### Design principles

- **Scripts do bookkeeping; LLMs do semantic merge.** State, schemas, dedup, hashing, IO are deterministic Python. Naming, gender attribution, alias judgment, conflict resolution are LLM calls.
- **Single writer for shared state.** Only the main agent writes `glossary.json` and `run_state.json`; sub-agents write per-chunk meta files. No locking needed.
- **Conservative merge.** New entities require evidence; alias merges need LLM judgment, not just string similarity; gender starts at `unknown` and only moves up under explicit evidence; canonical values aren't silently overwritten on conflict.
- **Three-layer state, three separate files.** `glossary.json` (canonical, sub-agents read), `output_chunkNNNN.meta.json` (raw per-chunk observations), `run_state.json` (orchestration).

### Phase 1 — Sub-agent feedback + glossary merge (shipped)

Closes the read+write loop. Glossary v2 adds `id`, `aliases`, `gender`, `confidence`, `evidence_refs`, `notes` (v1 files auto-upgrade on first load; the term table is now 3-col and aliases participate in selection). Sub-agents emit `output_chunkNNNN.meta.json` alongside each translated chunk. `scripts/merge_meta.py` (`prepare-merge` / `apply-merge` / `status`) merges per-batch with conservative rules: surface-form uniqueness enforced, malformed metas quarantined (warn + skip + count), confidence escalation via both `evidence_chunks` and `used_term_sources`, FIFO-cap at 5. See SKILL.md Step 4 / Step 4.5 / Step 5.

### Phase 2 — Neighbor context for pronouns (not started, independent of Phase 1)

Inject `prev_excerpt` (last ~300 chars of previous chunk) and `next_excerpt` (first ~300 chars of next chunk) into each sub-agent prompt as read-only context. No new state files. Pure prompt-assembly change.

### Phase 3 — Selective re-translation (not started, depends on Phase 1)

Phase 1's batch feedback only improves *forward*. Selective rerun closes the *backward* loop: new `scripts/run_state.py` + `run_state.json` schema; per-chunk tracking of `glossary_version_used`, `entity_ids_used`, `output_hash`; five decision rules for deciding which chunks need re-translation this run.

### Phase 4 — Bootstrap warm-up (experimental, gated on Phase 1 data)

Phase 1 grows the glossary batch-by-batch, so the first batch sees the smallest glossary and has the highest drift risk. Possible approaches: sequential bootstrap, variable concurrency, or skip entirely. Decision belongs to whoever has run the system on real books.

> The specific schemas and file layouts in each phase are illustrative — they may shift as Phase 1 hits real data. Phase 4 is gated on data; Phase 3 may be re-scoped or dropped if Phase 1 alone proves "good enough".

### Parallel track — Pipeline / UX backlog (not started, separate from issue #7)

Recent PR discussions also surfaced several useful workflow improvements, but these are broader than one-off patches and touch repo contracts (artifact names, temp-dir behavior, cleanup semantics, or EPUB compatibility scope). These are being tracked as maintainer-owned roadmap items rather than merged directly from the current PRs:

- **Explicit EPUB cover support.** Add `--cover <image>` and pass it through the HTML -> EPUB Calibre step. Keep `--cover-from <epub>` / EPUB cover auto-extraction out of scope until the project is ready to own EPUB parsing compatibility across different package layouts. (context: closed #3)
- **Configurable temp workspace location.** Keep the current cwd-local `{book_name}_temp/` default for compatibility. If this is revisited later, prefer an explicit `--temp-root` / `--work-dir` style option rather than silently changing the default location. (context: closed #4)
- **Safer Calibre/Pandoc artifact cleanup.** Continue improving cleanup rules incrementally under regression tests, while preserving the current page-number detection semantics and not stripping real display-math delimiters or content numbers. (context: closed #5)
- **Optional user-facing export names.** Keep canonical pipeline artifacts as `book.html`, `book_doc.html`, `book.docx`, `book.epub`, and `book.pdf`. If title-based filenames are added later, they should likely be optional exported aliases/copies rather than a silent replacement of the internal artifact contract. (context: closed #6)

## Star History

If you find this project helpful, please consider giving it a Star ⭐!

[![Star History Chart](https://api.star-history.com/svg?repos=deusyu/translate-book&type=Date)](https://star-history.com/#deusyu/translate-book&Date)

## Sponsor

If this project saves you time, consider sponsoring to keep it maintained and improved.

[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-pink?logo=github)](https://github.com/sponsors/deusyu)

## License

[MIT](LICENSE)


path: README.zh-CN.md
content: # Rainman Translate Book

[English](README.md) | 中文

Claude Code Skill，使用并行 subagent 将整本书（PDF/DOCX/EPUB）翻译成任意语言。

> 本项目受 [claude_translater](https://github.com/wizlijun/claude_translater) 启发。原项目以 shell 脚本为入口，配合 Claude CLI 和多个步骤脚本完成分块翻译；本项目则将流程重构为 Claude Code Skill，使用 subagent 按 chunk 并行翻译，并引入 manifest 驱动的完整性校验，将续跑和多格式输出整合为更统一的流水线。由于项目结构和实现方式均与原项目不同，本项目为独立实现，而非 fork。

---

## 工作原理

```
输入文件 (PDF/DOCX/EPUB)
  │
  ▼
Calibre ebook-convert → HTMLZ → HTML → Markdown
  │
  ▼
拆分为 chunk（chunk0001.md, chunk0002.md, ...）
  │  manifest.json 记录每个 chunk 的 SHA-256 hash
  ▼
并行 subagent 翻译（默认 8 路并发）
  │  每个 subagent：读取 1 个 chunk → 翻译 → 写入 output_chunk*.md
  │  分批执行，控制 API 速率
  ▼
校验（manifest hash 比对，源文件↔输出文件 1:1 匹配）
  │
  ▼
合并 → Pandoc → HTML（含目录）→ Calibre → DOCX / EPUB / PDF
```

每个 chunk 由独立的 subagent 翻译，拥有全新的上下文窗口。这避免了单次会话翻译整本书时的上下文堆积和输出截断问题。

## 功能特性

- **并行 subagent** — 每批 8 个并发翻译器，各自独立上下文
- **可续跑** — chunk 级续跑，重新运行时自动跳过已翻译的 chunk（元数据或模板变更建议全新运行）
- **Manifest 校验** — SHA-256 hash 追踪，防止过时或损坏的输出被合并
- **多格式输出** — HTML（含浮动目录）、DOCX、EPUB、PDF
- **多语言** — zh、en、ja、ko、fr、de、es（可扩展）
- **多格式输入** — PDF/DOCX/EPUB，Calibre 负责格式转换

## 前置要求

- **Claude Code CLI** — 已安装并完成认证
- **Calibre** — `ebook-convert` 命令可用（[下载](https://calibre-ebook.com/)）
- **Pandoc** — 用于 HTML↔Markdown 转换（[下载](https://pandoc.org/)）
- **Python 3**，需要：
  - `pypandoc` — 必需（`pip install pypandoc`）
  - `beautifulsoup4` — 可选，用于更好的目录生成（`pip install beautifulsoup4`）

## 快速开始

### 1. 安装 Skill

**方式 A：npx（推荐）**

```bash
npx skills add deusyu/translate-book -a claude-code -g
```

**方式 B：ClawHub**

```bash
clawhub install translate-book
```

**方式 C：Git 克隆**

```bash
git clone https://github.com/deusyu/translate-book.git ~/.claude/skills/translate-book
```


### 2. 翻译一本书

在 Claude Code 中直接说：

```
translate /path/to/book.pdf to Chinese
```

或使用斜杠命令：

```
/translate-book translate /path/to/book.pdf to Japanese
```

Skill 自动处理完整流程 — 转换、拆分、并行翻译、校验、合并、生成所有输出格式。

### 3. 查看输出

所有文件在 `{book_name}_temp/` 目录下：

| 文件 | 说明 |
|------|------|
| `output.md` | 合并后的翻译 Markdown |
| `book.html` | 网页版，含浮动目录 |
| `book.docx` | Word 文档 |
| `book.epub` | 电子书 |
| `book.pdf` | 可打印 PDF |

## 仓库测试资产

- 需要纳入仓库的基准书输入，统一放在 `tests/baselines/<book-id>/`。
- 完整流水线跑出来的产物统一放在 `tests/.artifacts/`，不提交到版本库。
- 由于 `scripts/convert.py` 会把 `{book_name}_temp/` 写到**当前工作目录**下，仓库内的 baseline 测试应从 `tests/.artifacts/` 目录里启动，这样生成文件不会散落到仓库根目录。

### 完整基准测试示例

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# 然后通过 skill 完成翻译
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "test"
```

## 流程详解

### 第一步：转换

```bash
python3 scripts/convert.py /path/to/book.pdf --olang zh
```

Calibre 将输入文件转为 HTMLZ，解压后转为 Markdown，再拆分为 chunk（每个约 6000 字符）。`manifest.json` 记录每个源 chunk 的 SHA-256 hash，用于后续校验。

### 第一步半：术语表（保证全书译名一致）

每个 chunk 由独立的 fresh-context subagent 翻译 — 这意味着同一个专有名词在 100 个 chunk 之间可能出现多种译法。为此，skill 在翻译前会先构建术语表：

1. 抽样 5 个 chunk（首章、末章、3 个均匀分布的中间章节）。
2. 提取专有名词和反复出现的领域术语，给每个术语确定一个标准译法。
3. 写入 `<temp_dir>/glossary.json`（schema 见下，可手动编辑）。
4. 运行 `python3 scripts/glossary.py count-frequencies <temp_dir>`，统计每个术语在全书的出现次数（ASCII 术语用单词边界正则，避免 `cat` 误匹配 `category`；中日韩术语用子串匹配；单字汉字术语会被拒绝以防过度匹配；别名也计入所属术语的频次）。
5. 翻译每个 chunk 之前，主 agent 调用 `python3 scripts/glossary.py print-terms-for-chunk <temp_dir> chunkNNNN.md`，将输出的 3 列（`原文 | 别名 | 译文`）markdown 表格作为硬性约束注入到该 chunk 的 prompt。术语选取 = (本 chunk 中出现原文或任一别名的术语) ∪ (全书出现频率 top-N 的术语)。

```json
{
  "version": 2,
  "terms": [
    {"id": "Manhattan", "source": "Manhattan", "target": "曼哈顿",
     "category": "place", "aliases": [], "gender": "unknown",
     "confidence": "medium", "frequency": 12,
     "evidence_refs": [], "notes": ""}
  ],
  "high_frequency_top_n": 20,
  "applied_meta_hashes": {}
}
```

已有的 v1 `glossary.json` 会在首次加载时自动升级为 v2。v2 禁止同一个表面词（原文或别名）同时归属于两个不同术语；如果 v1 文件存在同名（polysemous）的重复 source，升级会终止并给出消歧提示 — 手工修复后重新加载即可。

可在两次运行之间编辑 `glossary.json` 修正译法。已存在的 `glossary.json` 不会被覆盖 — 删除它才会重建。

> **关于增量重跑**：当前版本中，翻译完部分 chunk 之后再编辑 `glossary.json`，已翻译的 chunk **不会** 自动失效 — 它们仍保留旧译法。基于术语表变更的精确重跑会在下一个 commit 加入。在那之前，需要手动删除受影响的 `output_chunk*.md`（或整个 temp 目录）才能应用新的译法。

### 第二步：翻译（并行 subagent）

Skill 分批启动 subagent（默认 8 路并发）。每个 subagent：

1. 读取一个源 chunk（如 `chunk0042.md`）
2. 翻译为目标语言
3. 将结果写入 `output_chunk0042.md`

如果运行中断，重新运行会跳过已有合法输出的 chunk。翻译失败的 chunk 会自动重试一次。

### 第三步：合并与构建

```bash
python3 scripts/merge_and_build.py --temp-dir book_temp --title "《译后书名》"
```

合并前校验：
- 每个源 chunk 都有对应的输出文件（1:1 匹配）
- 源 chunk hash 与 manifest 一致（无过时输出）
- 输出文件不为空

校验通过后：合并 → Pandoc 生成 HTML → 注入目录 → Calibre 生成 DOCX、EPUB、PDF。

**注意：** `{book_name}_temp/` 是单次翻译运行的工作目录。如果修改了标题、作者、输出语言、模板或图片资源，建议使用新的 temp 目录，或先删除已有的最终产物（`output.md`、`book*.html`、`book.docx`、`book.epub`、`book.pdf`）再重跑。

## 项目结构

| 文件 | 用途 |
|------|------|
| `SKILL.md` | Claude Code Skill 定义 — 编排完整流程 |
| `scripts/convert.py` | PDF/DOCX/EPUB → Markdown chunks（经 Calibre HTMLZ） |
| `scripts/manifest.py` | Chunk manifest：SHA-256 追踪与合并校验 |
| `scripts/glossary.py` | 术语表管理：为每个 chunk 生成专属术语对照表，保证全书译名一致 |
| `scripts/meta.py` | 子 agent 单 chunk 观察文件 schema（`output_chunkNNNN.meta.json`） |
| `scripts/merge_meta.py` | 批次边界合并：子 agent 观察 → canonical 术语表 |
| `scripts/merge_and_build.py` | 合并 chunks → HTML → DOCX/EPUB/PDF |
| `scripts/calibre_html_publish.py` | Calibre 格式转换封装 |
| `scripts/template.html` | 网页 HTML 模板，含浮动目录 |
| `scripts/template_ebook.html` | 电子书 HTML 模板 |
| `tests/baselines/` | 纳入仓库的完整链路 baseline 输入 |
| `tests/.artifacts/` | 被忽略的完整链路测试产物 |

## 常见问题

| 问题 | 解决方案 |
|------|----------|
| `Calibre ebook-convert not found` | 安装 Calibre，确保 `ebook-convert` 在 PATH 中 |
| `Manifest validation failed` | 源 chunk 在拆分后被修改 — 重新运行 `convert.py` |
| `Missing source chunk` | 源文件被删除 — 重新运行 `convert.py` 重新生成 |
| 翻译不完整 | 重新运行 Skill，会从中断处继续 |
| 修改标题、模板或图片后输出未更新 | 删除 temp 目录中的 `output.md`、`book*.html`、`book.docx`、`book.epub`、`book.pdf`，然后重跑 `merge_and_build.py` |
| 想去掉 PDF 输出中的页码 | 默认会自动识别单调递增的页码序列（如 `1, 2, 3, ...`）并删除，同时保留年份（`1984`）、章节编号、引用编号等离散的独立数字行。若识别不到你的页码格式，可给 `convert.py` 加 `--strip-page-numbers`，强制删除所有独立数字行。该标志在检测到已缓存的 `input.md` 或 `chunk*.md` 时会直接报错 — 需先删除这些缓存，标志才会生效 |
| `output.md exists but manifest invalid` | 旧输出已过时 — 脚本会自动删除并重新合并 |
| `Glossary upgrade rejected: duplicate source` | v2 不允许两个术语共用同一个 source/alias 表面词。手工编辑 `glossary.json` 消歧（例如把一个 source 从 `Apple` 改为 `Apple (Inc.)`）后重新加载。 |
| PDF 生成失败 | 确认 Calibre 已安装且支持 PDF 输出 |

## 后续规划

跟踪 [issue #7](https://github.com/deusyu/translate-book/issues/7) — chunk 之间的人名/术语不一致以及代词/性别错误。当前的术语表功能已覆盖高频主实体（主角、主要地名、反复出现的领域术语），但低频角色、拼写变体、代词指代尚未覆盖。整体方案分为四个可独立交付的阶段。

### 设计原则

- **脚本做记账，LLM 做语义合并**。状态管理、schema 校验、去重、hash、IO 是确定性的 Python；命名、性别归属、别名判定、冲突解决交给 LLM。
- **共享状态单写者**。`glossary.json` 和 `run_state.json` 仅由主 agent 写入；子 agent 只读共享状态，并写入各自的 chunk meta 文件。无需加锁。
- **保守合并**。新实体必须有证据；别名合并需要 LLM 判断,不能仅靠字符串相似度；性别默认 `unknown`,仅在显式证据下才升级；canonical 值在冲突时不会被静默覆盖。
- **三层状态,三个独立文件**。`glossary.json`（canonical,子 agent 读取）、`output_chunkNNNN.meta.json`（子 agent 原始观察）、`run_state.json`（编排状态）。

### Phase 1 — 子 agent 反馈 + 术语表合并（已发布）

闭合读写回路。术语表 v2 新增 `id`、`aliases`、`gender`、`confidence`、`evidence_refs`、`notes`（v1 文件首次加载时自动升级；术语表现在是 3 列，`aliases` 参与选词链路）。子 agent 在输出译文的同时生成 `output_chunkNNNN.meta.json`。新增 `scripts/merge_meta.py`（`prepare-merge` / `apply-merge` / `status`）按批次执行保守合并：跨术语 surface form 唯一性、坏 meta 隔离（warn + skip + count）、`evidence_chunks` 与 `used_term_sources` 双路 confidence 升级、FIFO 上限 5。详见 SKILL.md Step 4 / Step 4.5 / Step 5。

### Phase 2 — 代词的邻居上下文（未开始,独立于 Phase 1）

为每个子 agent prompt 注入 `prev_excerpt`（上一个 chunk 末尾约 300 字）和 `next_excerpt`（下一个 chunk 开头约 300 字）,仅作只读上下文参考。不新增状态文件,纯 prompt 装配变更。

### Phase 3 — 精确重译（未开始,依赖 Phase 1）

Phase 1 的批次反馈只能向前优化。精确重译闭合向后的回路:新增 `scripts/run_state.py` 和 `run_state.json` schema；按 chunk 跟踪 `glossary_version_used`、`entity_ids_used`、`output_hash`；五条决策规则判断本次哪些 chunk 需要重译。

### Phase 4 — 冷启动预热（实验性,依赖 Phase 1 的实际数据）

Phase 1 让术语表按批次增长,因此第一批看到的术语表最小,drift 风险最高。可能的方案:顺序冷启动、可变并发、或跳过预热。决策权属于实际跑过完整书的人。

> 各阶段的具体 schema 和文件布局是示意性的,会在 Phase 1 接触真实数据后调整。Phase 4 取决于实际数据；如果 Phase 1 已经"够好",Phase 3 可能重新调整范围或被放弃。

### 平行线路 — Pipeline / UX backlog（未开始,独立于 issue #7）

最近几轮 PR 讨论也暴露出一些有价值的工作流改进,但它们都不属于“一次性小补丁”：会触及仓库契约（产物命名、temp-dir 行为、清理语义、或 EPUB 兼容性边界）。因此这些内容会被收敛为 maintainer 自己维护的 roadmap 项,而不是直接从当前 PR 合入:

- **显式 EPUB 封面支持**。增加 `--cover <image>`,并在 HTML -> EPUB 的 Calibre 步骤透传。`--cover-from <epub>` / EPUB 封面自动提取先不纳入当前范围,等项目准备好承担不同 EPUB 包布局的解析兼容性后再考虑。(context: closed #3)
- **可配置的 temp 工作目录位置**。当前默认的 cwd-local `{book_name}_temp/` 约定保持不变以兼容现有流程。如果后续要调整,更适合新增显式 `--temp-root` / `--work-dir` 一类参数,而不是静默改变默认位置。(context: closed #4)
- **更安全的 Calibre/Pandoc 噪声清理**。继续在回归测试保护下逐步增加清理规则,同时保留当前页码检测语义,避免误删真实数字内容或显示公式的定界符。(context: closed #5)
- **可选的面向用户导出文件名**。流水线内部的 canonical 产物仍保持 `book.html`、`book_doc.html`、`book.docx`、`book.epub`、`book.pdf`。如果后续支持按译后标题命名,更合理的方向可能是额外导出一份 alias/copy,而不是静默替换内部产物契约。(context: closed #6)

## Star History

如果这个项目对您有帮助，请考虑为其点亮一颗 Star ⭐！

[![Star History Chart](https://api.star-history.com/svg?repos=deusyu/translate-book&type=Date)](https://star-history.com/#deusyu/translate-book&Date)

## 赞助

如果这个项目帮你节省了时间，欢迎赞助支持后续维护和改进。

[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-pink?logo=github)](https://github.com/sponsors/deusyu)

## License

[MIT](LICENSE)


path: scripts/calibre_html_publish.py
content: #!/usr/bin/env python3
"""
HTML Publisher using Calibre
Unified script to convert HTML to DOCX, EPUB, and PDF formats
Usage: calibre_html_publish.py input.html -o output.docx/epub/pdf
"""

import os
import sys
import subprocess
import argparse
import tempfile
import shutil
from pathlib import Path
import signal
import re

def timeout_handler(signum, frame):
    """Handle timeout signal"""
    raise TimeoutError("Conversion timed out")

def find_calibre_convert():
    """Find ebook-convert command from Calibre installation"""
    possible_paths = [
        "/Applications/calibre.app/Contents/MacOS/ebook-convert",
        "/usr/bin/ebook-convert", 
        "/usr/local/bin/ebook-convert",
        "ebook-convert"  # If in PATH
    ]
    
    for path in possible_paths:
        try:
            result = subprocess.run([path, "--version"], capture_output=True, text=True, timeout=10)
            if result.returncode == 0:
                print(f"✓ Found Calibre ebook-convert: {path}")
                return path
        except (FileNotFoundError, subprocess.TimeoutExpired):
            continue
    
    return None

def extract_html_metadata(html_file):
    """Extract title and author from HTML file"""
    try:
        with open(html_file, 'r', encoding='utf-8') as f:
            content = f.read()
        
        # Extract title
        title_match = re.search(r'<title[^>]*>(.*?)</title>', content, re.IGNORECASE | re.DOTALL)
        if title_match:
            title = re.sub(r'<[^>]+>', '', title_match.group(1)).strip()
        else:
            # Try h1 tag
            h1_match = re.search(r'<h1[^>]*>(.*?)</h1>', content, re.IGNORECASE | re.DOTALL)
            if h1_match:
                title = re.sub(r'<[^>]+>', '', h1_match.group(1)).strip()
            else:
                title = os.path.splitext(os.path.basename(html_file))[0]
        
        # Extract author
        author_match = re.search(r'<meta[^>]*name=["\']author["\'][^>]*content=["\']([^"\']*)["\']', content, re.IGNORECASE)
        if author_match:
            author = author_match.group(1).strip()
        else:
            author = "Unknown Author"
        
        return title, author
        
    except Exception as e:
        print(f"Warning: Could not extract metadata: {e}")
        return os.path.splitext(os.path.basename(html_file))[0], "Unknown Author"

def _get_font_family_for_lang(lang):
    """Get appropriate font family CSS for the given language."""
    lang_lower = lang.lower()
    if lang_lower.startswith('zh'):
        return '"FangSong", "FangSong_GB2312", "仿宋", "仿宋_GB2312", "STFangSong", "SimSun", serif'
    elif lang_lower.startswith('ja'):
        return '"Hiragino Mincho ProN", "Yu Mincho", "MS Mincho", serif'
    elif lang_lower.startswith('ko'):
        return '"Nanum Myeongjo", "Batang", serif'
    else:
        return 'Georgia, "Times New Roman", Times, serif'


def _get_pdf_font_for_lang(lang):
    """Get PDF font name for the given language."""
    lang_lower = lang.lower()
    if lang_lower.startswith('zh'):
        return 'FangSong'
    elif lang_lower.startswith('ja'):
        return 'Hiragino Mincho ProN'
    elif lang_lower.startswith('ko'):
        return 'Nanum Myeongjo'
    else:
        return 'Georgia'


def prepare_html_for_conversion(input_html, temp_dir, lang="zh-CN"):
    """Prepare HTML file for conversion with font styling"""

    # Create working copy
    work_html = os.path.join(temp_dir, "work.html")
    shutil.copy2(input_html, work_html)

    font_family = _get_font_family_for_lang(lang)

    try:
        with open(work_html, 'r', encoding='utf-8') as f:
            content = f.read()

        # Add font styling CSS
        font_css = f"""
<style>
body {{
    font-family: {font_family};
    font-size: 12pt;
    line-height: 1.6;
    text-decoration: none;
}}
h1, h2, h3, h4, h5, h6 {{
    font-family: {font_family};
    font-weight: bold;
    text-decoration: none;
}}
p {{
    font-family: {font_family};
    text-decoration: none;
}}
a {{
    text-decoration: none;
    color: inherit;
}}
* {{
    text-decoration: none !important;
}}
</style>
"""
        
        # Insert CSS after <head> tag
        if re.search(r'<head[^>]*>', content, re.IGNORECASE):
            content = re.sub(r'(<head[^>]*>)', r'\1\n' + font_css, content, flags=re.IGNORECASE)
        else:
            # If no head tag, add one
            if '<html' in content.lower():
                content = re.sub(r'(<html[^>]*>)', r'\1\n<head>\n' + font_css + '\n</head>', content, flags=re.IGNORECASE)
            else:
                content = '<head>\n' + font_css + '\n</head>\n' + content
        
        # Remove underline styling but preserve links
        content = re.sub(r'text-decoration\s*:\s*underline\s*;?', '', content, flags=re.IGNORECASE)
        content = re.sub(r'style\s*=\s*["\'][^"\']*text-decoration\s*:\s*underline[^"\']*["\']', '', content, flags=re.IGNORECASE)
        
        with open(work_html, 'w', encoding='utf-8') as f:
            f.write(content)
        
        print("✓ Added font styling and removed underlines from HTML")
        return work_html
        
    except Exception as e:
        print(f"Warning: Could not add font styling: {e}")
        return work_html

def copy_images_if_needed(html_file, temp_dir):
    """Copy images directory if it exists alongside HTML"""
    html_dir = os.path.dirname(html_file)
    images_dirs = ['images', 'media', 'image', 'pics']
    
    total_image_count = 0
    
    # Copy all image directories found
    for img_dir_name in images_dirs:
        img_dir = os.path.join(html_dir, img_dir_name)
        if os.path.exists(img_dir):
            target_dir = os.path.join(temp_dir, img_dir_name)
            try:
                shutil.copytree(img_dir, target_dir, dirs_exist_ok=True)
                image_count = len([f for f in os.listdir(target_dir) 
                                 if f.lower().endswith(('.jpg', '.jpeg', '.png', '.gif', '.svg', '.webp', '.bmp'))])
                print(f"✓ Copied {image_count} images from {img_dir_name}/")
                total_image_count += image_count
            except Exception as e:
                print(f"Warning: Could not copy {img_dir_name}/: {e}")
    
    # Also copy any loose image files in the HTML directory
    try:
        for file in os.listdir(html_dir):
            if file.lower().endswith(('.jpg', '.jpeg', '.png', '.gif', '.svg', '.webp', '.bmp')):
                src_file = os.path.join(html_dir, file)
                dst_file = os.path.join(temp_dir, file)
                shutil.copy2(src_file, dst_file)
                total_image_count += 1
                print(f"✓ Copied loose image file: {file}")
    except Exception as e:
        print(f"Warning: Could not copy loose image files: {e}")
    
    if total_image_count == 0:
        print("ℹ No images found")
    else:
        print(f"✓ Total images copied: {total_image_count}")
    
    return total_image_count

def get_output_format(output_file):
    """Determine output format from file extension"""
    ext = os.path.splitext(output_file)[1].lower()
    format_map = {
        '.docx': 'docx',
        '.epub': 'epub',
        '.pdf': 'pdf'
    }
    return format_map.get(ext)

def convert_html_with_calibre(html_file, output_file, format_type, timeout=600, lang="zh-CN"):
    """Convert HTML to specified format using Calibre with timeout protection"""
    
    calibre_path = find_calibre_convert()
    if not calibre_path:
        raise RuntimeError("Calibre ebook-convert not found. Please install Calibre.")
    
    # Extract metadata
    title, author = extract_html_metadata(html_file)
    
    print(f"Converting HTML to {format_type.upper()} using Calibre...")
    print(f"Title: {title}")
    print(f"Author: {author}")
    
    # Prepare Calibre command
    cmd = [
        calibre_path,
        html_file,
        output_file,
        "--title", title,
        "--authors", author,
        "--language", lang,
        "--book-producer", "Claude Translator",
        "--preserve-cover-aspect-ratio",
        "--smarten-punctuation"
    ]
    
    # Add format-specific options
    if format_type == 'docx':
        cmd.extend([
            "--disable-font-rescaling"
        ])
    elif format_type == 'epub':
        cmd.extend([
            "--epub-version", "3"
        ])
    elif format_type == 'pdf':
        pdf_font = _get_pdf_font_for_lang(lang)
        cmd.extend([
            "--pdf-page-numbers",
            "--pdf-serif-family", pdf_font,
            "--pdf-sans-family", pdf_font,
            "--pdf-mono-family", pdf_font,
            "--pdf-default-font-size", "12",
            "--pdf-mono-font-size", "12"
        ])
    
    try:
        # Set up timeout signal
        signal.signal(signal.SIGALRM, timeout_handler)
        signal.alarm(timeout)
        
        print(f"Starting conversion (timeout: {timeout}s)...")
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
        
        # Cancel timeout
        signal.alarm(0)
        
        if result.returncode == 0:
            if os.path.exists(output_file):
                file_size = os.path.getsize(output_file)
                print(f"✓ {format_type.upper()} conversion successful: {output_file} ({file_size} bytes)")
                return True
            else:
                print(f"✗ {format_type.upper()} file was not created")
                return False
        else:
            print(f"✗ Calibre conversion failed: {result.stderr}")
            return False
            
    except subprocess.TimeoutExpired:
        print(f"✗ Conversion timed out after {timeout} seconds")
        return False
    except TimeoutError:
        print(f"✗ Conversion timed out after {timeout} seconds")
        return False
    except Exception as e:
        print(f"✗ Conversion error: {e}")
        return False
    finally:
        # Ensure timeout is cancelled
        signal.alarm(0)

def main():
    """Main function"""
    parser = argparse.ArgumentParser(description='Convert HTML to DOCX/EPUB/PDF using Calibre')
    parser.add_argument('input_html', help='Input HTML file')
    parser.add_argument('-o', '--output', required=True, help='Output file (.docx, .epub, or .pdf)')
    parser.add_argument('-t', '--timeout', type=int, default=600,
                       help='Conversion timeout in seconds (default: 600)')
    parser.add_argument('--lang', default='zh-CN',
                       help='Language code for output metadata (default: zh-CN)')
    
    args = parser.parse_args()
    
    input_html = args.input_html
    output_file = args.output
    
    # Check input file
    if not os.path.exists(input_html):
        print(f"Error: Input file not found: {input_html}")
        sys.exit(1)
    
    # Determine output format
    format_type = get_output_format(output_file)
    if not format_type:
        print(f"Error: Unsupported output format. Use .docx, .epub, or .pdf")
        sys.exit(1)
    
    # Always use the exact output path provided - 07_generate_formats.py already handles base_temp logic
    final_output = os.path.abspath(output_file)
    
    # Ensure output directory exists
    output_dir = os.path.dirname(final_output)
    os.makedirs(output_dir, exist_ok=True)
    
    print("=== HTML Publisher (Calibre) ===")
    print(f"Input: {input_html}")
    print(f"Output: {final_output}")
    print(f"Format: {format_type.upper()}")
    print(f"Timeout: {args.timeout} seconds")
    print()
    
    try:
        # Create temp directory in the same directory as input HTML
        input_dir = os.path.dirname(os.path.abspath(input_html))
        base_name = os.path.splitext(os.path.basename(input_html))[0]
        temp_dir = os.path.join(input_dir, f"{base_name}_conversion_temp")
        os.makedirs(temp_dir, exist_ok=True)
        
        print(f"Working directory: {temp_dir}")
        
        # Copy images if needed
        image_count = copy_images_if_needed(input_html, temp_dir)
        
        # Prepare HTML with styling
        work_html = prepare_html_for_conversion(input_html, temp_dir, args.lang)
        
        # Convert to specified format
        if convert_html_with_calibre(work_html, final_output, format_type, args.timeout, args.lang):
            print("\n" + "="*50)
            print(f"✅ Conversion completed successfully!")
            print(f"📁 File: {final_output}")
            
            # Copy images directory to the final output directory if they exist in temp
            image_count = 0
            if os.path.exists(temp_dir):
                output_dir = os.path.dirname(final_output)
                images_dirs = ['images', 'media', 'image', 'pics']
                
                for img_dir_name in images_dirs:
                    temp_img_dir = os.path.join(temp_dir, img_dir_name)
                    if os.path.exists(temp_img_dir):
                        target_img_dir = os.path.join(output_dir, img_dir_name)
                        try:
                            if os.path.exists(target_img_dir):
                                shutil.rmtree(target_img_dir)
                            shutil.copytree(temp_img_dir, target_img_dir)
                            img_count = len([f for f in os.listdir(target_img_dir) 
                                           if f.lower().endswith(('.jpg', '.jpeg', '.png', '.gif', '.svg', '.webp', '.bmp'))])
                            print(f"✓ Copied {img_count} images from {img_dir_name}/ directory to output location")
                            image_count += img_count
                        except Exception as e:
                            print(f"Warning: Could not copy {img_dir_name}/ to output: {e}")
                
                # Also copy loose image files
                try:
                    for file in os.listdir(temp_dir):
                        if file.lower().endswith(('.jpg', '.jpeg', '.png', '.gif', '.svg', '.webp', '.bmp')):
                            src_file = os.path.join(temp_dir, file)
                            dst_file = os.path.join(output_dir, file)
                            shutil.copy2(src_file, dst_file)
                            image_count += 1
                            print(f"✓ Copied loose image file: {file}")
                except Exception as e:
                    print(f"Warning: Could not copy loose images to output: {e}")
            
            if os.path.exists(final_output):
                file_size = os.path.getsize(final_output)
                print(f"💾 Size: {file_size:,} bytes")
            print(f"🖼️  Images: {image_count} files")
            print("🔤 Font: 仿宋体 (FangSong)")
        else:
            print(f"\n❌ Conversion to {format_type.upper()} failed!")
            sys.exit(1)
        
        # Clean up temp directory
        try:
            shutil.rmtree(temp_dir)
            print(f"🧹 Cleaned up temporary directory: {temp_dir}")
        except Exception as e:
            print(f"Warning: Could not clean up temp directory: {e}")
                
    except KeyboardInterrupt:
        print("\nConversion interrupted by user")
        sys.exit(1)
    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)

if __name__ == '__main__':
    main()


path: scripts/convert.py
content: #!/usr/bin/env python3
"""
convert.py - Convert PDF/DOCX/EPUB to Markdown chunks via Calibre HTMLZ
Combines the original steps 1-2 into a single script.
"""

import os
import sys
import subprocess
import zipfile
import shutil
import tempfile
import argparse
import bisect
import glob
import re

from manifest import create_manifest


def find_calibre_convert():
    """Find ebook-convert command from Calibre installation"""
    possible_paths = [
        "/Applications/calibre.app/Contents/MacOS/ebook-convert",
        "/usr/bin/ebook-convert",
        "/usr/local/bin/ebook-convert",
        "ebook-convert"  # If in PATH
    ]

    for path in possible_paths:
        try:
            result = subprocess.run([path, "--version"], capture_output=True, text=True, timeout=10)
            if result.returncode == 0:
                print(f"Found Calibre ebook-convert: {path}")
                return path
        except (FileNotFoundError, subprocess.TimeoutExpired):
            continue

    return None


def convert_to_htmlz(input_file, htmlz_file, calibre_path):
    """Convert input file to HTMLZ using Calibre"""
    try:
        print(f"Converting {input_file} to HTMLZ...")
        cmd = [calibre_path, input_file, htmlz_file]
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)

        if result.returncode == 0:
            file_size = os.path.getsize(htmlz_file)
            print(f"HTMLZ conversion successful: {htmlz_file} ({file_size} bytes)")
            return True
        else:
            print(f"HTMLZ conversion failed: {result.stderr}")
            return False
    except subprocess.TimeoutExpired:
        print("HTMLZ conversion timed out")
        return False
    except Exception as e:
        print(f"HTMLZ conversion error: {e}")
        return False


def extract_metadata_from_htmlz(extract_dir):
    """Extract metadata from metadata.opf file in HTMLZ"""
    try:
        import xml.etree.ElementTree as ET

        metadata_file = None
        for root, dirs, files in os.walk(extract_dir):
            for file in files:
                if file.lower() == 'metadata.opf':
                    metadata_file = os.path.join(root, file)
                    break
            if metadata_file:
                break

        if not metadata_file:
            return {}

        tree = ET.parse(metadata_file)
        root = tree.getroot()

        namespaces = {
            'opf': 'http://www.idpf.org/2007/opf',
            'dc': 'http://purl.org/dc/elements/1.1/',
            'dcterms': 'http://purl.org/dc/terms/'
        }

        metadata = {}

        title_elem = root.find('.//dc:title', namespaces)
        if title_elem is not None and title_elem.text:
            metadata['title'] = title_elem.text.strip()

        creator_elem = root.find('.//dc:creator', namespaces)
        if creator_elem is not None and creator_elem.text:
            metadata['creator'] = creator_elem.text.strip()

        publisher_elem = root.find('.//dc:publisher', namespaces)
        if publisher_elem is not None and publisher_elem.text:
            metadata['publisher'] = publisher_elem.text.strip()

        language_elem = root.find('.//dc:language', namespaces)
        if language_elem is not None and language_elem.text:
            metadata['language'] = language_elem.text.strip()

        return metadata

    except Exception as e:
        print(f"Warning: Error extracting metadata: {e}")
        return {}


def extract_htmlz(htmlz_file, temp_dir):
    """Extract HTMLZ file and return paths to HTML and images"""
    try:
        with zipfile.ZipFile(htmlz_file, 'r') as zip_file:
            zip_file.extractall(temp_dir)

        html_file = None
        images_dir = None

        for root, dirs, files in os.walk(temp_dir):
            for file in files:
                if file.lower() in ['index.html', 'index.htm']:
                    html_file = os.path.join(root, file)
                    break
            for dir_name in dirs:
                if dir_name.lower() in ['images', 'image', 'pics', 'pictures']:
                    images_dir = os.path.join(root, dir_name)
                    break

        if not html_file:
            for root, dirs, files in os.walk(temp_dir):
                for file in files:
                    if file.lower().endswith(('.html', '.htm')):
                        html_file = os.path.join(root, file)
                        break
                if html_file:
                    break

        return html_file, images_dir

    except Exception as e:
        print(f"Error extracting HTMLZ: {e}")
        return None, None


def setup_temp_directory(input_file, html_file, images_dir):
    """Setup temp directory with HTML and images"""
    try:
        base_name = os.path.splitext(os.path.basename(input_file))[0]
        temp_dir = f"{base_name}_temp"
        os.makedirs(temp_dir, exist_ok=True)

        input_html = os.path.join(temp_dir, "input.html")
        if os.path.exists(input_html):
            print(f"Skipping HTML copy - input.html already exists")
        else:
            shutil.copy2(html_file, input_html)
            print(f"Copied HTML to: {input_html}")

        if images_dir and os.path.exists(images_dir):
            target_images_dir = os.path.join(temp_dir, "images")
            if os.path.exists(target_images_dir):
                print(f"Skipping images copy - images directory already exists")
            else:
                shutil.copytree(images_dir, target_images_dir)
                print(f"Copied images to: {target_images_dir}")

        return temp_dir
    except Exception as e:
        print(f"Error setting up temp directory: {e}")
        return None


def convert_html_to_markdown(html_file, md_file, strip_page_numbers=False):
    """Convert HTML to Markdown using pandoc"""
    try:
        import pypandoc

        pypandoc.convert_file(
            html_file,
            'markdown',
            outputfile=md_file,
            extra_args=['--wrap=none']
        )

        if os.path.exists(md_file):
            with open(md_file, 'r', encoding='utf-8') as f:
                content = f.read()

            content = content.replace('\ufeff', '')
            content = content.replace('\u00a0', ' ')
            content = clean_calibre_markers(content, strip_page_numbers=strip_page_numbers)

            with open(md_file, 'w', encoding='utf-8') as f:
                f.write(content)

            print(f"Markdown conversion successful: {md_file}")
            return True
        else:
            print("Markdown file was not created")
            return False
    except ImportError:
        print("pypandoc not found. Install with: pip install pypandoc")
        return False
    except Exception as e:
        print(f"HTML to Markdown conversion failed: {e}")
        return False


_PAGE_SEQUENCE_MIN_LENGTH = 4
_PAGE_SEQUENCE_MIN_RATIO = 0.5


def _detect_page_number_lines(lines):
    """Detect standalone-digit lines that form a monotonic page-number sequence.

    Returns a set of line indices that should be dropped as page numbers.

    Algorithm: collect every standalone-digit line in document order, find the
    Longest Non-Decreasing Subsequence (LNDS) of their integer values via
    bisect_right with parent-pointer reconstruction. If the LNDS is long enough
    and covers a large enough fraction of all standalone digits, treat those
    elements as page numbers. Outliers (years like 1984, chapter numbers,
    citation indices) sit off the monotonic spine and stay preserved.
    """
    digit_indices = []
    digit_values = []
    for i, line in enumerate(lines):
        s = line.strip()
        if s.isdigit():
            digit_indices.append(i)
            digit_values.append(int(s))

    n = len(digit_values)
    if n < _PAGE_SEQUENCE_MIN_LENGTH:
        return set()

    tails = []
    tails_idx = []
    parents = [-1] * n

    for i, v in enumerate(digit_values):
        pos = bisect.bisect_right(tails, v)
        if pos > 0:
            parents[i] = tails_idx[pos - 1]
        if pos == len(tails):
            tails.append(v)
            tails_idx.append(i)
        else:
            tails[pos] = v
            tails_idx[pos] = i

    lnds = []
    cur = tails_idx[-1]
    while cur != -1:
        lnds.append(cur)
        cur = parents[cur]
    lnds.reverse()

    if len(lnds) < _PAGE_SEQUENCE_MIN_LENGTH:
        return set()
    if len(lnds) / n < _PAGE_SEQUENCE_MIN_RATIO:
        return set()

    return {digit_indices[i] for i in lnds}


def clean_calibre_markers(content, strip_page_numbers=False):
    """Clean up Calibre-specific markers from markdown content.

    Standalone digit lines are handled in two layers:
      1. If a line is adjacent to Calibre noise (::: fence, .ct}/.cn} marker),
         drop it — clearly leftover.
      2. Otherwise, run LNDS over all standalone digits to detect a monotonic
         page-number sequence and drop those. Outliers like years (1984),
         chapter numbers, and citation indices stay preserved.

    Pass strip_page_numbers=True to bypass both layers and aggressively delete
    every standalone-digit line (legacy behavior).
    """
    content = re.sub(r'\{\.calibre[^}]*\}', '', content)
    content = re.sub(r'\(#calibre_link-\d+\)', '', content)

    # Clean heading calibre attribute blocks: {#calibre_link-N .calibreN}
    content = re.sub(r'\s*\{#calibre_link-\d+[^}]*\}', '', content)

    # Clean [**text**] format to **text**
    content = re.sub(r'\[\*\*([^*]+)\*\*\]', r'**\1**', content)

    lines = content.split('\n')

    page_number_lines = set() if strip_page_numbers else _detect_page_number_lines(lines)

    def is_calibre_noise(line):
        s = line.strip()
        if not s:
            return False
        if s.startswith(':::'):
            return True
        if s.endswith('.ct}') or s.endswith('.cn}'):
            return True
        return False

    def prev_nonblank(idx):
        for j in range(idx - 1, -1, -1):
            if lines[j].strip():
                return lines[j]
        return None

    def next_nonblank(idx):
        for j in range(idx + 1, len(lines)):
            if lines[j].strip():
                return lines[j]
        return None

    cleaned_lines = []
    for i, line in enumerate(lines):
        stripped_line = line.strip()

        if stripped_line.startswith(':::'):
            continue
        if stripped_line.endswith('.ct}') or stripped_line.endswith('.cn}'):
            continue

        if re.match(r'^\s*\d+\s*$', line):
            if strip_page_numbers:
                continue
            if i in page_number_lines:
                continue
            prev = prev_nonblank(i)
            nxt = next_nonblank(i)
            if (prev is not None and is_calibre_noise(prev)) or \
               (nxt is not None and is_calibre_noise(nxt)):
                continue
            # else: preserve as real content

        cleaned_lines.append(line)

    content = '\n'.join(cleaned_lines)
    content = re.sub(r'\n{3,}', '\n\n', content)
    return content


# =============================================================================
# Structural block parsing and chunk splitting (Step 3)
# =============================================================================

def parse_structural_blocks(content):
    """Parse markdown into structural blocks that should not be split.

    Returns list of (text, block_type) tuples where block_type is one of:
    'heading', 'code_block', 'table', 'list', 'blockquote', 'image', 'paragraph'
    """
    blocks = []
    lines = content.split('\n')
    i = 0

    while i < len(lines):
        line = lines[i]
        stripped = line.strip()

        # Code block (fenced)
        if stripped.startswith('```'):
            block_lines = [line]
            i += 1
            while i < len(lines):
                block_lines.append(lines[i])
                if lines[i].strip().startswith('```') and len(block_lines) > 1:
                    i += 1
                    break
                i += 1
            blocks.append(('\n'.join(block_lines), 'code_block'))
            continue

        # Heading
        if re.match(r'^#{1,6}\s', stripped):
            blocks.append((line, 'heading'))
            i += 1
            continue

        # Blockquote
        if stripped.startswith('>'):
            block_lines = [line]
            i += 1
            while i < len(lines) and (lines[i].strip().startswith('>') or
                                       (lines[i].strip() and not re.match(r'^#{1,6}\s', lines[i].strip())
                                        and not lines[i].strip().startswith('```')
                                        and not lines[i].strip().startswith('|')
                                        and not re.match(r'^[-*+]\s', lines[i].strip())
                                        and not re.match(r'^\d+\.\s', lines[i].strip())
                                        and block_lines[-1].strip().startswith('>'))):
                block_lines.append(lines[i])
                i += 1
            blocks.append(('\n'.join(block_lines), 'blockquote'))
            continue

        # Table (lines starting with |)
        if stripped.startswith('|'):
            block_lines = [line]
            i += 1
            while i < len(lines) and lines[i].strip().startswith('|'):
                block_lines.append(lines[i])
                i += 1
            blocks.append(('\n'.join(block_lines), 'table'))
            continue

        # List (unordered or ordered)
        if re.match(r'^[-*+]\s', stripped) or re.match(r'^\d+\.\s', stripped):
            block_lines = [line]
            i += 1
            while i < len(lines):
                s = lines[i].strip()
                # Continue list: list items, indented continuation, or blank lines within list
                if (re.match(r'^[-*+]\s', s) or re.match(r'^\d+\.\s', s) or
                        (lines[i].startswith('  ') and s) or
                        (s == '' and i + 1 < len(lines) and
                         (re.match(r'^[-*+]\s', lines[i+1].strip()) or
                          re.match(r'^\d+\.\s', lines[i+1].strip()) or
                          lines[i+1].startswith('  ')))):
                    block_lines.append(lines[i])
                    i += 1
                else:
                    break
            blocks.append(('\n'.join(block_lines), 'list'))
            continue

        # Image line (standalone or with surrounding caption)
        if re.match(r'!\[', stripped):
            blocks.append((line, 'image'))
            i += 1
            continue

        # Empty line — just a paragraph separator
        if stripped == '':
            blocks.append((line, 'paragraph'))
            i += 1
            continue

        # Regular paragraph — collect contiguous non-empty, non-special lines
        block_lines = [line]
        i += 1
        while i < len(lines):
            s = lines[i].strip()
            if (s == '' or s.startswith('```') or re.match(r'^#{1,6}\s', s) or
                    s.startswith('>') or s.startswith('|') or
                    re.match(r'^[-*+]\s', s) or re.match(r'^\d+\.\s', s) or
                    re.match(r'!\[', s)):
                break
            block_lines.append(lines[i])
            i += 1
        blocks.append(('\n'.join(block_lines), 'paragraph'))
        continue

    return blocks


def merge_blocks_to_chunks(blocks, target_size=6000):
    """Merge structural blocks into chunks respecting target_size.

    Prefers to split at heading boundaries. Never splits within a single
    structural block unless the block itself exceeds target_size * 2.
    """
    chunks = []
    current_parts = []
    current_size = 0

    def flush():
        nonlocal current_parts, current_size
        if current_parts:
            chunks.append('\n'.join(current_parts))
            current_parts = []
            current_size = 0

    for text, btype in blocks:
        block_size = len(text)

        # If a single block is oversized, handle degradation
        if block_size > target_size * 2:
            flush()
            print(f"  WARNING: Oversized {btype} block ({block_size} chars), force-splitting")
            sub_chunks = _force_split_block(text, target_size)
            chunks.extend(sub_chunks)
            continue

        # Prefer to split at heading boundaries
        if btype == 'heading' and current_size > 0:
            flush()

        # Would adding this block exceed target?
        if current_size + block_size > target_size and current_parts:
            flush()

        current_parts.append(text)
        current_size += block_size

    flush()
    return chunks


def _force_split_block(text, target_size):
    """Force-split an oversized block by paragraph (empty lines), then by lines.

    For fenced code blocks, each resulting chunk gets proper opening/closing fences
    so it remains valid Markdown.
    """
    stripped = text.strip()
    is_fenced_code = stripped.startswith('```')

    # Extract fence info for code blocks
    fence_opener = ''
    if is_fenced_code:
        first_line = stripped.split('\n', 1)[0]
        fence_opener = first_line  # e.g. "```python"

    # Try splitting by empty lines first (not applicable for code blocks — no empty lines expected)
    if not is_fenced_code:
        paragraphs = re.split(r'\n\n+', text)
        if len(paragraphs) > 1:
            chunks = []
            current = []
            current_size = 0
            for para in paragraphs:
                para_size = len(para)
                if current_size + para_size > target_size and current:
                    chunks.append('\n\n'.join(current))
                    current = [para]
                    current_size = para_size
                else:
                    current.append(para)
                    current_size += para_size
            if current:
                chunks.append('\n\n'.join(current))
            return chunks

    # Split by lines
    lines = text.split('\n')

    # For code blocks, strip the opening and closing fences before splitting content
    if is_fenced_code:
        # Remove opening fence line
        content_lines = lines[1:]
        # Remove closing fence line if present
        if content_lines and content_lines[-1].strip().startswith('```'):
            content_lines = content_lines[:-1]
        lines = content_lines

    chunks = []
    current = []
    current_size = 0
    for line in lines:
        line_size = len(line) + 1
        if current_size + line_size > target_size and current:
            chunks.append('\n'.join(current))
            current = [line]
            current_size = line_size
        else:
            current.append(line)
            current_size += line_size
    if current:
        chunks.append('\n'.join(current))

    # Re-wrap each chunk in fences for code blocks
    if is_fenced_code:
        chunks = [f"{fence_opener}\n{chunk}\n```" for chunk in chunks]

    return chunks


def split_markdown_structured(md_file, temp_dir, target_size=6000):
    """Split markdown into structural chunks.

    Returns list of chunk filenames (e.g. ['chunk0001.md', ...]).
    """
    try:
        with open(md_file, 'r', encoding='utf-8') as f:
            content = f.read()

        blocks = parse_structural_blocks(content)
        chunk_texts = merge_blocks_to_chunks(blocks, target_size)

        chunk_files = []
        for i, chunk_text in enumerate(chunk_texts, 1):
            filename = f"chunk{i:04d}.md"
            chunk_file = os.path.join(temp_dir, filename)
            with open(chunk_file, 'w', encoding='utf-8') as f:
                f.write(chunk_text)
            chunk_files.append(filename)

        print(f"Split into {len(chunk_files)} chunks")
        for filename in chunk_files:
            filepath = os.path.join(temp_dir, filename)
            size = os.path.getsize(filepath)
            print(f"  {filename}: {size} characters")

        return chunk_files
    except Exception as e:
        print(f"Error splitting markdown: {e}")
        return []


def _find_existing_chunk_files(temp_dir):
    """Find existing chunk source files (excluding output_ prefixed).

    Returns (filenames_list, is_legacy=False).
    """
    chunk_files = glob.glob(os.path.join(temp_dir, 'chunk*.md'))
    chunk_files = [os.path.basename(f) for f in chunk_files if not os.path.basename(f).startswith('output_')]

    if chunk_files:
        return sorted(chunk_files), False
    return [], False


def create_config_file(temp_dir, input_file, input_lang, output_lang, metadata=None):
    """Create config.txt file for the pipeline"""
    try:
        config_file = os.path.join(temp_dir, "config.txt")

        config_content = f"""# Translation Configuration
input_file={input_file}
input_lang={input_lang}
output_lang={output_lang}
conversion_method=calibre_htmlz
"""
        if metadata:
            config_content += f"\n# Book Metadata\n"
            if 'title' in metadata:
                config_content += f"original_title={metadata['title']}\n"
            if 'creator' in metadata:
                config_content += f"creator={metadata['creator']}\n"
            if 'publisher' in metadata:
                config_content += f"publisher={metadata['publisher']}\n"
            if 'language' in metadata:
                config_content += f"source_language={metadata['language']}\n"

        with open(config_file, 'w', encoding='utf-8') as f:
            f.write(config_content)

        print(f"Created config file: {config_file}")
        return True
    except Exception as e:
        print(f"Error creating config file: {e}")
        return False


def _do_split_and_manifest(temp_dir, input_md, chunk_size):
    """Split markdown and create manifest. Returns chunk count or 0 on failure."""
    existing, is_legacy = _find_existing_chunk_files(temp_dir)
    if existing:
        print(f"Skipping markdown splitting - found {len(existing)} existing {'page' if is_legacy else 'chunk'} files")
        # Create/update manifest for existing files
        create_manifest(temp_dir, existing, input_md)
        return len(existing)

    chunk_files = split_markdown_structured(input_md, temp_dir, chunk_size)
    if not chunk_files:
        return 0
    create_manifest(temp_dir, chunk_files, input_md)
    return len(chunk_files)


def _check_strip_page_numbers_cache_conflict(strip_flag, temp_dir, input_md):
    """Return list of cached files that would silently neutralize --strip-page-numbers.

    The flag only takes effect inside clean_calibre_markers, which runs during
    HTML→Markdown conversion. If input.md or chunk*.md already exist from a
    prior run, both are reused as-is and the flag becomes a no-op. Surface
    that conflict so the user knows to clean up.
    """
    if not strip_flag:
        return []
    if not os.path.isdir(temp_dir):
        return []

    blockers = []
    if os.path.exists(input_md):
        blockers.append(input_md)

    existing_chunks = [
        f for f in glob.glob(os.path.join(temp_dir, 'chunk*.md'))
        if not os.path.basename(f).startswith('output_')
    ]
    if existing_chunks:
        blockers.append(f"{len(existing_chunks)} chunk file(s) under {temp_dir}/")

    return blockers


def _abort_on_strip_cache_conflict(blockers, temp_dir):
    if not blockers:
        return
    print("Error: --strip-page-numbers cannot take effect because cached files exist:")
    for b in blockers:
        print(f"  - {b}")
    print(f"Delete the cached files (or remove the entire {temp_dir}/ directory) and re-run.")
    sys.exit(1)


def main():
    """Main conversion function"""
    parser = argparse.ArgumentParser(description="Convert PDF/DOCX/EPUB to markdown chunks via HTMLZ")
    parser.add_argument("input_file", help="Input file (PDF, DOCX, or EPUB)")
    parser.add_argument("-l", "--ilang", default="auto", help="Input language (default: auto)")
    parser.add_argument("--olang", default="zh", help="Output language (default: zh)")
    parser.add_argument("--chunk-size", type=int, default=6000, help="Target chunk size in characters (default: 6000)")
    parser.add_argument(
        "--strip-page-numbers",
        action="store_true",
        help="Aggressively delete every standalone-digit line (legacy behavior). "
             "Default is off: standalone digits are preserved unless adjacent to Calibre noise.",
    )

    args = parser.parse_args()
    input_file = args.input_file

    if not os.path.exists(input_file):
        print(f"Error: Input file not found: {input_file}")
        sys.exit(1)

    file_ext = os.path.splitext(input_file)[1].lower()
    if file_ext not in ['.pdf', '.docx', '.epub']:
        print(f"Error: Unsupported file type: {file_ext}")
        sys.exit(1)

    print("=== File Conversion via Calibre HTMLZ ===")
    print(f"Input file: {input_file}")
    print(f"Target chunk size: {args.chunk_size} characters")

    calibre_path = find_calibre_convert()
    if not calibre_path:
        print("Error: Calibre ebook-convert not found")
        print("Please install Calibre: https://calibre-ebook.com/")
        sys.exit(1)

    htmlz_file = f"{os.path.splitext(input_file)[0]}.htmlz"

    try:
        base_name = os.path.splitext(os.path.basename(input_file))[0]
        temp_dir = f"{base_name}_temp"
        input_html_path = os.path.join(temp_dir, "input.html")

        if os.path.exists(input_html_path):
            print(f"Skipping HTMLZ conversion - input.html already exists")

            metadata = {}
            config_file = os.path.join(temp_dir, "config.txt")
            if os.path.exists(config_file):
                try:
                    with open(config_file, 'r', encoding='utf-8') as f:
                        for line in f:
                            if '=' in line:
                                key, value = line.strip().split('=', 1)
                                if key == 'original_title':
                                    metadata['title'] = value
                                elif key == 'creator':
                                    metadata['creator'] = value
                                elif key == 'publisher':
                                    metadata['publisher'] = value
                                elif key == 'source_language':
                                    metadata['language'] = value
                except Exception as e:
                    print(f"Warning: Could not read metadata from config: {e}")

            input_md = os.path.join(temp_dir, "input.md")
            _abort_on_strip_cache_conflict(
                _check_strip_page_numbers_cache_conflict(args.strip_page_numbers, temp_dir, input_md),
                temp_dir,
            )
            if os.path.exists(input_md):
                print(f"Skipping HTML to Markdown conversion - input.md already exists")
            else:
                if not convert_html_to_markdown(input_html_path, input_md, strip_page_numbers=args.strip_page_numbers):
                    sys.exit(1)

            chunk_count = _do_split_and_manifest(temp_dir, input_md, args.chunk_size)
            if chunk_count == 0:
                sys.exit(1)

            create_config_file(temp_dir, input_file, args.ilang, args.olang, metadata)
            print("Conversion completed successfully!")
            print(f"Temp directory: {temp_dir}")
            return

        if not convert_to_htmlz(input_file, htmlz_file, calibre_path):
            sys.exit(1)

        with tempfile.TemporaryDirectory() as extract_dir:
            html_file, images_dir = extract_htmlz(htmlz_file, extract_dir)
            if not html_file:
                sys.exit(1)

            metadata = extract_metadata_from_htmlz(extract_dir)

            temp_dir = setup_temp_directory(input_file, html_file, images_dir)
            if not temp_dir:
                sys.exit(1)

            input_html = os.path.join(temp_dir, "input.html")
            input_md = os.path.join(temp_dir, "input.md")

            _abort_on_strip_cache_conflict(
                _check_strip_page_numbers_cache_conflict(args.strip_page_numbers, temp_dir, input_md),
                temp_dir,
            )
            if os.path.exists(input_md):
                print(f"Skipping HTML to Markdown conversion - input.md already exists")
            else:
                if not convert_html_to_markdown(input_html, input_md, strip_page_numbers=args.strip_page_numbers):
                    sys.exit(1)

            chunk_count = _do_split_and_manifest(temp_dir, input_md, args.chunk_size)
            if chunk_count == 0:
                sys.exit(1)

            create_config_file(temp_dir, input_file, args.ilang, args.olang, metadata)

            print("Conversion completed successfully!")
            print(f"Temp directory: {temp_dir}")
            print(f"Markdown chunks: {chunk_count} files")

            if os.path.exists(htmlz_file):
                os.remove(htmlz_file)

    except KeyboardInterrupt:
        print("\nConversion interrupted by user")
        sys.exit(1)
    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/glossary.py
content: #!/usr/bin/env python3
"""
glossary.py - Term-consistency glossary for parallel-subagent translation.

A separate sub-agent translates each chunk with a fresh context. Without
shared state, the same proper noun can get translated three different ways
across a 100-chunk book. This module manages a hand-editable glossary.json
that the main agent injects into each sub-agent's prompt as a per-chunk term
table — so every sub-agent sees the same canonical translations for the
terms that matter to its chunk.

The schema (v2):

    {
      "version": 2,
      "terms": [
        {"id": "Manhattan", "source": "Manhattan", "target": "曼哈顿",
         "category": "place", "aliases": [], "gender": "unknown",
         "confidence": "medium", "frequency": 12,
         "evidence_refs": [], "notes": ""}
      ],
      "high_frequency_top_n": 20,
      "applied_meta_hashes": {}
    }

Lives at <temp_dir>/glossary.json and is meant to be hand-edited.
"""

import argparse
import hashlib
import json
import os
import re
import sys
import tempfile
from pathlib import Path


GLOSSARY_SCHEMA_VERSION = 2
DEFAULT_TOP_N = 20
DEFAULT_MAX_TERMS = 50

VALID_GENDERS = ('male', 'female', 'nonbinary', 'unknown')
VALID_CONFIDENCES = ('low', 'medium', 'high')

_HEX_RE = re.compile(r'^[0-9a-f]+$')

_CJK_RANGES = (
    (0x3400, 0x4DBF),   # CJK Extension A
    (0x4E00, 0x9FFF),   # CJK Unified Ideographs
    (0x3040, 0x309F),   # Hiragana
    (0x30A0, 0x30FF),   # Katakana
    (0xAC00, 0xD7AF),   # Hangul Syllables
)


def _contains_cjk(s):
    for c in s:
        cp = ord(c)
        for lo, hi in _CJK_RANGES:
            if lo <= cp <= hi:
                return True
    return False


def _canonical_json(data):
    """Stable JSON for hashing — sorted keys, no whitespace, unicode preserved."""
    return json.dumps(data, sort_keys=True, separators=(',', ':'), ensure_ascii=False)


def glossary_hash(glossary):
    """SHA-256 of the canonical glossary. Input order / whitespace insensitive."""
    return hashlib.sha256(_canonical_json(glossary).encode('utf-8')).hexdigest()


def term_hash(term):
    """SHA-256 of a single term's identifying fields."""
    payload = f"{term.get('source', '')}→{term.get('target', '')}|{term.get('category', '')}"
    return hashlib.sha256(payload.encode('utf-8')).hexdigest()


def _v2_term_defaults(term):
    """Apply v2 defaults to a term in place. Caller must have already
    validated v1-shape required fields."""
    term.setdefault('id', term.get('source', ''))
    term.setdefault('aliases', [])
    term.setdefault('gender', 'unknown')
    term.setdefault('confidence', 'medium')
    term.setdefault('evidence_refs', [])
    term.setdefault('notes', '')
    term.setdefault('frequency', 0)


def _validate_term_strict(term, idx, path):
    """Strict v2 validation for a single term. Raises ValueError mentioning
    the file `path` for context."""
    if not isinstance(term, dict):
        raise ValueError(f"Glossary term #{idx} in {path} must be an object")

    for required in ('source', 'target'):
        if required not in term:
            raise ValueError(
                f"Glossary term #{idx} in {path} missing required field '{required}'"
            )
        if not isinstance(term[required], str):
            raise ValueError(
                f"Glossary term #{idx} in {path}: field '{required}' must be a string, "
                f"got {type(term[required]).__name__}"
            )

    if 'category' in term and not isinstance(term['category'], str):
        raise ValueError(
            f"Glossary term #{idx} in {path}: field 'category' must be a string, "
            f"got {type(term['category']).__name__}"
        )

    if 'frequency' in term:
        if isinstance(term['frequency'], bool) or not isinstance(term['frequency'], int):
            raise ValueError(
                f"Glossary term #{idx} in {path}: field 'frequency' must be an integer, "
                f"got {type(term['frequency']).__name__}"
            )

    if 'id' not in term:
        raise ValueError(f"Glossary term #{idx} in {path} missing required field 'id'")
    if not isinstance(term['id'], str) or not term['id']:
        raise ValueError(
            f"Glossary term #{idx} in {path}: 'id' must be a non-empty string"
        )

    aliases = term.get('aliases', [])
    if not isinstance(aliases, list):
        raise ValueError(
            f"Glossary term #{idx} (id={term['id']!r}) in {path}: 'aliases' must be a list, "
            f"got {type(aliases).__name__}"
        )
    for a_idx, alias in enumerate(aliases):
        if not isinstance(alias, str):
            raise ValueError(
                f"Glossary term #{idx} (id={term['id']!r}) in {path}: alias #{a_idx} "
                f"must be a string, got {type(alias).__name__}"
            )
        if alias == '':
            raise ValueError(
                f"Glossary term id={term['id']!r} in {path}: alias #{a_idx} is the empty "
                f"string. Empty aliases match nothing — remove it."
            )
        if alias == term['source']:
            raise ValueError(
                f"Glossary term id={term['id']!r} in {path}: alias #{a_idx} ({alias!r}) "
                f"equals the term's own source. Drop the redundant alias."
            )
    if len(set(aliases)) != len(aliases):
        seen = set()
        dup = next((a for a in aliases if a in seen or seen.add(a)), None)
        raise ValueError(
            f"Glossary term id={term['id']!r} in {path}: alias {dup!r} is duplicated "
            f"within the term's aliases list. Each alias must be unique per term."
        )

    gender = term.get('gender')
    if gender not in VALID_GENDERS:
        raise ValueError(
            f"Glossary term id={term['id']!r} in {path}: 'gender' must be one of "
            f"{list(VALID_GENDERS)}, got {gender!r}"
        )

    confidence = term.get('confidence')
    if confidence not in VALID_CONFIDENCES:
        raise ValueError(
            f"Glossary term id={term['id']!r} in {path}: 'confidence' must be one of "
            f"{list(VALID_CONFIDENCES)}, got {confidence!r}"
        )

    evidence_refs = term.get('evidence_refs', [])
    if not isinstance(evidence_refs, list):
        raise ValueError(
            f"Glossary term id={term['id']!r} in {path}: 'evidence_refs' must be a list, "
            f"got {type(evidence_refs).__name__}"
        )
    for e_idx, ref in enumerate(evidence_refs):
        if not isinstance(ref, str):
            raise ValueError(
                f"Glossary term id={term['id']!r} in {path}: evidence_refs #{e_idx} "
                f"must be a string, got {type(ref).__name__}"
            )

    notes = term.get('notes', '')
    if not isinstance(notes, str):
        raise ValueError(
            f"Glossary term id={term['id']!r} in {path}: 'notes' must be a string, "
            f"got {type(notes).__name__}"
        )


def _validate_cross_term_invariants(data, path):
    """Term id uniqueness + surface-form uniqueness across terms +
    high_frequency_top_n shape + applied_meta_hashes shape. Run after every
    term has passed individual validation."""
    terms = data['terms']

    seen_ids = {}
    for idx, t in enumerate(terms):
        if t['id'] in seen_ids:
            raise ValueError(
                f"Glossary at {path}: duplicate term id {t['id']!r} "
                f"(terms #{seen_ids[t['id']]} and #{idx}). Each term must have a unique id."
            )
        seen_ids[t['id']] = idx

    surface_owner = {}  # surface -> (term_id, "source"|"alias")
    for t in terms:
        surfaces = [(t['source'], 'source')] + [(a, 'alias') for a in t.get('aliases', [])]
        for surface, role in surfaces:
            if surface in surface_owner:
                other_id, other_role = surface_owner[surface]
                raise ValueError(
                    f"Glossary at {path}: surface form {surface!r} appears as "
                    f"{other_role} of term id={other_id!r} AND as {role} of term id={t['id']!r}. "
                    f"v2 forbids the same surface form mapping to two different terms — "
                    f"the term table can only list one translation per word. "
                    f"Disambiguate by renaming one source, dropping one alias, or merging "
                    f"the two terms into a single entity."
                )
            surface_owner[surface] = (t['id'], role)

    if 'high_frequency_top_n' in data:
        top_n_val = data['high_frequency_top_n']
        if isinstance(top_n_val, bool) or not isinstance(top_n_val, int):
            raise ValueError(
                f"Glossary at {path}: 'high_frequency_top_n' must be an integer, "
                f"got {type(top_n_val).__name__}"
            )

    if 'applied_meta_hashes' in data:
        amh = data['applied_meta_hashes']
        if not isinstance(amh, dict):
            raise ValueError(
                f"Glossary at {path}: 'applied_meta_hashes' must be an object, "
                f"got {type(amh).__name__}"
            )
        for k, v in amh.items():
            if not isinstance(k, str):
                raise ValueError(
                    f"Glossary at {path}: 'applied_meta_hashes' key {k!r} must be a string"
                )
            if not isinstance(v, str) or not _HEX_RE.match(v):
                raise ValueError(
                    f"Glossary at {path}: 'applied_meta_hashes[{k!r}]' must be a hex string, "
                    f"got {v!r}"
                )


def _validate_v1_shape(terms, data, path):
    """Pre-upgrade validation for v1 inputs — catches bad source/target/etc.
    before defaults paper over them."""
    for i, t in enumerate(terms):
        if not isinstance(t, dict):
            raise ValueError(f"Glossary term #{i} in {path} must be an object")
        for required in ('source', 'target'):
            if required not in t:
                raise ValueError(
                    f"Glossary term #{i} in {path} missing required field '{required}'"
                )
            if not isinstance(t[required], str):
                raise ValueError(
                    f"Glossary term #{i} in {path}: field '{required}' must be a string, "
                    f"got {type(t[required]).__name__}"
                )
        if 'category' in t and not isinstance(t['category'], str):
            raise ValueError(
                f"Glossary term #{i} in {path}: field 'category' must be a string, "
                f"got {type(t['category']).__name__}"
            )
        if 'frequency' in t:
            if isinstance(t['frequency'], bool) or not isinstance(t['frequency'], int):
                raise ValueError(
                    f"Glossary term #{i} in {path}: field 'frequency' must be an integer, "
                    f"got {type(t['frequency']).__name__}"
                )
    if 'high_frequency_top_n' in data:
        top_n_val = data['high_frequency_top_n']
        if isinstance(top_n_val, bool) or not isinstance(top_n_val, int):
            raise ValueError(
                f"Glossary at {path}: 'high_frequency_top_n' must be an integer, "
                f"got {type(top_n_val).__name__}"
            )


def load_glossary(path):
    """Load and validate a glossary file. v1 files are auto-upgraded to v2
    on disk (atomic write-back) and a one-line stderr notice is emitted.
    Raises actionable errors on bad input."""
    if not os.path.exists(path):
        raise FileNotFoundError(f"Glossary not found: {path}")

    with open(path, 'r', encoding='utf-8') as f:
        try:
            data = json.load(f)
        except json.JSONDecodeError as e:
            raise ValueError(f"Glossary at {path} is not valid JSON: {e}") from e

    if not isinstance(data, dict):
        raise ValueError(f"Glossary at {path} must be a JSON object, got {type(data).__name__}")

    version = data.get('version')
    if version not in (1, 2):
        raise ValueError(
            f"Glossary schema version mismatch in {path}: "
            f"expected 1 or 2, got {version!r}. "
            f"Delete the file to rebuild, or migrate it by hand."
        )

    terms = data.get('terms')
    if not isinstance(terms, list):
        raise ValueError(f"Glossary at {path} must have a 'terms' array")

    if version == 1:
        _validate_v1_shape(terms, data, path)

        # v1 legitimately allows two terms sharing a source (differentiated by
        # category). v2 forbids it because the term table can only list one
        # translation per surface form. Reject the upgrade with disambiguation
        # guidance BEFORE writing anything to disk.
        src_groups = {}
        for idx, t in enumerate(terms):
            src_groups.setdefault(t['source'], []).append(idx)
        for src, idxs in src_groups.items():
            if len(idxs) > 1:
                cats = [terms[i].get('category', '') for i in idxs]
                raise ValueError(
                    f"v1→v2 upgrade aborted for {path}: {len(idxs)} terms share source "
                    f"{src!r} (categories: {cats}). v2 forbids polysemous surface forms because "
                    f"the term table can only list one translation per word. Disambiguate by "
                    f"hand (e.g., change one source to {src!r} + a qualifier such as "
                    f"'{src} (Inc.)') and re-load. The file on disk has NOT been modified."
                )

        # Apply defaults, then run strict v2 validation. If v2 validation fails
        # here, the file is NOT rewritten — the user gets the original v1 back.
        data['version'] = GLOSSARY_SCHEMA_VERSION
        for term in terms:
            _v2_term_defaults(term)
        data.setdefault('applied_meta_hashes', {})

        for i, t in enumerate(terms):
            _validate_term_strict(t, i, path)
        _validate_cross_term_invariants(data, path)

        save_glossary(path, data)
        sys.stderr.write(f"Upgraded glossary.json at {path} from v1 to v2.\n")
    else:
        for i, t in enumerate(terms):
            _validate_term_strict(t, i, path)
        _validate_cross_term_invariants(data, path)

    return data


def save_glossary(path, glossary):
    """Atomically write the glossary as v2. Re-asserts the v2 invariants
    (per-term shape, surface-form uniqueness, etc.) before writing."""
    glossary['version'] = GLOSSARY_SCHEMA_VERSION
    glossary.setdefault('applied_meta_hashes', {})
    for term in glossary.get('terms', []):
        _v2_term_defaults(term)

    for i, t in enumerate(glossary.get('terms', [])):
        _validate_term_strict(t, i, path)
    _validate_cross_term_invariants(glossary, path)

    dirname = os.path.dirname(os.path.abspath(path)) or '.'
    fd, tmp_path = tempfile.mkstemp(dir=dirname, prefix='.glossary-', suffix='.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            json.dump(glossary, f, ensure_ascii=False, indent=2, sort_keys=True)
            f.write('\n')
        os.replace(tmp_path, path)
    except Exception:
        if os.path.exists(tmp_path):
            try:
                os.unlink(tmp_path)
            except OSError:
                pass
        raise


def _count_in_text(source, text):
    """Count occurrences of source in text. CJK uses substring; ASCII uses
    boundary-aware regex so 'C++' and '.NET' work and 'cat' doesn't match
    'category'."""
    if not source:
        return 0
    if _contains_cjk(source):
        if len(source) <= 1:
            return 0
        return text.count(source)
    if source.isascii():
        escaped = re.escape(source)
        pattern = rf'(?<!\w){escaped}(?!\w)'
        return len(re.findall(pattern, text))
    return text.count(source)


def _appears_in_text(source, text):
    """Boundary-aware presence check. Single-CJK-char sources return False
    (would over-match)."""
    return _count_in_text(source, text) > 0


def _term_appears_in_text(term, text):
    """True if the term's source OR any alias appears in text. Each surface
    form is checked via the same boundary-aware matcher."""
    if _appears_in_text(term.get('source', ''), text):
        return True
    for alias in term.get('aliases', []):
        if _appears_in_text(alias, text):
            return True
    return False


def count_frequencies(glossary_path, chunks_dir):
    """Scan all source chunks and write per-term frequency back into the glossary.

    Frequency = sum of source-occurrences plus every alias-occurrence. Source
    chunk discovery excludes `output_chunk*.md` files so re-runs don't
    double-count translated text.
    """
    glossary = load_glossary(glossary_path)
    chunks_dir_path = Path(chunks_dir)

    chunk_paths = sorted(
        p for p in chunks_dir_path.glob('chunk*.md')
        if not p.name.startswith('output_')
    )

    if not chunk_paths:
        print(f"Warning: no chunk*.md files found under {chunks_dir}", file=sys.stderr)

    all_text_parts = []
    for p in chunk_paths:
        try:
            all_text_parts.append(p.read_text(encoding='utf-8'))
        except OSError as e:
            print(f"Warning: could not read {p}: {e}", file=sys.stderr)
    all_text = '\n'.join(all_text_parts)

    for term in glossary['terms']:
        total = 0
        for surface in [term.get('source', '')] + list(term.get('aliases', [])):
            if not surface:
                continue
            if _contains_cjk(surface) and len(surface) <= 1:
                print(
                    f"Warning: skipping frequency count for single-character CJK surface "
                    f"{surface!r} in term id={term.get('id')!r} (would over-match as substring)",
                    file=sys.stderr,
                )
                continue
            total += _count_in_text(surface, all_text)
        term['frequency'] = total

    save_glossary(glossary_path, glossary)


def select_terms_for_chunk(glossary, chunk_text, top_n=None, max_terms=DEFAULT_MAX_TERMS):
    """Return terms relevant to a single chunk: union of (terms whose source
    or any alias appears in chunk_text) and (top-N most-frequent terms across
    the whole book).

    Local hits are protected — if the union exceeds max_terms, local hits keep
    their slots and global top-N is what gets truncated.
    """
    if top_n is None:
        top_n = glossary.get('high_frequency_top_n', DEFAULT_TOP_N)

    terms = glossary.get('terms', [])
    if not terms:
        return []

    sort_key = lambda t: (-t.get('frequency', 0), t.get('source', ''))

    local_terms = [t for t in terms if t.get('source') and _term_appears_in_text(t, chunk_text)]
    local_terms.sort(key=sort_key)

    local_ids = {t.get('id', t.get('source')) for t in local_terms}
    top_pool = sorted(
        (t for t in terms if t.get('id', t.get('source')) not in local_ids),
        key=sort_key,
    )
    top_terms = top_pool[:max(0, top_n)]

    if max_terms <= 0:
        return []
    if len(local_terms) >= max_terms:
        return local_terms[:max_terms]
    remaining = max_terms - len(local_terms)
    return local_terms + top_terms[:remaining]


def format_terms_for_prompt(terms):
    """Render a 3-col markdown table: 原文 | 别名 | 译文.

    Empty input yields empty string so the caller can omit the rule line
    entirely. The 别名 column is empty for terms without aliases. Pipes in
    any field escape to \\|.
    """
    if not terms:
        return ''
    rows = ['| 原文 | 别名 | 译文 |', '|------|------|------|']
    for t in terms:
        source = t.get('source', '').replace('|', '\\|')
        aliases = t.get('aliases', []) or []
        alias_str = ', '.join(a.replace('|', '\\|') for a in aliases)
        target = t.get('target', '').replace('|', '\\|')
        rows.append(f"| {source} | {alias_str} | {target} |")
    return '\n'.join(rows)


def main():
    parser = argparse.ArgumentParser(description="Glossary management for translate-book")
    sub = parser.add_subparsers(dest='cmd', required=True)

    p_count = sub.add_parser('count-frequencies', help="Update frequencies in glossary.json")
    p_count.add_argument('temp_dir', help="Path to <book>_temp/ directory")

    p_print = sub.add_parser('print-terms-for-chunk', help="Print markdown table of terms for a chunk")
    p_print.add_argument('temp_dir')
    p_print.add_argument('chunk_filename', help="e.g. chunk0001.md")
    p_print.add_argument('--top-n', type=int, default=None,
                         help="Override high_frequency_top_n from glossary.json")
    p_print.add_argument('--max-terms', type=int, default=DEFAULT_MAX_TERMS,
                         help=f"Cap on terms in the table (default: {DEFAULT_MAX_TERMS})")

    p_hash = sub.add_parser('compute-hash', help="Print glossary_hash to stdout")
    p_hash.add_argument('temp_dir')

    args = parser.parse_args()
    glossary_path = os.path.join(args.temp_dir, 'glossary.json')

    if args.cmd == 'count-frequencies':
        count_frequencies(glossary_path, args.temp_dir)
    elif args.cmd == 'print-terms-for-chunk':
        glossary = load_glossary(glossary_path)
        chunk_path = os.path.join(args.temp_dir, args.chunk_filename)
        with open(chunk_path, 'r', encoding='utf-8') as f:
            chunk_text = f.read()
        terms = select_terms_for_chunk(
            glossary, chunk_text, top_n=args.top_n, max_terms=args.max_terms
        )
        table = format_terms_for_prompt(terms)
        if table:
            print(table)
    elif args.cmd == 'compute-hash':
        glossary = load_glossary(glossary_path)
        print(glossary_hash(glossary))


if __name__ == '__main__':
    main()


path: scripts/manifest.py
content: #!/usr/bin/env python3
"""
manifest.py - Manifest management for chunk tracking and merge validation.
"""

import os
import json
import hashlib


def file_hash(filepath):
    """Compute SHA-256 hash of a file."""
    h = hashlib.sha256()
    with open(filepath, 'rb') as f:
        for block in iter(lambda: f.read(8192), b''):
            h.update(block)
    return h.hexdigest()


def create_manifest(temp_dir, chunk_files, source_md_path):
    """Create manifest.json after splitting.

    Args:
        temp_dir: temp directory path
        chunk_files: list of chunk filenames (e.g. ['chunk0001.md', ...])
        source_md_path: path to the source input.md
    """
    source_hash = file_hash(source_md_path) if os.path.exists(source_md_path) else ""

    chunks = []
    for order, filename in enumerate(chunk_files, 1):
        filepath = os.path.join(temp_dir, filename)
        # Derive output filename: chunk0001.md -> output_chunk0001.md
        output_filename = f"output_{filename}"
        chunk_id = os.path.splitext(filename)[0]  # e.g. "chunk0001"

        chunks.append({
            "id": chunk_id,
            "order": order,
            "source_file": filename,
            "source_hash": file_hash(filepath) if os.path.exists(filepath) else "",
            "output_file": output_filename,
        })

    manifest = {
        "chunk_count": len(chunks),
        "source_hash": source_hash,
        "chunks": chunks,
    }

    manifest_path = os.path.join(temp_dir, "manifest.json")
    with open(manifest_path, 'w', encoding='utf-8') as f:
        json.dump(manifest, f, indent=2, ensure_ascii=False)

    print(f"Created manifest.json ({len(chunks)} chunks)")
    return manifest


def load_manifest(temp_dir):
    """Load manifest.json from temp_dir. Returns None if not found."""
    manifest_path = os.path.join(temp_dir, "manifest.json")
    if not os.path.exists(manifest_path):
        return None
    with open(manifest_path, 'r', encoding='utf-8') as f:
        return json.load(f)


def validate_for_merge(temp_dir):
    """Validate that all chunks have been translated before merging.

    Returns (ok, ordered_output_files, warnings) where:
        ok: True if merge can proceed
        ordered_output_files: list of output file paths in order
        warnings: list of warning strings
    """
    manifest = load_manifest(temp_dir)
    if manifest is None:
        # No manifest — fall back to legacy glob-based merge
        return True, None, ["No manifest.json found, using legacy merge"]

    errors = []
    warnings = []
    ordered_output_files = []

    for chunk in sorted(manifest["chunks"], key=lambda c: c["order"]):
        output_path = os.path.join(temp_dir, chunk["output_file"])
        source_path = os.path.join(temp_dir, chunk["source_file"])

        # Check source file exists — reject outputs without source chunks
        if not os.path.exists(source_path):
            errors.append(
                f"Missing source: {chunk['source_file']} (chunk {chunk['id']}) — "
                f"cannot verify output integrity without source chunk"
            )
            continue

        # Check source hash matches — detect stale outputs from changed sources
        if chunk.get("source_hash"):
            current_hash = file_hash(source_path)
            if current_hash != chunk["source_hash"]:
                errors.append(
                    f"Source changed since splitting: {chunk['source_file']} "
                    f"(chunk {chunk['id']}). "
                    f"Expected hash {chunk['source_hash'][:12]}..., "
                    f"got {current_hash[:12]}... — "
                    f"delete output and re-translate, or re-run convert.py to re-split"
                )
                continue

        # Check output exists
        if not os.path.exists(output_path):
            errors.append(f"Missing output: {chunk['output_file']} (chunk {chunk['id']})")
            continue

        # Check non-empty
        output_size = os.path.getsize(output_path)
        if output_size == 0:
            errors.append(f"Empty output: {chunk['output_file']} (chunk {chunk['id']})")
            continue

        # Check abnormally short
        if os.path.exists(source_path):
            source_size = os.path.getsize(source_path)
            if source_size > 0 and output_size < source_size * 0.1:
                warnings.append(
                    f"Suspiciously short: {chunk['output_file']} "
                    f"({output_size} bytes vs source {source_size} bytes)"
                )

        ordered_output_files.append(output_path)

    if errors:
        for e in errors:
            print(f"ERROR: {e}")
        return False, None, warnings

    for w in warnings:
        print(f"WARNING: {w}")

    return True, ordered_output_files, warnings


path: scripts/merge_and_build.py
content: #!/usr/bin/env python3
"""
merge_and_build.py - Merge translated pages and build final outputs
Combines original steps 4-7: merge -> HTML -> TOC -> DOCX/EPUB/PDF

Usage: merge_and_build.py --temp-dir <path> [--title <title>] [--author <author>] [--lang <lang>]
"""

import os
import sys
import re
import glob
import shutil
import subprocess
import argparse
import html as _html_lib
from collections import Counter
from html.parser import HTMLParser
from pathlib import Path

from manifest import validate_for_merge


# =============================================================================
# Image structure validation helpers
# =============================================================================

# Markdown image: `![alt](url)` or `![alt](url "title")`.
# - Negative lookbehind on `\` skips escaped `\![...]` (renders as literal text).
# - Closing `)` is required — a missing `)` means the image won't render, so
#   such a fragment must NOT count as a preserved image reference.
_MD_IMG_RE = re.compile(r'(?<!\\)!\[[^\]]*\]\(\s*([^)\s]+)[^)]*\)')
_VALID_ATTR_NAME_RE = re.compile(r'^[A-Za-z_][A-Za-z0-9_:.\-]*$')


class _ImgTagCollector(HTMLParser):
    """Collects every <img> tag found in fed text. Uses stdlib HTMLParser, which
    correctly handles `>` inside quoted attribute values — unlike a plain
    `<img\\b[^>]*>` regex, which would truncate at the first `>`."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.records = []  # list of (raw_tag_text, attrs_list)

    def handle_starttag(self, tag, attrs):
        if tag == 'img':
            self.records.append((self.get_starttag_text(), list(attrs)))

    handle_startendtag = handle_starttag


def _scan_img_tags(text):
    """Return (Counter of <img> srcs, list of (raw_tag, bad_attr_name) tuples).

    Feeds the entire text to HTMLParser rather than pre-extracting tags via regex,
    so quoted attribute values containing `>` are handled correctly."""
    src_counts = Counter()
    bad_attrs = []
    parser = _ImgTagCollector()
    try:
        parser.feed(text)
        parser.close()
    except Exception as e:
        bad_attrs.append(('<unparseable input>', f'<parser error: {e}>'))
        return src_counts, bad_attrs
    for raw_tag, attrs in parser.records:
        for name, _ in attrs:
            if not _VALID_ATTR_NAME_RE.match(name):
                bad_attrs.append((raw_tag, name))
        for name, val in attrs:
            if name == 'src' and val:
                src_counts[val] += 1
    return src_counts, bad_attrs


def _scan_image_refs(text):
    """Return (Counter html_srcs, Counter md_srcs, list bad_attrs)."""
    html_srcs, bad_attrs = _scan_img_tags(text)
    md_srcs = Counter(_MD_IMG_RE.findall(text))
    return html_srcs, md_srcs, bad_attrs


def _validate_chunk_images(temp_dir):
    """Verify each output_chunk*.md preserves the image structure of its chunk*.md.

    Bad-attribute detection uses a per-chunk DELTA: a malformed <img> attribute
    is flagged only if it appears in the output chunk but not in the source
    chunk. This avoids false positives on code blocks that legitimately contain
    deliberately-broken <img> examples — both chunks carry the same example, so
    the delta is empty.

    Returns False on any divergence; collects all errors and prints them
    together so an agent can fix many chunks in one pass.
    """
    temp_path = Path(temp_dir)
    errors = []
    for src_chunk in sorted(temp_path.glob('chunk*.md')):
        if src_chunk.name.startswith('output_'):
            continue
        out_chunk = temp_path / f'output_{src_chunk.name}'
        if not out_chunk.exists():
            continue  # missing-output is the manifest validator's job
        src_html, src_md, src_bad = _scan_image_refs(src_chunk.read_text(encoding='utf-8'))
        out_html, out_md, out_bad = _scan_image_refs(out_chunk.read_text(encoding='utf-8'))

        src_bad_counts = Counter(name for _, name in src_bad)
        out_bad_counts = Counter(name for _, name in out_bad)
        new_bad_counts = out_bad_counts - src_bad_counts
        if new_bad_counts:
            new_bad_examples = [
                (raw_tag, attr_name)
                for raw_tag, attr_name in out_bad
                if new_bad_counts.get(attr_name, 0) > 0
            ]
            for raw_tag, attr_name in new_bad_examples:
                errors.append(
                    f"ERROR: {out_chunk.name} introduced malformed <img> tag (not present in source)\n"
                    f"  tag: {raw_tag}\n"
                    f"  problem: attribute name '{attr_name}' is not a valid HTML identifier\n"
                    f"  likely cause: an unescaped quote inside alt=\"...\" or title=\"...\" closed the attribute early\n"
                    f"  fix: in {out_chunk.name}, replace the inner quote with a curly quote in the target language or with &quot; / &#39;\n"
                    f"  source chunk for reference: {src_chunk.name}"
                )

        if src_html != out_html or src_md != out_md:
            errors.append(
                f"ERROR: {out_chunk.name} image references diverge from {src_chunk.name}\n"
                f"  missing <img src> (count): {sorted((src_html - out_html).items()) or 'none'}\n"
                f"  extra   <img src> (count): {sorted((out_html - src_html).items()) or 'none'}\n"
                f"  missing ![](path) (count): {sorted((src_md - out_md).items()) or 'none'}\n"
                f"  extra   ![](path) (count): {sorted((out_md - src_md).items()) or 'none'}\n"
                f"  fix: restore the missing image refs in {out_chunk.name} from {src_chunk.name}"
            )

    if errors:
        print("\n=== Image validation failed ===")
        for e in errors:
            print(e)
            print()
        return False
    return True


def _check_generated_html_sanity(html_path):
    """Sanity-check generated HTML for malformed <img> tags. Returns False on problems.

    Note: we deliberately do NOT flag `&lt;img` in the rendered HTML — books that
    discuss HTML in prose or code blocks legitimately render escaped `<img>` text,
    and that's not a corruption signal. Real corruption produces a malformed
    actual `<img>` tag, which the attribute-name check catches."""
    try:
        text = Path(html_path).read_text(encoding='utf-8')
    except Exception as e:
        print(f"ERROR: cannot read {html_path}: {e}")
        return False

    _, bad_attrs = _scan_img_tags(text)
    if not bad_attrs:
        return True

    print(f"ERROR: image sanity check failed on {Path(html_path).name}")
    for raw_tag, attr_name in bad_attrs:
        print(f"  - malformed <img>: {raw_tag}")
        print(f"    bad attribute name: '{attr_name}'")
    print(
        "  fix: inspect output.md and the corresponding output_chunk*.md;\n"
        "       if alt text contains literal quotes, replace with curly quotes or HTML entity"
    )
    return False

# Try to import BeautifulSoup for TOC generation
try:
    from bs4 import BeautifulSoup
    BS4_AVAILABLE = True
except ImportError:
    BS4_AVAILABLE = False

# Try to import markdown
try:
    import markdown
    MARKDOWN_AVAILABLE = True
except ImportError:
    MARKDOWN_AVAILABLE = False

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))

# =============================================================================
# Language configuration — single source of truth for lang-dependent values
# =============================================================================

LANG_CONFIG = {
    'zh': {
        'lang_attr': 'zh-CN',
        'font_family': "'FangSong', '仿宋', 'STFangSong', '华文仿宋', serif",
        'font_family_ebook': '"FangSong", "FangSong_GB2312", "仿宋", "仿宋_GB2312", "STFangSong", "SimSun", serif',
        'toc_label': '目录',
        'pdf_font': 'FangSong',
    },
    'en': {
        'lang_attr': 'en',
        'font_family': "Georgia, 'Times New Roman', Times, serif",
        'font_family_ebook': 'Georgia, "Times New Roman", Times, serif',
        'toc_label': 'Contents',
        'pdf_font': 'Georgia',
    },
    'ja': {
        'lang_attr': 'ja',
        'font_family': "'Hiragino Mincho ProN', 'Yu Mincho', 'MS Mincho', serif",
        'font_family_ebook': '"Hiragino Mincho ProN", "Yu Mincho", "MS Mincho", serif',
        'toc_label': '目次',
        'pdf_font': 'Hiragino Mincho ProN',
    },
    'ko': {
        'lang_attr': 'ko',
        'font_family': "'Nanum Myeongjo', 'Batang', serif",
        'font_family_ebook': '"Nanum Myeongjo", "Batang", serif',
        'toc_label': '목차',
        'pdf_font': 'Nanum Myeongjo',
    },
    'fr': {
        'lang_attr': 'fr',
        'font_family': "Georgia, 'Times New Roman', Times, serif",
        'font_family_ebook': 'Georgia, "Times New Roman", Times, serif',
        'toc_label': 'Table des matières',
        'pdf_font': 'Georgia',
    },
    'de': {
        'lang_attr': 'de',
        'font_family': "Georgia, 'Times New Roman', Times, serif",
        'font_family_ebook': 'Georgia, "Times New Roman", Times, serif',
        'toc_label': 'Inhaltsverzeichnis',
        'pdf_font': 'Georgia',
    },
    'es': {
        'lang_attr': 'es',
        'font_family': "Georgia, 'Times New Roman', Times, serif",
        'font_family_ebook': 'Georgia, "Times New Roman", Times, serif',
        'toc_label': 'Índice',
        'pdf_font': 'Georgia',
    },
}

# Default fallback for unknown languages
_DEFAULT_LANG_CONFIG = {
    'lang_attr': 'en',
    'font_family': "Georgia, 'Times New Roman', Times, serif",
    'font_family_ebook': 'Georgia, "Times New Roman", Times, serif',
    'toc_label': 'Contents',
    'pdf_font': 'Georgia',
}


def get_lang_config(lang_code):
    """Get language config, falling back to defaults for unknown languages."""
    return LANG_CONFIG.get(lang_code, _DEFAULT_LANG_CONFIG)


def load_config(temp_dir):
    """Load configuration from config.txt"""
    config_file = os.path.join(temp_dir, 'config.txt')
    if not os.path.exists(config_file):
        print("Error: config.txt not found in temp directory.")
        sys.exit(1)

    config = {}
    with open(config_file, 'r', encoding='utf-8') as f:
        for line in f:
            if '=' in line and not line.strip().startswith('#'):
                key, value = line.strip().split('=', 1)
                config[key] = value
    return config


def natural_sort_key(text):
    """Natural sorting key for filenames with numbers"""
    return [int(t) if t.isdigit() else t.lower() for t in re.split(r'(\d+)', text)]


# =============================================================================
# Step 4: Merge translated markdown files
# =============================================================================

def merge_markdown_files(temp_dir):
    """Merge all translated output files into output.md"""
    print("=== Merging translated markdown files ===")

    output_md = os.path.join(temp_dir, 'output.md')

    # Always validate manifest, even if output.md exists (catch stale/corrupt outputs)
    ok, ordered_files, warnings = validate_for_merge(temp_dir)

    # Image structure validation runs unconditionally — bad chunks invalidate any cached output.md
    if not _validate_chunk_images(temp_dir):
        if os.path.exists(output_md):
            print("Removing stale output.md (built from chunks that failed image validation)")
            os.remove(output_md)
        return False

    if os.path.exists(output_md):
        if not ok:
            print(f"WARNING: output.md exists but manifest validation failed — deleting stale output.md")
            os.remove(output_md)
        else:
            # Check if any output_chunk is newer than output.md (re-translated chunks)
            output_md_mtime = os.path.getmtime(output_md)
            newer_chunks = []
            if ordered_files:
                newer_chunks = [
                    os.path.basename(f) for f in ordered_files
                    if os.path.getmtime(f) > output_md_mtime
                ]
            if newer_chunks:
                print(f"Re-merging — {len(newer_chunks)} chunk(s) newer than output.md: {', '.join(newer_chunks[:5])}{'...' if len(newer_chunks) > 5 else ''}")
                os.remove(output_md)
            else:
                print(f"Skipping merge - output.md already exists and is up to date")
                return True

    if not ok:
        print("ERROR: Merge validation failed. Fix the issues above before merging.")
        return False

    if ordered_files is not None:
        # Manifest-based merge
        print(f"Merging {len(ordered_files)} translated files (manifest-ordered)")
        merged_content = ""
        for file_path in ordered_files:
            try:
                with open(file_path, 'r', encoding='utf-8') as f:
                    content = f.read().strip()
                if content:
                    merged_content += content + "\n\n"
            except Exception as e:
                print(f"Error reading {os.path.basename(file_path)}: {e}")
    else:
        # Legacy fallback: glob-based merge (no manifest)
        print("WARNING: No manifest.json found — using legacy glob-based merge.")
        print("  For hash validation, re-run convert.py to generate manifest.json")

        # Match chunk output files
        output_files = glob.glob(os.path.join(temp_dir, 'output_chunk*.md'))

        # Count original source files
        original_files = glob.glob(os.path.join(temp_dir, 'chunk*.md'))
        original_files = [f for f in original_files if not os.path.basename(f).startswith('output_')]

        if not output_files:
            print("Error: No translated markdown files found.")
            return False

        # Build expected output filename for each source file and verify 1:1 match
        source_basenames = sorted(
            [os.path.basename(f) for f in original_files],
            key=natural_sort_key
        )
        expected_outputs = set(f"output_{name}" for name in source_basenames)
        actual_outputs = set(os.path.basename(f) for f in output_files)

        missing = expected_outputs - actual_outputs
        orphaned = actual_outputs - expected_outputs

        if missing or orphaned:
            if missing:
                print(f"ERROR: Missing translations for: {', '.join(sorted(missing, key=natural_sort_key))}")
            if orphaned:
                print(f"ERROR: Orphaned outputs (no matching source): {', '.join(sorted(orphaned, key=natural_sort_key))}")
            return False

        # Verify no empty output files
        for fp in output_files:
            if os.path.getsize(fp) == 0:
                print(f"ERROR: Empty output file: {os.path.basename(fp)}")
                return False

        # Use source order to determine merge order (via expected output names)
        output_files = [
            os.path.join(temp_dir, f"output_{name}")
            for name in source_basenames
        ]
        print(f"Merging {len(output_files)} translated files (legacy glob)")

        merged_content = ""
        for file_path in output_files:
            try:
                with open(file_path, 'r', encoding='utf-8') as f:
                    content = f.read().strip()
                if content:
                    merged_content += content + "\n\n"
            except Exception as e:
                print(f"Error reading {os.path.basename(file_path)}: {e}")

    try:
        with open(output_md, 'w', encoding='utf-8') as f:
            f.write(merged_content)
        file_size = os.path.getsize(output_md)
        print(f"Merged into output.md ({file_size:,} bytes)")
        return True
    except Exception as e:
        print(f"Error saving merged file: {e}")
        return False


# =============================================================================
# Step 5: Convert markdown to HTML
# =============================================================================

def check_pandoc_available():
    """Check if pandoc is available"""
    try:
        subprocess.run(['pandoc', '--version'], capture_output=True, check=True)
        return True
    except (subprocess.CalledProcessError, FileNotFoundError):
        return False


def convert_with_pandoc(md_file, html_file, title, lang_attr):
    """Convert markdown to HTML using pandoc"""
    cmd = [
        'pandoc', md_file, '-o', html_file,
        '--standalone',
        '--metadata', f'title={title}',
        '--metadata', f'lang={lang_attr}',
        '--from', 'markdown+smart+east_asian_line_breaks',
        '--to', 'html5'
    ]

    try:
        subprocess.run(cmd, capture_output=True, text=True, check=True)
        print("Converted with pandoc")
        return True
    except subprocess.CalledProcessError as e:
        print(f"Pandoc conversion failed: {e.stderr}")
        return False


def convert_with_python_markdown(md_file, html_file, title):
    """Convert markdown to HTML using python-markdown (fallback 1)"""
    if not MARKDOWN_AVAILABLE:
        return False

    try:
        with open(md_file, 'r', encoding='utf-8') as f:
            md_content = f.read()

        extensions = ['toc', 'tables', 'fenced_code', 'codehilite', 'nl2br']
        md = markdown.Markdown(extensions=extensions)
        html_content = md.convert(md_content)

        full_html = f"""<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>{title}</title>
</head>
<body>
{html_content}
</body>
</html>"""

        with open(html_file, 'w', encoding='utf-8') as f:
            f.write(full_html)

        print("Converted with python-markdown (fallback)")
        return True
    except Exception as e:
        print(f"python-markdown conversion failed: {e}")
        return False


def convert_with_basic_regex(md_file, html_file, title):
    """Convert markdown to HTML using basic regex (fallback 2)"""
    try:
        with open(md_file, 'r', encoding='utf-8') as f:
            md_content = f.read()

        html_content = md_content

        # Headers
        html_content = re.sub(r'^#### (.*?)$', r'<h4>\1</h4>', html_content, flags=re.MULTILINE)
        html_content = re.sub(r'^### (.*?)$', r'<h3>\1</h3>', html_content, flags=re.MULTILINE)
        html_content = re.sub(r'^## (.*?)$', r'<h2>\1</h2>', html_content, flags=re.MULTILINE)
        html_content = re.sub(r'^# (.*?)$', r'<h1>\1</h1>', html_content, flags=re.MULTILINE)

        # Bold and italic
        html_content = re.sub(r'\*\*(.*?)\*\*', r'<strong>\1</strong>', html_content)
        html_content = re.sub(r'\*(.*?)\*', r'<em>\1</em>', html_content)
        html_content = re.sub(r'_(.*?)_', r'<em>\1</em>', html_content)

        # Images — escape alt and src so quotes in alt text don't break the tag
        def _md_img_to_html(m):
            alt = _html_lib.escape(m.group(1), quote=True)
            src = _html_lib.escape(m.group(2), quote=True)
            return f'<img src="{src}" alt="{alt}">'
        html_content = re.sub(r'!\[([^\]]*)\]\(([^)]*)\)', _md_img_to_html, html_content)

        # Links
        html_content = re.sub(r'\[([^\]]*)\]\(([^)]*)\)', r'<a href="\2">\1</a>', html_content)

        # Lists and paragraphs
        lines = html_content.split('\n')
        result_lines = []
        in_list = False

        for line in lines:
            stripped = line.strip()
            if stripped.startswith('- '):
                if not in_list:
                    result_lines.append('<ul>')
                    in_list = 'ul'
                item = stripped[2:]
                result_lines.append(f'<li>{item}</li>')
            elif re.match(r'^\d+\. ', stripped):
                if not in_list:
                    result_lines.append('<ol>')
                    in_list = 'ol'
                item = re.sub(r'^\d+\. ', '', stripped)
                result_lines.append(f'<li>{item}</li>')
            else:
                if in_list:
                    result_lines.append(f'</{in_list}>')
                    in_list = False
                if stripped and not stripped.startswith('<'):
                    result_lines.append(f'<p>{line}</p>')
                else:
                    result_lines.append(line)

        if in_list:
            result_lines.append(f'</{in_list}>')

        html_content = '\n'.join(result_lines)

        # Page separators
        html_content = re.sub(r'<p>---</p>', '<div class="page-separator"></div>', html_content)

        full_html = f"""<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>{title}</title>
</head>
<body>
{html_content}
</body>
</html>"""

        with open(html_file, 'w', encoding='utf-8') as f:
            f.write(full_html)

        print("Converted with basic regex (fallback 2)")
        return True
    except Exception as e:
        print(f"Basic regex conversion failed: {e}")
        return False


def apply_template_to_html(html_content, template_file, output_file, title, lang_cfg, author=None):
    """Apply a template to HTML content with language-aware substitutions"""
    if not template_file or not os.path.exists(template_file):
        print(f"Warning: Template {template_file} not found")
        return False

    try:
        with open(template_file, 'r', encoding='utf-8') as f:
            template_content = f.read()

        if '$body$' in template_content:
            full_html = template_content.replace('$body$', html_content)
        elif '{{content}}' in template_content:
            full_html = template_content.replace('{{content}}', html_content)
        else:
            if '</body>' in template_content:
                full_html = template_content.replace('</body>', f'{html_content}\n</body>')
            else:
                full_html = template_content + html_content

        # Replace all template placeholders
        full_html = full_html.replace('$title$', title)
        full_html = full_html.replace('$lang$', lang_cfg['lang_attr'])
        full_html = full_html.replace('$body_font$', lang_cfg['font_family'])
        full_html = full_html.replace('$toc_label$', lang_cfg['toc_label'])

        # Inject author meta tag into <head> so calibre_html_publish.py can extract it
        if author:
            author_meta = f'<meta name="author" content="{author}">'
            if '<head>' in full_html or '<head ' in full_html:
                full_html = re.sub(
                    r'(<head[^>]*>)',
                    r'\1\n    ' + author_meta,
                    full_html,
                    count=1,
                    flags=re.IGNORECASE
                )

        with open(output_file, 'w', encoding='utf-8') as f:
            f.write(full_html)
        return True
    except Exception as e:
        print(f"Error applying template: {e}")
        return False


def process_html_separators(html_file):
    """Process page separators in HTML"""
    try:
        with open(html_file, 'r', encoding='utf-8') as f:
            content = f.read()

        content = re.sub(r'<hr\s*/?>', '<div class="page-separator"></div>', content)
        content = re.sub(r'<p>\s*---\s*</p>', '<div class="page-separator"></div>', content)

        with open(html_file, 'w', encoding='utf-8') as f:
            f.write(content)
    except Exception as e:
        print(f"Error processing separators: {e}")


def convert_md_to_html(temp_dir, title, lang_cfg, author=None):
    """Convert output.md to HTML with templates"""
    print("=== Converting markdown to HTML ===")

    md_file = os.path.join(temp_dir, 'output.md')
    if not os.path.exists(md_file):
        print("Error: output.md not found.")
        return False

    book_doc_file = os.path.join(temp_dir, 'book_doc.html')

    # Skip HTML generation if book_doc.html exists and is newer than output.md
    if os.path.exists(book_doc_file):
        if os.path.getmtime(book_doc_file) > os.path.getmtime(md_file):
            if _check_generated_html_sanity(book_doc_file):
                print("Skipping HTML generation - book_doc.html is up to date")
                return True
            print("Stale book_doc.html failed image sanity — regenerating")
            os.remove(book_doc_file)
        else:
            print("Re-generating HTML - output.md is newer")

    temp_html_file = os.path.join(temp_dir, 'output.html')

    # Try pandoc -> python-markdown -> basic regex
    success = False
    if check_pandoc_available():
        success = convert_with_pandoc(md_file, temp_html_file, title, lang_cfg['lang_attr'])

    if not success:
        success = convert_with_python_markdown(md_file, temp_html_file, title)

    if not success:
        success = convert_with_basic_regex(md_file, temp_html_file, title)

    if not success:
        print("Error: All markdown-to-HTML converters failed")
        return False

    process_html_separators(temp_html_file)

    # Extract body content
    try:
        with open(temp_html_file, 'r', encoding='utf-8') as f:
            html_content = f.read()
    except Exception as e:
        print(f"Error reading HTML file: {e}")
        return False

    body_match = re.search(r'<body[^>]*>(.*?)</body>', html_content, re.DOTALL | re.IGNORECASE)
    body_content = body_match.group(1).strip() if body_match else html_content

    # Generate book_doc.html with ebook template
    template_ebook = os.path.join(SCRIPT_DIR, 'template_ebook.html')
    book_doc_file = os.path.join(temp_dir, 'book_doc.html')
    apply_template_to_html(body_content, template_ebook, book_doc_file, title, lang_cfg, author)

    # Generate book.html with web template
    template_web = os.path.join(SCRIPT_DIR, 'template.html')
    book_file = os.path.join(temp_dir, 'book.html')
    apply_template_to_html(body_content, template_web, book_file, title, lang_cfg, author)

    if not _check_generated_html_sanity(book_doc_file):
        return False
    if not _check_generated_html_sanity(book_file):
        return False

    print(f"Generated: output.html, book_doc.html, book.html")
    return True


# =============================================================================
# Step 6: Add TOC
# =============================================================================

def generate_heading_id(text, existing_ids):
    """Generate unique ID for heading"""
    base_id = re.sub(r'[^\w\s-]', '', text.lower())
    base_id = re.sub(r'[-\s]+', '-', base_id)
    base_id = base_id.strip('-')

    if not base_id:
        base_id = 'heading'

    heading_id = base_id
    counter = 1
    while heading_id in existing_ids:
        heading_id = f"{base_id}-{counter}"
        counter += 1

    return heading_id


def generate_simple_toc_html(toc_data):
    """Generate simple HTML for table of contents"""
    if not toc_data:
        return ""

    toc_html = '<ul>\n'
    current_level = 1

    for item in toc_data:
        level = item['level']
        text = item['text']
        heading_id = item['id']

        if level > current_level:
            while current_level < level:
                toc_html += '<li><ul>\n'
                current_level += 1
        elif level < current_level:
            while current_level > level:
                toc_html += '</ul></li>\n'
                current_level -= 1

        toc_html += f'<li><a href="#{heading_id}">{text}</a></li>\n'

    while current_level > 1:
        toc_html += '</ul></li>\n'
        current_level -= 1

    toc_html += '</ul>\n'
    return toc_html


def insert_toc_with_bs4(html_file):
    """Insert TOC using BeautifulSoup"""
    try:
        with open(html_file, 'r', encoding='utf-8') as f:
            html_content = f.read()
    except Exception as e:
        print(f"Error reading HTML file: {e}")
        return False

    soup = BeautifulSoup(html_content, 'html.parser')

    toc_data = []
    existing_ids = []

    for heading in soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6']):
        level = int(heading.name[1])
        text = heading.get_text().strip()
        if not text:
            continue

        heading_id = generate_heading_id(text, existing_ids)
        existing_ids.append(heading_id)
        heading['id'] = heading_id
        toc_data.append({'level': level, 'text': text, 'id': heading_id})

    if not toc_data:
        print("No headings found for TOC")
        return False

    toc_html = generate_simple_toc_html(toc_data)

    toc_content_div = soup.find('div', class_='toc-content')
    if toc_content_div:
        toc_content_div.clear()
        toc_soup = BeautifulSoup(toc_html, 'html.parser')
        toc_content_div.append(toc_soup)
        print(f"TOC inserted ({len(toc_data)} headings)")
    else:
        print("Warning: .toc-content div not found, TOC not inserted")
        return False

    try:
        with open(html_file, 'w', encoding='utf-8') as f:
            f.write(str(soup))
        return True
    except Exception as e:
        print(f"Error saving HTML file: {e}")
        return False


def insert_toc_with_regex(html_file):
    """Insert TOC using regex (fallback)"""
    try:
        with open(html_file, 'r', encoding='utf-8') as f:
            html_content = f.read()
    except Exception as e:
        print(f"Error reading HTML file: {e}")
        return False

    heading_pattern = r'<(h[1-6])(?:[^>]*)>(.*?)</\1>'
    headings = re.findall(heading_pattern, html_content, re.IGNORECASE | re.DOTALL)

    if not headings:
        print("No headings found for TOC")
        return False

    toc_html = '<ul>\n'
    for i, (tag, text) in enumerate(headings):
        level = int(tag[1])
        clean_text = re.sub(r'<[^>]+>', '', text).strip()
        heading_id = f"heading-{i+1}"

        old_heading = f'<{tag}>{text}</{tag}>'
        new_heading = f'<{tag} id="{heading_id}">{text}</{tag}>'
        html_content = html_content.replace(old_heading, new_heading, 1)

        if level > 1:
            for _ in range(level - 1):
                toc_html += '  '
        toc_html += f'<li><a href="#{heading_id}">{clean_text}</a></li>\n'

    toc_html += '</ul>\n'

    toc_content_pattern = r'(<div[^>]*class="toc-content[^"]*"[^>]*>).*?(</div>)'
    if re.search(toc_content_pattern, html_content, re.DOTALL):
        html_content = re.sub(
            toc_content_pattern,
            r'\1' + toc_html + r'\2',
            html_content,
            flags=re.DOTALL
        )
        print(f"TOC inserted ({len(headings)} headings)")
    else:
        print("Warning: .toc-content div not found")
        return False

    try:
        with open(html_file, 'w', encoding='utf-8') as f:
            f.write(html_content)
        return True
    except Exception as e:
        print(f"Error saving HTML file: {e}")
        return False


def add_toc(temp_dir):
    """Add TOC to book.html"""
    print("=== Adding Table of Contents ===")

    book_file = os.path.join(temp_dir, 'book.html')
    if not os.path.exists(book_file):
        print("Warning: book.html not found, skipping TOC")
        return False

    if BS4_AVAILABLE:
        return insert_toc_with_bs4(book_file)
    else:
        return insert_toc_with_regex(book_file)


# =============================================================================
# Step 7: Generate DOCX/EPUB/PDF with error transparency
# =============================================================================

def generate_format(html_file, temp_dir, output_ext, lang_attr):
    """Generate a specific format using calibre_html_publish.py"""
    output_file = os.path.join(temp_dir, f"book{output_ext}")

    if os.path.exists(output_file):
        output_mtime = os.path.getmtime(output_file)

        # Check if source HTML is newer
        html_newer = os.path.getmtime(html_file) > output_mtime

        # Check if any image asset is newer (Calibre embeds these)
        images_newer = False
        images_dir = os.path.join(temp_dir, 'images')
        if os.path.isdir(images_dir):
            for img in os.listdir(images_dir):
                img_path = os.path.join(images_dir, img)
                if os.path.isfile(img_path) and os.path.getmtime(img_path) > output_mtime:
                    images_newer = True
                    break

        if not html_newer and not images_newer:
            file_size = os.path.getsize(output_file)
            print(f"Skipping {output_ext} - already exists and up to date ({file_size:,} bytes)")
            return output_file
        else:
            reasons = []
            if html_newer:
                reasons.append("source HTML changed")
            if images_newer:
                reasons.append("image assets changed")
            print(f"Rebuilding {output_ext} - {', '.join(reasons)}")

    publish_script = os.path.join(SCRIPT_DIR, "calibre_html_publish.py")
    if not os.path.exists(publish_script):
        print(f"calibre_html_publish.py not found at: {publish_script}")
        return None

    try:
        cmd = ["python3", publish_script, html_file, "-o", output_file, "--lang", lang_attr]
        result = subprocess.run(cmd, check=True, capture_output=True, text=True)

        if os.path.exists(output_file):
            file_size = os.path.getsize(output_file)
            return output_file
        else:
            print(f"Failed to generate {output_ext}")
            if result.stdout:
                print(f"  stdout: {result.stdout[-500:]}")
            return None
    except subprocess.CalledProcessError as e:
        print(f"Failed to generate {output_ext}")
        if e.stdout:
            print(f"  stdout: {e.stdout[-500:]}")
        if e.stderr:
            print(f"  stderr: {e.stderr[-500:]}")
        return None
    except Exception as e:
        print(f"Error generating {output_ext}: {e}")
        return None


def generate_formats(temp_dir, lang_attr):
    """Generate DOCX, EPUB, and PDF with result summary"""
    print("=== Generating output formats ===")

    html_file = os.path.join(temp_dir, "book_doc.html")
    if not os.path.exists(html_file):
        html_files = glob.glob(os.path.join(temp_dir, "*.html"))
        if html_files:
            html_file = max(html_files, key=os.path.getmtime)
        else:
            print("No HTML files found for format generation")
            return

    results = {}
    for ext in ['.docx', '.epub', '.pdf']:
        result = generate_format(html_file, temp_dir, ext, lang_attr)
        if result:
            file_size = os.path.getsize(result)
            results[ext] = ('OK', f"{file_size:,} bytes")
        else:
            results[ext] = ('FAILED', '')

    # Print summary table
    print("\nFormat results:")
    has_failures = False
    for ext, (status, detail) in results.items():
        if status == 'OK':
            print(f"  {ext}: {status} ({detail})")
        else:
            print(f"  {ext}: {status}")
            has_failures = True

    return not has_failures


# =============================================================================
# Main
# =============================================================================

def main():
    parser = argparse.ArgumentParser(description='Merge translated pages and build final outputs')
    parser.add_argument('--temp-dir', required=True, help='Temp directory path')
    parser.add_argument('--title', default=None, help='Translated book title (override config)')
    parser.add_argument('--author', default=None, help='Author name (override config)')
    parser.add_argument('--lang', default=None, help='Output language code (override config)')
    parser.add_argument('--cleanup', action='store_true', help='Remove intermediate artifacts after successful build')

    args = parser.parse_args()
    temp_dir = args.temp_dir

    if not os.path.isdir(temp_dir):
        print(f"Error: Temp directory not found: {temp_dir}")
        sys.exit(1)

    # Load config as base, CLI args override
    config = load_config(temp_dir)

    lang_code = args.lang or config.get('output_lang', 'zh')
    lang_cfg = get_lang_config(lang_code)

    title = args.title or config.get('original_title', 'Translated Book')
    author = args.author or config.get('creator', 'Unknown Author')

    print(f"=== Merge and Build ===")
    print(f"Temp directory: {temp_dir}")
    print(f"Title: {title}")
    print(f"Author: {author}")
    print(f"Language: {lang_code} (attr: {lang_cfg['lang_attr']})")

    # Step 4: Merge
    if not merge_markdown_files(temp_dir):
        sys.exit(1)

    # Step 5: Convert to HTML
    if not convert_md_to_html(temp_dir, title, lang_cfg, author):
        sys.exit(1)

    # Step 6: Add TOC
    add_toc(temp_dir)

    # Step 7: Generate formats
    all_formats_ok = generate_formats(temp_dir, lang_cfg['lang_attr'])

    print("\n=== Build Complete ===")
    print(f"All outputs saved to: {temp_dir}")

    # List generated files
    for ext in ['book.html', 'book_doc.html', 'book.docx', 'book.epub', 'book.pdf']:
        filepath = os.path.join(temp_dir, ext)
        if os.path.exists(filepath):
            size = os.path.getsize(filepath)
            print(f"  {ext}: {size:,} bytes")

    # Cleanup intermediate artifacts if requested (skip if any format failed)
    if args.cleanup:
        if all_formats_ok:
            cleanup_intermediate_files(temp_dir)
        else:
            print("\nSkipping cleanup — some formats failed. Intermediate files kept for diagnosis/retry.")


def cleanup_intermediate_files(temp_dir):
    """Remove intermediate artifacts, keeping only final outputs."""
    print("\n=== Cleaning up intermediate files ===")

    removed = []

    # Remove chunk*.md and output_chunk*.md
    for pattern in ['chunk*.md', 'output_chunk*.md']:
        for filepath in glob.glob(os.path.join(temp_dir, pattern)):
            os.remove(filepath)
            removed.append(os.path.basename(filepath))

    # Remove specific intermediate files
    for name in ['input.html', 'input.md', 'output.html']:
        filepath = os.path.join(temp_dir, name)
        if os.path.exists(filepath):
            os.remove(filepath)
            removed.append(name)

    if removed:
        print(f"Removed {len(removed)} intermediate file(s):")
        # Summarize chunk files instead of listing each one
        chunk_files = [f for f in removed if 'chunk' in f]
        other_files = [f for f in removed if 'chunk' not in f]
        if chunk_files:
            print(f"  {len(chunk_files)} chunk files (chunk*.md, output_chunk*.md)")
        for f in other_files:
            print(f"  {f}")
    else:
        print("No intermediate files to remove.")


if __name__ == "__main__":
    main()


path: scripts/merge_meta.py
content: #!/usr/bin/env python3
"""
merge_meta.py - Merge sub-agent observation files into the canonical glossary.

Three subcommands:

- `prepare-merge <temp_dir>` — scan all `output_chunk*.meta.json` files in
  the temp dir, filter to those whose content hash isn't already in
  `glossary["applied_meta_hashes"]`, classify findings against current
  glossary state, and emit a JSON proposal to stdout. Malformed meta files
  are quarantined (warn + skip + count separately) — they don't crash the
  step.

- `apply-merge <temp_dir>` — read a decisions JSON from stdin, conservatively
  apply the auto-applies and resolved decisions to the glossary, and write
  meta content hashes into `applied_meta_hashes` for every chunk in
  `consumed_chunk_ids`. Atomic save.

- `status <temp_dir>` — read-only observability snapshot for the final
  verification report.

Approach A: scripts NEVER invoke an LLM. All semantic decisions (alias yes/no,
conflict resolution, conflicting-proposal pick) are made by the main agent in
SKILL.md Step 4.5; this script just shapes data and applies pre-decided
choices.
"""

import argparse
import json
import os
import re
import sys
from pathlib import Path

import glossary as glossary_mod
import meta as meta_mod


META_GLOB = 'output_chunk*.meta.json'
TRANSLATED_GLOB = 'output_chunk*.md'
EVIDENCE_REFS_CAP = 5

# Decision kinds and their valid choice sets.
VALID_CHOICES_BY_KIND = {
    'alias': frozenset({'yes_alias', 'no_separate_entity', 'skip'}),
    'conflict': frozenset({'keep_current', 'accept_proposed', 'record_in_notes'}),
    # alias_or_new_entity, conflicting_new_entity_proposals,
    # new_entity_existing_alias, and existing_entity_conflict have dynamic
    # use_*_N choices generated from variant counts; validated at apply time
    # against the decision item's options array.
}

VALID_GENDER_VALUES = ('male', 'female', 'nonbinary', 'unknown')


def _glossary_path(temp_dir):
    return os.path.join(temp_dir, 'glossary.json')


def _meta_path(temp_dir, chunk_id):
    return os.path.join(temp_dir, f'output_{chunk_id}.meta.json')


def _try_load_meta(path):
    """Wrap meta.load_meta for graceful degradation. Returns (data, error_msg).

    Honors the "meta is non-blocking" promise: a malformed meta file is
    skipped, not raised, so a single bad sub-agent doesn't crash the merge.
    """
    try:
        return meta_mod.load_meta(path), None
    except (ValueError, OSError) as e:
        return None, str(e)


def _confidence_for_evidence_count(n):
    if n >= 3:
        return 'high'
    if n == 2:
        return 'medium'
    return 'low'


_CONFIDENCE_RANK = {'low': 0, 'medium': 1, 'high': 2}


def _promote_confidence(current, candidate):
    """Return the higher-ranked confidence. Never downgrade."""
    if _CONFIDENCE_RANK.get(candidate, -1) > _CONFIDENCE_RANK.get(current, -1):
        return candidate
    return current


def _append_evidence_ref(term, chunk_id):
    """Append chunk_id to term's evidence_refs (FIFO cap at 5, dedup), then
    promote confidence based on the new length. Returns True if anything
    changed."""
    refs = term.get('evidence_refs', [])
    if chunk_id in refs:
        return False
    refs.append(chunk_id)
    if len(refs) > EVIDENCE_REFS_CAP:
        refs = refs[-EVIDENCE_REFS_CAP:]
    term['evidence_refs'] = refs
    term['confidence'] = _promote_confidence(
        term.get('confidence', 'low'),
        _confidence_for_evidence_count(len(refs)),
    )
    return True


def _append_note(term, line):
    """Append a line to a term's notes, keeping it readable."""
    existing = term.get('notes', '') or ''
    term['notes'] = (existing + '\n' + line).lstrip('\n')


def _build_surface_index(glossary):
    """Map every surface form (source or alias) → (term_id, role)."""
    idx = {}
    for t in glossary['terms']:
        idx[t['source']] = (t['id'], 'source')
        for a in t.get('aliases', []):
            idx[a] = (t['id'], 'alias')
    return idx


def _find_term_by_id(glossary, entity_id):
    for t in glossary['terms']:
        if t['id'] == entity_id:
            return t
    return None


def _find_term_by_surface(glossary, surface):
    """Return the term whose source OR any alias equals `surface`."""
    for t in glossary['terms']:
        if t['source'] == surface or surface in t.get('aliases', []):
            return t
    return None


def cmd_prepare_merge(temp_dir):
    """Scan unmerged metas and propose a merge plan. Output JSON on stdout."""
    glossary_path = _glossary_path(temp_dir)
    glossary = glossary_mod.load_glossary(glossary_path)
    applied = glossary.get('applied_meta_hashes', {})

    meta_paths = sorted(Path(temp_dir).glob(META_GLOB))

    consumed_chunk_ids = []
    malformed_chunk_ids = []
    # Loaded metas keyed by chunk_id, only including non-malformed unmerged ones.
    loaded = {}

    for path in meta_paths:
        try:
            chunk_id = meta_mod.chunk_id_from_meta_path(str(path))
        except ValueError as e:
            sys.stderr.write(f"warn: {e}\n")
            continue

        data, err = _try_load_meta(str(path))
        if err is not None:
            sys.stderr.write(
                f"warn: malformed meta for {chunk_id} at {path} — quarantined ({err})\n"
            )
            malformed_chunk_ids.append(chunk_id)
            continue

        content_hash = meta_mod.meta_content_hash(data)
        if applied.get(chunk_id) == content_hash:
            continue  # already applied, skip silently

        loaded[chunk_id] = data
        consumed_chunk_ids.append(chunk_id)

    # Group new_entities across all loaded metas by source string.
    grouped_new_entities = {}
    for chunk_id, data in loaded.items():
        for entity in data.get('new_entities', []):
            src = entity['source']
            grouped_new_entities.setdefault(src, []).append({
                'chunk_id': chunk_id,
                'target_proposal': entity['target_proposal'],
                'category': entity.get('category', ''),
                'evidence': entity['evidence'],
            })

    surface_idx = _build_surface_index(glossary)

    auto_apply = []
    decisions_needed = []
    next_decision_id = 1

    def _new_decision_id():
        nonlocal next_decision_id
        did = f'd{next_decision_id}'
        next_decision_id += 1
        return did

    # Helper: build the standalone_variants array for a variant pulled out
    # of grouped_new_entities. Each (target, category) pair gets its own
    # variant entry with its evidence_chunks, parallel to conflicting_new_entity_proposals.
    def _standalone_variants_for(proposals):
        grouped_by_pair = {}
        for p in proposals:
            grouped_by_pair.setdefault((p['target_proposal'], p['category']), []).append(p)
        out = []
        for (target, category), ps in grouped_by_pair.items():
            out.append({
                'target_proposal': target,
                'category': category,
                'evidence': ps[0]['evidence'],
                'evidence_chunks': sorted({p['chunk_id'] for p in ps}),
            })
        return out

    # Group alias_hypotheses by variant across all loaded metas. Each variant
    # may have multiple alias candidates (e.g. chunk0002 says Taig→Tai while
    # chunk0003 says Taig→Tao); they're all options the orchestrator must
    # choose between. Multiple chunks proposing the same (variant, candidate)
    # pair contribute their evidence_chunks to a single candidate entry.
    grouped_alias_hyps = {}  # variant -> list of (chunk_id, ah)
    for chunk_id, data in loaded.items():
        for ah in data.get('alias_hypotheses', []):
            grouped_alias_hyps.setdefault(ah['variant'], []).append((chunk_id, ah))

    def _alias_candidates_for(hyps):
        """Aggregate (chunk_id, ah) tuples into per-candidate entries with
        combined evidence_chunks. Preserves the sub-agent's literal candidate
        string — apply-merge re-resolves at dispatch time."""
        by_candidate = {}
        for chunk_id, ah in hyps:
            by_candidate.setdefault(ah['may_be_alias_of_source'], []).append((chunk_id, ah))
        out = []
        for candidate, items in by_candidate.items():
            out.append({
                'candidate_source': candidate,
                'evidence': items[0][1]['evidence'],
                'evidence_chunks': sorted({cid for cid, _ in items}),
            })
        return out

    # Pass 1: variants that collide with same-batch standalone proposals.
    # Emit alias_or_new_entity exposing EVERY alias candidate AND every
    # standalone variant. Pop the variant from grouped_new_entities so it
    # doesn't also get auto_apply'd / decisioned below.
    consumed_alias_variants = set()  # variants already wrapped into a unified decision
    for variant in list(grouped_alias_hyps.keys()):
        if variant not in grouped_new_entities:
            continue
        if variant in surface_idx:
            # Variant is already a glossary surface; the
            # new_entity_existing_alias / conflict path owns it.
            continue
        alias_candidates = _alias_candidates_for(grouped_alias_hyps[variant])
        proposals = grouped_new_entities.pop(variant)
        standalone_variants = _standalone_variants_for(proposals)
        options = (
            [f'use_alias_{i}' for i in range(len(alias_candidates))]
            + [f'use_standalone_{i}' for i in range(len(standalone_variants))]
            + ['skip']
        )
        decisions_needed.append({
            'id': _new_decision_id(),
            'kind': 'alias_or_new_entity',
            'variant': variant,
            'alias_candidates': alias_candidates,
            'standalone_variants': standalone_variants,
            'options': options,
        })
        consumed_alias_variants.add(variant)

    # Pass 2: variants with 2+ alias candidates but NO standalone collision.
    # Without a unified decision the orchestrator could naively accept multiple
    # `yes_alias` decisions for the same variant — surface uniqueness violation.
    for variant in list(grouped_alias_hyps.keys()):
        if variant in consumed_alias_variants:
            continue
        if variant in surface_idx:
            continue
        alias_candidates = _alias_candidates_for(grouped_alias_hyps[variant])
        if len(alias_candidates) < 2:
            continue
        options = [f'use_alias_{i}' for i in range(len(alias_candidates))] + ['skip']
        decisions_needed.append({
            'id': _new_decision_id(),
            'kind': 'alias_or_new_entity',
            'variant': variant,
            'alias_candidates': alias_candidates,
            'standalone_variants': [],
            'options': options,
        })
        consumed_alias_variants.add(variant)

    for src, proposals in grouped_new_entities.items():
        target_cat_pairs = {(p['target_proposal'], p['category']) for p in proposals}
        if src in surface_idx:
            owner_id, owner_role = surface_idx[src]
            owner_term = _find_term_by_id(glossary, owner_id)
            if owner_role == 'source':
                # src is already canonical. For each distinct (target, category)
                # proposal across the batch that DIFFERS from the canonical
                # (target, category), expose it as a competing variant. If
                # every proposal matches canonical, silent no-op.
                canonical_pair = (
                    owner_term['target'],
                    owner_term.get('category', ''),
                )
                proposed_variants = _standalone_variants_for(proposals)
                differing = [
                    v for v in proposed_variants
                    if (v['target_proposal'], v['category']) != canonical_pair
                ]
                if differing:
                    options = (
                        ['keep_current']
                        + [f'use_variant_{i}' for i in range(len(differing))]
                        + ['record_in_notes']
                    )
                    decisions_needed.append({
                        'id': _new_decision_id(),
                        'kind': 'existing_entity_conflict',
                        'entity_source': src,
                        'current_target': owner_term['target'],
                        'current_category': owner_term.get('category', ''),
                        'proposed_variants': differing,
                        'options': options,
                    })
                # else: every proposal matched canonical — no decision needed.
            else:  # role == 'alias'
                # src is currently another term's alias; sub-agents propose to
                # promote it. Expose every distinct (target, category) variant
                # so the orchestrator can pick the right promoted form.
                promoted_variants = _standalone_variants_for(proposals)
                options = (
                    [f'use_variant_{i}' for i in range(len(promoted_variants))]
                    + ['keep_as_alias', 'skip']
                )
                decisions_needed.append({
                    'id': _new_decision_id(),
                    'kind': 'new_entity_existing_alias',
                    'proposed_source': src,
                    'currently_alias_of': owner_id,
                    'promoted_variants': promoted_variants,
                    'options': options,
                })
        elif len(target_cat_pairs) == 1:
            # All proposals agree → auto_apply with combined evidence.
            target, category = next(iter(target_cat_pairs))
            chunk_ids = sorted({p['chunk_id'] for p in proposals})
            auto_apply.append({
                'action': 'add_entity',
                'entity': {
                    'source': src,
                    'target_proposal': target,
                    'category': category,
                    'evidence': proposals[0]['evidence'],
                },
                'evidence_chunks': chunk_ids,
            })
        else:
            # Conflicting proposals across chunks — main agent must pick.
            variants = _standalone_variants_for(proposals)
            options = [f'use_variant_{i}' for i in range(len(variants))] + ['skip']
            decisions_needed.append({
                'id': _new_decision_id(),
                'kind': 'conflicting_new_entity_proposals',
                'source': src,
                'variants': variants,
                'options': options,
            })

    # Compute the set of every surface that's already a glossary source OR
    # could become a standalone source via this batch's auto_apply / decisions.
    # The variant collision check uses this — we don't emit a regular alias
    # decision for a variant that another decision kind already owns.
    potential_standalone_sources = set(surface_idx.keys())  # everything currently in glossary
    for entry in auto_apply:
        potential_standalone_sources.add(entry['entity']['source'])
    for d in decisions_needed:
        if d['kind'] == 'conflicting_new_entity_proposals':
            potential_standalone_sources.add(d['source'])
        elif d['kind'] == 'new_entity_existing_alias':
            potential_standalone_sources.add(d['proposed_source'])
        elif d['kind'] == 'alias_or_new_entity':
            potential_standalone_sources.add(d['variant'])

    # Pass 3: simple alias hypotheses (single candidate, no standalone collision).
    # Emit a decision if the candidate is — or could become — any glossary
    # surface. Variant must NOT also be a potential standalone source; those
    # collisions are owned by alias_or_new_entity / new_entity_existing_alias.
    #
    # Alias chains are handled by iterating until fixed point: a hypothesis
    # `Taighi → Taig` is moot when first inspected if Taig isn't anywhere yet,
    # but it becomes resolvable after a sibling hypothesis `Taig → Tai` is
    # emitted — Taig will be a glossary alias by apply time. We loop until no
    # new decisions get emitted in a pass.
    #
    # candidate_source is preserved verbatim — apply-merge re-resolves at
    # dispatch time, after all entity-creating decisions have run AND after
    # earlier alias attachments have made new surfaces resolvable.
    pending_alias_hyps = []
    for variant, hyps in grouped_alias_hyps.items():
        if variant in consumed_alias_variants:
            continue  # owned by alias_or_new_entity above
        # By construction (pass 2), any variant reaching here has exactly one
        # distinct alias candidate. Multiple chunks proposing the same
        # (variant, candidate) collapse to a single decision; we keep the
        # first hypothesis as the representative (its evidence quote shows
        # in the prompt; the other chunks' hashes are still recorded via
        # consumed_chunk_ids when the decision is applied).
        first_chunk_id, first_ah = hyps[0]
        pending_alias_hyps.append((first_chunk_id, first_ah))

    # potential_surfaces = anything that might be a resolvable surface at apply
    # time. Starts with current glossary surfaces + every potential standalone
    # source. Grows as we emit alias decisions (each new alias variant becomes
    # a resolvable surface once attached to its candidate's term).
    potential_surfaces = set(surface_idx.keys()) | potential_standalone_sources

    while pending_alias_hyps:
        next_pending = []
        progressed = False
        for chunk_id, ah in pending_alias_hyps:
            variant = ah['variant']
            candidate_source = ah['may_be_alias_of_source']
            if candidate_source not in potential_surfaces:
                next_pending.append((chunk_id, ah))
                continue
            if variant in potential_standalone_sources:
                # Variant is owned by a standalone-creating decision elsewhere;
                # silently skip (was the moot rule).
                continue
            decisions_needed.append({
                'id': _new_decision_id(),
                'kind': 'alias',
                'variant': variant,
                'candidate_source': candidate_source,
                'evidence': ah['evidence'],
                'options': ['yes_alias', 'no_separate_entity', 'skip'],
            })
            potential_surfaces.add(variant)  # downstream chains can now resolve through it
            progressed = True
        if not progressed:
            break  # remaining hypotheses are truly moot
        pending_alias_hyps = next_pending

    # Sub-agent-flagged conflicts.
    for chunk_id, data in loaded.items():
        for c in data.get('conflicts', []):
            entity_term = _find_term_by_surface(glossary, c['entity_source'])
            if entity_term is None:
                continue  # nothing to conflict with
            field = c['field']
            current_val = entity_term.get(field, '')
            decisions_needed.append({
                'id': _new_decision_id(),
                'kind': 'conflict',
                'entity_source': c['entity_source'],
                'field': field,
                'current': current_val,
                'proposed': c['observed_better'],
                'evidence': c['evidence'],
                'options': ['keep_current', 'accept_proposed', 'record_in_notes'],
            })

    output = {
        'auto_apply': auto_apply,
        'decisions_needed': decisions_needed,
        'consumed_chunk_ids': sorted(consumed_chunk_ids),
        'malformed_meta_chunk_ids': sorted(malformed_chunk_ids),
    }
    print(json.dumps(output, ensure_ascii=False, indent=2))


def cmd_apply_merge(temp_dir):
    """Read decisions JSON from stdin and apply to the glossary."""
    try:
        decisions_doc = json.load(sys.stdin)
    except json.JSONDecodeError as e:
        sys.stderr.write(f"error: decisions JSON on stdin is not valid: {e}\n")
        sys.exit(2)

    if not isinstance(decisions_doc, dict):
        sys.stderr.write("error: decisions JSON must be an object\n")
        sys.exit(2)

    auto_apply = decisions_doc.get('auto_apply', [])
    decisions = decisions_doc.get('decisions', [])
    consumed_chunk_ids = decisions_doc.get('consumed_chunk_ids', [])

    if not isinstance(auto_apply, list):
        sys.stderr.write("error: 'auto_apply' must be a list\n")
        sys.exit(2)
    if not isinstance(decisions, list):
        sys.stderr.write("error: 'decisions' must be a list\n")
        sys.exit(2)
    if not isinstance(consumed_chunk_ids, list):
        sys.stderr.write("error: 'consumed_chunk_ids' must be a list\n")
        sys.exit(2)

    glossary_path = _glossary_path(temp_dir)
    glossary = glossary_mod.load_glossary(glossary_path)

    auto_applied = 0
    decisions_resolved = 0
    errors = []

    # Phase 1: process every consumed meta — used_term_sources confidence escalation
    # AND content hash recording. This runs even with empty auto_apply / decisions.
    consumed_metas = {}  # chunk_id -> meta data
    for chunk_id in consumed_chunk_ids:
        meta_path = _meta_path(temp_dir, chunk_id)
        if not os.path.exists(meta_path):
            errors.append(f"no meta file for chunk_id={chunk_id} at {meta_path}")
            continue
        # Defensive: re-derive chunk_id from filename and check.
        derived = meta_mod.chunk_id_from_meta_path(meta_path)
        if derived != chunk_id:
            errors.append(
                f"chunk_id mismatch: consumed_chunk_ids has {chunk_id!r}, "
                f"filename derives {derived!r}"
            )
            continue
        data, err = _try_load_meta(meta_path)
        if err is not None:
            errors.append(f"meta for chunk_id={chunk_id} is malformed: {err}")
            continue
        consumed_metas[chunk_id] = data

    if errors:
        sys.stderr.write("error: apply-merge aborting before any state mutation:\n")
        for e in errors:
            sys.stderr.write(f"  - {e}\n")
        sys.exit(2)

    # Pre-validate every decision payload BEFORE we mutate. If any decision is
    # malformed (wrong kind, invalid choice for kind, missing fields) we abort
    # the entire transaction without writing any hashes — otherwise on retry
    # prepare-merge would skip the affected metas and the unresolved decisions
    # would be lost forever.
    pre_errors = []
    for d in decisions:
        d_id = d.get('id')
        kind = d.get('kind')
        choice = d.get('choice')
        if kind is None or choice is None:
            pre_errors.append(f"decision {d_id!r} missing 'kind' or 'choice'")
            continue
        if kind == 'conflicting_new_entity_proposals':
            variants = d.get('variants') or []
            valid = {f'use_variant_{i}' for i in range(len(variants))} | {'skip'}
            if choice not in valid:
                pre_errors.append(
                    f"decision {d_id!r} (kind={kind}): invalid choice {choice!r}, "
                    f"must be one of {sorted(valid)}"
                )
            continue
        if kind == 'alias_or_new_entity':
            alias_candidates = d.get('alias_candidates') or []
            standalone_variants = d.get('standalone_variants') or []
            valid = (
                {'skip'}
                | {f'use_alias_{i}' for i in range(len(alias_candidates))}
                | {f'use_standalone_{i}' for i in range(len(standalone_variants))}
            )
            if choice not in valid:
                pre_errors.append(
                    f"decision {d_id!r} (kind={kind}): invalid choice {choice!r}, "
                    f"must be one of {sorted(valid)}"
                )
            continue
        if kind == 'new_entity_existing_alias':
            promoted_variants = d.get('promoted_variants') or []
            valid = (
                {'keep_as_alias', 'skip'}
                | {f'use_variant_{i}' for i in range(len(promoted_variants))}
            )
            if choice not in valid:
                pre_errors.append(
                    f"decision {d_id!r} (kind={kind}): invalid choice {choice!r}, "
                    f"must be one of {sorted(valid)}"
                )
            continue
        if kind == 'existing_entity_conflict':
            proposed_variants = d.get('proposed_variants') or []
            valid = (
                {'keep_current', 'record_in_notes'}
                | {f'use_variant_{i}' for i in range(len(proposed_variants))}
            )
            if choice not in valid:
                pre_errors.append(
                    f"decision {d_id!r} (kind={kind}): invalid choice {choice!r}, "
                    f"must be one of {sorted(valid)}"
                )
            continue
        valid = VALID_CHOICES_BY_KIND.get(kind)
        if valid is None:
            pre_errors.append(f"decision {d_id!r}: unknown kind {kind!r}")
            continue
        if choice not in valid:
            pre_errors.append(
                f"decision {d_id!r} (kind={kind}): invalid choice {choice!r}, "
                f"must be one of {sorted(valid)}"
            )

    if pre_errors:
        sys.stderr.write("error: apply-merge aborting before any state mutation:\n")
        for e in pre_errors:
            sys.stderr.write(f"  - {e}\n")
        sys.exit(2)

    # Now safe to mutate. Process used_term_sources first (purely additive).
    for chunk_id, data in consumed_metas.items():
        for src in data.get('used_term_sources', []):
            term = _find_term_by_surface(glossary, src)
            if term is None:
                continue
            _append_evidence_ref(term, chunk_id)

    # Process attribute_hypotheses (currently: gender). Roadmap rule:
    # unknown → first explicit evidence sets value + promotes confidence;
    # corroborating evidence keeps value + promotes confidence; conflicting
    # evidence resets to unknown and records both sides in notes.
    for chunk_id, data in consumed_metas.items():
        for ah in data.get('attribute_hypotheses', []):
            entity_source = ah.get('entity_source')
            attribute = ah.get('attribute')
            value = ah.get('value')
            evidence = ah.get('evidence', '')
            term = _find_term_by_surface(glossary, entity_source)
            if term is None:
                continue
            if attribute == 'gender':
                if value not in VALID_GENDER_VALUES:
                    # Schema validation accepted any string; we silently no-op
                    # on values the glossary doesn't model.
                    continue
                current = term.get('gender', 'unknown')
                if current == 'unknown':
                    term['gender'] = value
                    _append_evidence_ref(term, chunk_id)
                    _append_note(
                        term,
                        f"[gender] {value} set (chunk {chunk_id}); evidence={evidence!r}"
                    )
                elif current == value:
                    # Corroborating — promote entity confidence.
                    _append_evidence_ref(term, chunk_id)
                else:
                    # Conflict: revert to unknown, record both observations.
                    term['gender'] = 'unknown'
                    _append_note(
                        term,
                        f"[gender] conflict — was {current!r}, observed {value!r} "
                        f"in chunk {chunk_id}; evidence={evidence!r}; reverted to 'unknown'"
                    )
            # Other attributes: log to notes but don't mutate canonical fields.
            else:
                _append_note(
                    term,
                    f"[attr {attribute!r}] observed {value!r} in chunk {chunk_id}; "
                    f"evidence={evidence!r}"
                )

    # Phase 2: auto_apply new entities.
    for entry in auto_apply:
        if entry.get('action') != 'add_entity':
            errors.append(f"unknown auto_apply action: {entry.get('action')!r}")
            continue
        entity = entry['entity']
        evidence_chunks = list(entry.get('evidence_chunks', []))[:EVIDENCE_REFS_CAP]
        new_term = {
            'id': entity['source'],
            'source': entity['source'],
            'target': entity['target_proposal'],
            'category': entity.get('category', ''),
            'aliases': [],
            'gender': 'unknown',
            'confidence': _confidence_for_evidence_count(len(evidence_chunks)),
            'frequency': 0,
            'evidence_refs': evidence_chunks,
            'notes': '',
        }
        glossary['terms'].append(new_term)
        auto_applied += 1

    # Phase 3: process decisions. All choices have been pre-validated above.
    #
    # Dispatch order is INDEPENDENT of input order:
    #   Pass A — entity-creating decisions, in input order
    #   Pass B — non-alias followers (conflicts + no-op skips), in input order
    #   Pass C — alias-attachers (alias.yes_alias, alias_or_new_entity.yes_alias),
    #            iterated to fixed point so chains like Taighi → Taig → Tai
    #            resolve regardless of which alias decision came first.
    #
    # Any referential failure (entity not in glossary, etc.) is collected; if
    # non-empty after all passes, the entire transaction aborts so the
    # orchestrator can re-attempt.

    def _is_creator(d):
        kind = d['kind']
        choice = d['choice']
        if kind == 'new_entity_existing_alias':
            return choice.startswith('use_variant_')
        if kind == 'alias_or_new_entity':
            return choice.startswith('use_standalone_')
        if kind == 'conflicting_new_entity_proposals':
            return choice.startswith('use_variant_')
        return False

    def _is_alias_attacher(d):
        kind = d['kind']
        choice = d['choice']
        if kind == 'alias':
            return choice == 'yes_alias'
        if kind == 'alias_or_new_entity':
            return choice.startswith('use_alias_')
        return False

    def _dispatch(d):
        """Apply one decision. Returns (changed, err_or_None). `changed` is
        True if the dispatch did meaningful work (or was a real no-op like
        keep_current), False only if a referential lookup failed and the
        caller should retry later."""
        d_id = d.get('id')
        kind = d['kind']
        choice = d['choice']

        if kind == 'alias':
            if choice == 'yes_alias':
                variant = d.get('variant')
                candidate_source = d.get('candidate_source')
                term = _find_term_by_surface(glossary, candidate_source)
                if term is None:
                    return False, (
                        f"decision {d_id!r}: candidate source {candidate_source!r} "
                        f"not in glossary"
                    )
                if variant not in term.get('aliases', []):
                    term.setdefault('aliases', []).append(variant)
            return True, None

        if kind == 'conflict':
            entity_source = d.get('entity_source')
            field = d.get('field')
            current = d.get('current')
            proposed = d.get('proposed')
            evidence = d.get('evidence', '')
            term = _find_term_by_surface(glossary, entity_source)
            if term is None:
                return False, (
                    f"decision {d_id!r}: entity_source {entity_source!r} not in glossary"
                )
            if choice == 'record_in_notes':
                _append_note(
                    term,
                    f"[conflict] {field}: current={current!r} "
                    f"observed_better={proposed!r} evidence={evidence!r}",
                )
            elif choice == 'accept_proposed':
                old_value = term.get(field)
                term[field] = proposed
                _append_note(
                    term,
                    f"[updated] {field}: was={old_value!r} now={proposed!r}",
                )
            # keep_current: no-op
            return True, None

        if kind == 'new_entity_existing_alias':
            promoted_variants = d.get('promoted_variants') or []
            if choice in ('keep_as_alias', 'skip'):
                return True, None
            m = re.match(r'^use_variant_(\d+)$', choice)
            idx = int(m.group(1))
            chosen = promoted_variants[idx]
            proposed_source = d.get('proposed_source')
            host_id = d.get('currently_alias_of')
            host = _find_term_by_id(glossary, host_id)
            if host is None:
                return False, (
                    f"decision {d_id!r}: host term id={host_id!r} not in glossary"
                )
            if proposed_source in host.get('aliases', []):
                host['aliases'] = [a for a in host['aliases'] if a != proposed_source]
            combined_chunks = sorted({
                cid for v in promoted_variants for cid in v.get('evidence_chunks', [])
            })[:EVIDENCE_REFS_CAP]
            glossary['terms'].append({
                'id': proposed_source,
                'source': proposed_source,
                'target': chosen['target_proposal'],
                'category': chosen.get('category', ''),
                'aliases': [],
                'gender': 'unknown',
                'confidence': _confidence_for_evidence_count(len(combined_chunks)),
                'frequency': 0,
                'evidence_refs': combined_chunks,
                'notes': f'promoted from alias of {host_id!r}',
            })
            return True, None

        if kind == 'existing_entity_conflict':
            entity_source = d.get('entity_source')
            term = _find_term_by_surface(glossary, entity_source)
            if term is None:
                return False, (
                    f"decision {d_id!r}: entity_source {entity_source!r} not in glossary"
                )
            if choice == 'keep_current':
                return True, None
            proposed_variants = d.get('proposed_variants') or []
            if choice == 'record_in_notes':
                for v in proposed_variants:
                    _append_note(
                        term,
                        f"[conflict] target={v['target_proposal']!r} "
                        f"category={v['category']!r} evidence={v['evidence']!r}",
                    )
                return True, None
            m = re.match(r'^use_variant_(\d+)$', choice)
            idx = int(m.group(1))
            chosen = proposed_variants[idx]
            old_target = term.get('target')
            old_category = term.get('category', '')
            term['target'] = chosen['target_proposal']
            term['category'] = chosen.get('category', '')
            _append_note(
                term,
                f"[updated] target {old_target!r} → {chosen['target_proposal']!r}, "
                f"category {old_category!r} → {chosen.get('category', '')!r}",
            )
            return True, None

        if kind == 'alias_or_new_entity':
            variant = d.get('variant')
            alias_candidates = d.get('alias_candidates') or []
            standalone_variants = d.get('standalone_variants') or []
            if choice == 'skip':
                return True, None
            m_alias = re.match(r'^use_alias_(\d+)$', choice)
            if m_alias:
                idx = int(m_alias.group(1))
                candidate_source = alias_candidates[idx]['candidate_source']
                term = _find_term_by_surface(glossary, candidate_source)
                if term is None:
                    return False, (
                        f"decision {d_id!r}: candidate source {candidate_source!r} "
                        f"not in glossary"
                    )
                if variant not in term.get('aliases', []):
                    term.setdefault('aliases', []).append(variant)
                return True, None
            # use_standalone_N
            m = re.match(r'^use_standalone_(\d+)$', choice)
            idx = int(m.group(1))
            chosen = standalone_variants[idx]
            combined_chunks = sorted({
                cid for v in standalone_variants for cid in v.get('evidence_chunks', [])
            })[:EVIDENCE_REFS_CAP]
            glossary['terms'].append({
                'id': variant,
                'source': variant,
                'target': chosen['target_proposal'],
                'category': chosen.get('category', ''),
                'aliases': [],
                'gender': 'unknown',
                'confidence': _confidence_for_evidence_count(len(combined_chunks)),
                'frequency': 0,
                'evidence_refs': combined_chunks,
                'notes': '',
            })
            return True, None

        if kind == 'conflicting_new_entity_proposals':
            variants = d.get('variants') or []
            if choice == 'skip':
                return True, None
            m = re.match(r'^use_variant_(\d+)$', choice)
            idx = int(m.group(1))
            chosen = variants[idx]
            combined_chunks = sorted({
                cid for v in variants for cid in v.get('evidence_chunks', [])
            })[:EVIDENCE_REFS_CAP]
            glossary['terms'].append({
                'id': d['source'],
                'source': d['source'],
                'target': chosen['target_proposal'],
                'category': chosen.get('category', ''),
                'aliases': [],
                'gender': 'unknown',
                'confidence': _confidence_for_evidence_count(len(combined_chunks)),
                'frequency': 0,
                'evidence_refs': combined_chunks,
                'notes': '',
            })
            return True, None

        # Should never reach here — pre-validation rejects unknown kinds.
        return True, None

    creators = [d for d in decisions if _is_creator(d)]
    attachers = [d for d in decisions if _is_alias_attacher(d)]
    others = [d for d in decisions if not _is_creator(d) and not _is_alias_attacher(d)]

    # Pass A: entity-creators (in input order).
    for d in creators:
        changed, err = _dispatch(d)
        if err is not None:
            errors.append(err)
        else:
            decisions_resolved += 1

    # Pass B: non-alias followers (in input order).
    for d in others:
        changed, err = _dispatch(d)
        if err is not None:
            errors.append(err)
        else:
            decisions_resolved += 1

    # Pass C: alias-attachers iterated to fixed point. A chain like
    # Taighi → Taig → Tai: when Taig is itself a yes_alias decision, the first
    # iteration may not be able to resolve Taighi → Taig because Taig isn't a
    # surface yet. The next iteration succeeds after Taig → Tai attaches.
    pending_attachers = list(attachers)
    while pending_attachers:
        next_pending = []
        progressed = False
        for d in pending_attachers:
            changed, err = _dispatch(d)
            if err is None:
                decisions_resolved += 1
                progressed = True
            else:
                next_pending.append((d, err))
        if not progressed:
            for _, err in next_pending:
                errors.append(err)
            break
        pending_attachers = [d for d, _ in next_pending]

    if errors:
        # Don't write the glossary, don't record any hashes. Orchestrator
        # retries with fixed payload; prepare-merge will surface the same
        # decisions again because no hashes were recorded.
        sys.stderr.write("error: apply-merge failed during decision dispatch; "
                         "aborting transaction (no glossary or hash mutations persisted):\n")
        for e in errors:
            sys.stderr.write(f"  - {e}\n")
        sys.exit(2)

    # Phase 4: record content hashes for all consumed metas. Reached only
    # when every decision succeeded.
    glossary.setdefault('applied_meta_hashes', {})
    for chunk_id, data in consumed_metas.items():
        glossary['applied_meta_hashes'][chunk_id] = meta_mod.meta_content_hash(data)

    # Validate before save (save_glossary will re-validate too — this surfaces
    # surface-uniqueness violations from a bad alias decision before we touch disk).
    try:
        glossary_mod.save_glossary(glossary_path, glossary)
    except ValueError as e:
        sys.stderr.write(f"error: post-merge glossary failed validation; aborting: {e}\n")
        sys.exit(2)

    summary = {
        'auto_applied': auto_applied,
        'decisions_resolved': decisions_resolved,
        'consumed_chunks': len(consumed_metas),
        'errors': errors,
    }
    print(json.dumps(summary, ensure_ascii=False, indent=2))


def cmd_status(temp_dir):
    """Read-only summary for the final verification report."""
    glossary_path = _glossary_path(temp_dir)
    if os.path.exists(glossary_path):
        glossary = glossary_mod.load_glossary(glossary_path)
        applied = glossary.get('applied_meta_hashes', {})
    else:
        applied = {}

    translated_paths = sorted(Path(temp_dir).glob(TRANSLATED_GLOB))
    # Strip the .meta.json variants that the glob also catches if any.
    translated_paths = [p for p in translated_paths if not p.name.endswith('.meta.json')]
    translated_chunk_ids = set()
    for p in translated_paths:
        m = re.match(r'^output_(chunk\d+)\.md$', p.name)
        if m:
            translated_chunk_ids.add(m.group(1))

    meta_paths = sorted(Path(temp_dir).glob(META_GLOB))
    found_chunk_ids = set()
    consumed_chunk_ids = set()
    malformed_chunk_ids = []
    unmerged_chunk_ids = []
    for path in meta_paths:
        try:
            chunk_id = meta_mod.chunk_id_from_meta_path(str(path))
        except ValueError:
            continue
        found_chunk_ids.add(chunk_id)
        data, err = _try_load_meta(str(path))
        if err is not None:
            malformed_chunk_ids.append(chunk_id)
            continue
        if applied.get(chunk_id) == meta_mod.meta_content_hash(data):
            consumed_chunk_ids.add(chunk_id)
        else:
            unmerged_chunk_ids.append(chunk_id)

    missing_meta = sorted(translated_chunk_ids - found_chunk_ids)

    out = {
        'translated_chunks': len(translated_chunk_ids),
        'meta_files_found': len(found_chunk_ids),
        'meta_files_consumed': len(consumed_chunk_ids),
        'unmerged_meta_files': len(unmerged_chunk_ids),
        'malformed_meta_files': len(malformed_chunk_ids),
        'missing_meta_chunk_ids': missing_meta[:10],
        'malformed_meta_chunk_ids': sorted(malformed_chunk_ids)[:10],
    }
    print(json.dumps(out, ensure_ascii=False, indent=2))


def main():
    parser = argparse.ArgumentParser(description="Merge sub-agent meta files into glossary")
    sub = parser.add_subparsers(dest='cmd', required=True)

    p_prep = sub.add_parser('prepare-merge', help="Scan unmerged metas; emit merge proposal")
    p_prep.add_argument('temp_dir')

    p_app = sub.add_parser('apply-merge', help="Apply merge decisions from stdin")
    p_app.add_argument('temp_dir')

    p_stat = sub.add_parser('status', help="Read-only observability snapshot")
    p_stat.add_argument('temp_dir')

    args = parser.parse_args()

    if args.cmd == 'prepare-merge':
        cmd_prepare_merge(args.temp_dir)
    elif args.cmd == 'apply-merge':
        cmd_apply_merge(args.temp_dir)
    elif args.cmd == 'status':
        cmd_status(args.temp_dir)


if __name__ == '__main__':
    main()


path: scripts/meta.py
content: #!/usr/bin/env python3
"""
meta.py - Per-chunk sub-agent observation file.

Each translation sub-agent emits an `output_chunk<NNNN>.meta.json` alongside
its translated chunk. The main agent reads these after each batch and merges
them into the canonical glossary (see merge_meta.py).

The schema (v1):

    {
      "schema_version": 1,
      "new_entities":         [{"source": "...", "target_proposal": "...",
                                "category": "...", "evidence": "..."}],
      "alias_hypotheses":     [{"variant": "...", "may_be_alias_of_source": "...",
                                "evidence": "..."}],
      "attribute_hypotheses": [{"entity_source": "...", "attribute": "...",
                                "value": "...", "confidence": "...",
                                "evidence": "..."}],
      "used_term_sources":    ["..."],
      "conflicts":            [{"entity_source": "...", "field": "...",
                                "injected": "...", "observed_better": "...",
                                "evidence": "..."}]
    }

`chunk_id` is intentionally NOT a field — chunk identity is derived from the
filename (`output_chunk<NNNN>.meta.json` → `chunk<NNNN>`). Letting the
sub-agent fill it would create a hallucination hole.
"""

import hashlib
import json
import os
import re
import tempfile


META_SCHEMA_VERSION = 1
EVIDENCE_MAX_LEN = 200

VALID_TOP_LEVEL_KEYS = frozenset({
    'schema_version',
    'new_entities',
    'alias_hypotheses',
    'attribute_hypotheses',
    'used_term_sources',
    'conflicts',
})

VALID_ATTRIBUTE_CONFIDENCES = ('low', 'medium', 'high')

_CHUNK_ID_RE = re.compile(r'^output_(chunk\d+)\.meta\.json$')


def chunk_id_from_meta_path(path):
    """Extract the chunk identifier from a meta filename.

    Filename is the authoritative source of chunk identity — never trust the
    payload to provide it.
    """
    basename = os.path.basename(path)
    m = _CHUNK_ID_RE.match(basename)
    if not m:
        raise ValueError(
            f"Meta filename {basename!r} does not match expected pattern "
            f"'output_chunk<NNNN>.meta.json'. Cannot derive chunk_id."
        )
    return m.group(1)


def _canonical_json(data):
    return json.dumps(data, sort_keys=True, separators=(',', ':'), ensure_ascii=False)


def meta_content_hash(data):
    """SHA-256 of canonical JSON. Used by merge_meta.py to detect whether a
    given meta has already been applied to the glossary."""
    return hashlib.sha256(_canonical_json(data).encode('utf-8')).hexdigest()


def _check_evidence(value, where, path):
    if not isinstance(value, str):
        raise ValueError(
            f"Meta at {path}: 'evidence' in {where} must be a string, "
            f"got {type(value).__name__}"
        )
    if len(value) > EVIDENCE_MAX_LEN:
        raise ValueError(
            f"Meta at {path}: 'evidence' in {where} is {len(value)} chars; "
            f"limit is {EVIDENCE_MAX_LEN}. Quote a shorter excerpt."
        )


def _require_str(value, field, where, path):
    if not isinstance(value, str):
        raise ValueError(
            f"Meta at {path}: {where} field {field!r} must be a string, "
            f"got {type(value).__name__}"
        )


def _validate_array(data, key, path):
    val = data.get(key, [])
    if not isinstance(val, list):
        raise ValueError(
            f"Meta at {path}: {key!r} must be a list, got {type(val).__name__}"
        )
    return val


def validate_meta(data, path='<meta>'):
    """Strict v1 validation. Raises ValueError with actionable messages."""
    if not isinstance(data, dict):
        raise ValueError(
            f"Meta at {path} must be a JSON object, got {type(data).__name__}"
        )

    schema_version = data.get('schema_version')
    if schema_version != META_SCHEMA_VERSION:
        raise ValueError(
            f"Meta at {path}: schema_version mismatch — expected "
            f"{META_SCHEMA_VERSION}, got {schema_version!r}."
        )

    extras = set(data.keys()) - VALID_TOP_LEVEL_KEYS
    if extras:
        # 'chunk_id' is the most likely offender; call it out specifically so
        # the fix is obvious.
        if 'chunk_id' in extras:
            raise ValueError(
                f"Meta at {path}: 'chunk_id' field is not allowed in the meta "
                f"payload — chunk identity is derived from the filename. Remove it."
            )
        raise ValueError(
            f"Meta at {path}: unknown top-level key(s) {sorted(extras)!r}. "
            f"Allowed keys: {sorted(VALID_TOP_LEVEL_KEYS)!r}."
        )

    for entity in _validate_array(data, 'new_entities', path):
        if not isinstance(entity, dict):
            raise ValueError(
                f"Meta at {path}: each new_entities entry must be an object, "
                f"got {type(entity).__name__}"
            )
        for required in ('source', 'target_proposal', 'evidence'):
            if required not in entity:
                raise ValueError(
                    f"Meta at {path}: new_entities entry missing required field "
                    f"{required!r}"
                )
        _require_str(entity['source'], 'source', 'new_entities', path)
        _require_str(entity['target_proposal'], 'target_proposal', 'new_entities', path)
        if 'category' in entity:
            _require_str(entity['category'], 'category', 'new_entities', path)
        _check_evidence(entity['evidence'], 'new_entities', path)

    for alias in _validate_array(data, 'alias_hypotheses', path):
        if not isinstance(alias, dict):
            raise ValueError(
                f"Meta at {path}: each alias_hypotheses entry must be an object"
            )
        for required in ('variant', 'may_be_alias_of_source', 'evidence'):
            if required not in alias:
                raise ValueError(
                    f"Meta at {path}: alias_hypotheses entry missing field {required!r}"
                )
        _require_str(alias['variant'], 'variant', 'alias_hypotheses', path)
        _require_str(alias['may_be_alias_of_source'], 'may_be_alias_of_source',
                     'alias_hypotheses', path)
        _check_evidence(alias['evidence'], 'alias_hypotheses', path)

    for attr in _validate_array(data, 'attribute_hypotheses', path):
        if not isinstance(attr, dict):
            raise ValueError(
                f"Meta at {path}: each attribute_hypotheses entry must be an object"
            )
        for required in ('entity_source', 'attribute', 'value', 'confidence', 'evidence'):
            if required not in attr:
                raise ValueError(
                    f"Meta at {path}: attribute_hypotheses entry missing field {required!r}"
                )
        _require_str(attr['entity_source'], 'entity_source', 'attribute_hypotheses', path)
        _require_str(attr['attribute'], 'attribute', 'attribute_hypotheses', path)
        _require_str(attr['value'], 'value', 'attribute_hypotheses', path)
        if attr['confidence'] not in VALID_ATTRIBUTE_CONFIDENCES:
            raise ValueError(
                f"Meta at {path}: attribute_hypotheses 'confidence' must be one of "
                f"{list(VALID_ATTRIBUTE_CONFIDENCES)}, got {attr['confidence']!r}"
            )
        _check_evidence(attr['evidence'], 'attribute_hypotheses', path)

    for s_idx, src in enumerate(_validate_array(data, 'used_term_sources', path)):
        if not isinstance(src, str):
            raise ValueError(
                f"Meta at {path}: used_term_sources #{s_idx} must be a string, "
                f"got {type(src).__name__}"
            )

    for conflict in _validate_array(data, 'conflicts', path):
        if not isinstance(conflict, dict):
            raise ValueError(
                f"Meta at {path}: each conflicts entry must be an object"
            )
        for required in ('entity_source', 'field', 'injected', 'observed_better', 'evidence'):
            if required not in conflict:
                raise ValueError(
                    f"Meta at {path}: conflicts entry missing field {required!r}"
                )
        _require_str(conflict['entity_source'], 'entity_source', 'conflicts', path)
        _require_str(conflict['field'], 'field', 'conflicts', path)
        _require_str(conflict['injected'], 'injected', 'conflicts', path)
        _require_str(conflict['observed_better'], 'observed_better', 'conflicts', path)
        _check_evidence(conflict['evidence'], 'conflicts', path)


def load_meta(path):
    """Load and strictly validate a meta file. Raises ValueError on any
    problem. Callers that want non-blocking behavior should wrap this in a
    quarantine wrapper (see merge_meta.py)."""
    if not os.path.exists(path):
        raise FileNotFoundError(f"Meta not found: {path}")
    with open(path, 'r', encoding='utf-8') as f:
        try:
            data = json.load(f)
        except json.JSONDecodeError as e:
            raise ValueError(f"Meta at {path} is not valid JSON: {e}") from e
    validate_meta(data, path)
    return data


def save_meta(path, data):
    """Atomically write a meta file. Validates before writing."""
    validate_meta(data, path)

    dirname = os.path.dirname(os.path.abspath(path)) or '.'
    fd, tmp_path = tempfile.mkstemp(dir=dirname, prefix='.meta-', suffix='.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            json.dump(data, f, ensure_ascii=False, indent=2, sort_keys=True)
            f.write('\n')
        os.replace(tmp_path, path)
    except Exception:
        if os.path.exists(tmp_path):
            try:
                os.unlink(tmp_path)
            except OSError:
                pass
        raise


path: scripts/template.html
content: <!DOCTYPE html>
<html lang="$lang$">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>$title$</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }

        html {
            font-size: 16px;
            scroll-behavior: smooth;
        }

        body {
            font-family: $body_font$;
            line-height: 1.8;
            color: #333333;
            background-color: #fefefe;
            max-width: 700px;
            margin: 0 auto;
            padding: 3rem 2rem;
            font-size: 1.1rem;
            text-align: justify;
        }

        /* 浮动目录按钮 */
        .toc-toggle {
            position: fixed;
            left: 20px;
            top: 50%;
            transform: translateY(-50%);
            z-index: 1000;
            background: rgba(96, 165, 250, 0.5);
            color: white;
            border: none;
            border-radius: 8px;
            width: 48px;
            height: 48px;
            font-size: 20px;
            cursor: pointer;
            box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2);
            transition: all 0.3s ease;
        }

        .toc-toggle:hover {
            background: rgba(96, 165, 250, 0.8);
            transform: translateY(-50%) scale(1.1);
        }

        /* 浮动目录面板 */
        .toc-sidebar {
            position: fixed;
            left: -320px;
            top: 0;
            width: 300px;
            height: 100vh;
            background: #ffffff;
            border-right: 1px solid #e2e8f0;
            box-shadow: 4px 0 12px rgba(0, 0, 0, 0.1);
            z-index: 999;
            transition: left 0.3s ease;
            overflow-y: auto;
            padding: 2rem 1rem;
        }

        .toc-sidebar.active {
            left: 0;
        }

        .toc-sidebar h3 {
            font-size: 1.2rem;
            color: #1a202c;
            margin-bottom: 1rem;
            text-align: center;
            border-bottom: 2px solid #3182ce;
            padding-bottom: 0.5rem;
        }

        /* 目录区域 - 用于插入目录内容 */
        .toc-content {
            /* 目录内容将插入到这里 */
        }

        /* 遮罩层 */
        .toc-overlay {
            position: fixed;
            top: 0;
            left: 0;
            width: 100%;
            height: 100%;
            background: rgba(0, 0, 0, 0.3);
            z-index: 998;
            opacity: 0;
            visibility: hidden;
            transition: all 0.3s ease;
        }

        .toc-overlay.active {
            opacity: 1;
            visibility: visible;
        }

        /* 简洁的目录样式 */
        .simple-toc ul {
            list-style: none;
            padding-left: 0;
            margin: 0;
        }

        .simple-toc ul ul {
            padding-left: 1rem;
            margin-top: 0.25rem;
        }

        .simple-toc li {
            margin-bottom: 0.25rem;
        }

        .simple-toc a {
            display: block;
            padding: 0.5rem 0.75rem;
            color: #4a5568;
            text-decoration: none;
            border-radius: 4px;
            transition: all 0.2s ease;
            font-size: 0.9rem;
            line-height: 1.4;
        }

        .simple-toc a:hover {
            background-color: #edf2f7;
            color: #3182ce;
            transform: translateX(4px);
        }

        /* 标题样式 */
        h1, h2, h3, h4, h5, h6 {
            font-family: $body_font$;
            color: #2d3748;
            margin-top: 2rem;
            margin-bottom: 1.2rem;
            line-height: 1.4;
            font-weight: normal;
            text-align: center;
        }

        h1 {
            font-size: 1.8rem;
            border-bottom: 2px solid #718096;
            padding-bottom: 0.8rem;
            margin-top: 0;
            margin-bottom: 2.5rem;
        }

        h2 {
            font-size: 1.5rem;
            border-bottom: 1px solid #cbd5e0;
            padding-bottom: 0.6rem;
            margin-top: 2.5rem;
        }

        h3 {
            font-size: 1.3rem;
            color: #4a5568;
        }

        h4 {
            font-size: 1.2rem;
            color: #4a5568;
        }

        h5 {
            font-size: 1.1rem;
            color: #4a5568;
        }

        h6 {
            font-size: 1rem;
            color: #718096;
            font-weight: normal;
        }

        /* 段落样式 */
        p {
            margin-bottom: 1.5rem;
            text-align: justify;
            word-break: break-word;
            text-indent: 2em;
            font-family: $body_font$;
        }

        /* 链接样式 */
        a {
            color: #3182ce;
            text-decoration: none;
            transition: color 0.2s ease;
        }

        a:hover {
            color: #2c5282;
            text-decoration: underline;
        }

        /* 图片样式 - 居中显示 */
        img {
            max-width: 100%;
            height: auto;
            display: block;
            margin: 2rem auto;
            border-radius: 8px;
            box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
        }

        /* 列表样式 */
        ul, ol {
            margin: 1.25rem 0;
            padding-left: 2rem;
        }

        li {
            margin-bottom: 0.5rem;
        }

        ul ul, ol ol, ul ol, ol ul {
            margin: 0.5rem 0;
        }

        /* 代码样式 */
        code {
            font-family: 'Courier New', 'Monaco', 'Consolas', 'Ubuntu Mono', monospace;
            background-color: #f8f9fa;
            color: #d63384;
            padding: 0.2rem 0.4rem;
            border-radius: 3px;
            font-size: 0.9rem;
        }

        pre {
            background-color: #f8f9fa;
            border: 1px solid #e9ecef;
            border-radius: 6px;
            padding: 1.5rem;
            margin: 2rem 0;
            overflow-x: auto;
            font-family: 'Courier New', 'Monaco', 'Consolas', 'Ubuntu Mono', monospace;
            font-size: 0.9rem;
            line-height: 1.6;
        }

        pre code {
            background: none;
            color: inherit;
            padding: 0;
            border-radius: 0;
            font-size: inherit;
        }

        /* 引用样式 */
        blockquote {
            border-left: 4px solid #a0aec0;
            margin: 2rem 0;
            padding: 1.2rem 1.8rem;
            background-color: #f7fafc;
            color: #4a5568;
            font-style: italic;
            border-radius: 0 6px 6px 0;
            font-family: $body_font$;
        }

        blockquote p:last-child {
            margin-bottom: 0;
        }

        /* 表格样式 */
        table {
            width: 100%;
            border-collapse: collapse;
            margin: 1.5rem 0;
            background-color: #ffffff;
            border-radius: 6px;
            overflow: hidden;
            box-shadow: 0 1px 3px rgba(0, 0, 0, 0.1);
        }

        th, td {
            padding: 0.75rem 1rem;
            text-align: left;
            border-bottom: 1px solid #e2e8f0;
        }

        th {
            background-color: #f7fafc;
            font-weight: 600;
            color: #2d3748;
        }

        tr:hover {
            background-color: #f7fafc;
        }

        /* 分隔线 */
        hr {
            border: none;
            border-top: 2px solid #e2e8f0;
            margin: 3rem 0;
        }

        /* 页面分隔器 */
        .page-separator {
            border-top: 2px solid #e2e8f0;
            margin: 3rem 0;
            text-align: center;
            position: relative;
        }

        .page-separator::after {
            content: "• • •";
            background-color: #ffffff;
            color: #a0aec0;
            padding: 0 1rem;
            position: relative;
            top: -0.75rem;
        }

        /* 强调样式 */
        strong, b {
            font-weight: 700;
            color: #1a202c;
        }

        em, i {
            font-style: italic;
            color: #4a5568;
        }

        /* 目标标题高亮 */
        h1:target, h2:target, h3:target, h4:target, h5:target, h6:target {
            background-color: #fff3cd;
            padding: 1rem;
            border-radius: 6px;
            margin: 1.5rem 0;
            border-left: 4px solid #ffc107;
            animation: highlight 2s ease-in-out;
        }

        @keyframes highlight {
            0% {
                background-color: #fff3cd;
                transform: scale(1.02);
            }
            100% {
                background-color: transparent;
                transform: scale(1);
            }
        }

        /* 响应式设计 */
        @media (max-width: 768px) {
            body {
                padding: 1.5rem;
                font-size: 1rem;
                max-width: 100%;
            }

            .toc-toggle {
                left: 10px;
                width: 40px;
                height: 40px;
                font-size: 16px;
            }

            .toc-sidebar {
                width: 280px;
                left: -300px;
                padding: 1rem;
            }

            h1 {
                font-size: 1.6rem;
            }

            h2 {
                font-size: 1.4rem;
            }

            h3 {
                font-size: 1.2rem;
            }

            h4 {
                font-size: 1.1rem;
            }

            table {
                font-size: 0.9rem;
            }

            th, td {
                padding: 0.6rem 0.8rem;
            }

            pre {
                padding: 1.2rem;
                font-size: 0.85rem;
            }

            blockquote {
                padding: 1rem 1.2rem;
                margin: 1.5rem 0;
            }
        }

        @media (max-width: 480px) {
            body {
                padding: 1.2rem;
                font-size: 0.95rem;
            }

            .toc-sidebar {
                width: 260px;
                left: -280px;
            }

            h1, h2, h3, h4, h5, h6 {
                margin-top: 1.5rem;
            }

            ul, ol {
                padding-left: 1.5rem;
            }
            
            p {
                text-indent: 1.5em;
            }
        }

        /* 打印样式 */
        @media print {
            .toc-toggle,
            .toc-sidebar,
            .toc-overlay {
                display: none !important;
            }

            body {
                max-width: none;
                padding: 0;
                font-size: 12pt;
                line-height: 1.5;
                color: #000;
            }

            h1, h2, h3, h4, h5, h6 {
                page-break-after: avoid;
            }

            img {
                max-width: 100% !important;
                page-break-inside: avoid;
            }

            blockquote, pre {
                page-break-inside: avoid;
            }

            a {
                color: #000;
                text-decoration: underline;
            }
        }
    </style>
</head>
<body>
    <!-- 浮动目录按钮 -->
    <button class="toc-toggle" onclick="toggleToc()" title="打开/关闭目录">
        ☰
    </button>

    <!-- 浮动目录面板 -->
    <div class="toc-sidebar" id="tocSidebar">
        <h3>$toc_label$</h3>
        <div class="toc-content simple-toc">
            <!-- 目录内容将插入到这里 -->
        </div>
    </div>

    <!-- 遮罩层 -->
    <div class="toc-overlay" id="tocOverlay" onclick="closeToc()"></div>

    <!-- 主要内容 -->
    $body$

    <script>
        function toggleToc() {
            const sidebar = document.getElementById('tocSidebar');
            const overlay = document.getElementById('tocOverlay');
            
            if (sidebar.classList.contains('active')) {
                closeToc();
            } else {
                openToc();
            }
        }

        function openToc() {
            const sidebar = document.getElementById('tocSidebar');
            const overlay = document.getElementById('tocOverlay');
            
            sidebar.classList.add('active');
            overlay.classList.add('active');
            document.body.style.overflow = 'hidden';
        }

        function closeToc() {
            const sidebar = document.getElementById('tocSidebar');
            const overlay = document.getElementById('tocOverlay');
            
            sidebar.classList.remove('active');
            overlay.classList.remove('active');
            document.body.style.overflow = 'auto';
        }

        // ESC键关闭目录
        document.addEventListener('keydown', function(e) {
            if (e.key === 'Escape') {
                closeToc();
            }
        });

        // 点击目录链接后自动关闭目录（移动端）
        document.addEventListener('click', function(e) {
            if (e.target.matches('.simple-toc a') && window.innerWidth <= 768) {
                setTimeout(closeToc, 300);
            }
        });
    </script>
</body>
</html>

path: scripts/template_ebook.html
content: <!DOCTYPE html>
<html lang="$lang$">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>$title$</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }

        html {
            font-size: 16px;
            scroll-behavior: smooth;
        }

        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';
            line-height: 1.7;
            color: #2c3e50;
            background-color: #ffffff;
            max-width: 800px;
            margin: 0 auto;
            padding: 2rem 1.5rem;
            font-size: 1rem;
        }

        /* 标题样式 */
        h1, h2, h3, h4, h5, h6 {
            font-weight: 600;
            margin-top: 2.5rem;
            margin-bottom: 1rem;
            color: #1a202c;
            line-height: 1.3;
        }

        h1 {
            font-size: 2.25rem;
            border-bottom: 3px solid #3182ce;
            padding-bottom: 0.75rem;
            margin-top: 0;
            margin-bottom: 2rem;
        }

        h2 {
            font-size: 1.875rem;
            border-bottom: 1px solid #e2e8f0;
            padding-bottom: 0.5rem;
            margin-top: 3rem;
        }

        h3 {
            font-size: 1.5rem;
            color: #2d3748;
        }

        h4 {
            font-size: 1.25rem;
            color: #4a5568;
        }

        h5 {
            font-size: 1.125rem;
            color: #4a5568;
        }

        h6 {
            font-size: 1rem;
            color: #718096;
            font-weight: 500;
        }

        /* 段落样式 */
        p {
            margin-bottom: 1.25rem;
            text-align: justify;
            word-break: break-word;
        }

        /* 链接样式 */
        a {
            color: #3182ce;
            text-decoration: none;
            transition: color 0.2s ease;
        }

        a:hover {
            color: #2c5282;
            text-decoration: underline;
        }

        /* 图片样式 - 居中显示 */
        img {
            max-width: 100%;
            height: auto;
            display: block;
            margin: 2rem auto;
            border-radius: 8px;
            box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
        }

        /* 列表样式 */
        ul, ol {
            margin: 1.25rem 0;
            padding-left: 2rem;
        }

        li {
            margin-bottom: 0.5rem;
        }

        ul ul, ol ol, ul ol, ol ul {
            margin: 0.5rem 0;
        }

        /* 代码样式 */
        code {
            font-family: 'Fira Code', 'Monaco', 'Consolas', 'Ubuntu Mono', monospace;
            background-color: #f7fafc;
            color: #e53e3e;
            padding: 0.125rem 0.375rem;
            border-radius: 3px;
            font-size: 0.875rem;
        }

        pre {
            background-color: #f7fafc;
            border: 1px solid #e2e8f0;
            border-radius: 6px;
            padding: 1.25rem;
            margin: 1.5rem 0;
            overflow-x: auto;
            font-family: 'Fira Code', 'Monaco', 'Consolas', 'Ubuntu Mono', monospace;
            font-size: 0.875rem;
            line-height: 1.5;
        }

        pre code {
            background: none;
            color: inherit;
            padding: 0;
            border-radius: 0;
            font-size: inherit;
        }

        /* 引用样式 */
        blockquote {
            border-left: 4px solid #3182ce;
            margin: 1.5rem 0;
            padding: 1rem 1.5rem;
            background-color: #f7fafc;
            color: #4a5568;
            font-style: italic;
            border-radius: 0 6px 6px 0;
        }

        blockquote p:last-child {
            margin-bottom: 0;
        }

        /* 表格样式 */
        table {
            width: 100%;
            border-collapse: collapse;
            margin: 1.5rem 0;
            background-color: #ffffff;
            border-radius: 6px;
            overflow: hidden;
            box-shadow: 0 1px 3px rgba(0, 0, 0, 0.1);
        }

        th, td {
            padding: 0.75rem 1rem;
            text-align: left;
            border-bottom: 1px solid #e2e8f0;
        }

        th {
            background-color: #f7fafc;
            font-weight: 600;
            color: #2d3748;
        }

        tr:hover {
            background-color: #f7fafc;
        }

        /* 分隔线 */
        hr {
            border: none;
            border-top: 2px solid #e2e8f0;
            margin: 3rem 0;
        }

        /* 页面分隔器 */
        .page-separator {
            border-top: 2px solid #e2e8f0;
            margin: 3rem 0;
            text-align: center;
            position: relative;
        }

        .page-separator::after {
            content: "• • •";
            background-color: #ffffff;
            color: #a0aec0;
            padding: 0 1rem;
            position: relative;
            top: -0.75rem;
        }

        /* 强调样式 */
        strong, b {
            font-weight: 700;
            color: #1a202c;
        }

        em, i {
            font-style: italic;
            color: #4a5568;
        }

        /* 响应式设计 */
        @media (max-width: 768px) {
            body {
                padding: 1rem;
                font-size: 0.9rem;
            }

            h1 {
                font-size: 1.875rem;
            }

            h2 {
                font-size: 1.5rem;
            }

            h3 {
                font-size: 1.25rem;
            }

            h4 {
                font-size: 1.125rem;
            }

            table {
                font-size: 0.875rem;
            }

            th, td {
                padding: 0.5rem 0.75rem;
            }

            pre {
                padding: 1rem;
                font-size: 0.8rem;
            }

            blockquote {
                padding: 0.75rem 1rem;
                margin: 1rem 0;
            }
        }

        @media (max-width: 480px) {
            body {
                padding: 0.75rem;
            }

            h1, h2, h3, h4, h5, h6 {
                margin-top: 1.5rem;
            }

            ul, ol {
                padding-left: 1.5rem;
            }
        }

        /* 打印样式 */
        @media print {
            body {
                max-width: none;
                padding: 0;
                font-size: 12pt;
                line-height: 1.5;
                color: #000;
            }

            h1, h2, h3, h4, h5, h6 {
                page-break-after: avoid;
            }

            img {
                max-width: 100% !important;
                page-break-inside: avoid;
            }

            blockquote, pre {
                page-break-inside: avoid;
            }

            a {
                color: #000;
                text-decoration: underline;
            }
        }
    </style>
</head>
<body>
    $body$
</body>
</html>

path: tests/baselines/README.md
content: # Baselines

Checked-in full-pipeline test inputs live under `tests/baselines/<book-id>/`.

Each baseline directory should contain:

- the original input book file (`.epub`, `.pdf`, or `.docx`)
- a short `SOURCE.md` describing the source and why the baseline exists

Generated outputs do not belong here. Put them under `tests/.artifacts/`.

## Tiers

| Tier | Baseline | Chunks (target_size=6000) | Inline images | When to run |
|---|---|---|---|---|
| Smoke | `sleepy-hollow` | ~21 | 0 (cover only) | Default CI / quick iteration |
| Gold | `standard-alice` | ~38 | ~45 | Release-gate / image-path coverage |
| Alias-stress (opt-in) | `diligent-dick` | ~57 | 5 | When changing `merge_meta` or `glossary` |

The smoke baseline is the cheapest way to exercise convert → chunk → glossary → merge → build. It deliberately gives up inline-image coverage; use the gold baseline whenever image handling could be affected. The alias-stress baseline is intentionally larger than the gold one because its value lies in `Dick / Richard / Stuart` co-occurring across ~9 consecutive chunks — a contested-variant cross-batch scenario the smaller baselines cannot reproduce.

## Expected outcome convention

Each `SOURCE.md` documents an `## Expected outcome` section with three tiers:

- **Measured** — a value directly observable in the current `tests/.artifacts/<baseline>_temp/`. Pass/fail today against that artifact.
- **Expected target (unverified)** — structurally deterministic but no run has confirmed it yet. Forward-looking spec, not pass/fail until a run produces evidence.
- **Drift indicator** — LLM-dependent content (e.g. specific translation strings, gender attributions). Recorded for spot-checking; never pass/fail.

A `Measured` assertion that no longer holds against the current artifact is a regression. An `Expected target` that fails on first run means either the target was wrong (update `SOURCE.md`) or the pipeline regressed (fix code) — judgement call. Promote `Expected target` items to `Measured` once a run validates them.

For values that depend on first-run measurement (e.g. exact `manifest.chunk_count`, image counts not pinned by the source file), write the `Expected target` as approximate or as "first run records the exact value; subsequent runs assert equality against the recorded value". Do not pin a precise number until a run produces it. Otherwise an estimation correction on first run will be misread as a pipeline failure.

Field-name vocabulary follows the existing test suite (`target`, `confidence`, `evidence_refs`, `aliases`, `auto_apply`, `consumed_chunk_ids` — see `tests/test_merge_meta.py` and `tests/test_glossary.py`). Schema references: `scripts/glossary.py` (entity shape), `scripts/manifest.py` (manifest), `scripts/meta.py` (per-chunk meta), `scripts/merge_meta.py` (merge-time confidence promotion rule at `:76-91`).

## When adding a new baseline

A baseline that has never been run is not a baseline — it's just a checked-in book. Before merging:

1. Run the full pipeline on it once (`convert.py` → translate via the skill → `merge_and_build.py`).
2. Capture the real chunk count, image count, and any cross-chunk entity / alias evidence in the new `SOURCE.md`. Replace any estimates you used while picking the book.
3. Note coverage gaps explicitly (e.g. "cover only, no inline images" for sleepy-hollow). Future readers should not have to re-derive what this baseline does and does not exercise.
4. Add an `## Expected outcome` section using the three-tier format above. Anything not present in your initial run goes under **Expected target (unverified)** with that label, not **Measured**. Use approximate values (or "first run records exact value") for unmeasured numerical assertions like `chunk_count`.

Numbers in `SOURCE.md` come from a measured run, not from Project Gutenberg metadata or word-count rules of thumb.


path: tests/baselines/diligent-dick/SOURCE.md
content: # diligent-dick

Opt-in alias-stress baseline — exercises the `merge_meta` contested-variant and alias-chain paths under realistic cross-chunk pressure. **Not part of default CI.** Run when changing `scripts/merge_meta.py` or `scripts/glossary.py`.

- File: `diligent-dick.epub`
- Source: Project Gutenberg #71493 *Diligent Dick; or, The Young Farmer* (`epub3.images`)
- Why this baseline: the protagonist appears as `Dick` / `Richard` / `Stuart` (with `Mr. Stuart` ambiguating father vs. son) and all three variants co-occur across ~9 consecutive chunks (chunks 30/33/36/38/40/42/48/50/53/54 in a fresh convert). Single-chunk Brom-style aliases — which Sleepy Hollow only provides intra-chunk — appear here cross-batch, which is exactly what the recent `merge_meta` "Surface ALL competing proposals" / "Surface ALL alias candidates" fixes are meant to protect.
- Cost note: ~57 chunks (≈1.5× `standard-alice`), and ~30 of those are tiny heading- or image-only chunks created by the chunker's heading-boundary preference on this book's many short Sunday-school chapters. Real story content is ~27 chunks. Treat the high chunk count as expected, not a bug.
- Image coverage: 5 inline images (`image001`–`image005`); modest, do not rely on this baseline for image-path coverage.

Run from `tests/.artifacts/` so generated files stay out of the repo root:

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/diligent-dick/diligent-dick.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir diligent-dick_temp --title "勤奋的迪克（Alias-Stress Baseline）"
```

## Expected outcome

See `tests/baselines/README.md` for the three-tier convention.

*Forward-looking targets, not yet validated. The first full pipeline run after this section was added is the validation event.*

### Measured

None — no `tests/.artifacts/diligent-dick_temp/` exists yet.

### Expected target (unverified — needs first full pipeline run)

- `manifest.chunk_count` is approximately 57 (this `SOURCE.md` says `~57 chunks (≈1.5× standard-alice)` above, with the heading-boundary explanation for why so many are tiny). First run records the exact value; subsequent runs assert equality against the recorded value. Variance of a few chunks is estimation drift, not regression.
- All `chunk*.md` and matching `output_chunk*.md` exist; count matches `manifest.chunk_count`
- 5 inline images preserved (`image001`–`image005`, source-file property — exact)
- All 4 final formats generated: `book.html`, `book.docx`, `book.epub`, `book.pdf`
- **Primary alias assertion**: `Dick` and `Richard` resolve to a *single* glossary entity, both forms present in `aliases`. (Unambiguous nickname relationship.)
- **Critical ambiguity flag** — `Stuart` / `Mr. Stuart` (father vs. son): the disambiguation must surface in `merge_meta prepare-merge` output as **any** decision item (any `kind` — `alias`, `alias_or_new_entity`, `conflicting_new_entity_proposals`, `new_entity_existing_alias`, or `existing_entity_conflict` — whichever fires depends on run state). The non-silent criterion is "main agent gets to judge", regardless of decision kind. It must **NOT** be silently merged into `Dick`; silent merge = loss of evidence = regression.
  - Do **NOT** assert it must appear in the per-chunk meta `conflicts` array; that array is specifically for "injected canonical field vs observed-better value" comparisons (see `scripts/meta.py:21-23`), not general entity ambiguity.
- Once the protagonist has been seen in the first two co-occurrence chunks (chunks 30 and 33 per the documented evidence above — *not* "batch 2", which lands on chunks 9–16 at default concurrency and contains no protagonist), the entity must reach `confidence != "low"`. The recent `merge_meta` "Surface ALL competing proposals" / "Surface ALL alias candidates" fixes (commits `73b5f82`, `bbee9c9`) exist to drive convergence here without waiting for many additional chunks.

### Drift indicator (record current values, do not pass/fail)

- `Dick`/`Richard` final translation: a single Chinese form (not `迪克` and `理查德` mixed in the same output)
- `Mr. Stuart`'s rendering matches whichever resolution wins (consistency only, no specific target value)


path: tests/baselines/sleepy-hollow/SOURCE.md
content: # sleepy-hollow

Smoke baseline — cheapest full-pipeline input that still exercises convert, chunking, glossary, merge, and final format generation.

- File: `sleepy-hollow.epub`
- Source: Project Gutenberg #41 *The Legend of Sleepy Hollow* (`epub3.images`)
- Why this baseline: at `target_size=6000` the input splits into ~21 chunks (about 13 story chunks + 8 PG boilerplate chunks), roughly 55% of the cost of `standard-alice`. The `Headless Horseman / Galloping Hessian / spectre / goblin` alias chain spans ~9 chunks, so cross-chunk entity tracking still gets exercised.
- Coverage gap: only 1 image (the cover) — does **not** exercise inline-image preservation. Use `standard-alice` for image-path coverage.

Run from `tests/.artifacts/` so generated files stay out of the repo root:

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/sleepy-hollow/sleepy-hollow.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir sleepy-hollow_temp --title "睡谷传奇（Smoke Baseline）"
```

## Expected outcome

See `tests/baselines/README.md` for the three-tier convention.

*Forward-looking targets, not yet validated. The first full pipeline run after this section was added is the validation event — promote items to **Measured** as a run confirms them, or correct the targets if the run reveals the spec was wrong.*

### Measured

None — no `tests/.artifacts/sleepy-hollow_temp/` exists yet.

### Expected target (unverified — needs first full pipeline run)

- `manifest.chunk_count` is approximately 21 (this `SOURCE.md` says `~21` above). First run records the exact value; subsequent runs assert equality against the recorded value, not against `21`. A first-run result of 20 or 22 is an estimation correction, not a regression.
- All `chunk*.md` and matching `output_chunk*.md` exist; count matches `manifest.chunk_count`
- 1 image (cover) preserved end-to-end into the final EPUB
- All 4 final formats generated: `book.html`, `book.docx`, `book.epub`, `book.pdf`
- Glossary entries for `Ichabod Crane`, `Headless Horseman`, `Brom Bones`, `Katrina Van Tassel` reach `confidence == "high"` (≥3 evidence_refs each, per `scripts/merge_meta.py:76-91`)
- **Primary assertion** (the reason this baseline exists beyond smoke coverage): `Headless Horseman` and `Galloping Hessian` resolve to a *single* glossary entity, with both surface forms present in `aliases`. If they split into two entities, the cross-chunk alias coverage failed.

### Drift indicator (record current values, do not pass/fail)

- `Ichabod Crane.target`: a single Chinese form across all chunks (consistency only, not a specific value)
- `Brom Bones`'s alias chain (`Abraham`, `Brom Van Brunt`) collapses into one entity


path: tests/baselines/standard-alice/SOURCE.md
content: # standard-alice

Full-pipeline baseline input for this repository.

- File: `standard-alice.epub`
- Source: Standard Ebooks edition of *Alice's Adventures in Wonderland*
- Why this baseline: stable EPUB structure, chaptered prose, recurring entities, and many illustrations, which makes it useful for exercising convert, chunking, translation, merge, and final format generation.

Run the repository baseline test from `tests/.artifacts/` so generated files stay out of the repo root:

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "爱丽丝梦游仙境（Baseline Test）"
```

## Expected outcome

See `tests/baselines/README.md` for the three-tier convention (Measured / Expected target / Drift indicator).

### Measured (from current `tests/.artifacts/standard-alice_temp/`)

Full pipeline run completed at `target_size=6000`, default concurrency 8, 4 batches.

`manifest.json`:
- `chunk_count == 38`
- `source_hash == "6b1ea8ca29311ff8c43740b5a87d3fcfe0b906eb69bdcc74585d81a280f40c09"`
- `chunks[]` has 38 entries, sequential `chunk0001`–`chunk0038`, each with a non-empty `source_hash` and matching `output_file`

Files on disk:
- 38 `chunk0001.md`–`chunk0038.md` source chunks exist
- 38 `output_chunk0001.md`–`output_chunk0038.md` translated chunks exist
- 38 `output_chunk*.meta.json` per-chunk meta files exist
- 45 image files in `images/`

`merge_meta status`:
- `translated_chunks: 38`, `meta_files_found: 38`, `meta_files_consumed: 38`
- `unmerged_meta_files: 0`, `malformed_meta_files: 0`

`glossary.json` (version 2):
- `applied_meta_hashes` has 38 entries (all chunks merged)
- 95 total terms; 22 reach `confidence: "high"` (≥3 `evidence_refs`)
- Major character entities (sample of high-confidence with stable aliases):

| `source` | `target` | `confidence` | `evidence_refs` | `aliases` |
|---|---|---|---|---|
| Alice | 爱丽丝 | high | 5 | [] |
| White Rabbit | 白兔 | high | 5 | ["the Rabbit", "W. Rabbit"] |
| Cheshire Cat | 柴郡猫 | high | 5 | ["Cheshire Puss", "Cat"] |
| Queen of Hearts | 红心皇后 | high | 5 | ["Queen", "the Queen"] |
| King of Hearts | 红心国王 | high | 5 | ["the King"] |
| Bill | 比尔 | high | 5 | ["Lizard"] |
| Caterpillar | 毛毛虫 | high | 5 | [] |
| Hatter | 帽匠 | high | 5 | [] |
| March Hare | 三月兔 | high | 5 | [] |
| Dormouse | 睡鼠 | high | 5 | [] |
| Mock Turtle | 素甲鱼 | high | 5 | [] |
| Gryphon | 鹰头狮 | high | 5 | [] |

Note on the auto-applied `King` (`国王`) entity, separate from `King of Hearts` (`红心国王`): `merge_meta` saw `King` as a new entity in chunks 23/24 with no canonical conflict, so it auto-applied. This creates a (semantically redundant) standalone entity alongside `King of Hearts` (which has `the King` as alias). The pipeline does not currently merge two existing entities; this redundancy is expected output, not a bug. Recorded so a future reader doesn't "fix" it without considering the apply-merge contract.

Note on `Wonderland`, `Frog-Footman`: a few entities carry `confidence: high` without enough `evidence_refs` to satisfy `merge_meta._confidence_for_evidence_count`. This is **not** an anomaly — `scripts/glossary.py` validates `confidence` against the enum only; the merge-time promotion rule applies only inside `_append_evidence_ref`. Seeded or reused entries can legitimately carry high confidence with sparse `evidence_refs`.

Final-format outputs:
- `output.md`: 160,705 bytes (merged translated markdown)
- `book.html`: 189,824 bytes (web version with floating TOC)
- `book_doc.html`: 182,797 bytes (ebook-style version)
- `book.docx`: 9,809,493 bytes
- `book.epub`: 9,861,786 bytes (43 of 45 source images preserved end-to-end; 2-image gap is Calibre cover-handling / dedup behavior, not a content regression)
- `book.pdf`: 14,665,137 bytes
- TOC inserted with 18 headings

### Expected target (unverified)

None — all previous targets validated by the run above.

### Drift indicator (record current values, do not pass/fail)

LLM-dependent, may differ across runs/models. These are *current observed values*, not assertions:
- `Alice.target` = `"爱丽丝"`
- `White Rabbit.target` = `"白兔"`
- `Cheshire Cat.target` = `"柴郡猫"`
- `Queen of Hearts.target` = `"红心皇后"`
- `Mock Turtle.target` = `"素甲鱼"` (chosen over alternative `"假海龟"` via `conflicting_new_entity_proposals` decision; 赵元任 1922 译本经典渲染)
- `Mouse.gender` = `"male"` (Carroll's narrator uses "his")
- `Dinah.gender` = `"female"` (Alice refers to her cat as "she")
- Cross-batch convergence: `Alice`, `White Rabbit`, `Mouse`, `Dinah`, `Lory` reached `confidence: "high"` within the chunks 4–11 prior partial run; remaining 22 high-confidence entities accumulated across batches 1–4 of this completion run. Major-character convergence held at first batch — no late-batch alias drift observed.


path: tests/test_calibre_html_publish.py
content: import inspect
import sys
import tempfile
import unittest
import zipfile
from pathlib import Path
from unittest import mock


SCRIPT_DIR = Path(__file__).resolve().parents[1] / "scripts"
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

import calibre_html_publish  # noqa: E402
import merge_and_build  # noqa: E402


class ConvertHtmlWithCalibreTests(unittest.TestCase):
    def test_builds_expected_epub_command(self):
        with tempfile.TemporaryDirectory() as temp_dir:
            input_html = Path(temp_dir) / "input.html"
            output_file = Path(temp_dir) / "output.epub"
            input_html.write_text("<html><head><title>Book</title></head></html>", encoding="utf-8")

            def fake_run(cmd, capture_output, text, timeout):
                output_file.write_text("epub", encoding="utf-8")
                return mock.Mock(returncode=0, stderr="")

            with mock.patch.object(
                calibre_html_publish, "find_calibre_convert", return_value="/usr/bin/ebook-convert"
            ), mock.patch.object(
                calibre_html_publish, "extract_html_metadata", return_value=("Book", "Author")
            ), mock.patch.object(
                calibre_html_publish.subprocess, "run", side_effect=fake_run
            ) as run_mock, mock.patch.object(
                calibre_html_publish.signal, "signal"
            ), mock.patch.object(
                calibre_html_publish.signal, "alarm"
            ):
                ok = calibre_html_publish.convert_html_with_calibre(
                    str(input_html), str(output_file), "epub", timeout=12, lang="ja"
                )

            self.assertTrue(ok)
            cmd = run_mock.call_args.args[0]
            self.assertEqual(cmd[0], "/usr/bin/ebook-convert")
            self.assertEqual(cmd[1], str(input_html))
            self.assertEqual(cmd[2], str(output_file))
            self.assertIn("--title", cmd)
            self.assertIn("--authors", cmd)
            self.assertIn("--language", cmd)
            self.assertIn("ja", cmd)
            self.assertIn("--epub-version", cmd)
            self.assertIn("3", cmd)
            self.assertNotIn("--disable-font-rescaling", cmd)

    @unittest.skipUnless(
        "cover" in inspect.signature(calibre_html_publish.convert_html_with_calibre).parameters,
        "cover support not merged yet",
    )
    def test_includes_cover_argument_when_requested(self):
        with tempfile.TemporaryDirectory() as temp_dir:
            input_html = Path(temp_dir) / "input.html"
            output_file = Path(temp_dir) / "output.epub"
            cover_file = Path(temp_dir) / "cover.jpg"
            input_html.write_text("<html><head><title>Book</title></head></html>", encoding="utf-8")
            cover_file.write_text("img", encoding="utf-8")

            def fake_run(cmd, capture_output, text, timeout):
                output_file.write_text("epub", encoding="utf-8")
                return mock.Mock(returncode=0, stderr="")

            with mock.patch.object(
                calibre_html_publish, "find_calibre_convert", return_value="/usr/bin/ebook-convert"
            ), mock.patch.object(
                calibre_html_publish, "extract_html_metadata", return_value=("Book", "Author")
            ), mock.patch.object(
                calibre_html_publish.subprocess, "run", side_effect=fake_run
            ) as run_mock, mock.patch.object(
                calibre_html_publish.signal, "signal"
            ), mock.patch.object(
                calibre_html_publish.signal, "alarm"
            ):
                ok = calibre_html_publish.convert_html_with_calibre(
                    str(input_html),
                    str(output_file),
                    "epub",
                    timeout=12,
                    lang="ja",
                    cover=str(cover_file),
                )

            self.assertTrue(ok)
            cmd = run_mock.call_args.args[0]
            self.assertIn("--cover", cmd)
            self.assertIn(str(cover_file), cmd)

class ExtractCoverFromEpubTests(unittest.TestCase):
    @unittest.skipUnless(
        hasattr(merge_and_build, "extract_cover_from_epub"),
        "cover extraction support not merged yet",
    )
    def test_extracts_cover_from_opf_cover_image_item(self):
        with tempfile.TemporaryDirectory() as temp_dir:
            epub_file = Path(temp_dir) / "source.epub"
            output_dir = Path(temp_dir) / "extract"
            cover_bytes = b"fake-image"

            with zipfile.ZipFile(epub_file, "w") as zf:
                zf.writestr(
                    "META-INF/container.xml",
                    """<?xml version="1.0" encoding="utf-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
  <rootfiles>
    <rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/>
  </rootfiles>
</container>
""",
                )
                zf.writestr(
                    "decoy/unused.opf",
                    """<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
  <metadata />
  <manifest />
</package>
""",
                )
                zf.writestr(
                    "OEBPS/content.opf",
                    """<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
  <metadata />
  <manifest>
    <item id="cover-image" href="images/cover.jpg" media-type="image/jpeg" />
  </manifest>
</package>
""",
                )
                zf.writestr("OEBPS/images/cover.jpg", cover_bytes)

            extracted = merge_and_build.extract_cover_from_epub(
                str(epub_file), str(output_dir)
            )

            self.assertIsNotNone(extracted)
            self.assertTrue(Path(extracted).exists())
            self.assertEqual(Path(extracted).read_bytes(), cover_bytes)

    @unittest.skipUnless(
        hasattr(merge_and_build, "extract_cover_from_epub"),
        "cover extraction support not merged yet",
    )
    def test_extracts_cover_via_meta_cover_fallback(self):
        with tempfile.TemporaryDirectory() as temp_dir:
            epub_file = Path(temp_dir) / "source.epub"
            output_dir = Path(temp_dir) / "extract"
            cover_bytes = b"fallback-image"

            with zipfile.ZipFile(epub_file, "w") as zf:
                zf.writestr(
                    "META-INF/container.xml",
                    """<?xml version="1.0" encoding="utf-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
  <rootfiles>
    <rootfile full-path="OPS/package.opf" media-type="application/oebps-package+xml"/>
  </rootfiles>
</container>
""",
                )
                zf.writestr(
                    "decoy/unused.opf",
                    """<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
  <metadata />
  <manifest />
</package>
""",
                )
                zf.writestr(
                    "OPS/package.opf",
                    """<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
  <metadata>
    <meta name="cover" content="cover-item" />
  </metadata>
  <manifest>
    <item id="cover-item" href="images/fallback.jpg" media-type="image/jpeg" />
  </manifest>
</package>
""",
                )
                zf.writestr("OPS/images/fallback.jpg", cover_bytes)

            extracted = merge_and_build.extract_cover_from_epub(
                str(epub_file), str(output_dir)
            )

            self.assertIsNotNone(extracted)
            self.assertTrue(Path(extracted).exists())
            self.assertEqual(Path(extracted).read_bytes(), cover_bytes)


if __name__ == "__main__":
    unittest.main()


path: tests/test_convert.py
content: import os
import sys
import tempfile
import unittest
from pathlib import Path


SCRIPT_DIR = Path(__file__).resolve().parents[1] / "scripts"
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

import convert  # noqa: E402


class CleanCalibreMarkersTests(unittest.TestCase):
    def test_removes_known_calibre_artifacts(self):
        content = "\n".join(
            [
                "## Heading {#calibre_link-12 .calibre3}",
                "[**Chapter One**]",
                "Paragraph text{.calibre5} (#calibre_link-2)",
                "::: {.calibre1}",
                "42",
                "broken.ct}",
                "Regular paragraph.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertIn("## Heading", cleaned)
        self.assertIn("**Chapter One**", cleaned)
        self.assertIn("Paragraph text", cleaned)
        self.assertIn("Regular paragraph.", cleaned)
        self.assertNotIn(".calibre", cleaned)
        self.assertNotIn("(#calibre_link-", cleaned)
        self.assertNotIn(":::", cleaned)
        # 42 sits between ::: noise and broken.ct} noise, both calibre artifacts.
        # Context-aware cleaner still drops it — but only because of the neighbors.
        self.assertNotIn("\n42\n", f"\n{cleaned}\n")
        self.assertNotIn("broken.ct}", cleaned)

    def test_preserves_year_in_paragraph(self):
        content = "\n".join(
            [
                "He was born in",
                "1984",
                "and died later.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertIn("1984", cleaned)
        self.assertIn("He was born in", cleaned)
        self.assertIn("and died later.", cleaned)

    def test_preserves_chapter_number_after_heading(self):
        content = "\n".join(
            [
                "## Chapter",
                "",
                "3",
                "",
                "Introduction text follows.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertIn("\n3\n", f"\n{cleaned}\n")
        self.assertIn("## Chapter", cleaned)
        self.assertIn("Introduction text follows.", cleaned)

    def test_drops_digit_line_inside_calibre_fence(self):
        content = "\n".join(
            [
                "Some real paragraph.",
                "::: {.calibre1}",
                "42",
                ":::",
                "More real paragraph.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertNotIn("42", cleaned)
        self.assertNotIn(":::", cleaned)
        self.assertIn("Some real paragraph.", cleaned)
        self.assertIn("More real paragraph.", cleaned)

    def test_drops_digit_line_adjacent_to_ct_marker(self):
        content = "\n".join(
            [
                "Real paragraph above.",
                "7",
                "broken.ct}",
                "Real paragraph below.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertNotIn("\n7\n", f"\n{cleaned}\n")
        self.assertNotIn("broken.ct}", cleaned)

    def test_drops_sequential_page_numbers(self):
        # Six paragraphs separated by sequential page-number footers — clear
        # monotonic spine should be detected and dropped.
        content = "\n".join(
            [
                "Para one.",
                "",
                "1",
                "",
                "Para two.",
                "",
                "2",
                "",
                "Para three.",
                "",
                "3",
                "",
                "Para four.",
                "",
                "4",
                "",
                "Para five.",
                "",
                "5",
                "",
                "Para six.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        for n in ("1", "2", "3", "4", "5"):
            self.assertNotIn(f"\n{n}\n", f"\n{cleaned}\n")
        for p in ("Para one.", "Para two.", "Para three.", "Para four.", "Para five.", "Para six."):
            self.assertIn(p, cleaned)

    def test_preserves_year_among_page_numbers(self):
        # Same page-number spine plus a year (1984) sitting between two page
        # numbers — LNDS picks 1..5 and skips 1984, which stays as content.
        content = "\n".join(
            [
                "Para one.",
                "",
                "1",
                "",
                "Para two.",
                "",
                "2",
                "",
                "He was born in 1984.",
                "Standalone year:",
                "1984",
                "and continued.",
                "",
                "3",
                "",
                "Para three.",
                "",
                "4",
                "",
                "Para four.",
                "",
                "5",
                "",
                "Para five.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        self.assertIn("\n1984\n", f"\n{cleaned}\n")
        for n in ("1", "2", "3", "4", "5"):
            self.assertNotIn(f"\n{n}\n", f"\n{cleaned}\n")

    def test_few_digits_not_treated_as_page_numbers(self):
        # Only three standalone digits — below the LNDS minimum length, so all
        # are preserved (assuming no calibre-noise neighbors).
        content = "\n".join(
            [
                "Intro paragraph.",
                "",
                "1",
                "",
                "Body paragraph.",
                "",
                "2",
                "",
                "More body.",
                "",
                "3",
                "",
                "Closing paragraph.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        for n in ("1", "2", "3"):
            self.assertIn(f"\n{n}\n", f"\n{cleaned}\n")

    def test_non_monotonic_digits_preserved(self):
        # Five standalone digits but no monotonic spine — LNDS coverage too low
        # to trigger; everything preserved.
        content = "\n".join(
            [
                "Intro.",
                "",
                "1984",
                "",
                "Para A.",
                "",
                "42",
                "",
                "Para B.",
                "",
                "7",
                "",
                "Para C.",
                "",
                "1066",
                "",
                "Para D.",
                "",
                "3",
                "",
                "Closing.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content)

        for n in ("1984", "42", "7", "1066", "3"):
            self.assertIn(f"\n{n}\n", f"\n{cleaned}\n")

    def test_strip_page_numbers_flag_restores_legacy(self):
        content = "\n".join(
            [
                "He was born in",
                "1984",
                "and died later.",
                "",
                "## Chapter",
                "",
                "3",
                "",
                "Introduction text follows.",
            ]
        )

        cleaned = convert.clean_calibre_markers(content, strip_page_numbers=True)

        self.assertNotIn("1984", cleaned)
        self.assertNotIn("\n3\n", f"\n{cleaned}\n")
        self.assertIn("He was born in", cleaned)
        self.assertIn("Introduction text follows.", cleaned)


class StripPageNumbersCacheConflictTests(unittest.TestCase):
    def test_no_blockers_when_flag_off(self):
        with tempfile.TemporaryDirectory() as tmp:
            input_md = os.path.join(tmp, "input.md")
            with open(input_md, "w", encoding="utf-8") as f:
                f.write("placeholder")
            with open(os.path.join(tmp, "chunk0001.md"), "w", encoding="utf-8") as f:
                f.write("placeholder")

            blockers = convert._check_strip_page_numbers_cache_conflict(
                strip_flag=False, temp_dir=tmp, input_md=input_md
            )

            self.assertEqual(blockers, [])

    def test_no_blockers_when_temp_dir_missing(self):
        missing_dir = os.path.join(tempfile.gettempdir(), "definitely-not-here-xyz-123")
        # Make extra sure it really doesn't exist.
        self.assertFalse(os.path.isdir(missing_dir))
        input_md = os.path.join(missing_dir, "input.md")

        blockers = convert._check_strip_page_numbers_cache_conflict(
            strip_flag=True, temp_dir=missing_dir, input_md=input_md
        )

        self.assertEqual(blockers, [])

    def test_aborts_when_input_md_cached(self):
        with tempfile.TemporaryDirectory() as tmp:
            input_md = os.path.join(tmp, "input.md")
            with open(input_md, "w", encoding="utf-8") as f:
                f.write("cached markdown")

            blockers = convert._check_strip_page_numbers_cache_conflict(
                strip_flag=True, temp_dir=tmp, input_md=input_md
            )

            self.assertEqual(len(blockers), 1)
            self.assertIn("input.md", blockers[0])

    def test_aborts_when_chunks_cached(self):
        with tempfile.TemporaryDirectory() as tmp:
            input_md = os.path.join(tmp, "input.md")  # absent
            for i in range(1, 4):
                with open(os.path.join(tmp, f"chunk{i:04d}.md"), "w", encoding="utf-8") as f:
                    f.write("chunk")
            # output_chunk*.md files must not be counted as source chunks.
            with open(os.path.join(tmp, "output_chunk0001.md"), "w", encoding="utf-8") as f:
                f.write("translated")

            blockers = convert._check_strip_page_numbers_cache_conflict(
                strip_flag=True, temp_dir=tmp, input_md=input_md
            )

            self.assertEqual(len(blockers), 1)
            self.assertIn("3 chunk file(s)", blockers[0])


if __name__ == "__main__":
    unittest.main()


path: tests/test_meta.py
content: import json
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock


SCRIPT_DIR = Path(__file__).resolve().parents[1] / "scripts"
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

import meta  # noqa: E402


def minimal_meta():
    return {
        'schema_version': 1,
        'new_entities': [],
        'alias_hypotheses': [],
        'attribute_hypotheses': [],
        'used_term_sources': [],
        'conflicts': [],
    }


def full_meta():
    return {
        'schema_version': 1,
        'new_entities': [
            {'source': 'Taig', 'target_proposal': '泰格', 'category': 'person',
             'evidence': 'Taig walked in.'},
        ],
        'alias_hypotheses': [
            {'variant': 'Taig', 'may_be_alias_of_source': 'Tai',
             'evidence': 'Taig nodded — Tai must have heard.'},
        ],
        'attribute_hypotheses': [
            {'entity_source': 'Tai', 'attribute': 'gender', 'value': 'male',
             'confidence': 'high', 'evidence': 'He smiled at Tai.'},
        ],
        'used_term_sources': ['Tai', 'Manhattan'],
        'conflicts': [
            {'entity_source': 'Tai', 'field': 'target', 'injected': '泰',
             'observed_better': '太一', 'evidence': 'Context implies 太一.'},
        ],
    }


class RoundTripTests(unittest.TestCase):
    def test_save_then_load_round_trip(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = os.path.join(tmp, 'output_chunk0001.meta.json')
            data = full_meta()
            meta.save_meta(path, data)
            loaded = meta.load_meta(path)
            self.assertEqual(loaded, data)


class ValidateMetaTests(unittest.TestCase):
    def test_validate_minimal_meta_with_empty_arrays(self):
        meta.validate_meta(minimal_meta())

    def test_validate_full_meta_with_one_of_each_entity_type(self):
        meta.validate_meta(full_meta())

    def test_rejects_invalid_attribute_confidence(self):
        data = minimal_meta()
        data['attribute_hypotheses'] = [{
            'entity_source': 'Tai', 'attribute': 'gender', 'value': 'male',
            'confidence': 'maybe', 'evidence': '...',
        }]
        with self.assertRaises(ValueError) as ctx:
            meta.validate_meta(data)
        self.assertIn("'confidence'", str(ctx.exception))

    def test_rejects_overlong_evidence_field(self):
        data = minimal_meta()
        data['new_entities'] = [{
            'source': 'X', 'target_proposal': 'x', 'category': 'p',
            'evidence': 'A' * (meta.EVIDENCE_MAX_LEN + 1),
        }]
        with self.assertRaises(ValueError) as ctx:
            meta.validate_meta(data)
        self.assertIn("limit is", str(ctx.exception))

    def test_load_rejects_unknown_top_level_key(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = os.path.join(tmp, 'output_chunk0001.meta.json')
            data = minimal_meta()
            data['surprise'] = 'oops'
            with open(path, 'w', encoding='utf-8') as f:
                json.dump(data, f)
            with self.assertRaises(ValueError) as ctx:
                meta.load_meta(path)
            self.assertIn('surprise', str(ctx.exception))

    def test_load_rejects_chunk_id_in_payload(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = os.path.join(tmp, 'output_chunk0001.meta.json')
            data = minimal_meta()
            data['chunk_id'] = 'chunk0001'
            with open(path, 'w', encoding='utf-8') as f:
                json.dump(data, f)
            with self.assertRaises(ValueError) as ctx:
                meta.load_meta(path)
            msg = str(ctx.exception)
            self.assertIn('chunk_id', msg)
            self.assertIn('filename', msg)

    def test_rejects_wrong_schema_version(self):
        data = minimal_meta()
        data['schema_version'] = 99
        with self.assertRaises(ValueError) as ctx:
            meta.validate_meta(data)
        self.assertIn('schema_version', str(ctx.exception))


class HashTests(unittest.TestCase):
    def test_meta_content_hash_stable_across_key_order(self):
        a = full_meta()
        b = json.loads(json.dumps(a, sort_keys=False))
        self.assertEqual(meta.meta_content_hash(a), meta.meta_content_hash(b))

    def test_meta_content_hash_changes_when_content_changes(self):
        a = minimal_meta()
        b = minimal_meta()
        b['used_term_sources'] = ['Manhattan']
        self.assertNotEqual(meta.meta_content_hash(a), meta.meta_content_hash(b))


class ChunkIdFromPathTests(unittest.TestCase):
    def test_chunk_id_from_meta_path_extracts_basename(self):
        self.assertEqual(meta.chunk_id_from_meta_path('/tmp/output_chunk0042.meta.json'),
                         'chunk0042')
        self.assertEqual(meta.chunk_id_from_meta_path('output_chunk9999.meta.json'),
                         'chunk9999')

    def test_chunk_id_from_meta_path_rejects_unexpected_basename(self):
        with self.assertRaises(ValueError):
            meta.chunk_id_from_meta_path('/tmp/random.json')
        with self.assertRaises(ValueError):
            meta.chunk_id_from_meta_path('/tmp/chunk0001.meta.json')  # missing 'output_'


class AtomicWriteTests(unittest.TestCase):
    def test_atomic_write_survives_simulated_interrupt(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = os.path.join(tmp, 'output_chunk0001.meta.json')
            original = minimal_meta()
            original['used_term_sources'] = ['Original']
            meta.save_meta(path, original)

            doomed = minimal_meta()
            doomed['used_term_sources'] = ['Doomed']
            with mock.patch('meta.json.dump', side_effect=OSError("disk full")):
                with self.assertRaises(OSError):
                    meta.save_meta(path, doomed)

            with open(path, 'r', encoding='utf-8') as f:
                on_disk = json.load(f)
            self.assertEqual(on_disk['used_term_sources'], ['Original'])

            leftovers = [f for f in os.listdir(tmp) if f.startswith('.meta-')]
            self.assertEqual(leftovers, [])


if __name__ == '__main__':
    unittest.main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)