Skill: rizmoon/finance-ocr-pro (version 1.0.7)

--- name: finance-ocr-pro version: 1.0.7 description: "Use this skill when the user asks to OCR, transcribe, extract, or convert the contents of a scanned PDF, image, or office document into Markdown, HTML, DOCX, or Excel. This workflow sends page images and OCR prompts to a configured OpenAI-compatible VLM endpoint, requires API_KEY, BASE_URL, and VLM_MODEL, bundles HTML report assets locally so no runtime CDN downloads are needed, and produces improved native Word equation output for recognized LaTeX formulas. It is especially valuable for financial documents and other visually complex materials with dense tables, charts, graphs, and multi-part layouts. Prefer durable background jobs for long-running OCR work." metadata: openclaw: requires: env: - API_KEY - BASE_URL - VLM_MODEL anyBins: - python3 - python primaryEnv: API_KEY --- # Finance OCR Pro Version 1.0.7 update: Word/DOCX output now includes optimized and expanded LaTeX formula restoration for recognized formulas. Run this skill only after OCR intent from the user. This skill is especially helpful for financial reports, annual reports, prospectuses, investor presentations, regulatory filings, research reports, and other documents with complicated structure, charts, graphs, tables, and mixed layout elements. ## Security And Privacy Before running OCR, make the operating model clear: - This skill requires three environment variables, all of which must be configured before OCR can run: - `API_KEY` (sensitive) -- the API key for authenticating with the VLM endpoint. - `BASE_URL` -- the base URL of the OpenAI-compatible VLM endpoint. All page images and OCR prompts are transmitted to this URL. - `VLM_MODEL` -- the vision-capable model identifier. Must support image inputs; text-only models will not work. - OCR sends rendered page images and structured prompts to `BASE_URL`. This is the primary data-transmission path. Users must verify that the endpoint is trusted before processing sensitive documents. - If the user wants offline or local-only OCR, `BASE_URL` must point to a local VLM service. Do not run this skill against an external endpoint with sensitive documents unless the provider is trusted. - `scripts/ocr_setup.py` checks dependencies and creates `.env` templates, but it never installs Python packages automatically. Users must review and run dependency installation themselves. - HTML report generation uses vendored Mermaid and MathJax files from `scripts/vendor/` and does not download frontend assets from a CDN at runtime. - Local subprocess usage is limited to starting the local OCR worker and invoking document-conversion tools such as LibreOffice or `osascript`. Commands are executed with explicit argument lists rather than shell strings. - Never commit a populated `.env` file. Use `.env.example` as a template and keep real credentials local. ## Pre-Run Notice After the user asks for OCR or extraction, give a short notice that includes: - whether `BASE_URL` is local or remote - which `VLM_MODEL` will be used - which execution mode will be used - where results will be written - that the skill supports multi-thread OCR and the thread count can be increased when the user's API endpoint, rate limits, and plan support parallel OCR requests - that page images and prompts will be transmitted to the configured endpoint Proceed automatically unless the user asks to change those defaults. ## Defaults To Announce - Running mode: background job by default - Model: `VLM_MODEL` - Threads: `1` If the user's API endpoint or plan supports safe parallel OCR requests, tell them they can choose a higher thread count. - Result path: - background: `~/.semantic-ocr/jobs/<job_id>/results/` - synchronous: `ocr_output/OCR_<filename>/results/` ## Setup Use the skill-local virtual environment if present. - macOS/Linux: `.venv/bin/python` - Windows: `.venv/Scripts/python.exe` - Fallback: `python3` on macOS/Linux, `python` on Windows Before running any command, resolve the interpreter and reuse it for the rest of the session: - macOS/Linux: `PYTHON="${PYTHON:-$( [ -x .venv/bin/python ] && printf .venv/bin/python || printf python3 )}"` - Windows: use `.venv\Scripts\python.exe` when present, otherwise `python` Run: ```bash $PYTHON scripts/ocr_setup.py --check ``` If setup is incomplete, run: ```bash $PYTHON scripts/ocr_setup.py ``` ## Preferred Execution By default, start a background worker: ```bash $PYTHON scripts/ocrctl.py --json start /path/to/document.pdf ``` If the provider supports concurrency and the user wants faster OCR, offer a higher thread count such as: ```bash $PYTHON scripts/ocrctl.py --json start -t 4 /path/to/document.pdf ``` Then inspect progress and outputs: ```bash $PYTHON scripts/ocrctl.py --json status <job_id> $PYTHON scripts/ocrctl.py --json artifacts <job_id> $PYTHON scripts/ocrctl.py --json tail <job_id> ``` Use synchronous mode only when the user explicitly wants inline execution: ```bash $PYTHON scripts/ocr_main.py /path/to/document.pdf ``` ## Notes - Inputs: PDF, common office documents, Apple office formats, and images. - Outputs: merged Markdown, HTML review report, DOCX with improved native Word equation output for recognized LaTeX formulas, and Excel. - OCR requires `API_KEY`, `BASE_URL`, and `VLM_MODEL` to be configured before running. - The default page-rendering resolution is `200` DPI. - The skill supports multi-thread OCR. Keep the default at `1` unless the user's API endpoint, rate limits, and plan support concurrent OCR requests. - Sensitive document pages are transmitted to the configured endpoint during OCR unless the endpoint is a local service. - Best suited for financial documents and other visually dense materials with tables, charts, graphs, and complex page structure. - Office-document conversion may require LibreOffice. - OCR extraction by the VLM model may be time-consuming; check the status regularly.

path: requirements.txt
content: # AI model client
openai>=1.0
python-dotenv>=0.19

# PDF / image processing
PyMuPDF>=1.21
Pillow>=9.0
pdf2image>=1.16

# Markdown → HTML rendering
markdown2>=2.4

# Markdown → DOCX rendering
python-docx>=0.8.11
beautifulsoup4>=4.12
lxml>=4.9
latex2mathml>=3.75

# Markdown → Excel rendering
openpyxl>=3.1
unicodeit>=0.7

# Windows-only Office COM conversion fallback
pywin32>=306; platform_system == "Windows"


path: agents/openai.yaml
content: interface:
  display_name: "Finance OCR Pro"
  short_description: "Extract structured Markdown, HTML, DOCX, and Excel from documents, with improved Word equation output for LaTeX formulas"
  default_prompt: >
    Use this skill when the user asks to OCR, transcribe,
    extract, or convert a document's contents. This skill requires API_KEY
    (secret), BASE_URL, and VLM_MODEL. Page images are transmitted to
    BASE_URL during OCR. It outputs Markdown, HTML, DOCX with improved native
    Word equation restoration for recognized LaTeX formulas, and Excel.
    Announce the default mode, model, thread count, result path, and
    remote-processing implications, then proceed. Also tell the user that the
    skill supports multi-thread OCR, but higher thread counts should only be
    used when the API endpoint, rate limits, and subscription plan support
    parallel OCR requests. First resolve the
    interpreter path: on macOS/Linux use `.venv/bin/python` if present,
    otherwise `python3`; on Windows use `.venv/Scripts/python.exe` if
    present, otherwise `python`. Prefer
    `<python> scripts/ocrctl.py --json start ...` for long jobs and
    `<python> scripts/ocr_main.py ...` for small inline jobs. If dependencies
    are missing, direct the user to the manual install command printed by
    `scripts/ocr_setup.py` rather than attempting automatic installation.
  required_env:
    - API_KEY
    - BASE_URL
    - VLM_MODEL
  primary_credential: API_KEY


path: THIRD_PARTY_NOTICES.md
content: # Third-Party Notices

This repository vendors the following frontend assets so generated HTML reports
do not download code from third-party CDNs at runtime:

- Mermaid `11.14.0`
  - file: `scripts/vendor/mermaid-11.14.0/mermaid.min.js`
  - license file: `scripts/vendor/mermaid-11.14.0/LICENSE`
- MathJax `3.2.2`
  - file: `scripts/vendor/mathjax-3.2.2/tex-svg-full.js`
  - license file: `scripts/vendor/mathjax-3.2.2/LICENSE`


path: README.md
content: # Finance OCR Pro

Finance OCR Pro is a cross-platform OCR skill and local runtime for agent
environments such as Codex, Claude Code, and OpenClaw. The runtime executes on
the user's machine, but the OCR step sends rendered page images and OCR prompts
to a configured OpenAI-compatible Vision-Language Model (VLM) endpoint unless
the user points `BASE_URL` to a trusted local service. HTML report assets are
bundled locally, and the setup flow prints manual dependency instructions
instead of installing packages automatically.

The project reconstructs each page into structured Markdown and then produces a
reviewable HTML report, an editable DOCX file, and an Excel workbook of
extracted tables.

It is especially useful for financial documents and other visually complex
materials where the page contains dense tables, charts, graphs, footnotes,
multi-column layouts, and mixed text/figure structure that simple text
extraction often handles poorly.

This repository is organized as a reusable skill plus a local background-job
runtime. The skill layer tells an agent when and how to run OCR. The runtime
layer lets long OCR jobs continue safely on the user's machine even if the
agent session is interrupted or times out.

## Version Notes

### 1.0.7

Update: LaTeX formula restoration for Word/DOCX output has been optimized and
expanded. DOCX generation now preserves more native Word equation structures,
including improved handling for common functions, continuous identifiers,
subscripts attached to OCR text, n-ary operators such as sums, stretch
delimiters, and cases-style formulas.

## Security And Data Handling

Before using this project, understand the following:

- The OCR pipeline sends rendered page images and structured OCR prompts to
  the configured `BASE_URL`. This is the primary data-transmission path.
- Those page images may contain sensitive content such as financial statements,
  contracts, IDs, signatures, or internal reports.
- The project requires three user-supplied environment variables, all of which
  must be configured before OCR can run:
  - `API_KEY` (sensitive) -- API key for authenticating with the VLM endpoint.
    This is the primary credential. Treat it as a secret.
  - `BASE_URL` -- base URL of the OpenAI-compatible VLM endpoint
    (e.g. `https://api.openai.com/v1`). All page images are transmitted here.
  - `VLM_MODEL` -- vision-capable model identifier (e.g. `gpt-4o`). Must
    support image inputs; text-only models will not work.
- These same environment variables are declared in the platform manifest
  (`skill.yaml`), in `SKILL.md` front matter (`metadata.openclaw.requires.env`),
  and in `agents/openai.yaml` so that agent platforms and security scanners
  can verify the requirements before installation.
- Users should verify the exact endpoint they are sending data to and confirm
  that the provider is appropriate for the document sensitivity level.
- If offline or local-only processing is required, configure `BASE_URL` to
  point to a trusted local VLM service. Do not use an external endpoint for
  sensitive documents unless the provider is trusted.
- `scripts/ocr_setup.py` does not install dependencies automatically. It only
  checks readiness and prints the manual install command for the current
  Python interpreter.
- HTML report generation uses vendored Mermaid and MathJax assets from
  `scripts/vendor/` and performs no runtime CDN fetches.
- Local subprocess usage is restricted to starting the background OCR worker
  and calling known document-conversion helpers such as LibreOffice and
  `osascript`. Commands are passed as explicit argument arrays; the project
  does not use `shell=True`.
- Do not commit a populated `.env` file. Keep secrets local and distribute
  `.env.example` only.

## Agent Activation Policy

This skill is intentionally designed to require explicit OCR intent from the
user because Finance OCR Pro can be slow, costly, and may transmit sensitive
page images to a model endpoint.

- Run the skill when the user asks to OCR, extract, transcribe, 
  or convert the document contents.
- If a file is uploaded for another purpose, such as review, summary,
  translation, comparison, or general analysis, agents should not start OCR
  unless the user clearly requests extraction or the process needs an extraction.

Once the user has requested OCR, the agent should present a short
pre-run notice with the planned defaults and remote-processing implications,
allow the user to change them if desired, and otherwise proceed automatically
without waiting for a second confirmation.

That notice should explicitly state:

- whether `BASE_URL` points to a remote or local endpoint
- which `VLM_MODEL` will be used
- which execution mode will be used
- where results will be written
- that multi-thread OCR is supported and a higher thread count can be offered
  when the user's API endpoint, rate limits, and plan support concurrent OCR
  requests
- that page images and OCR prompts will be sent to the configured endpoint
- extraction by VLM models may be time-consuming

## Project Purpose

The project is intended for scanned or visually complex documents where simple
text extraction is not enough. Its goals are:

- extract readable, structured content from scanned pages
- preserve layout semantics such as headings, lists, tables, and some diagrams
- generate outputs that are useful both for review and downstream editing
- support long-running document jobs in agent-driven environments

It is particularly well suited to:

- annual reports, interim reports, earnings releases, and investor presentations
- prospectuses, offering memoranda, and other transaction documents
- regulatory filings and compliance materials
- research reports
- and other documents with complicated tables, charts, and graphs

Typical input examples include PDFs, office documents, presentations, and image
files. Typical output examples include a merged Markdown transcription, a
side-by-side HTML review report, a styled Word document, and an Excel workbook
of extracted tables.

## What Makes This Project Different

This is not just a batch OCR script. It has three notable characteristics:

1. Semantic extraction
Each page is rendered to an image and sent to a vision-capable model with a
strict OCR prompt. The model is instructed to reconstruct content faithfully in
Markdown, use HTML tables for tabular content, preserve math, and represent
certain diagrams with Mermaid when appropriate.

This is especially valuable in financial and analytical documents where
important meaning lives inside tables, ownership charts, figure captions,
footnotes, and structured page layout rather than plain paragraph text alone.

2. Review-friendly output
The project does not stop at raw text extraction. It also generates:

- a combined Markdown document
- an HTML comparison report that places source page images beside extracted content
- a DOCX document intended for editing and handoff, with native Word equation
  output for recognized LaTeX formulas
- an Excel workbook for extracted tables

3. Agent-safe long-running execution
The repository now includes a durable local job controller so an agent can
start OCR, poll progress, fetch artifacts, and resume/cancel work later without
holding the actual OCR process open.

## High-Level Workflow

At a high level, the OCR flow is:

1. Preflight and configuration check
The project validates Python dependencies and model configuration.

2. Document-to-image conversion
The input document is converted into one image per page.

3. Page-by-page semantic OCR
Each page image is sent to the configured VLM and written to per-page Markdown.

4. Combined document assembly
Page outputs are merged into one Markdown file.

5. Review artifact generation
The project creates:

- an HTML report
- a DOCX document
- an Excel workbook

## Cross-Platform Interpreter Notes

All scripts are intended to run on Windows, macOS, and Linux.

- Use the skill-local virtualenv interpreter when present:
  - macOS/Linux: `.venv/bin/python`
  - Windows: `.venv/Scripts/python.exe`
- If the virtualenv has not been created yet, use `python3` on macOS/Linux or
  `python` on Windows for setup and local testing.

On macOS/Linux, a reliable pattern is:

```bash
PYTHON="${PYTHON:-$( [ -x .venv/bin/python ] && printf .venv/bin/python || printf python3 )}"
```

Then reuse `$PYTHON` for the remaining commands. On Windows, use
`.venv\Scripts\python.exe` when present, otherwise `python`.

## How The Project Works Internally

### 1. Skill Layer

The skill-facing metadata lives in:

- `skill.yaml` -- OpenClaw platform manifest. Declares the skill's identity,
  required permissions (`network`, `filesystem`, `shell`), configuration
  options (`API_KEY` with `secret: true`, `BASE_URL`, `VLM_MODEL`), and entry
  point. This is the file Clawhub reads for registry metadata and what the
  security scanner checks for declared credentials.
- `SKILL.md` -- agent-facing instructions. The YAML front matter declares
  `metadata.openclaw.requires.env` and `metadata.openclaw.requires.anyBins`
  using standard multi-line YAML so the OpenClaw skill loader and ClawHub
  security scanner can gate the skill correctly. The body tells agents when
  and how to run OCR.
- `agents/openai.yaml` -- OpenAI-agent-specific interface metadata.

For distribution, ship `.env.example` as the editable template and keep the
real `.env` local-only.

These files explain when an agent should use the project, what credentials are
required, what data leaves the machine during OCR, and which commands to run.

### 2. OCR Pipeline Layer

The main OCR pipeline lives in the `scripts/` directory.

Core modules:

- `scripts/ocr_main.py`
  The main synchronous pipeline orchestrator.
- `scripts/docs_to_image.py`
  Converts source documents into per-page images.
- `scripts/image_to_md.py`
  Performs batch OCR from images to Markdown.
- `scripts/ai_service_vlm.py`
  Sends text + image requests to an OpenAI-compatible VLM endpoint.
- `scripts/ocr_prompt.py`
  Contains the structured OCR prompt used for page reconstruction.
- `scripts/md_to_html.py`
  Builds the HTML review report.
- `scripts/md_to_docx.py`
  Converts Markdown into a styled Word document, including native Word equation
  output for recognized LaTeX formulas.
- `scripts/md_to_excel.py`
  Extracts HTML tables from Markdown into a formatted Excel workbook.

### 3. Background Job Layer

To support long-running OCR in agent environments, the repository includes a
disk-backed job runtime:

- `scripts/ocrctl.py`
  User-facing controller for starting and managing jobs.
- `scripts/ocr_worker.py`
  Detached worker process that executes one OCR job.
- `scripts/ocr_jobs.py`
  Job manifest, progress tracking, artifact tracking, and event logging.
- `scripts/ocr_runtime.py`
  Shared progress and cancellation interfaces used by the pipeline.

This layer is especially important for OpenClaw-like environments, where a
single agent run may be shorter than a large OCR task.

## Execution Modes

### Synchronous Mode

Used when an agent or user is comfortable waiting for the full job inline.

Example:

```bash
$PYTHON scripts/ocr_main.py /path/to/document.pdf
```

### Durable Background Job Mode

Used when the OCR task may outlive an agent session.

Examples:

```bash
$PYTHON scripts/ocrctl.py --json start /path/to/document.pdf
$PYTHON scripts/ocrctl.py --json status <job_id>
$PYTHON scripts/ocrctl.py --json artifacts <job_id>
$PYTHON scripts/ocrctl.py --json cancel <job_id>
$PYTHON scripts/ocrctl.py --json resume <job_id>
```

In background mode, OCR state is persisted under:

```text
~/.semantic-ocr/jobs/<job_id>/
```

Each job directory contains:

- `manifest.json`
  Immutable job definition and input parameters.
- `progress.json`
  Current status snapshot.
- `events.jsonl`
  Append-only event history.
- `worker.log`
  Console output from the detached worker.
- `results/`
  Generated output files.

On successful completion, intermediate `images/` and `markdowns/` directories
are removed automatically to save disk space. Failed or cancelled jobs keep
their intermediates so they can be resumed.

## Default Run Parameters To Show Before Starting

After the user has explicitly requested OCR or extraction, the hosting agent
should tell the user the defaults it plans to use:

- **Running mode**: background job by default
- **Vision model**: the current configured or user-provided `VLM_MODEL`
- **Results folder**:
  - background mode default: `~/.semantic-ocr/jobs/<job_id>/results/`
  - synchronous mode default: `ocr_output/OCR_<filename>/results/`
- **Threads**: `1` by default
  - The agent should also tell the user that this skill supports multi-thread
    OCR and can use a higher thread count when the API endpoint, rate limits,
    and subscription plan support parallel OCR requests.

If the user does not change these defaults, the agent should proceed
automatically. If the user explicitly requests inline execution, the agent
should switch the running mode and corresponding default results folder.

## Mode Selection By Host

The recommended execution mode depends on the host environment:

- **OpenClaw**
  Use durable background-job mode by default. This is the recommended and
  expected mode for normal OCR requests because long OCR tasks may outlive a
  single OpenClaw agent run.
- **Codex**
  Synchronous mode is acceptable for small jobs. Durable background-job mode is
  preferred for large or long-running documents.
- **Claude Code**
  Synchronous mode is acceptable for small jobs. Durable background-job mode is
  preferred for large or long-running documents.
- **Direct human terminal use**
  Synchronous mode is convenient for quick local tests. Background-job mode is
  better when the user wants to disconnect and check back later.

## Inputs And Outputs

### Supported Inputs

The project is designed to accept:

- PDF
- DOC / DOCX
- PPT / PPTX
- ODT / ODP / RTF / WPS
- KEY / PAGES
- PNG / JPG / JPEG / WEBP / BMP / TIF / TIFF

Some office-format conversions depend on LibreOffice being installed locally.

### Primary Outputs

For a source file named `report.pdf`, the main outputs are:

- `report_combined.md`
- `report.html`
- `report.docx`
- `report.xlsx`

In background mode, the worker records artifact paths in `progress.json`.

## Configuration Requirements

The project expects an OpenAI-compatible VLM endpoint. Three environment
variables must be set before OCR can run:

| Variable    | Required | Sensitive | Purpose |
|-------------|----------|-----------|---------|
| `API_KEY`   | Yes      | Yes       | API key for authenticating with the VLM endpoint (primary credential) |
| `BASE_URL`  | Yes      | No        | Base URL of the VLM endpoint. Page images are transmitted here during OCR |
| `VLM_MODEL` | Yes      | No        | Vision-capable model identifier. Must support image inputs |

These are the same variables declared in `skill.yaml` (platform manifest),
`SKILL.md` front matter (`metadata.openclaw.requires.env`), and
`agents/openai.yaml`.

Important:

- The configured model must be vision-capable. Text-only models will not work
  because the pipeline sends page images to the API.
- `BASE_URL` determines where document images are sent. Review it carefully
  before processing sensitive files.
- A missing or placeholder `.env` means the OCR pipeline is not configured yet.

First-run setup is handled by:

```bash
$PYTHON scripts/ocr_setup.py
```

If Python dependencies are missing, the setup script will print a manual
install command similar to:

```bash
$PYTHON -m pip install -r requirements.txt
```

For redistribution, do not ship a populated `.env` file. Ship
`.env.example` instead and let each user create their own local `.env`.

Example template:

```dotenv
[REDACTED_SECRET]
BASE_URL=https://your-api-endpoint/v1
VLM_MODEL=your-vision-model-here
```

## Key Review Points

For a project review, the most important aspects are:

### A. Separation of concerns

The repository clearly separates:

- skill instructions for agent use
- OCR pipeline implementation
- job-control/runtime logic for long tasks

### B. Operational resilience

The background-job model is intended to reduce fragility in agent workflows.
An agent starts a job and later reconnects to check status or retrieve results
instead of owning the entire OCR runtime.

### C. Structured outputs

The project emphasizes outputs that can be reviewed and reused downstream,
rather than returning plain text only.

### D. Recoverability

The pipeline now supports resume-aware behavior for existing page markdown and
job-state persistence on disk.

## Known Limitations And Considerations

- OCR quality depends on the chosen VLM, endpoint behavior, and the source
  document quality.
- Page rendering now defaults to 200 DPI to balance OCR quality, speed, and
  disk usage across hosts.
- Very large documents can still take significant time, network transfer, and
  API cost.
- Office document conversion may require local system dependencies.
- HTML report generation depends on the vendored Mermaid and MathJax assets in
  `scripts/vendor/`.
- Cancellation is cooperative, which means it is most responsive between page
  operations rather than in the middle of a single OCR request.

## Release Checklist

- Include `skill.yaml`, `SKILL.md`, `agents/openai.yaml`, `README.md`,
  `requirements.txt`, `scripts/`, `.env.example`, and `LICENSE`.
- Ensure `skill.yaml`, `SKILL.md` front matter, `agents/openai.yaml`, and
  `README.md` all consistently declare the required env vars (`API_KEY`,
  `BASE_URL`, `VLM_MODEL`), the primary credential (`API_KEY`), and the
  data-transmission disclosure.
- Ensure `README.md` and `SKILL.md` clearly state that OCR sends page images
  and prompts to the configured VLM endpoint unless that endpoint is local.
- Exclude developer-local files such as `.env`, `.venv/`, `__pycache__/`,
  `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`, `.DS_Store`,
  `ocr_output/`, `scripts/image_outputs_*/`, and ad-hoc
  test folders.
- Include `scripts/vendor/`, `THIRD_PARTY_NOTICES.md`, and `SECURITY.md` in
  distribution packages.
- Expect PDF/image OCR to work with Python dependencies alone.
- Expect Office-document OCR to require LibreOffice locally.

## Suggested Review Summary

If a reviewer wants a short summary:

Finance OCR Pro is a local, agent-friendly document extraction project that uses a
vision model to convert scanned and visually complex documents into structured
Markdown, reviewable HTML, editable DOCX, and reusable Excel outputs. Its main
engineering feature is the addition of a durable background-job runtime, which
allows agent tools to manage long OCR tasks safely on the user's machine
instead of tying them to one fragile interactive session.


path: scripts/md_to_docx.py
content: """
Markdown to DOCX Converter

Converts Markdown files (with HTML tables and LaTeX formulas)
into professionally styled A4 DOCX documents with multilingual support.

Handles:
    - Headings (# through ######)
    - Paragraphs with inline formatting (bold, italic, code)
    - Inline LaTeX ($...$) and block LaTeX ($$...$$)
    - HTML tables with colspan/rowspan and LaTeX in cells
    - Ordered and unordered lists
    - Fenced code blocks (```language ... ```)
    - Embedded images (when source file exists)

Requirements:
    pip install python-docx beautifulsoup4 lxml latex2mathml

Usage:
    from pathlib import Path
    markdown_to_docx(Path("input.md"), Path("output.docx"))
"""

from __future__ import annotations

import re
from pathlib import Path

from docx import Document
from docx.shared import Pt, Cm, RGBColor, Twips
from docx.enum.text import WD_ALIGN_PARAGRAPH, WD_BREAK
from docx.enum.table import WD_ROW_HEIGHT_RULE
from docx.oxml.ns import qn
from docx.oxml import OxmlElement
from bs4 import BeautifulSoup, NavigableString, Tag
import latex2mathml.converter
from lxml import etree


# ═══════════════════════════════════════════════════════════════════════════════
# Configuration
# ═══════════════════════════════════════════════════════════════════════════════

OMML_NS = "http://schemas.openxmlformats.org/officeDocument/2006/math"
WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
M = "{" + OMML_NS + "}"
W = "{" + WORD_NS + "}"

FONT_BODY = "Arial"
FONT_MATH = "Cambria Math"
FONT_CODE = "Consolas"

FONT_SIZE_BODY = Pt(11)
FONT_SIZE_TABLE = Pt(9.5)
FONT_SIZE_CODE = Pt(10)

COLOR_HEADING = RGBColor(0x1F, 0x49, 0x7D)
COLOR_BODY = RGBColor(0x1A, 0x1A, 0x1A)
COLOR_MUTED = RGBColor(0x88, 0x88, 0x88)

TABLE_HEADER_BG = "D6E4F0"
TABLE_ALT_ROW_BG = "F5F8FC"
TABLE_BORDER_COLOR = "A6A6A6"
TABLE_CELL_MARGIN_DXA = 72

CODE_BG_COLOR = "F2F2F2"
CODE_BORDER_COLOR = "CCCCCC"

MARGIN_NARROW = Cm(1.8)

HEADING_SIZES = {1: 20, 2: 16, 3: 14, 4: 12, 5: 11, 6: 11}


# ═══════════════════════════════════════════════════════════════════════════════
# Font Utilities
# ═══════════════════════════════════════════════════════════════════════════════

def _set_rfonts(rpr_element, font_name: str):
    """Set w:rFonts for all four script types (Latin, East Asian, Complex Script)."""
    rf = rpr_element.find(qn('w:rFonts'))
    if rf is None:
        rf = OxmlElement('w:rFonts')
        rpr_element.insert(0, rf)
    for attr in ('w:ascii', 'w:hAnsi', 'w:eastAsia', 'w:cs'):
        rf.set(qn(attr), font_name)


def set_run_fonts(run, font_name: str = FONT_BODY, size=FONT_SIZE_BODY,
                  bold: bool = False, italic: bool = False,
                  color: RGBColor | None = None):
    """Configure a run's font for full multilingual compatibility."""
    run.font.name = font_name
    run.font.size = size
    run.bold = bold
    run.italic = italic
    if color:
        run.font.color.rgb = color
    _set_rfonts(run._r.get_or_add_rPr(), font_name)


# ═══════════════════════════════════════════════════════════════════════════════
# Preprocessing
# ═══════════════════════════════════════════════════════════════════════════════

_PAGE_ORDER_RE = re.compile(r'^Page_Order_\d+$', re.IGNORECASE | re.MULTILINE)
_PAGE_NUMBER_RE = re.compile(r'Page\s+Number\s+\d+\s*:\s*\n?', re.IGNORECASE)
_PAGE_SEP_RE = re.compile(r'\n\s*---\s*\n')
_MULTI_BLANK_RE = re.compile(r'\n{3,}')


def preprocess_content(content: str) -> str:
    """Remove OCR noise, separators, and collapse excess blank lines.

    ``Page_Order_N`` sentinels are preserved so the document builder can
    insert page breaks at the correct positions.
    """
    content = _PAGE_NUMBER_RE.sub('', content)
    content = _PAGE_SEP_RE.sub('\n\n', content)
    content = _MULTI_BLANK_RE.sub('\n\n', content)
    return content.strip()


# ═══════════════════════════════════════════════════════════════════════════════
# MathML to OMML Conversion
# ═══════════════════════════════════════════════════════════════════════════════

_NARY_OPERATORS = frozenset({"∑", "∏", "∐", "∫", "∬", "∭", "∮", "⋂", "⋃"})
_COMMON_FUNCTIONS = frozenset({
    "arccos", "arcsin", "arctan", "arg", "cos", "cosh", "cot", "coth",
    "csc", "deg", "det", "dim", "exp", "gcd", "hom", "inf", "ker",
    "lg", "lim", "liminf", "limsup", "ln", "log", "max", "min", "mod",
    "Pr", "sec", "sin", "sinh", "sup", "tan", "tanh",
})


def _omml_run(text: str, *, italic: bool = False) -> etree._Element:
    """Create an OMML run element containing the given text."""
    r = etree.Element(M + "r")
    if italic:
        rPr = etree.SubElement(r, M + "rPr")
        sty = etree.SubElement(rPr, M + "sty")
        sty.set(M + "val", "i")
    t = etree.SubElement(r, M + "t")
    t.set("{http://www.w3.org/XML/1998/namespace}space", "preserve")
    t.text = text if text else ""
    return r


def _tag_local(element) -> str:
    """Strip namespace URI from an element's tag name."""
    tag = element.tag
    return tag.split('}')[1] if '}' in tag else tag


def _simple_text(element) -> str:
    """Return direct text for simple MathML token elements."""
    if len(element):
        return ""
    return (element.text or "").strip()


def _is_fence_mo(element, form: str | None = None) -> bool:
    """Return True for MathML fence operators produced by \\left/\\right."""
    if _tag_local(element) != 'mo':
        return False
    if element.get('fence') != 'true':
        return False
    if form is not None and element.get('form') != form:
        return False
    return bool((element.text or "").strip())


def _append_delimiter(parent_omml, begin: str, end: str, enclosed_children: list) -> None:
    """Append an OMML delimiter around already grouped MathML children."""
    d = etree.SubElement(parent_omml, M + "d")
    dPr = etree.SubElement(d, M + "dPr")
    beg = etree.SubElement(dPr, M + "begChr")
    beg.set(M + "val", begin)
    end_el = etree.SubElement(dPr, M + "endChr")
    end_el.set(M + "val", end)
    e = etree.SubElement(d, M + "e")
    _convert_child_sequence(enclosed_children, e)


def _append_identifier_run(parent_omml, text: str) -> None:
    """Append a compact identifier run instead of spacing letters apart."""
    if not text:
        return
    italic = len(text) == 1 and text.isalpha() and text not in _COMMON_FUNCTIONS
    parent_omml.append(_omml_run(text, italic=italic))


def _append_nary(parent_omml, operator: str, sub_el=None, sup_el=None,
                 body_children: list | None = None) -> None:
    """Append an OMML n-ary operator with optional lower/upper limits."""
    nary = etree.SubElement(parent_omml, M + "nary")
    naryPr = etree.SubElement(nary, M + "naryPr")
    ch = etree.SubElement(naryPr, M + "chr")
    ch.set(M + "val", operator)
    limLoc = etree.SubElement(naryPr, M + "limLoc")
    limLoc.set(M + "val", "undOvr" if operator != "∫" else "subSup")
    sub = etree.SubElement(nary, M + "sub")
    sup = etree.SubElement(nary, M + "sup")
    e = etree.SubElement(nary, M + "e")
    if sub_el is not None:
        _mml_to_omml(sub_el, sub)
    if sup_el is not None:
        _mml_to_omml(sup_el, sup)
    if body_children:
        _convert_child_sequence(body_children, e)


def _append_script(parent_omml, tag: str, base_text: str, sub_el=None, sup_el=None) -> None:
    """Append a subscript/superscript whose base was assembled from text."""
    if tag == 'msup':
        ss = etree.SubElement(parent_omml, M + "sSup")
        etree.SubElement(ss, M + "sSupPr")
        e = etree.SubElement(ss, M + "e")
        sup = etree.SubElement(ss, M + "sup")
        _append_identifier_run(e, base_text)
        if sup_el is not None:
            _mml_to_omml(sup_el, sup)
        return

    if tag == 'msub':
        ss = etree.SubElement(parent_omml, M + "sSub")
        etree.SubElement(ss, M + "sSubPr")
        e = etree.SubElement(ss, M + "e")
        sub = etree.SubElement(ss, M + "sub")
        _append_identifier_run(e, base_text)
        if sub_el is not None:
            _mml_to_omml(sub_el, sub)
        return

    ss = etree.SubElement(parent_omml, M + "sSubSup")
    etree.SubElement(ss, M + "sSubSupPr")
    e = etree.SubElement(ss, M + "e")
    sub = etree.SubElement(ss, M + "sub")
    sup = etree.SubElement(ss, M + "sup")
    _append_identifier_run(e, base_text)
    if sub_el is not None:
        _mml_to_omml(sub_el, sub)
    if sup_el is not None:
        _mml_to_omml(sup_el, sup)


def _nary_base_operator(element) -> str | None:
    """Return the n-ary operator represented by a MathML base element."""
    text = _simple_text(element)
    if text in _NARY_OPERATORS:
        return text
    if len(element) == 1:
        return _nary_base_operator(list(element)[0])
    return None


def _scripted_nary_parts(element) -> tuple[str, object | None, object | None] | None:
    """Return operator, subscript, and superscript for scripted n-ary MathML."""
    tag = _tag_local(element)
    children = list(element)
    if tag not in {'msub', 'msup', 'msubsup'} or not children:
        return None

    op = _nary_base_operator(children[0])
    if not op:
        return None

    sub_el = children[1] if tag in {'msub', 'msubsup'} and len(children) >= 2 else None
    sup_index = 1 if tag == 'msup' else 2
    sup_el = children[sup_index] if tag in {'msup', 'msubsup'} and len(children) > sup_index else None
    return op, sub_el, sup_el


def _append_mathml_tail(element, parent_omml) -> None:
    """Append an element tail in document order when MathML provides one."""
    if element.tail and element.tail.strip():
        parent_omml.append(_omml_run(element.tail.strip()))


def _collect_nary_body(children: list, start: int) -> tuple[list, int]:
    """Collect the operand following a scripted n-ary operator."""
    body: list = []
    depth = 0
    i = start

    while i < len(children):
        child = children[i]
        tag = _tag_local(child)
        text = (child.text or "").strip()

        if body and depth == 0 and tag == 'mo' and text in {'·', '×', '+', '-', '−', '=', '<', '>', '≤', '≥', '≈', ','}:
            break

        body.append(child)

        if tag == 'mo':
            if text in {'(', '[', '{'}:
                depth += 1
            elif text in {')', ']', '}'} and depth > 0:
                depth -= 1

        i += 1
        if body and depth == 0 and tag not in {'mi', 'mn', 'msub', 'msup', 'msubsup', 'mrow', 'mfenced'}:
            # Keep scanning through simple function arguments, but stop before the
            # next explicit binary operator handled above.
            continue

    return body, i


def _convert_child_sequence(children: list, parent_omml) -> None:
    """Convert a child sequence, coalescing adjacent identifiers."""
    i = 0
    while i < len(children):
        child = children[i]
        nary_parts = _scripted_nary_parts(child)
        if nary_parts:
            body, next_i = _collect_nary_body(children, i + 1)
            if body:
                op, sub_el, sup_el = nary_parts
                _append_nary(parent_omml, op, sub_el=sub_el, sup_el=sup_el, body_children=body)
                i = next_i
                continue

        if _tag_local(child) == 'mi' and _simple_text(child):
            parts = [_simple_text(child)]
            j = i + 1
            while j < len(children) and _tag_local(children[j]) == 'mi' and _simple_text(children[j]):
                parts.append(_simple_text(children[j]))
                j += 1
            if j < len(children) and _tag_local(children[j]) in {'msub', 'msup', 'msubsup'}:
                script_children = list(children[j])
                if script_children and _tag_local(script_children[0]) == 'mi' and _simple_text(script_children[0]):
                    base_text = ''.join(parts) + _simple_text(script_children[0])
                    sub_el = None
                    sup_el = None
                    script_tag = _tag_local(children[j])
                    if script_tag in {'msub', 'msubsup'} and len(script_children) >= 2:
                        sub_el = script_children[1]
                    if script_tag == 'msup' and len(script_children) >= 2:
                        sup_el = script_children[1]
                    elif script_tag == 'msubsup' and len(script_children) >= 3:
                        sup_el = script_children[2]
                    _append_script(parent_omml, script_tag, base_text, sub_el=sub_el, sup_el=sup_el)
                    _append_mathml_tail(children[j], parent_omml)
                    i = j + 1
                    continue
            _append_identifier_run(parent_omml, ''.join(parts))
            _append_mathml_tail(children[j - 1], parent_omml)
            i = j
            continue

        _mml_to_omml(child, parent_omml)
        _append_mathml_tail(child, parent_omml)
        i += 1


def _convert_children(mml_el, parent_omml):
    """Recursively convert all children of a MathML element to OMML."""
    if mml_el.text and mml_el.text.strip():
        parent_omml.append(_omml_run(mml_el.text.strip()))
    _convert_child_sequence(list(mml_el), parent_omml)


_MML_PASSTHROUGH_TAGS = frozenset(('mstyle', 'mpadded', 'mphantom'))
_MML_TEXT_TAGS = frozenset(('mn', 'mo', 'mtext'))


def _mml_to_omml(mml_el, parent_omml):
    """Recursively convert a single MathML element to its OMML equivalent."""
    tag = _tag_local(mml_el)
    text = (mml_el.text or "").strip()
    children = list(mml_el)

    if tag == 'math':
        _convert_children(mml_el, parent_omml)

    elif tag == 'semantics':
        for child in children:
            if not _tag_local(child).startswith('annotation'):
                _mml_to_omml(child, parent_omml)
                break

    elif tag == 'mrow':
        if len(children) >= 2 and _is_fence_mo(children[0], 'prefix') and _is_fence_mo(children[-1], 'postfix'):
            _append_delimiter(
                parent_omml,
                (children[0].text or "").strip(),
                (children[-1].text or "").strip(),
                children[1:-1],
            )
        elif children and _is_fence_mo(children[0], 'prefix') and any(_tag_local(child) == 'mtable' for child in children[1:]):
            _append_delimiter(parent_omml, (children[0].text or "").strip(), "", children[1:])
        else:
            _convert_children(mml_el, parent_omml)

    elif tag == 'mi':
        _append_identifier_run(parent_omml, text)

    elif tag in _MML_TEXT_TAGS:
        parent_omml.append(_omml_run(text))

    elif tag == 'mspace':
        parent_omml.append(_omml_run(" "))

    elif tag == 'mfrac':
        f = etree.SubElement(parent_omml, M + "f")
        etree.SubElement(f, M + "fPr")
        num = etree.SubElement(f, M + "num")
        den = etree.SubElement(f, M + "den")
        if len(children) >= 1:
            _mml_to_omml(children[0], num)
        if len(children) >= 2:
            _mml_to_omml(children[1], den)

    elif tag == 'msup':
        ss = etree.SubElement(parent_omml, M + "sSup")
        etree.SubElement(ss, M + "sSupPr")
        e = etree.SubElement(ss, M + "e")
        sup = etree.SubElement(ss, M + "sup")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)
        if len(children) >= 2:
            _mml_to_omml(children[1], sup)

    elif tag == 'msub':
        ss = etree.SubElement(parent_omml, M + "sSub")
        etree.SubElement(ss, M + "sSubPr")
        e = etree.SubElement(ss, M + "e")
        sub = etree.SubElement(ss, M + "sub")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)
        if len(children) >= 2:
            _mml_to_omml(children[1], sub)

    elif tag == 'msubsup':
        ss = etree.SubElement(parent_omml, M + "sSubSup")
        etree.SubElement(ss, M + "sSubSupPr")
        e = etree.SubElement(ss, M + "e")
        sub = etree.SubElement(ss, M + "sub")
        sup = etree.SubElement(ss, M + "sup")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)
        if len(children) >= 2:
            _mml_to_omml(children[1], sub)
        if len(children) >= 3:
            _mml_to_omml(children[2], sup)

    elif tag == 'msqrt':
        rad = etree.SubElement(parent_omml, M + "rad")
        radPr = etree.SubElement(rad, M + "radPr")
        dh = etree.SubElement(radPr, M + "degHide")
        dh.set(M + "val", "1")
        etree.SubElement(rad, M + "deg")
        e = etree.SubElement(rad, M + "e")
        _convert_children(mml_el, e)

    elif tag == 'mroot':
        rad = etree.SubElement(parent_omml, M + "rad")
        etree.SubElement(rad, M + "radPr")
        deg = etree.SubElement(rad, M + "deg")
        e = etree.SubElement(rad, M + "e")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)
        if len(children) >= 2:
            _mml_to_omml(children[1], deg)

    elif tag == 'mfenced':
        d = etree.SubElement(parent_omml, M + "d")
        dPr = etree.SubElement(d, M + "dPr")
        beg = etree.SubElement(dPr, M + "begChr")
        beg.set(M + "val", mml_el.get('open', '('))
        end = etree.SubElement(dPr, M + "endChr")
        end.set(M + "val", mml_el.get('close', ')'))
        e = etree.SubElement(d, M + "e")
        _convert_children(mml_el, e)

    elif tag == 'mtable':
        m_el = etree.SubElement(parent_omml, M + "m")
        etree.SubElement(m_el, M + "mPr")
        for child in children:
            if _tag_local(child) == 'mtr':
                mr = etree.SubElement(m_el, M + "mr")
                for cell in child:
                    if _tag_local(cell) == 'mtd':
                        e = etree.SubElement(mr, M + "e")
                        _convert_children(cell, e)

    elif tag == 'mover':
        acc = etree.SubElement(parent_omml, M + "acc")
        accPr = etree.SubElement(acc, M + "accPr")
        if len(children) >= 2:
            ch = etree.SubElement(accPr, M + "chr")
            ch.set(M + "val", children[1].text if children[1].text else "^")
        e = etree.SubElement(acc, M + "e")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)

    elif tag == 'munder':
        ll = etree.SubElement(parent_omml, M + "limLow")
        etree.SubElement(ll, M + "limLowPr")
        e = etree.SubElement(ll, M + "e")
        lim = etree.SubElement(ll, M + "lim")
        if len(children) >= 1:
            _mml_to_omml(children[0], e)
        if len(children) >= 2:
            _mml_to_omml(children[1], lim)

    elif tag == 'munderover':
        nary = etree.SubElement(parent_omml, M + "nary")
        naryPr = etree.SubElement(nary, M + "naryPr")
        if len(children) >= 1:
            first = children[0]
            op = first.text
            if not op and len(list(first)) > 0:
                op = list(first)[0].text
            if op:
                ch = etree.SubElement(naryPr, M + "chr")
                ch.set(M + "val", op)
        limLoc = etree.SubElement(naryPr, M + "limLoc")
        limLoc.set(M + "val", "undOvr")
        sub = etree.SubElement(nary, M + "sub")
        sup = etree.SubElement(nary, M + "sup")
        e = etree.SubElement(nary, M + "e")
        if len(children) >= 2:
            _mml_to_omml(children[1], sub)
        if len(children) >= 3:
            _mml_to_omml(children[2], sup)

    elif tag in _MML_PASSTHROUGH_TAGS:
        _convert_children(mml_el, parent_omml)

    else:
        if text:
            parent_omml.append(_omml_run(text))
        _convert_children(mml_el, parent_omml)


def latex_to_omml(latex_str: str) -> etree._Element | None:
    """Convert a LaTeX string to an OMML <m:oMath> element, or None on failure."""
    try:
        latex_str = latex_str.strip()
        if not latex_str:
            return None
        mathml_str = latex2mathml.converter.convert(latex_str)
        mathml_str = re.sub(r'\s+xmlns(:[a-z]+)?="[^"]*"', '', mathml_str)
        mml_tree = etree.fromstring(mathml_str.encode('utf-8'))
        omath = etree.Element(M + "oMath")
        _mml_to_omml(mml_tree, omath)
        return omath
    except Exception as exc:
        preview = latex_str[:80] + ('...' if len(latex_str) > 80 else '')
        print(f"  [LaTeX warning] {exc} — expression: {preview}")
        return None


# ═══════════════════════════════════════════════════════════════════════════════
# Inline Content Parsing
# ═══════════════════════════════════════════════════════════════════════════════

_INLINE_RE = re.compile(
    r'(\$\$[\s\S]*?\$\$)'
    r'|(\$(?!\$)[^\$\n]+?\$)'
    r'|(\*\*\*(.+?)\*\*\*)'
    r'|(\*\*(.+?)\*\*)'
    r'|(\*(?!\*)(.+?)(?<!\*)\*)'
    r'|(`[^`\n]+?`)'
)
_TRAILING_MATH_BASE_RE = re.compile(r'([A-Za-z][A-Za-z0-9]*)$')


def _attach_leading_script_to_previous_text(parts: list[dict], latex: str) -> str:
    """Turn text like ``VaR$_{t-1}$`` into one LaTeX expression."""
    if not latex.startswith(('_', '^')) or not parts:
        return latex
    prev = parts[-1]
    if prev.get('type') != 'text':
        return latex
    match = _TRAILING_MATH_BASE_RE.search(prev.get('content', ''))
    if not match:
        return latex

    base = match.group(1)
    prev['content'] = prev['content'][:match.start(1)]
    if not prev['content']:
        parts.pop()
    return f"{base}{latex}"


def parse_inline_content(text: str) -> list[dict]:
    """
    Split text into typed segments for rendering.

    Segment types: text, bold, bold_italic, code, latex_inline, latex_block.
    """
    if not text:
        return []

    parts: list[dict] = []
    last = 0

    for m in _INLINE_RE.finditer(text):
        if m.start() > last:
            parts.append({'type': 'text', 'content': text[last:m.start()]})

        s = m.group(0)
        if s.startswith('$$') and s.endswith('$$'):
            c = s[2:-2].strip()
            if c:
                parts.append({'type': 'latex_block', 'content': c})
        elif s.startswith('$') and s.endswith('$'):
            c = s[1:-1].strip()
            if c:
                c = _attach_leading_script_to_previous_text(parts, c)
                parts.append({'type': 'latex_inline', 'content': c})
        elif s.startswith('***') and s.endswith('***'):
            parts.append({'type': 'bold_italic', 'content': s[3:-3]})
        elif s.startswith('**') and s.endswith('**'):
            parts.append({'type': 'bold', 'content': s[2:-2]})
        elif s.startswith('*') and s.endswith('*') and not s.startswith('**'):
            parts.append({'type': 'italic', 'content': s[1:-1]})
        elif s.startswith('`') and s.endswith('`'):
            parts.append({'type': 'code', 'content': s[1:-1]})

        last = m.end()

    if last < len(text):
        parts.append({'type': 'text', 'content': text[last:]})

    if not parts and text:
        parts.append({'type': 'text', 'content': text})

    return parts


# ═══════════════════════════════════════════════════════════════════════════════
# HTML Table Parsing
# ═══════════════════════════════════════════════════════════════════════════════

def _cell_text(cell_tag) -> str:
    """Extract all text from a BeautifulSoup table cell, collapsing whitespace."""
    fragments: list[str] = []

    def _walk(node):
        if isinstance(node, NavigableString):
            t = re.sub(r'\s+', ' ', str(node))
            if t:
                fragments.append(t)
        elif isinstance(node, Tag):
            if node.name == 'br':
                fragments.append('\n')
            for child in node.children:
                _walk(child)

    _walk(cell_tag)
    return ''.join(fragments).strip()


def parse_html_table(table_html: str) -> tuple[list[dict], int, int]:
    """
    Parse an HTML table into cell data with merge information.

    Returns (cells_data, num_rows, num_cols).
    Each cell dict has: row, col, rowspan, colspan, content (list of segments),
    is_header (bool).
    """
    soup = BeautifulSoup(table_html, 'lxml')
    table = soup.find('table')
    if not table:
        return [], 0, 0

    rows = table.find_all('tr')
    if not rows:
        return [], 0, 0

    num_rows = len(rows)
    grid: dict[tuple[int, int], bool] = {}
    max_cols = 0

    for ri, row in enumerate(rows):
        ci = 0
        for cell in row.find_all(['td', 'th']):
            while (ri, ci) in grid:
                ci += 1
            cs = int(cell.get('colspan', 1))
            rs = int(cell.get('rowspan', 1))
            for r in range(ri, ri + rs):
                for c in range(ci, ci + cs):
                    grid[(r, c)] = True
            ci += cs
        max_cols = max(max_cols, ci)

    cells_data: list[dict] = []
    grid = {}

    for ri, row in enumerate(rows):
        ci = 0
        for cell in row.find_all(['td', 'th']):
            while (ri, ci) in grid:
                ci += 1
            cs = int(cell.get('colspan', 1))
            rs = int(cell.get('rowspan', 1))
            content_text = _cell_text(cell)
            cells_data.append({
                'row': ri,
                'col': ci,
                'rowspan': rs,
                'colspan': cs,
                'content': parse_inline_content(content_text),
                'is_header': cell.name == 'th',
            })
            for r in range(ri, ri + rs):
                for c in range(ci, ci + cs):
                    grid[(r, c)] = True
            ci += cs

    return cells_data, num_rows, max_cols


# ═══════════════════════════════════════════════════════════════════════════════
# Table Rendering in DOCX
# ═══════════════════════════════════════════════════════════════════════════════

def _set_table_borders(table, color: str = TABLE_BORDER_COLOR, size: str = "4"):
    """Apply uniform thin borders to all table edges."""
    tblPr = table._tbl.tblPr
    if tblPr is None:
        tblPr = OxmlElement('w:tblPr')
        table._tbl.insert(0, tblPr)

    for old in tblPr.findall(qn('w:tblBorders')):
        tblPr.remove(old)

    borders = OxmlElement('w:tblBorders')
    for name in ('top', 'left', 'bottom', 'right', 'insideH', 'insideV'):
        b = OxmlElement(f'w:{name}')
        b.set(qn('w:val'), 'single')
        b.set(qn('w:sz'), size)
        b.set(qn('w:space'), '0')
        b.set(qn('w:color'), color)
        borders.append(b)
    tblPr.append(borders)


def _set_cell_shading(cell, hex_color: str):
    """Set the background fill color of a table cell."""
    tcPr = cell._tc.get_or_add_tcPr()
    for old in tcPr.findall(qn('w:shd')):
        tcPr.remove(old)
    shd = OxmlElement('w:shd')
    shd.set(qn('w:val'), 'clear')
    shd.set(qn('w:color'), 'auto')
    shd.set(qn('w:fill'), hex_color)
    tcPr.append(shd)


def _set_cell_margins(cell, dxa: int = TABLE_CELL_MARGIN_DXA):
    """Set uniform internal margins (padding) for a table cell."""
    tcPr = cell._tc.get_or_add_tcPr()
    mar = OxmlElement('w:tcMar')
    for side in ('top', 'left', 'bottom', 'right'):
        el = OxmlElement(f'w:{side}')
        el.set(qn('w:w'), str(dxa))
        el.set(qn('w:type'), 'dxa')
        mar.append(el)
    tcPr.append(mar)


def _set_cell_vertical_alignment(cell, val: str = "center"):
    """Vertically align cell content (top, center, bottom)."""
    tcPr = cell._tc.get_or_add_tcPr()
    va = OxmlElement('w:vAlign')
    va.set(qn('w:val'), val)
    tcPr.append(va)


def _render_segments(para, segments: list[dict], *,
                     font_size=FONT_SIZE_BODY, bold: bool = False,
                     color: RGBColor | None = None):
    """Render a list of inline segments (text, bold, LaTeX, code) into a paragraph."""
    for seg in segments:
        stype = seg['type']

        if stype == 'text':
            run = para.add_run(seg['content'])
            set_run_fonts(run, size=font_size, bold=bold, color=color)

        elif stype == 'bold':
            run = para.add_run(seg['content'])
            set_run_fonts(run, size=font_size, bold=True, color=color)

        elif stype == 'italic':
            run = para.add_run(seg['content'])
            set_run_fonts(run, size=font_size, italic=True, color=color)

        elif stype == 'bold_italic':
            run = para.add_run(seg['content'])
            set_run_fonts(run, size=font_size, bold=True, italic=True, color=color)

        elif stype == 'code':
            run = para.add_run(seg['content'])
            set_run_fonts(run, font_name=FONT_CODE, size=FONT_SIZE_CODE, color=color)

        elif stype in ('latex_inline', 'latex_block'):
            omml = latex_to_omml(seg['content'])
            if omml is not None:
                para._p.append(omml)
            else:
                run = para.add_run(f"${seg['content']}$")
                set_run_fonts(run, size=font_size, italic=True, color=color)


def create_docx_table(doc: Document, cells_data: list[dict],
                      num_rows: int, num_cols: int) -> None:
    """Build a professionally styled DOCX table from parsed HTML cell data."""
    if num_rows == 0 or num_cols == 0:
        return

    table = doc.add_table(rows=num_rows, cols=num_cols)
    table.style = 'Table Grid'

    tblPr = table._tbl.tblPr
    if tblPr is None:
        tblPr = OxmlElement('w:tblPr')
        table._tbl.insert(0, tblPr)
    tblW = OxmlElement('w:tblW')
    tblW.set(qn('w:type'), 'pct')
    tblW.set(qn('w:w'), '5000')
    tblPr.append(tblW)
    jc = OxmlElement('w:jc')
    jc.set(qn('w:val'), 'center')
    tblPr.append(jc)

    _set_table_borders(table)

    tblLayout = OxmlElement('w:tblLayout')
    tblLayout.set(qn('w:type'), 'fixed')
    tblPr.append(tblLayout)

    processed: set = set()

    for cd in cells_data:
        ri, ci = cd['row'], cd['col']
        if (ri, ci) in processed:
            continue

        try:
            cell = table.cell(ri, ci)
        except IndexError:
            continue

        rs, cs = cd['rowspan'], cd['colspan']
        if rs > 1 or cs > 1:
            er = min(ri + rs - 1, num_rows - 1)
            ec = min(ci + cs - 1, num_cols - 1)
            try:
                cell.merge(table.cell(er, ec))
                for r in range(ri, er + 1):
                    for c in range(ci, ec + 1):
                        processed.add((r, c))
            except Exception:
                pass

        processed.add((ri, ci))

        para = cell.paragraphs[0] if cell.paragraphs else cell.add_paragraph()
        para.clear()
        _render_segments(para, cd['content'],
                         font_size=FONT_SIZE_TABLE, bold=cd['is_header'])
        para.alignment = WD_ALIGN_PARAGRAPH.CENTER

        if cd['is_header']:
            _set_cell_shading(cell, TABLE_HEADER_BG)
        elif ri % 2 == 1:
            _set_cell_shading(cell, TABLE_ALT_ROW_BG)

        _set_cell_margins(cell)
        _set_cell_vertical_alignment(cell)

    for row in table.rows:
        row.height = Twips(360)
        row.height_rule = WD_ROW_HEIGHT_RULE.AT_LEAST


# ═══════════════════════════════════════════════════════════════════════════════
# Document Styling & Page Layout
# ═══════════════════════════════════════════════════════════════════════════════

def _setup_styles(doc: Document) -> None:
    """Configure Normal and Heading styles for a professional business look."""
    normal = doc.styles['Normal']
    normal.font.name = FONT_BODY
    normal.font.size = FONT_SIZE_BODY
    normal.font.color.rgb = COLOR_BODY
    normal.paragraph_format.space_after = Pt(8)
    normal.paragraph_format.line_spacing = 1.15
    _set_rfonts(normal.element.get_or_add_rPr(), FONT_BODY)

    for level, size in HEADING_SIZES.items():
        name = f'Heading {level}'
        if name not in doc.styles:
            continue
        hs = doc.styles[name]
        hs.font.name = FONT_BODY
        hs.font.size = Pt(size)
        hs.font.bold = (level <= 5)
        hs.font.color.rgb = COLOR_HEADING
        hs.paragraph_format.space_before = Pt(14)
        hs.paragraph_format.space_after = Pt(8)
        hs.paragraph_format.line_spacing = 1.2
        _set_rfonts(hs.element.get_or_add_rPr(), FONT_BODY)


def _setup_page(doc: Document) -> None:
    """Set A4 page size with narrow margins."""
    for section in doc.sections:
        section.page_width = Cm(21.0)
        section.page_height = Cm(29.7)
        section.left_margin = MARGIN_NARROW
        section.right_margin = MARGIN_NARROW
        section.top_margin = MARGIN_NARROW
        section.bottom_margin = MARGIN_NARROW


# ═══════════════════════════════════════════════════════════════════════════════
# Markdown Element Parsing
# ═══════════════════════════════════════════════════════════════════════════════

_HEADING_RE = re.compile(r'^(#{1,6})\s+(.+)$')
_UL_RE = re.compile(r'^[\-\*\+□☐]\s+(.+)$')
_OL_RE = re.compile(r'^\d+[\.\)]\s+(.+)$')
_IMG_RE = re.compile(r'^!\[([^\]]*)\]\(([^\)]+)\)')
_FENCE_RE = re.compile(r'^(`{3,}|~{3,})\s*(.*)')
_PAGE_BREAK_RE = re.compile(r'^Page_Order_\d+$', re.IGNORECASE)


def parse_markdown_elements(content: str) -> list[dict]:
    """
    Parse preprocessed markdown text into a flat list of typed elements.

    Element types: heading, paragraph, table, latex_block,
                   unordered_list, ordered_list, image, code_block.
    """
    elements: list[dict] = []
    lines = content.split('\n')
    i, n = 0, len(lines)

    while i < n:
        line = lines[i]
        stripped = line.strip()

        if not stripped:
            i += 1
            continue

        # ── Page break sentinel from OCR page markers ──
        if _PAGE_BREAK_RE.match(stripped):
            elements.append({'type': 'page_break'})
            i += 1
            continue

        # ── HTML table ──
        if '<table' in line.lower():
            buf = [line]
            depth = line.lower().count('<table') - line.lower().count('</table>')
            i += 1
            while i < n and depth > 0:
                depth += lines[i].lower().count('<table')
                depth -= lines[i].lower().count('</table>')
                buf.append(lines[i])
                i += 1
            elements.append({'type': 'table', 'content': '\n'.join(buf)})
            continue

        # ── Fenced code block (``` or ~~~) ──
        fence_m = _FENCE_RE.match(stripped)
        if fence_m:
            fence_char = fence_m.group(1)[0]
            fence_len = len(fence_m.group(1))
            lang = fence_m.group(2).strip().lower()
            buf = []
            i += 1
            while i < n:
                close_m = _FENCE_RE.match(lines[i].strip())
                if close_m and close_m.group(1)[0] == fence_char and len(close_m.group(1)) >= fence_len:
                    i += 1
                    break
                buf.append(lines[i])
                i += 1
            code_content = '\n'.join(buf)
            elements.append({'type': 'code_block', 'content': code_content, 'language': lang})
            continue

        # ── Heading ──
        hm = _HEADING_RE.match(stripped)
        if hm:
            elements.append({
                'type': 'heading',
                'level': len(hm.group(1)),
                'content': hm.group(2).strip(),
            })
            i += 1
            continue

        # ── Block LaTeX $$...$$ ──
        if stripped.startswith('$$'):
            if stripped.endswith('$$') and len(stripped) > 4:
                elements.append({'type': 'latex_block', 'content': stripped[2:-2].strip()})
                i += 1
                continue
            buf = [line]
            i += 1
            while i < n:
                buf.append(lines[i])
                if lines[i].strip().endswith('$$'):
                    i += 1
                    break
                i += 1
            joined = '\n'.join(buf)
            m = re.search(r'\$\$([\s\S]*?)\$\$', joined)
            if m:
                elements.append({'type': 'latex_block', 'content': m.group(1).strip()})
            continue

        # ── Unordered list ──
        um = _UL_RE.match(stripped)
        if um:
            items = []
            while i < n:
                raw_line = lines[i]
                ls = raw_line.strip()
                m = _UL_RE.match(ls)
                if m:
                    items.append(m.group(1))
                    i += 1
                elif ls and (raw_line.startswith('  ') or raw_line.startswith('\t')):
                    if items:
                        items[-1] += ' ' + ls
                    i += 1
                else:
                    break
            if items:
                elements.append({'type': 'unordered_list', 'items': items})
            continue

        # ── Ordered list ──
        om = _OL_RE.match(stripped)
        if om:
            items = []
            while i < n:
                ls = lines[i].strip()
                m = _OL_RE.match(ls)
                if not m:
                    break
                items.append(m.group(1))
                i += 1
            if items:
                elements.append({'type': 'ordered_list', 'items': items})
            continue

        # ── Image ──
        im = _IMG_RE.match(stripped)
        if im:
            elements.append({'type': 'image', 'alt': im.group(1), 'src': im.group(2)})
            i += 1
            continue

        # ── Paragraph (may span consecutive non-blank lines) ──
        buf = [line]
        i += 1
        while i < n:
            ns = lines[i].strip()
            if (not ns
                    or ns.startswith('#')
                    or ns.startswith('$$')
                    or _UL_RE.match(ns)
                    or _OL_RE.match(ns)
                    or '<table' in lines[i].lower()
                    or _IMG_RE.match(ns)
                    or _FENCE_RE.match(ns)
                    or _PAGE_BREAK_RE.match(ns)):
                break
            buf.append(lines[i])
            i += 1
        para_text = ' '.join(l.strip() for l in buf if l.strip())
        if para_text:
            elements.append({'type': 'paragraph', 'content': para_text})

    return elements


# ═══════════════════════════════════════════════════════════════════════════════
# Document Building Helpers
# ═══════════════════════════════════════════════════════════════════════════════

def _add_paragraph(doc: Document, text: str) -> None:
    """Add a body paragraph with mixed text / LaTeX / bold / code segments."""
    para = doc.add_paragraph()
    _render_segments(para, parse_inline_content(text))


def _add_heading(doc: Document, text: str, level: int) -> None:
    """Add a heading, supporting LaTeX content within the heading text."""
    segments = parse_inline_content(text)
    has_special = any(s['type'] != 'text' for s in segments)

    if not has_special:
        h = doc.add_heading(text, level=min(level, 6))
        for run in h.runs:
            set_run_fonts(run, size=Pt(HEADING_SIZES.get(level, 11)),
                          bold=(level <= 5), color=COLOR_HEADING)
        return

    para = doc.add_paragraph()
    para.style = f'Heading {min(level, 6)}'
    _render_segments(para, segments,
                     font_size=Pt(HEADING_SIZES.get(level, 11)),
                     bold=(level <= 5), color=COLOR_HEADING)


def _add_block_equation(doc: Document, latex_content: str) -> None:
    """Add a centered block-display LaTeX equation."""
    para = doc.add_paragraph()
    para.alignment = WD_ALIGN_PARAGRAPH.CENTER
    para.paragraph_format.space_before = Pt(6)
    para.paragraph_format.space_after = Pt(6)

    omml = latex_to_omml(latex_content)
    if omml is not None:
        omp = etree.Element(M + "oMathPara")
        ompp = etree.SubElement(omp, M + "oMathParaPr")
        jc = etree.SubElement(ompp, M + "jc")
        jc.set(M + "val", "center")
        omp.append(omml)
        para._p.append(omp)
    else:
        run = para.add_run(f"$${latex_content}$$")
        set_run_fonts(run, font_name=FONT_MATH, italic=True)


def _add_list(doc: Document, items: list[str], ordered: bool = False) -> None:
    """Add ordered or unordered list items, each supporting inline LaTeX/bold."""
    style_name = 'List Number' if ordered else 'List Bullet'
    for item_text in items:
        para = doc.add_paragraph(style=style_name)
        _render_segments(para, parse_inline_content(item_text))


def _add_image(doc: Document, md_dir: Path, alt: str, src: str) -> None:
    """Embed an image if the file exists, otherwise insert a placeholder."""
    img_path = None
    src_p = Path(src)

    if src_p.is_absolute() and src_p.exists():
        img_path = src_p
    else:
        candidate = md_dir / src
        if candidate.exists():
            img_path = candidate

    if img_path and img_path.exists():
        try:
            usable_width = Cm(21.0 - 2 * 1.8)
            para = doc.add_paragraph()
            para.alignment = WD_ALIGN_PARAGRAPH.CENTER
            run = para.add_run()
            run.add_picture(str(img_path), width=usable_width)
            return
        except Exception:
            pass

    para = doc.add_paragraph()
    para.alignment = WD_ALIGN_PARAGRAPH.CENTER
    run = para.add_run(f"[Image: {alt or src}]")
    set_run_fonts(run, size=Pt(10), italic=True, color=COLOR_MUTED)


def _set_paragraph_shading(para, hex_color: str):
    """Apply background shading to a paragraph."""
    pPr = para._p.get_or_add_pPr()
    shd = OxmlElement('w:shd')
    shd.set(qn('w:val'), 'clear')
    shd.set(qn('w:color'), 'auto')
    shd.set(qn('w:fill'), hex_color)
    pPr.append(shd)


def _set_paragraph_borders(para, hex_color: str = CODE_BORDER_COLOR,
                           sides: tuple[str, ...] = ('top', 'left', 'bottom', 'right')):
    """Apply a thin border to specified sides of a paragraph."""
    pPr = para._p.get_or_add_pPr()
    pBdr = OxmlElement('w:pBdr')
    for side in sides:
        b = OxmlElement(f'w:{side}')
        b.set(qn('w:val'), 'single')
        b.set(qn('w:sz'), '4')
        b.set(qn('w:space'), '4')
        b.set(qn('w:color'), hex_color)
        pBdr.append(b)
    pPr.append(pBdr)


def _add_code_block(doc: Document, code: str, language: str = "") -> None:
    """Render a fenced code block with monospace font and shaded background."""
    code_lines = code.split('\n')
    total = len(code_lines)

    for line_idx, code_line in enumerate(code_lines):
        para = doc.add_paragraph()
        para.paragraph_format.space_before = Pt(0)
        para.paragraph_format.space_after = Pt(0)
        para.paragraph_format.line_spacing = 1.0

        _set_paragraph_shading(para, CODE_BG_COLOR)

        sides = ['left', 'right']
        if line_idx == 0:
            sides.append('top')
            para.paragraph_format.space_before = Pt(4)
        if line_idx == total - 1:
            sides.append('bottom')
            para.paragraph_format.space_after = Pt(4)
        _set_paragraph_borders(para, sides=tuple(sides))

        run = para.add_run(code_line if code_line else " ")
        set_run_fonts(run, font_name=FONT_CODE, size=FONT_SIZE_CODE)


# ═══════════════════════════════════════════════════════════════════════════════
# Main Conversion
# ═══════════════════════════════════════════════════════════════════════════════

def markdown_to_docx(markdown_path: Path, output_path: Path) -> None:
    """
    Convert a Markdown file to a professionally styled DOCX document.

    The Markdown may contain:
      - Headers (# through ######)
      - Regular paragraphs with inline LaTeX / bold / code
      - Block LaTeX equations ($$...$$)
      - HTML tables with colspan / rowspan and LaTeX in cells
      - Ordered and unordered lists
      - Fenced code blocks (```language ... ```)
      - Image references ![alt](path)

    OCR artifacts (page separators, page markers) are automatically removed.

    Args:
        markdown_path: Path to the source .md file.
        output_path:   Path for the output .docx file.
    """
    content = Path(markdown_path).read_text(encoding='utf-8')
    content = preprocess_content(content)

    doc = Document()
    _setup_styles(doc)
    _setup_page(doc)

    md_dir = Path(markdown_path).parent
    elements = parse_markdown_elements(content)

    seen_page_break = False
    for elem in elements:
        t = elem['type']

        if t == 'page_break':
            if seen_page_break:
                para = doc.add_paragraph()
                para.add_run().add_break(WD_BREAK.PAGE)
            seen_page_break = True
            continue

        if t == 'heading':
            _add_heading(doc, elem['content'], elem['level'])

        elif t == 'paragraph':
            _add_paragraph(doc, elem['content'])

        elif t == 'table':
            cells, nr, nc = parse_html_table(elem['content'])
            if nr > 0 and nc > 0:
                create_docx_table(doc, cells, nr, nc)
                doc.add_paragraph()

        elif t == 'latex_block':
            _add_block_equation(doc, elem['content'])

        elif t == 'unordered_list':
            _add_list(doc, elem['items'], ordered=False)

        elif t == 'ordered_list':
            _add_list(doc, elem['items'], ordered=True)

        elif t == 'image':
            _add_image(doc, md_dir, elem.get('alt', ''), elem.get('src', ''))

        elif t == 'code_block':
            _add_code_block(doc, elem['content'], elem.get('language', ''))

    output_path.parent.mkdir(parents=True, exist_ok=True)
    doc.save(str(output_path))

    print(f"Converted: {markdown_path} → {output_path}")


# ═══════════════════════════════════════════════════════════════════════════════
# CLI Entry Point
# ═══════════════════════════════════════════════════════════════════════════════

if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser(
        description="Convert Markdown (with HTML tables, LaTeX) to DOCX.",
    )
    parser.add_argument("input", type=Path, help="Path to the source .md file.")
    parser.add_argument(
        "output", nargs="?", type=Path, default=None,
        help="Output .docx path (default: same stem as input).",
    )
    args = parser.parse_args()

    dst = args.output if args.output else args.input.with_suffix('.docx')
    markdown_to_docx(args.input, dst)


path: scripts/ocr_jobs.py
content: """Disk-backed OCR job store and progress reporter."""

from __future__ import annotations

import json
import os
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

from ocr_runtime import ProgressReporter

JOBS_ROOT = Path.home() / ".semantic-ocr" / "jobs"

PIPELINE_STEPS: list[tuple[str, str]] = [
    ("preflight", "Preflight checks"),
    ("render_pages", "Converting document to images"),
    ("ocr_pages", "Extracting page content via VLM"),
    ("build_html", "Generating HTML comparison report"),
    ("build_docx", "Generating DOCX document"),
    ("build_excel", "Generating Excel workbook"),
    ("finalize", "Finalizing outputs"),
]

TERMINAL_STATUSES = {"completed", "failed", "cancelled"}


def utc_now() -> str:
    """Return a compact UTC timestamp suitable for JSON payloads."""
    return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")


def create_job_id() -> str:
    """Create a stable, human-readable OCR job id."""
    timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
    suffix = uuid.uuid4().hex[:6]
    return f"ocr_{timestamp}_{suffix}"


def _atomic_write_json(path: Path, payload: dict[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    tmp_path = path.with_suffix(path.suffix + ".tmp")
    tmp_path.write_text(
        json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )
    tmp_path.replace(path)


def _load_json(path: Path) -> dict[str, Any]:
    if not path.exists():
        return {}
    return json.loads(path.read_text(encoding="utf-8"))


def _build_steps() -> list[dict[str, str]]:
    return [{"id": step_id, "label": label, "status": "pending"} for step_id, label in PIPELINE_STEPS]


def _is_pid_alive(pid: int | None) -> bool:
    if not pid:
        return False
    try:
        os.kill(pid, 0)
    except OSError:
        return False
    return True


class JobStore:
    """Owns the on-disk state for a single OCR background job."""

    def __init__(self, job_dir: str | Path):
        self.job_dir = Path(job_dir).expanduser().resolve()
        self.manifest_path = self.job_dir / "manifest.json"
        self.progress_path = self.job_dir / "progress.json"
        self.events_path = self.job_dir / "events.jsonl"
        self.log_path = self.job_dir / "worker.log"
        self.pid_path = self.job_dir / "worker.pid"
        self.cancel_path = self.job_dir / "cancel.flag"

    @classmethod
    def jobs_root(cls) -> Path:
        JOBS_ROOT.mkdir(parents=True, exist_ok=True)
        return JOBS_ROOT

    @property
    def job_id(self) -> str:
        return self.job_dir.name

    def exists(self) -> bool:
        return self.manifest_path.exists() and self.progress_path.exists()

    def create(self, input_path: str | Path, *, params: dict[str, Any]) -> None:
        input_file = Path(input_path).expanduser().resolve()
        created_at = utc_now()

        manifest = {
            "job_id": self.job_id,
            "version": 1,
            "created_at": created_at,
            "input_path": str(input_file),
            "input_name": input_file.name,
            "params": params,
        }
        progress = {
            "job_id": self.job_id,
            "version": 1,
            "status": "queued",
            "message": "Queued for OCR processing.",
            "error": None,
            "created_at": created_at,
            "started_at": None,
            "finished_at": None,
            "updated_at": created_at,
            "current_step": None,
            "step_label": None,
            "steps": _build_steps(),
            "total_pages": None,
            "processed_pages": 0,
            "failed_pages": 0,
            "completed_pages": [],
            "failed_page_numbers": [],
            "artifacts": {},
            "worker_pid": None,
        }

        self.job_dir.mkdir(parents=True, exist_ok=True)
        _atomic_write_json(self.manifest_path, manifest)
        _atomic_write_json(self.progress_path, progress)
        self.append_event("job_created", input_path=str(input_file))

    def load_manifest(self) -> dict[str, Any]:
        data = _load_json(self.manifest_path)
        if not data:
            raise FileNotFoundError(f"Job manifest not found: {self.manifest_path}")
        return data

    def load_progress(self) -> dict[str, Any]:
        data = _load_json(self.progress_path)
        if not data:
            raise FileNotFoundError(f"Job progress not found: {self.progress_path}")
        return data

    def save_progress(self, progress: dict[str, Any]) -> dict[str, Any]:
        progress["updated_at"] = utc_now()
        _atomic_write_json(self.progress_path, progress)
        return progress

    def append_event(self, event_type: str, **payload: Any) -> None:
        record = {"ts": utc_now(), "type": event_type, **payload}
        self.events_path.parent.mkdir(parents=True, exist_ok=True)
        with self.events_path.open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")

    def snapshot(self, *, refresh_liveness: bool = False) -> dict[str, Any]:
        if refresh_liveness:
            self.refresh_liveness()
        manifest = self.load_manifest()
        progress = self.load_progress()
        artifacts = {
            name: path
            for name, path in progress.get("artifacts", {}).items()
            if Path(path).exists()
        }
        return {
            **progress,
            "input_path": manifest.get("input_path"),
            "input_name": manifest.get("input_name"),
            "params": manifest.get("params", {}),
            "job_dir": str(self.job_dir),
            "artifacts": artifacts,
            "artifacts_ready": sorted(artifacts.keys()),
        }

    def set_worker_pid(self, pid: int) -> None:
        progress = self.load_progress()
        progress["worker_pid"] = pid
        self.pid_path.write_text(f"{pid}\n", encoding="utf-8")
        self.save_progress(progress)

    def mark_started(self, *, pid: int | None = None) -> None:
        progress = self.load_progress()
        progress["status"] = "running"
        progress["message"] = "OCR worker is running."
        progress["error"] = None
        progress["finished_at"] = None
        if progress.get("started_at") is None:
            progress["started_at"] = utc_now()
        if pid is not None:
            progress["worker_pid"] = pid
            self.pid_path.write_text(f"{pid}\n", encoding="utf-8")
        self.save_progress(progress)
        self.append_event("job_started", pid=progress.get("worker_pid"))

    def set_step(self, step_id: str, status: str) -> None:
        progress = self.load_progress()
        for step in progress.get("steps", []):
            if step["id"] == step_id:
                step["status"] = status
                progress["current_step"] = step_id
                progress["step_label"] = step["label"]
                break
        self.save_progress(progress)
        self.append_event("step_updated", step_id=step_id, status=status)

    def record_pages_discovered(self, total_pages: int) -> None:
        progress = self.load_progress()
        progress["total_pages"] = total_pages
        self.save_progress(progress)
        self.append_event("pages_discovered", total_pages=total_pages)

    def _update_page_sets(
        self,
        *,
        completed_page: int | None = None,
        failed_page: int | None = None,
    ) -> dict[str, Any]:
        progress = self.load_progress()
        completed = set(progress.get("completed_pages", []))
        failed = set(progress.get("failed_page_numbers", []))

        if completed_page is not None:
            completed.add(completed_page)
            failed.discard(completed_page)
        if failed_page is not None:
            failed.add(failed_page)
            completed.discard(failed_page)

        progress["completed_pages"] = sorted(completed)
        progress["failed_page_numbers"] = sorted(failed)
        progress["processed_pages"] = len(progress["completed_pages"])
        progress["failed_pages"] = len(progress["failed_page_numbers"])
        return self.save_progress(progress)

    def record_page_done(self, page_num: int, *, skipped: bool = False) -> None:
        self._update_page_sets(completed_page=page_num)
        self.append_event("page_done", page=page_num, skipped=skipped)

    def record_page_failed(self, page_num: int, error: str) -> None:
        self._update_page_sets(failed_page=page_num)
        self.append_event("page_failed", page=page_num, error=error)

    def record_artifact(self, name: str, path: str | Path) -> None:
        artifact_path = str(Path(path).resolve())
        progress = self.load_progress()
        progress.setdefault("artifacts", {})[name] = artifact_path
        self.save_progress(progress)
        self.append_event("artifact_ready", name=name, path=artifact_path)

    def log_message(self, message: str, *, warning: bool = False) -> None:
        progress = self.load_progress()
        progress["message"] = message
        self.save_progress(progress)
        self.append_event("warning" if warning else "log", message=message)

    def mark_completed(self, message: str = "OCR job completed.") -> None:
        progress = self.load_progress()
        progress["status"] = "completed"
        progress["message"] = message
        progress["error"] = None
        progress["finished_at"] = utc_now()
        progress["current_step"] = None
        progress["step_label"] = None
        for step in progress.get("steps", []):
            if step["status"] == "running":
                step["status"] = "completed"
        self.save_progress(progress)
        self.append_event("job_completed")

    def mark_failed(self, error: str) -> None:
        progress = self.load_progress()
        progress["status"] = "failed"
        progress["message"] = "OCR job failed."
        progress["error"] = error
        progress["finished_at"] = utc_now()
        self.save_progress(progress)
        self.append_event("job_failed", error=error)

    def mark_cancel_requested(self) -> None:
        self.cancel_path.touch()
        progress = self.load_progress()
        if progress.get("status") not in TERMINAL_STATUSES:
            progress["status"] = "cancelling"
            progress["message"] = "Cancellation requested."
            self.save_progress(progress)
        self.append_event("cancel_requested")

    def clear_cancel_requested(self) -> None:
        if self.cancel_path.exists():
            self.cancel_path.unlink()

    def is_cancel_requested(self) -> bool:
        return self.cancel_path.exists()

    def mark_cancelled(self) -> None:
        progress = self.load_progress()
        progress["status"] = "cancelled"
        progress["message"] = "OCR job cancelled."
        progress["finished_at"] = utc_now()
        progress["current_step"] = None
        progress["step_label"] = None
        self.save_progress(progress)
        self.append_event("job_cancelled")

    def prepare_for_resume(self) -> None:
        progress = self.load_progress()
        if progress.get("status") == "running":
            raise RuntimeError("Cannot resume a running OCR job.")
        progress["status"] = "queued"
        progress["message"] = "Resume requested."
        progress["error"] = None
        progress["finished_at"] = None
        progress["current_step"] = None
        progress["step_label"] = None
        for step in progress.get("steps", []):
            if step["status"] in {"running", "failed"}:
                step["status"] = "pending"
        self.save_progress(progress)
        self.append_event("resume_requested")

    def refresh_liveness(self) -> None:
        progress = self.load_progress()
        status = progress.get("status")
        if status in TERMINAL_STATUSES:
            return
        pid = progress.get("worker_pid")
        if pid and not _is_pid_alive(pid):
            progress["status"] = "failed"
            progress["message"] = "OCR worker is no longer running."
            progress["error"] = "Background OCR worker exited unexpectedly."
            progress["finished_at"] = utc_now()
            self.save_progress(progress)
            self.append_event("job_failed", error=progress["error"])

    def tail_log(self, lines: int = 100) -> list[str]:
        if not self.log_path.exists():
            return []
        content = self.log_path.read_text(encoding="utf-8", errors="replace").splitlines()
        if lines <= 0:
            return content
        return content[-lines:]


class DiskProgressReporter(ProgressReporter):
    """Writes OCR progress updates into a JobStore."""

    def __init__(self, store: JobStore):
        self.store = store

    def job_started(self) -> None:
        self.store.log_message("OCR pipeline started.")

    def step_started(self, step_id: str, label: str) -> None:
        self.store.set_step(step_id, "running")
        self.store.log_message(label)

    def step_completed(self, step_id: str) -> None:
        self.store.set_step(step_id, "completed")

    def pages_discovered(self, total_pages: int) -> None:
        self.store.record_pages_discovered(total_pages)

    def page_done(self, page_num: int, total_pages: int, *, skipped: bool = False) -> None:
        self.store.record_pages_discovered(total_pages)
        self.store.record_page_done(page_num, skipped=skipped)

    def page_failed(self, page_num: int, total_pages: int, error: str) -> None:
        self.store.record_pages_discovered(total_pages)
        self.store.record_page_failed(page_num, error)

    def artifact_ready(self, name: str, path: str | Path) -> None:
        self.store.record_artifact(name, path)

    def warning(self, message: str) -> None:
        self.store.log_message(message, warning=True)

    def log(self, message: str) -> None:
        self.store.log_message(message)

    def is_cancelled(self) -> bool:
        return self.store.is_cancel_requested()


def list_jobs(*, state: str | None = None) -> list[dict[str, Any]]:
    jobs_root = JobStore.jobs_root()
    rows: list[dict[str, Any]] = []
    for job_dir in sorted((p for p in jobs_root.iterdir() if p.is_dir()), reverse=True):
        store = JobStore(job_dir)
        if not store.exists():
            continue
        snapshot = store.snapshot(refresh_liveness=True)
        if state and snapshot.get("status") != state:
            continue
        rows.append(snapshot)
    return rows


path: scripts/md_to_excel.py
content: """
Markdown folder to Excel converter.

Extracts HTML tables from OCR-generated Markdown page files and writes them into
one formatted `.xlsx` workbook.

Usage:
    python md_to_excel.py /path/to/markdowns /path/to/output.xlsx
"""

from __future__ import annotations

import argparse
from pathlib import Path
import re
from typing import Optional, Tuple, Dict
from bs4 import BeautifulSoup
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
from openpyxl.cell.cell import MergedCell

# For LaTeX to human-readable conversion
try:
    import unicodeit

    HAS_UNICODEIT = True
except ImportError:
    HAS_UNICODEIT = False


def convert_latex_to_readable(text: str) -> str:
    """
    Convert LaTeX expressions in text to human-readable Unicode math formulas.

    Handles both display math ($$...$$) and inline math ($...$).

    Args:
        text: Text that may contain LaTeX expressions

    Returns:
        Text with LaTeX expressions converted to readable Unicode
    """
    if not text or not isinstance(text, str):
        return text

    # Check if there's any LaTeX to convert
    if '$' not in text and '\\' not in text:
        return text

    def convert_single_latex(latex_content: str) -> str:
        """Convert a single LaTeX expression to readable text."""

        # Try unicodeit first if available (best results for scientific notation)
        if HAS_UNICODEIT:
            try:
                result = unicodeit.replace(latex_content)
                # Clean up any remaining LaTeX artifacts
                result = _cleanup_latex_artifacts(result)
                return result.strip()
            except Exception:
                pass

        # Fallback to comprehensive manual conversion
        return _manual_latex_convert(latex_content)

    # Pattern to match display math ($$...$$ and \[...\]) and inline math ($...$ or \(...\))
    # Process display math first ($$...$$ and \[...\])
    display_math_pattern = r'\$\$([\s\S]*?)\$\$'
    bracket_display_pattern = r'\\\[\s*([\s\S]*?)\s*\\\]'

    def replace_display_math(match):
        latex_content = match.group(1)
        converted = convert_single_latex(latex_content)
        return f' {converted} '  # Add spaces for readability

    result = re.sub(display_math_pattern, replace_display_math, text, flags=re.DOTALL)
    result = re.sub(bracket_display_pattern, replace_display_math, result, flags=re.DOTALL)

    # Process inline math ($...$ and \(...\))
    inline_math_pattern = r'\$([^$]+)\$'
    paren_inline_pattern = r'\\\(\s*([^)]*?)\s*\\\)'

    def replace_inline_math(match):
        latex_content = match.group(1)
        return convert_single_latex(latex_content)

    result = re.sub(inline_math_pattern, replace_inline_math, result)
    result = re.sub(paren_inline_pattern, replace_inline_math, result)

    # Normalize superscript/subscript runs for Excel display
    result = _normalize_supsub_for_excel(result)

    # Clean up multiple spaces
    result = re.sub(r' +', ' ', result)
    result = result.strip()

    return result


def _cleanup_latex_artifacts(text: str) -> str:
    """Clean up remaining LaTeX artifacts after conversion."""
    result = text

    # Remove \left and \right
    result = re.sub(r'\\left\s*\.?', '', result)
    result = re.sub(r'\\right\s*\.?', '', result)

    # Remove text formatting commands but keep content
    result = re.sub(r'\\(?:text|mathrm|mathbf|mathit|textbf|textit)\s*\{([^{}]*)\}', r'\1', result)

    # Clean up spacing commands
    result = result.replace(r'\quad', '  ')
    result = result.replace(r'\qquad', '   ')
    result = re.sub(r'\\[,;:!]', ' ', result)

    # Remove remaining unknown commands
    result = re.sub(r'\\[a-zA-Z]+(?:\s*\{[^{}]*\})*', '', result)

    # Clean up braces
    result = result.replace('{', '').replace('}', '')

    return result


def _normalize_supsub_for_excel(text: str) -> str:
    """
    Excel sometimes fails to render sub/superscript letters. Keep numeric runs
    in Unicode (they render well), but rewrite any letter-containing runs to
    ASCII fallbacks like _(...) or ^(...).
    """
    if not text:
        return text

    sub_map = {
        '₀': '0', '₁': '1', '₂': '2', '₃': '3', '₄': '4',
        '₅': '5', '₆': '6', '₇': '7', '₈': '8', '₉': '9',
        '₊': '+', '₋': '-', '₌': '=', '₍': '(', '₎': ')',
        'ₐ': 'a', 'ₑ': 'e', 'ₕ': 'h', 'ᵢ': 'i', 'ⱼ': 'j', 'ₖ': 'k',
        'ₗ': 'l', 'ₘ': 'm', 'ₙ': 'n', 'ₒ': 'o', 'ₚ': 'p', 'ᵣ': 'r',
        'ₛ': 's', 'ₜ': 't', 'ᵤ': 'u', 'ᵥ': 'v', 'ₓ': 'x',
    }
    sup_map = {
        '⁰': '0', '¹': '1', '²': '2', '³': '3', '⁴': '4',
        '⁵': '5', '⁶': '6', '⁷': '7', '⁸': '8', '⁹': '9',
        '⁺': '+', '⁻': '-', '⁼': '=', '⁽': '(', '⁾': ')',
        'ᵃ': 'a', 'ᵇ': 'b', 'ᶜ': 'c', 'ᵈ': 'd', 'ᵉ': 'e', 'ᶠ': 'f',
        'ᵍ': 'g', 'ʰ': 'h', 'ⁱ': 'i', 'ʲ': 'j', 'ᵏ': 'k', 'ˡ': 'l',
        'ᵐ': 'm', 'ⁿ': 'n', 'ᵒ': 'o', 'ᵖ': 'p', 'ʳ': 'r', 'ˢ': 's',
        'ᵗ': 't', 'ᵘ': 'u', 'ᵛ': 'v', 'ʷ': 'w', 'ˣ': 'x', 'ʸ': 'y',
        'ᶻ': 'z', 'ᴬ': 'A', 'ᴮ': 'B', 'ᴰ': 'D', 'ᴱ': 'E', 'ᴳ': 'G',
        'ᴴ': 'H', 'ᴵ': 'I', 'ᴶ': 'J', 'ᴷ': 'K', 'ᴸ': 'L', 'ᴹ': 'M',
        'ᴺ': 'N', 'ᴼ': 'O', 'ᴾ': 'P', 'ᴿ': 'R', 'ᵀ': 'T', 'ᵁ': 'U',
        'ⱽ': 'V', 'ᵂ': 'W',
    }

    sub_letters = set(ch for ch in sub_map if ch.isalpha())
    sup_letters = set(ch for ch in sup_map if ch.isalpha())

    def replace_sub(match):
        run = match.group(1)
        has_letter = any(ch in sub_letters for ch in run)
        if not has_letter:
            return run  # purely numeric/symbolic run renders fine
        plain = ''.join(sub_map.get(ch, '') or ch for ch in run)
        return f'_({plain})' if len(plain) > 1 else f'_{plain}'

    def replace_sup(match):
        run = match.group(1)
        has_letter = any(ch in sup_letters for ch in run)
        if not has_letter:
            return run  # keep numeric-only superscripts
        plain = ''.join(sup_map.get(ch, '') or ch for ch in run)
        return f'^({plain})' if len(plain) > 1 else f'^{plain}'

    # Replace letter-containing sub/superscript runs
    text = re.sub(r'([₀-₉₊₋₌₍₎ₐₑₕᵢⱼₖₗₘₙₒₚᵣₛₜᵤᵥₓ]+)', replace_sub, text)
    text = re.sub(r'([⁰¹²³⁴⁵⁶⁷⁸⁹⁺⁻⁼⁽⁾ᵃᵇᶜᵈᵉᶠᵍʰⁱʲᵏˡᵐⁿᵒᵖʳˢᵗᵘᵛʷˣʸᶻᴬᴮᴰᴱᴳᴴᴵᴶᴷᴸᴹᴺᴼᴾᴿᵀᵁⱽᵂ]+)', replace_sup, text)
    return text


def _manual_latex_convert(latex_content: str) -> str:
    """Comprehensive manual LaTeX to Unicode conversion."""
    result = latex_content

    # ==================== Greek Letters ====================
    greek_lower = {
        r'\alpha': 'α', r'\beta': 'β', r'\gamma': 'γ', r'\delta': 'δ',
        r'\epsilon': 'ε', r'\varepsilon': 'ε', r'\zeta': 'ζ', r'\eta': 'η',
        r'\theta': 'θ', r'\vartheta': 'ϑ', r'\iota': 'ι', r'\kappa': 'κ',
        r'\lambda': 'λ', r'\mu': 'μ', r'\nu': 'ν', r'\xi': 'ξ',
        r'\omicron': 'ο', r'\pi': 'π', r'\varpi': 'ϖ', r'\rho': 'ρ',
        r'\varrho': 'ϱ', r'\sigma': 'σ', r'\varsigma': 'ς', r'\tau': 'τ',
        r'\upsilon': 'υ', r'\phi': 'φ', r'\varphi': 'ϕ', r'\chi': 'χ',
        r'\psi': 'ψ', r'\omega': 'ω',
    }
    greek_upper = {
        r'\Gamma': 'Γ', r'\Delta': 'Δ', r'\Theta': 'Θ', r'\Lambda': 'Λ',
        r'\Xi': 'Ξ', r'\Pi': 'Π', r'\Sigma': 'Σ', r'\Upsilon': 'Υ',
        r'\Phi': 'Φ', r'\Psi': 'Ψ', r'\Omega': 'Ω',
    }

    # ==================== Math Operators ====================
    operators = {
        r'\times': '×', r'\div': '÷', r'\pm': '±', r'\mp': '∓',
        r'\cdot': '·', r'\ast': '∗', r'\star': '⋆', r'\circ': '∘',
        r'\bullet': '•', r'\oplus': '⊕', r'\ominus': '⊖', r'\otimes': '⊗',
    }

    # ==================== Relations ====================
    relations = {
        r'\leq': '≤', r'\le': '≤', r'\geq': '≥', r'\ge': '≥',
        r'\neq': '≠', r'\ne': '≠', r'\approx': '≈', r'\cong': '≅',
        r'\equiv': '≡', r'\sim': '∼', r'\simeq': '≃', r'\propto': '∝',
        r'\ll': '≪', r'\gg': '≫', r'\prec': '≺', r'\succ': '≻',
        r'\preceq': '⪯', r'\succeq': '⪰',
    }

    # ==================== Set Theory ====================
    sets = {
        r'\subset': '⊂', r'\supset': '⊃', r'\subseteq': '⊆', r'\supseteq': '⊇',
        r'\in': '∈', r'\ni': '∋', r'\notin': '∉', r'\cup': '∪', r'\cap': '∩',
        r'\emptyset': '∅', r'\varnothing': '∅', r'\setminus': '∖',
    }

    # ==================== Logic & Quantifiers ====================
    logic = {
        r'\forall': '∀', r'\exists': '∃', r'\nexists': '∄',
        r'\neg': '¬', r'\lnot': '¬', r'\land': '∧', r'\lor': '∨',
        r'\wedge': '∧', r'\vee': '∨', r'\implies': '⟹', r'\iff': '⟺',
    }

    # ==================== Arrows ====================
    arrows = {
        r'\rightarrow': '→', r'\to': '→', r'\leftarrow': '←',
        r'\leftrightarrow': '↔', r'\Rightarrow': '⇒', r'\Leftarrow': '⇐',
        r'\Leftrightarrow': '⇔', r'\mapsto': '↦', r'\uparrow': '↑',
        r'\downarrow': '↓', r'\nearrow': '↗', r'\searrow': '↘',
        r'\longrightarrow': '⟶', r'\longleftarrow': '⟵',
    }

    # ==================== Calculus & Analysis ====================
    calculus = {
        r'\infty': '∞', r'\partial': '∂', r'\nabla': '∇',
        r'\sum': '∑', r'\prod': '∏', r'\coprod': '∐',
        r'\int': '∫', r'\iint': '∬', r'\iiint': '∭', r'\oint': '∮',
        r'\prime': '′', r'\dprime': '″',
    }

    # ==================== Delimiters ====================
    # Use placeholders for literal braces to protect them during cleanup
    LBRACE_PLACEHOLDER = '\x00LBRACE\x00'
    RBRACE_PLACEHOLDER = '\x00RBRACE\x00'

    delimiters = {
        r'\langle': '⟨', r'\rangle': '⟩', r'\lfloor': '⌊', r'\rfloor': '⌋',
        r'\lceil': '⌈', r'\rceil': '⌉', r'\lvert': '|', r'\rvert': '|',
        r'\lVert': '‖', r'\rVert': '‖',
        r'\{': LBRACE_PLACEHOLDER, r'\}': RBRACE_PLACEHOLDER,
        r'\lbrace': LBRACE_PLACEHOLDER, r'\rbrace': RBRACE_PLACEHOLDER,
        r'\lbrack': '[', r'\rbrack': ']',
    }

    # ==================== Miscellaneous ====================
    misc = {
        r'\ldots': '…', r'\cdots': '⋯', r'\vdots': '⋮', r'\ddots': '⋱',
        r'\degree': '°', r'\angle': '∠', r'\measuredangle': '∡',
        r'\perp': '⊥', r'\parallel': '∥', r'\triangle': '△',
        r'\square': '□', r'\diamond': '◇', r'\clubsuit': '♣',
        r'\spadesuit': '♠', r'\heartsuit': '♡', r'\diamondsuit': '♢',
        r'\aleph': 'ℵ', r'\hbar': 'ℏ', r'\ell': 'ℓ', r'\wp': '℘',
        r'\Re': 'ℜ', r'\Im': 'ℑ',
    }

    # ==================== Spacing ====================
    spacing = {
        r'\quad': '  ', r'\qquad': '    ', r'\,': ' ', r'\;': ' ',
        r'\:': ' ', r'\!': '', r'\ ': ' ', r'~': ' ',
    }

    # Apply all symbol replacements (order matters - longer patterns first)
    all_symbols = {}
    all_symbols.update(greek_lower)
    all_symbols.update(greek_upper)
    all_symbols.update(operators)
    all_symbols.update(relations)
    all_symbols.update(sets)
    all_symbols.update(logic)
    all_symbols.update(arrows)
    all_symbols.update(calculus)
    all_symbols.update(delimiters)
    all_symbols.update(misc)
    all_symbols.update(spacing)

    # Sort by length (longest first) to avoid partial replacements
    for latex, unicode_char in sorted(all_symbols.items(), key=lambda x: -len(x[0])):
        result = result.replace(latex, unicode_char)

    # ==================== Handle Complex Structures ====================

    def extract_braced_content(s: str, start: int) -> Tuple[str, int]:
        """Extract content within braces, handling nested braces.
        Returns (content, end_position)."""
        if start >= len(s) or s[start] != '{':
            return '', start
        depth = 0
        content_start = start + 1
        i = start
        while i < len(s):
            if s[i] == '{':
                depth += 1
            elif s[i] == '}':
                depth -= 1
                if depth == 0:
                    return s[content_start:i], i + 1
            i += 1
        return s[content_start:], len(s)

    # Handle fractions: \frac{a}{b} -> (a)/(b)
    def process_fractions(s: str) -> str:
        """Process all \frac commands in the string."""
        result_str = s
        while r'\frac' in result_str:
            idx = result_str.find(r'\frac')
            if idx == -1:
                break

            # Find start of first brace
            brace_start = idx + 5
            while brace_start < len(result_str) and result_str[brace_start] in ' \t':
                brace_start += 1

            if brace_start >= len(result_str) or result_str[brace_start] != '{':
                break

            # Extract numerator
            num, num_end = extract_braced_content(result_str, brace_start)

            # Find start of second brace
            den_start = num_end
            while den_start < len(result_str) and result_str[den_start] in ' \t':
                den_start += 1

            if den_start >= len(result_str) or result_str[den_start] != '{':
                break

            # Extract denominator
            den, den_end = extract_braced_content(result_str, den_start)

            # Recursively process numerator and denominator
            num_converted = process_fractions(num) if r'\frac' in num else num
            den_converted = process_fractions(den) if r'\frac' in den else den

            # Format the fraction
            if len(num_converted) > 1 and not num_converted.replace('.', '').replace('-', '').isalnum():
                num_converted = f'({num_converted})'
            if len(den_converted) > 1 and not den_converted.replace('.', '').replace('-', '').isalnum():
                den_converted = f'({den_converted})'

            fraction_str = f'{num_converted}/{den_converted}'

            # Replace in result
            result_str = result_str[:idx] + fraction_str + result_str[den_end:]

        return result_str

    result = process_fractions(result)

    # Handle square roots: \sqrt{x} -> √(x), \sqrt[n]{x} -> ⁿ√(x)
    def process_sqrt(s: str) -> str:
        """Process all \\sqrt commands."""
        result_str = s
        # Handle \sqrt[n]{x} first
        while r'\sqrt[' in result_str:
            match = re.search(r'\\sqrt\s*\[([^\]]*)\]\s*\{', result_str)
            if not match:
                break
            n_val = match.group(1)
            brace_start = match.end() - 1
            content, end_pos = extract_braced_content(result_str, brace_start)
            replacement = f'{n_val}√({content})'
            result_str = result_str[:match.start()] + replacement + result_str[end_pos:]

        # Handle \sqrt{x}
        while r'\sqrt' in result_str:
            match = re.search(r'\\sqrt\s*\{', result_str)
            if not match:
                # Try \sqrt x (without braces)
                match2 = re.search(r'\\sqrt\s+(\w)', result_str)
                if match2:
                    result_str = result_str[:match2.start()] + f'√{match2.group(1)}' + result_str[match2.end():]
                else:
                    break
                continue
            brace_start = match.end() - 1
            content, end_pos = extract_braced_content(result_str, brace_start)
            replacement = f'√({content})'
            result_str = result_str[:match.start()] + replacement + result_str[end_pos:]

        return result_str

    result = process_sqrt(result)

    # ==================== Superscripts ====================
    superscript_map = {
        '0': '⁰', '1': '¹', '2': '²', '3': '³', '4': '⁴',
        '5': '⁵', '6': '⁶', '7': '⁷', '8': '⁸', '9': '⁹',
        '+': '⁺', '-': '⁻', '=': '⁼', '(': '⁽', ')': '⁾',
        'a': 'ᵃ', 'b': 'ᵇ', 'c': 'ᶜ', 'd': 'ᵈ', 'e': 'ᵉ',
        'f': 'ᶠ', 'g': 'ᵍ', 'h': 'ʰ', 'i': 'ⁱ', 'j': 'ʲ',
        'k': 'ᵏ', 'l': 'ˡ', 'm': 'ᵐ', 'n': 'ⁿ', 'o': 'ᵒ',
        'p': 'ᵖ', 'r': 'ʳ', 's': 'ˢ', 't': 'ᵗ', 'u': 'ᵘ',
        'v': 'ᵛ', 'w': 'ʷ', 'x': 'ˣ', 'y': 'ʸ', 'z': 'ᶻ',
        'A': 'ᴬ', 'B': 'ᴮ', 'D': 'ᴰ', 'E': 'ᴱ', 'G': 'ᴳ',
        'H': 'ᴴ', 'I': 'ᴵ', 'J': 'ᴶ', 'K': 'ᴷ', 'L': 'ᴸ',
        'M': 'ᴹ', 'N': 'ᴺ', 'O': 'ᴼ', 'P': 'ᴾ', 'R': 'ᴿ',
        'T': 'ᵀ', 'U': 'ᵁ', 'V': 'ⱽ', 'W': 'ᵂ',
    }

    def convert_superscript(content: str) -> str:
        """Convert content to superscript Unicode if possible."""
        # First, process any nested ^n patterns within the content
        processed = content
        # Handle simple nested superscripts like q^2 -> q²
        for char, sup in superscript_map.items():
            if len(char) == 1 and char.isdigit():
                processed = re.sub(rf'\^{char}(?![0-9])', sup, processed)

        converted = []
        i = 0
        while i < len(processed):
            char = processed[i]
            if char in superscript_map:
                converted.append(superscript_map[char])
            elif char == ' ':
                converted.append(' ')
            elif char == '/':
                converted.append('/')  # Keep division in superscript
            elif char in '²³⁴⁵⁶⁷⁸⁹⁰¹⁺⁻':
                converted.append(char)  # Already converted superscript
            else:
                return f'^({processed})'  # Can't fully convert, use fallback
            i += 1
        return ''.join(converted)

    # Handle ^{...} and ^x patterns
    def replace_superscript(match):
        content = match.group(1) if match.group(1) else match.group(2)
        return convert_superscript(content)

    result = re.sub(r'\^{([^{}]*)}|\^(\w)', replace_superscript, result)

    # ==================== Subscripts ====================
    subscript_map = {
        '0': '₀', '1': '₁', '2': '₂', '3': '₃', '4': '₄',
        '5': '₅', '6': '₆', '7': '₇', '8': '₈', '9': '₉',
        '+': '₊', '-': '₋', '=': '₌', '(': '₍', ')': '₎',
    }

    def convert_subscript(content: str) -> str:
        """Convert content to subscript Unicode where broadly supported; otherwise keep ASCII."""
        # If the subscript contains letters, stick to ASCII so Excel never drops glyphs
        contains_letters = any(ch.isalpha() for ch in content)
        if contains_letters:
            return f'_({content})' if len(content) > 1 else f'_{content}'

        converted = []
        for char in content:
            if char in subscript_map:
                converted.append(subscript_map[char])
            else:
                # Unknown symbol: fall back to readable ASCII
                return f'_({content})' if len(content) > 1 else f'_{content}'

        return ''.join(converted)

    # Handle _{...} and _x patterns
    def replace_subscript(match):
        content = match.group(1) if match.group(1) else match.group(2)
        return convert_subscript(content)

    result = re.sub(r'_{([^{}]*)}|_(\w)', replace_subscript, result)

    # ==================== Math Functions ====================
    math_functions = [
        'sin', 'cos', 'tan', 'cot', 'sec', 'csc',
        'arcsin', 'arccos', 'arctan', 'arccot',
        'sinh', 'cosh', 'tanh', 'coth',
        'log', 'ln', 'lg', 'exp',
        'lim', 'sup', 'inf', 'max', 'min',
        'det', 'dim', 'ker', 'deg', 'arg', 'gcd', 'lcm',
        'mod', 'bmod', 'pmod',
    ]
    for func in math_functions:
        result = re.sub(rf'\\{func}\b', func, result)

    # ==================== Accents/Modifiers ====================
    # Use postfix modifier characters instead of combining diacritics for Excel compatibility
    # Combining characters don't display well in Excel cells
    accent_patterns = [
        # Hat/circumflex: use modifier letter circumflex (ˆ U+02C6) as postfix
        (r'\\hat\s*\{([^{}]+)\}', r'\1ˆ'),
        (r'\\hat\s*(\w)', r'\1ˆ'),
        # Bar/overline: use macron (¯ U+00AF) as postfix
        (r'\\bar\s*\{([^{}]+)\}', r'\1¯'),
        (r'\\bar\s*(\w)', r'\1¯'),
        (r'\\overline\s*\{([^{}]*)\}', r'\1¯'),
        # Tilde: use small tilde (˜ U+02DC) as postfix
        (r'\\tilde\s*\{([^{}]+)\}', r'\1˜'),
        (r'\\tilde\s*(\w)', r'\1˜'),
        # Dot: use middle dot (· U+00B7) as superscript-like postfix
        (r'\\dot\s*\{([^{}]+)\}', r'\1̇'),
        (r'\\dot\s*(\w)', r'\1̇'),
        # Double dot: use diaeresis (¨ U+00A8) as postfix
        (r'\\ddot\s*\{([^{}]+)\}', r'\1¨'),
        (r'\\ddot\s*(\w)', r'\1¨'),
        # Vector: use arrow (→) as postfix
        (r'\\vec\s*\{([^{}]+)\}', r'\1→'),
        (r'\\vec\s*(\w)', r'\1→'),
        # Underline: use low line
        (r'\\underline\s*\{([^{}]*)\}', r'_\1_'),
    ]
    for pattern, replacement in accent_patterns:
        result = re.sub(pattern, replacement, result)

    # ==================== Text Formatting ====================
    result = re.sub(r'\\(?:text|mathrm|mathbf|mathit|textbf|textit|mbox|hbox)\s*\{([^{}]*)\}', r'\1', result)
    result = re.sub(r'\\mathbb\s*\{([^{}]*)\}', r'\1', result)  # Could map to double-struck if needed
    result = re.sub(r'\\mathcal\s*\{([^{}]*)\}', r'\1', result)
    result = re.sub(r'\\mathfrak\s*\{([^{}]*)\}', r'\1', result)

    # ==================== Delimiters ====================
    result = re.sub(r'\\left\s*([(\[{|.])', r'\1', result)
    result = re.sub(r'\\right\s*([)\]}|.])', r'\1', result)
    result = result.replace(r'\left.', '').replace(r'\right.', '')
    result = result.replace(r'\bigl', '').replace(r'\bigr', '')
    result = result.replace(r'\Bigl', '').replace(r'\Bigr', '')
    result = result.replace(r'\biggl', '').replace(r'\biggr', '')
    result = result.replace(r'\Biggl', '').replace(r'\Biggr', '')

    # ==================== Clean Up ====================
    # Remove any remaining backslash commands
    result = re.sub(r'\\[a-zA-Z]+\s*', '', result)

    # Clean up structural braces (not literal braces)
    result = result.replace('{', '').replace('}', '')

    # Restore literal braces from placeholders
    result = result.replace(LBRACE_PLACEHOLDER, '{').replace(RBRACE_PLACEHOLDER, '}')

    # Clean up multiple spaces
    result = re.sub(r' +', ' ', result)

    return result.strip()


def contains_math_unicode(text: str) -> bool:
    """
    Check if text contains Unicode mathematical symbols that need special font support.

    Args:
        text: Text to check

    Returns:
        True if text contains special Unicode math characters
    """
    if not text or not isinstance(text, str):
        return False

    # Check for combining diacritical marks (like hat, bar, tilde accents)
    # Range U+0300 to U+036F
    for char in text:
        code = ord(char)
        # Latin-1 Supplement superscripts: ¹²³ (U+00B2, U+00B3, U+00B9)
        if char in '¹²³':
            return True
        # Latin-1 Supplement: ± × ÷ ¯ ¨ (operators and modifiers)
        if char in '±×÷¯¨':
            return True
        # Modifier letters: ˆ ˜ (U+02C6, U+02DC) used for hat and tilde
        if char in 'ˆ˜':
            return True
        # Combining diacritical marks
        if 0x0300 <= code <= 0x036F:
            return True
        # Greek letters (U+0370-U+03FF)
        if 0x0370 <= code <= 0x03FF:
            return True
        # Superscripts and subscripts (U+2070-U+209F)
        if 0x2070 <= code <= 0x209F:
            return True
        # General punctuation (includes prime ′)
        if 0x2032 <= code <= 0x2037:
            return True
        # Mathematical operators (U+2200-U+22FF)
        if 0x2200 <= code <= 0x22FF:
            return True
        # Miscellaneous mathematical symbols (U+2A00-U+2AFF)
        if 0x2A00 <= code <= 0x2AFF:
            return True
        # Letterlike symbols (U+2100-U+214F)
        if 0x2100 <= code <= 0x214F:
            return True
        # Arrows (U+2190-U+21FF)
        if 0x2190 <= code <= 0x21FF:
            return True
        # Modifier letters (superscript/subscript letters)
        if 0x1D00 <= code <= 0x1DBF:
            return True
        # Latin Extended-D (more subscript chars)
        if 0xA720 <= code <= 0xA7FF:
            return True

    return False


def contains_latex_markup(text: str) -> bool:
    """
    Lightweight check for LaTeX-style markup so we can switch to a math-friendly font
    even if the conversion produces mostly ASCII characters.
    """
    if not text or not isinstance(text, str):
        return False
    return bool(re.search(r'(\$[^$]*\$)|\\[a-zA-Z]+', text, flags=re.DOTALL))


def markdown_folder_to_excel(markdown_folder: Path, excel_path: Path) -> None:
    """
    Extract HTML tables from markdown files and export to a formatted Excel file.

    Args:
        markdown_folder: Path to folder containing .md files named like XXXXX_00001.md
        excel_path: Path for the output Excel file
    """

    # ==================== Helper Functions ====================

    def parse_filename(filepath: Path) -> Tuple[Optional[str], Optional[int]]:
        """Extract report name and page number from filename like 'Report_Name_00001.md'."""
        match = re.match(r"(.+)_(\d+)\.md$", filepath.name)
        if match:
            return match.group(1), int(match.group(2))
        return None, None

    def convert_to_number(value: str):
        """
        Convert string to int or float if it represents a number.
        Float numbers are rounded to 2 decimal places.
        """
        if not isinstance(value, str):
            return value

        original = value.strip()
        if not original:
            return original

        # Remove common formatting: currency symbols, commas, spaces, parentheses for negatives
        cleaned = original

        # Handle negative numbers in parentheses: (123.45) -> -123.45
        is_negative_parens = cleaned.startswith('(') and cleaned.endswith(')')
        if is_negative_parens:
            cleaned = cleaned[1:-1]

        # Remove currency symbols and thousand separators
        cleaned = re.sub(r'[$€£¥₹,\s]', '', cleaned)

        # Handle percentage
        is_percent = cleaned.endswith('%')
        if is_percent:
            cleaned = cleaned[:-1]

        # Handle trailing negative sign
        if cleaned.endswith('-'):
            cleaned = '-' + cleaned[:-1]

        try:
            if '.' in cleaned:
                num = float(cleaned)
            else:
                # Check if it's actually an integer
                num = int(cleaned)
                if is_negative_parens:
                    num = -num
                if is_percent:
                    return round(num / 100, 2)
                return num

            if is_negative_parens:
                num = -num
            if is_percent:
                num = num / 100

            # Round floats to 2 decimal places
            return round(num, 2)
        except ValueError:
            return original

    def estimate_text_width(value) -> float:
        """Estimate the display width of a cell value."""
        if value is None:
            return 0
        text = str(value)
        # Rough estimation: normal chars = 1, wide chars = 1.5
        width = 0
        for char in text:
            if ord(char) > 255:  # Non-ASCII (likely CJK)
                width += 1.8
            else:
                width += 1
        return width

    # ==================== Get and Sort Files ====================

    md_files = list(markdown_folder.glob("*.md"))

    files_with_info = []
    for f in md_files:
        name, page_num = parse_filename(f)
        if page_num is not None:
            files_with_info.append((f, name, page_num))

    # Sort by page number
    files_with_info.sort(key=lambda x: x[2])

    if not files_with_info:
        raise ValueError(f"No valid markdown files found in {markdown_folder}")

    # ==================== Setup Workbook and Styles ====================

    wb = Workbook()
    ws = wb.active
    ws.title = "Extracted Tables"

    # Color definitions (Business Dark Blue theme)
    BUSINESS_DARK_BLUE = "1F4E79"
    WHITE = "FFFFFF"
    HIGHLIGHT_GOLD = "FFD966"
    GRAY_TEXT = "666666"
    LIGHT_GRAY_BORDER = "BFBFBF"

    # Style objects
    # Use a wide-coverage font for mathematical Unicode symbols
    # Arial Unicode MS covers most sub/superscripts better than Cambria Math in cells
    MATH_FONT_NAME = "Arial Unicode MS"

    header_fill = PatternFill(start_color=BUSINESS_DARK_BLUE, end_color=BUSINESS_DARK_BLUE, fill_type="solid")
    header_font = Font(color=WHITE, bold=True, size=10)
    header_font_math = Font(name=MATH_FONT_NAME, color=WHITE, bold=True, size=10)
    header_alignment = Alignment(horizontal='center', vertical='center', wrap_text=True)

    white_fill = PatternFill(start_color=WHITE, end_color=WHITE, fill_type="solid")

    page_header_fill = PatternFill(start_color=HIGHLIGHT_GOLD, end_color=HIGHLIGHT_GOLD, fill_type="solid")
    page_header_font = Font(bold=True, size=11, color="000000")

    no_table_font = Font(italic=True, color=GRAY_TEXT, size=10)

    # Font for cells containing mathematical Unicode
    math_font = Font(name=MATH_FONT_NAME, size=10)
    math_font_bold = Font(name=MATH_FONT_NAME, bold=True, size=10)

    cell_border = Border(
        left=Side(style='thin', color=LIGHT_GRAY_BORDER),
        right=Side(style='thin', color=LIGHT_GRAY_BORDER),
        top=Side(style='thin', color=LIGHT_GRAY_BORDER),
        bottom=Side(style='thin', color=LIGHT_GRAY_BORDER)
    )

    left_align = Alignment(horizontal='left', vertical='center', wrap_text=True)
    right_align = Alignment(horizontal='right', vertical='center')

    # Track column widths across all tables
    col_max_widths: Dict[int, float] = {}

    current_row = 1

    # ==================== Table Writing Function ====================

    def write_table(table_html: str, start_row: int) -> Tuple[int, int]:
        """
        Parse an HTML table and write it to the worksheet.

        Returns:
            Tuple of (next_available_row, max_column_used)
        """
        soup = BeautifulSoup(table_html, 'html.parser')
        table = soup.find('table')
        if not table:
            return start_row, 1

        all_rows = table.find_all('tr')
        if not all_rows:
            return start_row, 1

        # Identify header rows (rows in <thead> or rows containing only <th> elements)
        thead = table.find('thead')
        thead_trs = set(thead.find_all('tr')) if thead else set()

        # Grid to track cells occupied by rowspan
        # Key: (local_row_offset, column) -> True if occupied
        occupied: Dict[Tuple[int, int], bool] = {}

        # Track merge operations to apply styling to merged cells later
        merge_info = []  # List of (start_row, start_col, end_row, end_col, is_header, value)

        local_row = 0
        max_col_used = 1

        for row_idx, tr in enumerate(all_rows):
            cells = tr.find_all(['th', 'td'])

            # Determine if this is a header row
            is_header_row = (
                    tr in thead_trs or
                    all(cell.name == 'th' for cell in cells)
            )

            col = 1

            for cell_elem in cells:
                # Skip columns occupied by previous rowspans
                while (local_row, col) in occupied:
                    col += 1

                # Get span attributes
                colspan = int(cell_elem.get('colspan', 1))
                rowspan = int(cell_elem.get('rowspan', 1))

                # Extract and convert cell value
                raw_text = cell_elem.get_text(strip=True)
                had_latex_markup = contains_latex_markup(raw_text)
                # Convert LaTeX expressions to human-readable format
                raw_text = convert_latex_to_readable(raw_text)
                value = convert_to_number(raw_text)

                # Determine if this specific cell should be styled as header
                is_header_cell = is_header_row or cell_elem.name == 'th'

                # Calculate Excel positions
                excel_row = start_row + local_row
                excel_col = col

                # Write the cell value
                cell = ws.cell(row=excel_row, column=excel_col, value=value)

                # Check if value contains math Unicode characters
                has_math = contains_math_unicode(str(value)) if value else False
                if not has_math and had_latex_markup:
                    # Even if conversion is mostly ASCII, ensure math font for better fidelity
                    has_math = True

                # Apply base styling
                cell.border = cell_border

                if is_header_cell:
                    cell.fill = header_fill
                    cell.font = header_font_math if has_math else header_font
                    cell.alignment = header_alignment
                else:
                    # Use math font for cells with mathematical symbols
                    if has_math:
                        cell.font = math_font
                    if isinstance(value, (int, float)):
                        cell.alignment = right_align
                    else:
                        cell.alignment = left_align

                # Track column width
                text_width = estimate_text_width(value)
                if excel_col not in col_max_widths:
                    col_max_widths[excel_col] = 0
                col_max_widths[excel_col] = max(col_max_widths[excel_col], text_width)

                # Handle cell merging
                if colspan > 1 or rowspan > 1:
                    end_excel_row = excel_row + rowspan - 1
                    end_excel_col = excel_col + colspan - 1

                    # Store merge info for styling
                    merge_info.append((
                        excel_row, excel_col,
                        end_excel_row, end_excel_col,
                        is_header_cell
                    ))

                    # Merge the cells
                    ws.merge_cells(
                        start_row=excel_row,
                        start_column=excel_col,
                        end_row=end_excel_row,
                        end_column=end_excel_col
                    )

                    # Mark cells as occupied for rowspan tracking
                    for r_offset in range(rowspan):
                        for c_offset in range(colspan):
                            if r_offset > 0 or c_offset > 0:
                                occupied[(local_row + r_offset, col + c_offset)] = True

                max_col_used = max(max_col_used, col + colspan - 1)
                col += colspan

            local_row += 1

        # Apply borders to all cells in merged ranges
        for (sr, sc, er, ec, is_hdr) in merge_info:
            for r in range(sr, er + 1):
                for c in range(sc, ec + 1):
                    cell = ws.cell(row=r, column=c)
                    if not isinstance(cell, MergedCell):
                        cell.border = cell_border
                        if is_hdr:
                            cell.fill = header_fill

        return start_row + local_row, max_col_used

    # ==================== Process Each File ====================

    total_files = len(files_with_info)

    for file_idx, (filepath, report_name, page_num) in enumerate(files_with_info):
        # Read file content
        try:
            content = filepath.read_text(encoding='utf-8')
        except UnicodeDecodeError:
            try:
                content = filepath.read_text(encoding='latin-1')
            except Exception as e:
                print(f"Warning: Could not read {filepath}: {e}")
                continue
        except Exception as e:
            print(f"Warning: Could not read {filepath}: {e}")
            continue

        # Find all HTML tables using regex (handles nested tags)
        table_pattern = r'<table[^>]*>.*?</table>'
        tables = re.findall(table_pattern, content, re.DOTALL | re.IGNORECASE)

        # Write page header with highlighting
        page_header_cell = ws.cell(
            row=current_row,
            column=1,
            value=f"Tables from Page {page_num}"
        )
        page_header_cell.fill = page_header_fill
        page_header_cell.font = page_header_font
        page_header_cell.alignment = Alignment(horizontal='left', vertical='center')
        current_row += 1

        if not tables:
            # No tables found on this page
            no_table_cell = ws.cell(
                row=current_row,
                column=1,
                value="(This page does not have explicit tables)"
            )
            no_table_cell.font = no_table_font
            current_row += 1
        else:
            # Process each table on the page
            for table_idx, table_html in enumerate(tables):
                current_row, _ = write_table(table_html, current_row)

                # Add 2 empty lines between tables (not after the last table)
                if table_idx < len(tables) - 1:
                    current_row += 2

        # Add 3 blank lines between pages (not after the last page)
        if file_idx < total_files - 1:
            current_row += 3

    # ==================== Post-Processing ====================

    max_row = ws.max_row
    max_col = ws.max_column

    # Set white background for ALL cells to hide grid lines
    for row in range(1, max_row + 50):  # Add some buffer rows
        for col in range(1, max(max_col + 10, 20)):  # Add some buffer columns
            cell = ws.cell(row=row, column=col)
            if not isinstance(cell, MergedCell):
                current_fill = cell.fill
                # Only apply white fill if cell doesn't have a custom fill
                if (current_fill.patternType is None or
                        current_fill.fgColor.rgb in ('00000000', None)):
                    cell.fill = white_fill

    # Adjust column widths (elegant but not too wide)
    MAX_COL_WIDTH = 45
    MIN_COL_WIDTH = 8
    DEFAULT_WIDTH = 12

    for col in range(1, max_col + 1):
        if col in col_max_widths:
            # Add padding and apply constraints
            width = col_max_widths[col] + 3
            width = max(MIN_COL_WIDTH, min(width, MAX_COL_WIDTH))
        else:
            width = DEFAULT_WIDTH

        ws.column_dimensions[get_column_letter(col)].width = width

    # Set elegant row heights
    DEFAULT_ROW_HEIGHT = 18
    for row in range(1, max_row + 1):
        ws.row_dimensions[row].height = DEFAULT_ROW_HEIGHT

    # ==================== Save Workbook ====================

    excel_path.parent.mkdir(parents=True, exist_ok=True)
    wb.save(excel_path)

    print(f"Successfully extracted tables to: {excel_path}")
    print(f"  - Processed {total_files} page(s)")
    print(f"  - Output has {max_row} rows and {max_col} columns")


def _build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Convert OCR Markdown page files with HTML tables into one Excel workbook.",
    )
    parser.add_argument(
        "markdown_folder",
        type=Path,
        help="Directory containing per-page Markdown files such as report_00001.md.",
    )
    parser.add_argument(
        "output",
        type=Path,
        help="Output .xlsx file path.",
    )
    return parser


if __name__ == "__main__":
    args = _build_parser().parse_args()
    markdown_folder_to_excel(args.markdown_folder, args.output)


path: scripts/md_to_html.py
content: """
OCR Markdown (with HTML tables) → Side-by-side HTML Comparison Report
with Mermaid Diagram Rendering

Generates ONE self-contained HTML file that shows:
- Left:  source scanned page image (embedded as base64 data URL)
- Right: rendered OCR-extracted Markdown (OCR tables are already HTML <table> blocks)

Mermaid ```mermaid``` fenced code blocks are rendered as interactive SVG
diagrams via mermaid.js.

The output HTML is fully self-contained by embedding bundled local
MathJax and Mermaid assets inline. No network requests are performed
during report generation or while viewing the generated report.

File naming convention:
    Images:   <name>_<page>.png|jpeg|jpg|webp|gif   e.g. 陕国投年报_00001.png
    Markdown: <name>_<page>.md|markdown              e.g. 陕国投年报_00001.md
    where <name> is an arbitrary-length stem (may contain underscores) and
    <page> is a zero-padded page number (typically 5 digits).

Cross-platform: works on macOS, Windows, and Linux.

Dependencies: markdown2
Bundled locally: MathJax 3.2.2 (tex-svg-full), Mermaid 11.14.0
"""

from __future__ import annotations

import argparse
import base64
import html
import re
from pathlib import Path

import markdown2


# ==============================================================================
# Filename Parsing / Scanning Utilities
# ==============================================================================

_PATTERN_FILENAME = re.compile(
    r"^(?P<base>.+)_(?P<num>\d+)\.(?P<ext>[^.]+)$",
    re.IGNORECASE,
)

_IMG_EXTS = {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".tif", ".tiff"}
_MD_EXTS = {".md", ".markdown"}

_EXT_PRIORITY = {
    ".png": 0,
    ".webp": 1,
    ".jpg": 2,
    ".jpeg": 3,
    ".gif": 4,
    ".md": 0,
    ".markdown": 1,
}


def _escape_attr(text: str) -> str:
    return html.escape(text, quote=True)


def _escape_text(text: str) -> str:
    return html.escape(text, quote=False)


def _scan_pages_by_base_and_num(folder: Path, allowed_exts: set) -> dict[tuple[str, str], Path]:
    """
    Scan a directory for files like XXXXX_00001.png / XXXXX_00001.md

    Returns:
        mapping[(base, page_num_str)] = Path
    """
    mapping: dict[tuple[str, str], Path] = {}
    if not folder.exists():
        return mapping

    for file_path in folder.iterdir():
        if not file_path.is_file():
            continue

        suffix = file_path.suffix.lower()
        if suffix not in allowed_exts:
            continue

        m = _PATTERN_FILENAME.match(file_path.name)
        if not m:
            continue

        base_name = m.group("base")
        num = m.group("num")

        key = (base_name, num)
        if key not in mapping:
            mapping[key] = file_path
        else:
            cur = mapping[key]
            cur_pri = _EXT_PRIORITY.get(cur.suffix.lower(), 999)
            new_pri = _EXT_PRIORITY.get(suffix, 999)
            if new_pri < cur_pri:
                mapping[key] = file_path

    return mapping


def _sorted_keys(keys: list[tuple[str, str]]) -> list[tuple[str, str]]:
    def _k(item: tuple[str, str]) -> tuple[str, int, str]:
        base_name, num = item
        try:
            n = int(num)
        except ValueError:
            n = 10**18
        return (base_name.lower(), n, num)

    return sorted(keys, key=_k)


# ==============================================================================
# Image Processing
# ==============================================================================

_MIME_MAP = {
    ".png": "image/png",
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".webp": "image/webp",
    ".gif": "image/gif",
    ".bmp": "image/bmp",
    ".tif": "image/tiff",
    ".tiff": "image/tiff",
}


def _image_to_base64_data_url(img_path: Path) -> str:
    mime = _MIME_MAP.get(img_path.suffix.lower(), "image/png")
    b64 = base64.b64encode(img_path.read_bytes()).decode("ascii")
    return f"data:{mime};base64,{b64}"


# ==============================================================================
# Markdown Preprocessing
# ==============================================================================

_FIGURE_PLACEHOLDER_LINE = re.compile(r"^\s*!\[(?P<desc>.*?)\]\s*$")


def _ensure_blank_lines_around_tables(md: str) -> str:
    """
    Ensure blank lines surround <table>...</table> blocks so that
    Markdown parsers treat them as block-level HTML (avoids <p><table>…).
    """
    lines = md.splitlines()
    out: list[str] = []
    i = 0
    while i < len(lines):
        line = lines[i]
        stripped = line.strip().lower()

        if stripped.startswith("<table>") or stripped.startswith("<table "):
            if out and out[-1].strip() != "":
                out.append("")
            out.append(line)

            if "</table>" not in stripped:
                i += 1
                while i < len(lines):
                    out.append(lines[i])
                    if "</table>" in lines[i].strip().lower():
                        i += 1
                        break
                    i += 1
            else:
                i += 1

            if i < len(lines) and out and out[-1].strip() != "":
                out.append("")
            continue

        out.append(line)
        i += 1

    return "\n".join(out)


def _convert_ocr_figure_placeholders(md: str) -> str:
    """
    OCR prompt uses a non-standard placeholder line:
      ![Concise description ≤20 words]
    Render it human-readably without requiring a URL.
    """
    out_lines: list[str] = []
    for line in md.splitlines():
        m = _FIGURE_PLACEHOLDER_LINE.match(line)
        if m:
            desc = m.group("desc")
            out_lines.append(f'> **Figure:** {desc}')
        else:
            out_lines.append(line)
    return "\n".join(out_lines)


def _normalize_indentation(markdown_text: str) -> str:
    """
    Prevent accidental code blocks from OCR-added indentation.

    Additionally: if a line looks like an HTML table tag but is indented 4+ spaces,
    de-indent it to avoid Markdown treating it as code.
    """
    lines = markdown_text.split("\n")
    result: list[str] = []
    in_fenced_code = False

    for line in lines:
        stripped = line.strip()

        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fenced_code = not in_fenced_code
            result.append(line)
            continue

        if in_fenced_code:
            result.append(line)
            continue

        indent_match = re.match(r"^( {4,}|\t+)", line)
        if indent_match:
            content = line.lstrip()

            if not content:
                result.append(line)
                continue

            if re.match(r"^</?(table|thead|tbody|tr|th|td)\b", content.strip(), re.IGNORECASE):
                result.append(content)
                continue

            if _is_likely_prose(content):
                result.append(content)
            else:
                result.append(line)
        else:
            result.append(line)

    return "\n".join(result)


_RE_NON_LATIN = re.compile(
    r"["
    r"\u0400-\u04FF"    # Cyrillic
    r"\u0600-\u06FF"    # Arabic
    r"\u0900-\u097F"    # Devanagari
    r"\u0980-\u09FF"    # Bengali
    r"\u0A00-\u0A7F"    # Gurmukhi
    r"\u0E00-\u0E7F"    # Thai
    r"\u1000-\u109F"    # Myanmar
    r"\u3040-\u309F"    # Hiragana
    r"\u30A0-\u30FF"    # Katakana
    r"\u4E00-\u9FFF"    # CJK Unified Ideographs
    r"\uAC00-\uD7AF"    # Hangul Syllables
    r"]"
)


_PROSE_PATTERNS = [re.compile(p) for p in (
    r"^注[*\d\s]*[:：]",
    r"^(资料|数据)?来源[:：]",
    r"^[备说]注[:：]",
    r"^[（\(][一二三四五六七八九十\d]+[）\)]",
    r"^[一二三四五六七八九十]+[、.]",
    r"^[\d]+[、.\)）]\s*\S",
    r"^[•·\-\*]\s+\S",
    r"^\*+\s*\S",
    r"^##+\s",
    r"^[A-Z][a-z].*[.!?]$",
)]

_CODE_PATTERNS = [re.compile(p, re.IGNORECASE) for p in (
    r"^(def|class|import|from|if|for|while|return|function|var|let|const|public|private)\s",
    r"^[{}\[\]<>]",
    r"[{}\[\];]$",
    r"^//",
    r"^\s*@\w+",
    r"^(SELECT|INSERT|UPDATE|DELETE|CREATE|DROP)\s",
    r"^\$\w+",
    r"^<\?",
    r"^<!",
)]

_RE_LETTERS = re.compile(r"[a-zA-Z]")


def _is_likely_prose(content: str) -> bool:
    """Heuristic: return True if *content* looks like natural-language prose rather than code."""
    if re.search(r"(?<!\\)\$", content):
        return True
    if re.search(r"\\\(|\\\)|\\\[|\\\]", content):
        return True

    has_non_latin = bool(_RE_NON_LATIN.search(content))

    if any(p.match(content) for p in _PROSE_PATTERNS):
        return True
    if any(p.search(content) for p in _CODE_PATTERNS):
        return False
    if has_non_latin:
        return True

    letter_ratio = len(_RE_LETTERS.findall(content)) / max(len(content), 1)
    return letter_ratio > 0.6


# ==============================================================================
# Math Protection (for Markdown conversion stability)
# ==============================================================================

def _protect_math_expressions(md: str) -> tuple[str, list[tuple[str, str]]]:
    """
    Shield LaTeX math ($...$ and $$...$$) from Markdown parsing, then restore as HTML
    wrappers so MathJax can reliably typeset.

    Important: do NOT match escaped dollars (\\$) used for currency/literals.
    """
    math_blocks: list[tuple[str, str]] = []
    protected = md

    def _ph(i: int) -> str:
        return f"\uE000MATH{i}\uE001"

    display_re = re.compile(r"(?<!\\)\$\$(.+?)(?<!\\)\$\$", re.DOTALL)
    matches = list(display_re.finditer(protected))
    for m in reversed(matches):
        idx = len(math_blocks)
        placeholder = _ph(idx)
        original = m.group(0)
        escaped = html.escape(original, quote=False)
        html_elem = f'<div class="math-display">{escaped}</div>'
        math_blocks.append((placeholder, html_elem))
        protected = protected[: m.start()] + placeholder + protected[m.end() :]

    inline_re = re.compile(r"(?<!\\)\$(?!\$)([^\$\n]+?)(?<!\\)\$")
    matches = list(inline_re.finditer(protected))
    for m in reversed(matches):
        idx = len(math_blocks)
        placeholder = _ph(idx)
        original = m.group(0)
        escaped = html.escape(original, quote=False)
        html_elem = f'<span class="math-inline">{escaped}</span>'
        math_blocks.append((placeholder, html_elem))
        protected = protected[: m.start()] + placeholder + protected[m.end() :]

    display_bracket_re = re.compile(r"^\\\[\s*\n(.*?)\n\s*\\\]\s*$", re.DOTALL | re.MULTILINE)
    matches = list(display_bracket_re.finditer(protected))
    for m in reversed(matches):
        idx = len(math_blocks)
        placeholder = _ph(idx)
        original = m.group(0)
        escaped = html.escape(original, quote=False)
        html_elem = f'<div class="math-display">{escaped}</div>'
        math_blocks.append((placeholder, html_elem))
        protected = protected[: m.start()] + placeholder + protected[m.end() :]

    inline_paren_re = re.compile(r"\\\((.+?)\\\)")
    matches = list(inline_paren_re.finditer(protected))
    for m in reversed(matches):
        idx = len(math_blocks)
        placeholder = _ph(idx)
        original = m.group(0)
        escaped = html.escape(original, quote=False)
        html_elem = f'<span class="math-inline">{escaped}</span>'
        math_blocks.append((placeholder, html_elem))
        protected = protected[: m.start()] + placeholder + protected[m.end() :]

    return protected, math_blocks


def _restore_math_expressions(html_content: str, math_blocks: list[tuple[str, str]]) -> str:
    for placeholder, html_elem in math_blocks:
        escaped_ph = html.escape(placeholder, quote=False)
        if html_elem.startswith("<div"):
            html_content = html_content.replace(f"<p>{placeholder}</p>", html_elem)
            html_content = html_content.replace(f"<p>{escaped_ph}</p>", html_elem)
        html_content = html_content.replace(placeholder, html_elem)
        html_content = html_content.replace(escaped_ph, html_elem)
    return html_content


# ==============================================================================
# Table Post-processing (wrapping + optional numeric classification + widths)
# ==============================================================================

def _get_text_display_length(text: str) -> float:
    """Approximate display width: wide characters (CJK, fullwidth, etc.) count as 1.8."""
    length = 0.0
    for char in text:
        cp = ord(char)
        if (
            0x4E00 <= cp <= 0x9FFF      # CJK Unified Ideographs
            or 0x3040 <= cp <= 0x309F    # Hiragana
            or 0x30A0 <= cp <= 0x30FF    # Katakana
            or 0xAC00 <= cp <= 0xD7AF    # Hangul Syllables
            or 0xFF00 <= cp <= 0xFFEF    # Fullwidth Forms
            or 0x3000 <= cp <= 0x303F    # CJK Symbols and Punctuation
            or 0x2E80 <= cp <= 0x2EFF    # CJK Radicals Supplement
            or 0xF900 <= cp <= 0xFAFF    # CJK Compatibility Ideographs
        ):
            length += 1.8
        else:
            length += 1.0
    return length


def _extract_cell_contents(table_html: str) -> tuple[int, list[list[str]]]:
    rows: list[list[str]] = []
    row_matches = re.findall(r"<tr[^>]*>(.*?)</tr>", table_html, re.DOTALL | re.IGNORECASE)

    for row_html in row_matches:
        cells: list[str] = []
        cell_matches = re.findall(r"<(?:th|td)[^>]*>(.*?)</(?:th|td)>", row_html, re.DOTALL | re.IGNORECASE)
        for cell_content in cell_matches:
            plain = re.sub(r"<[^>]+>", "", cell_content)
            plain = html.unescape(plain).strip()
            cells.append(plain)
        if cells:
            rows.append(cells)

    col_count = max((len(r) for r in rows), default=0)
    return col_count, rows


def _calculate_column_widths(col_count: int, rows: list[list[str]]) -> list[float]:
    if col_count == 0:
        return []

    effective: list[float] = []
    for c in range(col_count):
        lens: list[float] = []
        for r in rows:
            if c < len(r):
                t = r[c].strip()
                if t:
                    lens.append(_get_text_display_length(t))
        if lens:
            avg_len = sum(lens) / len(lens)
            max_len = max(lens)
            eff = avg_len * 0.7 + max_len * 0.3
        else:
            eff = 2.0
        effective.append(eff)

    total = sum(effective) or col_count
    raw = [(e / total) * 100 for e in effective]

    if col_count >= 5:
        min_w = 50 / col_count
        max_w = 250 / col_count
    else:
        min_w = 10
        max_w = 70

    clamped = [max(min_w, min(max_w, w)) for w in raw]
    s = sum(clamped)
    return [(w / s) * 100 for w in clamped] if s > 0 else [100 / col_count] * col_count


def _apply_column_widths_to_table(table_html: str, widths: list[float]) -> str:
    if not widths:
        return table_html
    col_elements = "".join(f'<col style="width:{w:.2f}%">' for w in widths)
    colgroup = f"<colgroup>{col_elements}</colgroup>"
    return re.sub(r"(<table[^>]*>)", r"\1" + colgroup, table_html, count=1, flags=re.IGNORECASE)


def _enhance_tables(html_content: str) -> str:
    def process_table(m: re.Match) -> str:
        table_html = m.group(0)
        col_count, rows = _extract_cell_contents(table_html)

        wrapper_open = '<div class="table-wrapper">'
        wrapper_close = "</div>"

        if col_count > 0:
            widths = _calculate_column_widths(col_count, rows)
            table_html2 = _apply_column_widths_to_table(table_html, widths)
            table_html2 = re.sub(
                r"<table([^>]*)>",
                rf'<table\1 data-cols="{col_count}">',
                table_html2,
                count=1,
                flags=re.IGNORECASE,
            )
        else:
            table_html2 = table_html

        return f"{wrapper_open}{table_html2}{wrapper_close}"

    return re.sub(r"<table[^>]*>.*?</table>", process_table, html_content, flags=re.DOTALL | re.IGNORECASE)


def _classify_numeric_cells(html_content: str) -> str:
    def classify(m: re.Match) -> str:
        tag = m.group(1)
        attrs = m.group(2) or ""
        content = m.group(3)

        if re.match(r"^[\s\-+$€¥£]?\(?[\d,]+(?:\.\d+)?\)?%?$", content.strip()):
            if 'class="' in attrs:
                attrs = re.sub(r'class="([^"]*)"', r'class="\1 numeric"', attrs)
            else:
                attrs = f'{attrs} class="numeric"'

        return f"<{tag}{attrs}>{content}</{tag}>"

    return re.sub(r"<(td|th)([^>]*)>([^<]*)</\1>", classify, html_content, flags=re.IGNORECASE)


# ==============================================================================
# JS Library Embedding (self-contained HTML — no external requests)
# ==============================================================================

_SCRIPT_DIR = Path(__file__).resolve().parent
_BUNDLED_JS_DIR = _SCRIPT_DIR / "vendor"
_BUNDLED_JS_FILES = {
    "mermaid.min.js": _BUNDLED_JS_DIR / "mermaid-11.14.0" / "mermaid.min.js",
    "tex-svg-full.js": _BUNDLED_JS_DIR / "mathjax-3.2.2" / "tex-svg-full.js",
}
# Embedded company logo so generated HTML stays self-contained even if repo assets are removed.
_COMPANY_LOGO_DATA_URL = (
    'data:image/webp;base64,UklGRio/AABXRUJQVlA4IB4/AADwyQCdASpEAtMAPjEWikOiISESScXkIAMEs7d6PRn4Kp07w1EZj'
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
    [REDACTED_SECRET]
)

def _load_bundled_js(filename: str) -> str:
    """Return bundled JS content for inline embedding."""
    asset_path = _BUNDLED_JS_FILES[filename]
    if not asset_path.is_file():
        raise FileNotFoundError(
            f"Bundled JavaScript asset not found: {asset_path}. "
            "Reinstall or repackage the skill with scripts/vendor included."
        )
    return asset_path.read_text(encoding="utf-8")


def _safe_inline_js(js_content: str) -> str:
    r"""Escape sequences that would break an inline <script> block."""
    return (
        js_content
        .replace("</script>", r"<\/script>")
        .replace("</Script>", r"<\/Script>")
        # Vendored MathJax embeds legacy jsDelivr URL strings for optional SRE assets.
        # Neutralize them so generated HTML contains no third-party CDN references.
        .replace("https://cdn.jsdelivr.net/", "data:text/plain,")
    )


def _get_company_logo_data_url() -> str | None:
    """
    Return the company logo as a data URL.

    Priority:
      1) scripts/logo.png (hardcoded into HTML at generation time)
      2) built-in embedded fallback
    """
    candidate_paths = [
        _SCRIPT_DIR / "logo.png",
        _SCRIPT_DIR.parent / "logo.png",
    ]
    for logo_png in candidate_paths:
        if logo_png.exists() and logo_png.is_file():
            try:
                return _image_to_base64_data_url(logo_png)
            except Exception:
                continue
    return _COMPANY_LOGO_DATA_URL


# ==============================================================================
# Mermaid Block Protection / Restoration
# ==============================================================================

_MERMAID_FENCE_RE = re.compile(
    r"^(`{3,}|~{3,})\s*mermaid\s*\n(.*?)^\1\s*$",
    re.DOTALL | re.MULTILINE,
)


def _protect_mermaid_blocks(md: str) -> tuple[str, list[tuple[str, str]]]:
    """
    Extract ```mermaid ... ``` blocks, replace with PUA placeholders
    so they survive Markdown parsing intact.
    """
    blocks: list[tuple[str, str]] = []

    def _replace(m: re.Match) -> str:
        idx = len(blocks)
        ph = f"\uE000MERMAID{idx}\uE001"
        code = m.group(2).strip()
        blocks.append((ph, code))
        return ph

    protected = _MERMAID_FENCE_RE.sub(_replace, md)
    return protected, blocks


def _restore_mermaid_blocks(html_content: str, blocks: list[tuple[str, str]]) -> str:
    """Replace mermaid placeholders with <pre class="mermaid"> elements."""
    for ph, code in blocks:
        mermaid_html = (
            '<div class="mermaid-wrapper">'
            f'<pre class="mermaid">{html.escape(code)}</pre>'
            '</div>'
        )
        html_content = html_content.replace(f"<p>{ph}</p>", mermaid_html)
        html_content = html_content.replace(ph, mermaid_html)
        escaped_ph = html.escape(ph, quote=False)
        html_content = html_content.replace(f"<p>{escaped_ph}</p>", mermaid_html)
        html_content = html_content.replace(escaped_ph, mermaid_html)
    return html_content


# ==============================================================================
# Markdown → HTML Conversion (with Mermaid support)
# ==============================================================================

def _convert_markdown_to_html(raw_md: str) -> str:
    md = raw_md or ""
    md = _normalize_indentation(md)
    md = _convert_ocr_figure_placeholders(md)
    md = _ensure_blank_lines_around_tables(md)

    md, mermaid_blocks = _protect_mermaid_blocks(md)
    protected, math_blocks = _protect_math_expressions(md)

    processor = markdown2.Markdown(
        extras=[
            "fenced-code-blocks",
            "code-friendly",
            "cuddled-lists",
            "header-ids",
            "strike",
            "tables",
            "task_list",
        ]
    )

    html_content = processor.convert(protected)
    html_content = _restore_math_expressions(html_content, math_blocks)
    html_content = _restore_mermaid_blocks(html_content, mermaid_blocks)
    html_content = _enhance_tables(html_content)
    html_content = _classify_numeric_cells(html_content)
    return html_content


# ==============================================================================
# CSS / JS
# ==============================================================================

def _get_css_styles() -> str:
    return r''':root {
  --bg-primary: #eef3f7;
  --bg-secondary: #f7fbff;
  --surface: rgba(255, 255, 255, 0.72);
  --surface-strong: rgba(255, 255, 255, 0.88);
  --surface-soft: rgba(255, 255, 255, 0.58);
  --text: #0e1a26;
  --text-secondary: #314457;
  --muted: #536171;
  --brand-primary: #1e6ee8;
  --brand-secondary: #00c2b2;
  --brand-soft: rgba(30, 110, 232, 0.10);
  --brand-soft-2: rgba(0, 194, 178, 0.12);
  --border: rgba(14, 26, 38, 0.12);
  --border-strong: rgba(30, 110, 232, 0.18);
  --glass-stroke: rgba(255, 255, 255, 0.42);
  --shadow: 0 22px 48px rgba(8, 28, 61, 0.14);
  --shadow-soft: 0 10px 28px rgba(8, 28, 61, 0.10);
  --radius-xl: 22px;
  --radius-lg: 18px;
  --radius-md: 14px;
  --radius-sm: 10px;
  --maxw: 1680px;
  --header-maxw: 1160px;
  --code-bg: rgba(16, 33, 54, 0.05);
  --code-border: rgba(17, 54, 93, 0.12);
  --code-surface: linear-gradient(180deg, #1d2d43 0%, #142133 100%);
  --font-ui: "Avenir Next", "Segoe UI Variable", "IBM Plex Sans", "Helvetica Neue", Arial, sans-serif;
  --font-display: "Avenir Next", "Segoe UI Variable", "Trebuchet MS", sans-serif;
  --font-code: "JetBrains Mono", "SFMono-Regular", Menlo, Monaco, Consolas, monospace;
}

*, *::before, *::after { box-sizing: border-box; }

html, body {
  margin: 0;
  padding: 0;
  min-height: 100%;
}

body {
  font-family: var(--font-ui);
  font-size: 14px;
  line-height: 1.6;
  color: var(--text);
  background:
    radial-gradient(900px 560px at 8% 12%, rgba(0, 194, 178, 0.12), transparent 55%),
    radial-gradient(960px 620px at 88% -8%, rgba(30, 110, 232, 0.18), transparent 52%),
    linear-gradient(180deg, var(--bg-secondary) 0%, var(--bg-primary) 100%);
  background-attachment: fixed;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  overflow-x: hidden;
}

.blob {
  position: fixed;
  border-radius: 50%;
  filter: blur(88px);
  opacity: 0.58;
  pointer-events: none;
  z-index: 0;
}

.blob {
  width: 560px;
  height: 560px;
  right: -140px;
  top: -120px;
  background: linear-gradient(135deg, rgba(30, 110, 232, 0.30), rgba(0, 194, 178, 0.16));
  animation: float1 18s ease-in-out infinite;
}

.blob-2 {
  width: 520px;
  height: 520px;
  left: -120px;
  bottom: -180px;
  background: linear-gradient(135deg, rgba(0, 194, 178, 0.26), rgba(30, 110, 232, 0.16));
  animation: float2 22s ease-in-out infinite;
}

@keyframes float1 {
  0%, 100% { transform: translate(0, 0) scale(1); }
  50% { transform: translate(-18px, 22px) scale(1.05); }
}

@keyframes float2 {
  0%, 100% { transform: translate(0, 0) scale(1); }
  50% { transform: translate(22px, -16px) scale(0.96); }
}

@keyframes logoSheen {
  0% { transform: translateX(-120%) rotate(16deg); opacity: 0; }
  22% { opacity: 0.55; }
  100% { transform: translateX(320%) rotate(16deg); opacity: 0; }
}

.wrap {
  position: relative;
  z-index: 1;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
}

header {
  position: sticky;
  top: 0;
  z-index: 20;
  width: 100%;
  padding: 7px clamp(16px, 3vw, 30px);
  background: rgba(255, 255, 255, 0.44);
  border-bottom: 1px solid var(--glass-stroke);
  -webkit-backdrop-filter: blur(18px) saturate(180%);
  backdrop-filter: blur(18px) saturate(180%);
}

.header-content {
  max-width: var(--header-maxw);
  margin: 0 auto;
  display: flex;
  flex-direction: column;
  align-items: center;
  gap: 10px;
  padding: 0;
}

.brand {
  display: inline-flex;
  align-items: center;
  gap: 12px;
  text-decoration: none;
  color: inherit;
  min-width: 0;
}

.logo {
  width: 222px;
  height: 76px;
  display: flex;
  align-items: center;
  justify-content: center;
  padding: 0 18px;
  border-radius: 18px;
  background: linear-gradient(180deg, rgba(255, 255, 255, 0.97), rgba(244, 249, 255, 0.92));
  border: 1px solid rgba(255, 255, 255, 0.72);
  box-shadow: 0 14px 34px rgba(8, 28, 61, 0.11),
              0 1px 0 rgba(255, 255, 255, 0.82) inset;
  overflow: hidden;
  position: relative;
  transition: transform 0.2s cubic-bezier(0.4, 0, 0.2, 1),
              box-shadow 0.2s cubic-bezier(0.4, 0.0, 0.2, 1);
}

.logo::before {
  content: "";
  position: absolute;
  inset: -40% auto -40% -70%;
  width: 48%;
  transform: rotate(16deg);
  background: linear-gradient(90deg, transparent, rgba(255, 255, 255, 0.45), transparent);
  animation: logoSheen 2.8s ease-out 0.15s 1;
  pointer-events: none;
}

.logo:hover {
  transform: scale(1.03);
  box-shadow: 0 18px 42px rgba(8, 28, 61, 0.14),
              0 1px 0 rgba(255, 255, 255, 0.88) inset;
}

.logo img {
  display: block;
  width: 100%;
  height: 100%;
  object-fit: contain;
  object-position: center;
  transform: scale(1.08);
  transform-origin: center;
  opacity: 0;
  transition: opacity 0.56s ease, transform 0.72s cubic-bezier(0.22, 1, 0.36, 1);
}

.logo img.is-loaded {
  opacity: 1;
  transform: scale(1);
}

.logo.logo-error img {
  display: none;
}

.logo.logo-error::after {
  content: "";
  display: block;
  width: 84%;
  height: 18px;
  border-radius: 999px;
  background: linear-gradient(135deg, var(--brand-primary), var(--brand-secondary));
}

.logo-fallback {
  display: block;
  width: 100%;
  height: 22px;
  border-radius: 999px;
  background: linear-gradient(135deg, var(--brand-primary), var(--brand-secondary));
}

.brand-name {
  font-family: var(--font-display);
  font-size: clamp(20px, 2.4vw, 24px);
  font-weight: 700;
  line-height: 1.05;
  letter-spacing: 0.02em;
  background: linear-gradient(135deg, var(--brand-primary), var(--brand-secondary));
  -webkit-background-clip: text;
  -webkit-text-fill-color: transparent;
  background-clip: text;
}

.domain {
  margin-top: 4px;
  color: var(--muted);
  font-size: 12px;
  font-weight: 600;
  letter-spacing: 0.06em;
  text-transform: uppercase;
}

.subtitle-badge {
  display: inline-flex;
  align-items: center;
  justify-content: center;
  padding: 8px 20px;
  border-radius: 9999px;
  background: linear-gradient(135deg, rgba(255, 255, 255, 0.94), rgba(236, 246, 255, 0.92));
  box-shadow: 0 10px 24px rgba(8, 28, 61, 0.09);
  border: 1px solid rgba(30, 110, 232, 0.10);
  backdrop-filter: blur(8px);
  -webkit-backdrop-filter: blur(8px);
  font-size: 12px;
  letter-spacing: 0.02em;
  transition: transform 0.2s ease;
}

.subtitle-badge:hover {
  transform: translateY(-1px);
  box-shadow: 0 12px 28px rgba(8, 28, 61, 0.12);
}

.subtitle-badge .text {
  font-family: "Avenir Next", "Segoe UI Variable", "IBM Plex Sans", "Helvetica Neue", Arial, sans-serif;
  font-size: 12px;
  font-weight: 600;
  letter-spacing: 0.045em;
  text-transform: none;
  color: #29506f;
  background: linear-gradient(135deg, #2c5f88 0%, #157d91 100%);
  -webkit-background-clip: text;
  -webkit-text-fill-color: transparent;
  background-clip: text;
}

main {
  flex: 1;
  width: 100%;
  max-width: var(--maxw);
  margin: 0 auto;
  padding: 30px clamp(16px, 3vw, 32px) 44px;
}

.page-section {
  position: relative;
  margin-bottom: 24px;
  overflow: hidden;
  border-radius: var(--radius-xl);
  background: var(--surface);
  border: 1px solid var(--glass-stroke);
  box-shadow: var(--shadow);
  -webkit-backdrop-filter: blur(18px) saturate(160%);
  backdrop-filter: blur(18px) saturate(160%);
}

.page-section::before {
  content: "";
  position: absolute;
  inset: 0;
  background:
    radial-gradient(circle at top right, rgba(0, 194, 178, 0.12), transparent 28%),
    radial-gradient(circle at bottom left, rgba(30, 110, 232, 0.08), transparent 32%);
  pointer-events: none;
}

.page-header {
  position: relative;
  z-index: 1;
  display: flex;
  align-items: center;
  gap: 12px;
  padding: 14px 24px;
  background: linear-gradient(135deg, var(--brand-primary), var(--brand-secondary));
  color: white;
  font-size: 13px;
  font-weight: 700;
  letter-spacing: 0.09em;
  text-transform: uppercase;
  border-radius: 0 0 18px 18px;
  box-shadow: 0 4px 14px rgba(8, 28, 61, 0.11);
}

.page-indicator {
  width: 10px;
  height: 10px;
  border-radius: 50%;
  background: white;
  box-shadow: 0 0 0 6px rgba(255, 255, 255, 0.18);
  flex: 0 0 auto;
}

.page-content {
  position: relative;
  z-index: 1;
  display: grid;
  grid-template-columns: minmax(320px, 39%) minmax(0, 1fr);
  gap: 18px;
  padding: 18px;
  align-items: start;
}

.image-panel {
  position: sticky;
  top: 104px;
  align-self: start;
  overflow: hidden;
  border-radius: var(--radius-lg);
  border: 1px solid var(--border);
  background: linear-gradient(180deg, rgba(255, 255, 255, 0.96), rgba(244, 250, 255, 0.90));
  box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.72), var(--shadow-soft);
}

.image-panel img {
  display: block;
  width: 100%;
  height: auto;
  object-fit: contain;
  background: white;
}

.image-missing {
  padding: 20px;
  color: var(--muted);
  font-size: 13px;
}

.markdown-panel {
  min-height: 0;
  padding: 18px 20px;
  border-radius: var(--radius-lg);
  border: 1px solid rgba(14, 26, 38, 0.08);
  background: linear-gradient(180deg, rgba(255, 255, 255, 0.94), rgba(248, 251, 255, 0.92));
  box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.84);
}

.markdown-content {
  color: var(--text);
  font-family: var(--font-ui);
  font-size: 14px;
  line-height: 1.72;
  word-wrap: break-word;
  overflow-wrap: break-word;
}

.markdown-content h1,
.markdown-content h2,
.markdown-content h3,
.markdown-content h4,
.markdown-content h5,
.markdown-content h6 {
  margin: 1.45em 0 0.6em;
  font-family: var(--font-display);
  line-height: 1.25;
  color: var(--text);
}

.markdown-content h1:first-child,
.markdown-content h2:first-child,
.markdown-content h3:first-child {
  margin-top: 0;
}

.markdown-content h1 {
  font-size: 1.9em;
  font-weight: 800;
  letter-spacing: -0.02em;
  padding-bottom: 0.28em;
  border-bottom: 1px solid rgba(30, 110, 232, 0.14);
}

.markdown-content h2 {
  font-size: 1.5em;
  font-weight: 700;
  padding-bottom: 0.24em;
  border-bottom: 1px solid rgba(14, 26, 38, 0.08);
}

.markdown-content h3 { font-size: 1.24em; font-weight: 700; }
.markdown-content h4 { font-size: 1.08em; font-weight: 700; }
.markdown-content h5 { font-size: 1em; font-weight: 700; }
.markdown-content h6 { font-size: 0.95em; color: var(--text-secondary); }

.markdown-content p { margin: 0 0 1em; }
.markdown-content p:last-child { margin-bottom: 0; }

.markdown-content a {
  color: var(--brand-primary);
  text-decoration: none;
  font-weight: 600;
}

.markdown-content a:hover { text-decoration: underline; }

.markdown-content strong { color: var(--text); font-weight: 700; }
.markdown-content em { font-style: italic; }

.markdown-content blockquote {
  margin: 1.15em 0;
  padding: 0.85em 1.1em;
  border-left: 4px solid var(--brand-primary);
  border-radius: 0 var(--radius-sm) var(--radius-sm) 0;
  background: linear-gradient(135deg, rgba(30, 110, 232, 0.08), rgba(0, 194, 178, 0.08));
  color: var(--text-secondary);
}

.markdown-content blockquote p { margin: 0; }

.markdown-content ul,
.markdown-content ol {
  margin: 0.8em 0;
  padding-left: 1.75em;
}

.markdown-content li { margin: 0.3em 0; }
.markdown-content li > ul,
.markdown-content li > ol { margin: 0.3em 0; }

.markdown-content ul.task-list {
  list-style: none;
  padding-left: 0;
}

.markdown-content .task-list-item {
  display: flex;
  align-items: flex-start;
  gap: 0.5em;
}

.markdown-content .task-list-item input[type="checkbox"] {
  margin-top: 0.25em;
  accent-color: var(--brand-primary);
}

.markdown-content code {
  font-family: var(--font-code);
  font-size: 0.9em;
  padding: 0.18em 0.42em;
  color: #103760;
  background: rgba(30, 110, 232, 0.07);
  border: 1px solid rgba(30, 110, 232, 0.10);
  border-radius: 6px;
}

.markdown-content pre {
  margin: 1.15em 0;
  padding: 1.05em 1.25em;
  overflow-x: auto;
  border-radius: var(--radius-md);
  border: 1px solid rgba(17, 54, 93, 0.18);
  background: var(--code-surface);
  color: #dbe9f7;
  box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.06);
  font-size: 13px;
  line-height: 1.55;
}

.markdown-content pre code {
  padding: 0;
  border: none;
  background: transparent;
  color: inherit;
  font-size: inherit;
  border-radius: 0;
}

.markdown-content hr {
  margin: 1.6em 0;
  border: none;
  border-top: 1px solid rgba(14, 26, 38, 0.10);
}

.markdown-content img {
  max-width: 100%;
  height: auto;
  border-radius: var(--radius-sm);
}

.markdown-content del {
  color: var(--muted);
  text-decoration: line-through;
}

.table-wrapper {
  margin: 1.15em 0;
  overflow-x: auto;
  border-radius: 16px;
  border: 1px solid rgba(30, 110, 232, 0.12);
  background: rgba(255, 255, 255, 0.92);
  box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.80);
}

.markdown-content table {
  width: 100%;
  border-collapse: collapse;
  table-layout: fixed;
  font-size: 13px;
  line-height: 1.5;
}

.markdown-content thead {
  background: linear-gradient(135deg, rgba(30, 110, 232, 0.11), rgba(0, 194, 178, 0.10));
}

.markdown-content th,
.markdown-content td {
  padding: 9px 14px;
  text-align: left;
  word-wrap: break-word;
  overflow-wrap: break-word;
}

.markdown-content th {
  color: var(--text);
  font-weight: 700;
  border-bottom: 1px solid rgba(30, 110, 232, 0.14);
  border-right: 1px solid rgba(14, 26, 38, 0.08);
}

.markdown-content td {
  color: var(--text);
  border-bottom: 1px solid rgba(14, 26, 38, 0.07);
  border-right: 1px solid rgba(14, 26, 38, 0.05);
  vertical-align: top;
}

.markdown-content th:last-child,
.markdown-content td:last-child { border-right: none; }

.markdown-content tbody tr:last-child td { border-bottom: none; }

.markdown-content tbody tr:nth-child(even) {
  background: rgba(30, 110, 232, 0.025);
}

.markdown-content tbody tr:hover {
  background: rgba(0, 194, 178, 0.06);
}

.markdown-content .numeric {
  font-family: var(--font-code);
  font-variant-numeric: tabular-nums;
}

.math-inline {
  display: inline-block;
  max-width: 100%;
  vertical-align: middle;
  padding: 0 2px;
  font-family: "Times New Roman", serif;
}

.math-display {
  display: block;
  margin: 1.25em 0;
  padding: 1em;
  text-align: center;
  overflow-x: auto;
  border-radius: var(--radius-md);
  border: 1px solid rgba(30, 110, 232, 0.12);
  background: linear-gradient(180deg, rgba(255, 255, 255, 0.94), rgba(241, 249, 255, 0.92));
}

mjx-container { display: inline-block !important; margin: 0 !important; }
mjx-container[display="true"] { display: block !important; margin: 1em 0 !important; }

.mermaid-wrapper {
  margin: 1.4em 0;
  padding: 1.15em;
  overflow-x: auto;
  text-align: center;
  border-radius: 18px;
  border: 1px solid rgba(30, 110, 232, 0.14);
  background:
    radial-gradient(circle at top right, rgba(0, 194, 178, 0.10), transparent 30%),
    linear-gradient(180deg, rgba(255, 255, 255, 0.96), rgba(240, 248, 255, 0.92));
  box-shadow: var(--shadow-soft);
}

.mermaid-wrapper pre.mermaid {
  margin: 0;
  padding: 0;
  background: transparent;
  border: none;
  color: var(--text);
  text-align: center;
  overflow: visible;
  font-size: 14px;
  line-height: 1.6;
}

.mermaid-wrapper pre.mermaid-rendered {
  padding: 14px 16px;
  overflow-x: auto;
  border-radius: 14px;
  border: 1px solid rgba(14, 26, 38, 0.08);
  background: rgba(255, 255, 255, 0.88);
}

.mermaid-wrapper svg {
  max-width: 100%;
  height: auto;
}

.mermaid-fallback {
  margin: 1em 0;
  overflow: hidden;
  border-radius: 14px;
  border: 1px solid rgba(30, 110, 232, 0.16);
}

.mermaid-fallback-label {
  padding: 7px 12px;
  color: white;
  font-size: 11px;
  font-family: var(--font-code);
  letter-spacing: 0.08em;
  text-transform: uppercase;
  background: linear-gradient(135deg, var(--brand-primary), var(--brand-secondary));
}

.mermaid-fallback pre {
  margin: 0;
  padding: 1em 1.2em;
  overflow-x: auto;
  font-size: 13px;
  line-height: 1.5;
  color: #15304b;
  background: rgba(255, 255, 255, 0.94);
  border-top: 1px solid rgba(30, 110, 232, 0.10);
}

.mermaid-fallback pre code {
  padding: 0;
  border: none;
  background: transparent;
  color: inherit;
  font-size: inherit;
  border-radius: 0;
  font-family: var(--font-code);
}

footer {
  margin-top: auto;
  padding: 0 clamp(16px, 3vw, 32px) 34px;
  color: var(--muted);
}

.foot {
  max-width: var(--maxw);
  margin: 0 auto;
  padding-top: 18px;
  text-align: center;
  font-size: 13px;
  border-top: 1px solid rgba(14, 26, 38, 0.08);
}

.foot-copy { color: var(--muted); }
.foot-copy strong { color: var(--text); }

.loading {
  padding: 3em;
  text-align: center;
  color: var(--muted);
  font-size: 14px;
}

@media (max-width: 1360px) {
  .page-content {
    grid-template-columns: minmax(280px, 38%) minmax(0, 1fr);
  }
}

@media (max-width: 1024px) {
  header {
    position: relative;
    top: auto;
  }

  .header-content {
    justify-content: center;
  }

  .page-content {
    grid-template-columns: 1fr;
  }

  .image-panel {
    position: static;
    top: auto;
    max-width: 860px;
    margin: 0 auto;
  }
}

@media (max-width: 768px) {
  header {
    padding: 12px 16px;
  }

  .brand {
    width: 100%;
    justify-content: center;
    flex-wrap: wrap;
    text-align: center;
  }

  .logo {
    width: 166px;
    height: 68px;
  }

  .subtitle-badge {
    width: 100%;
    margin-top: 8px;
  }

  .page-content {
    padding: 16px;
    gap: 16px;
  }

  .markdown-panel {
    padding: 16px;
  }

  .markdown-content {
    font-size: 13px;
  }

  .markdown-content table {
    font-size: 11px;
  }

  .markdown-content th,
  .markdown-content td {
    padding: 7px 10px;
  }
}

@media print {
  .blob, .blob-2 { display: none; }
  body { background: white; }
  header {
    position: static;
    background: white;
    border-bottom: 1px solid #ddd;
    -webkit-backdrop-filter: none;
    backdrop-filter: none;
  }
  .page-section {
    box-shadow: none;
    border: 1px solid #d8e1ea;
    background: white;
    page-break-inside: avoid;
  }
  .page-content { grid-template-columns: 1fr; }
  .image-panel {
    position: static;
    max-height: 400px;
    box-shadow: none;
  }
}

@media (prefers-reduced-motion: reduce) {
  *, *::before, *::after {
    animation-duration: 0.01ms !important;
    transition-duration: 0.01ms !important;
  }
}

@supports not ((-webkit-backdrop-filter: blur(10px)) or (backdrop-filter: blur(10px))) {
  header,
  .page-section {
    background: rgba(255, 255, 255, 0.94);
  }
}'''


def _get_mathjax_config() -> str:
    return r'''window.MathJax = {
  tex: {
    inlineMath: [['$', '$'], ['\\(', '\\)']],
    displayMath: [['$$', '$$'], ['\\[', '\\]']],
    processEscapes: true,
    processEnvironments: true,
    tags: 'ams',
    packages: {'[+]': ['ams', 'newcommand', 'configmacros']}
  },
  options: {
    skipHtmlTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'],
    ignoreHtmlClass: 'tex2jax_ignore',
    processHtmlClass: 'tex2jax_process'
  },
  svg: {
    fontCache: 'global',
    scale: 1,
    displayAlign: 'center'
  },
  startup: {
    pageReady: () => MathJax.startup.defaultPageReady()
  }
};'''


def _get_page_scripts() -> str:
    return r'''document.getElementById('year').textContent = new Date().getFullYear();

document.addEventListener('DOMContentLoaded', function() {
  if (window.MathJax && window.MathJax.typesetPromise) {
    window.MathJax.typesetPromise().catch(function(err) {
      console.warn('MathJax typeset error:', err);
    });
  }
});

(function() {
  var prefersReduced = window.matchMedia('(prefers-reduced-motion: reduce)');
  function toggleMotion(disable) {
    document.querySelectorAll('.blob, .blob-2').forEach(function(el) {
      el.style.animationPlayState = disable ? 'paused' : 'running';
    });
  }
  toggleMotion(prefersReduced.matches);
  prefersReduced.addEventListener('change', function(e) {
    toggleMotion(e.matches);
  });
})();

document.addEventListener('DOMContentLoaded', function() {
  document.querySelectorAll('.logo').forEach(function(container) {
    var img = container.querySelector('img');
    if (!img) return;

    function markLoaded() {
      img.classList.add('is-loaded');
    }
    function markError() {
      container.classList.add('logo-error');
    }

    if (img.complete) {
      if (img.naturalWidth > 0) markLoaded();
      else markError();
    } else {
      img.addEventListener('load', markLoaded, { once: true });
      img.addEventListener('error', markError, { once: true });
    }
  });
});'''


_MERMAID_INIT_JS = r"""mermaid.initialize({
  startOnLoad: false,
  theme: 'base',
  securityLevel: 'loose',
  themeVariables: {
    background: '#f7fbff',
    primaryColor: '#ffffff',
    primaryBorderColor: '#1e6ee8',
    primaryTextColor: '#0e1a26',
    secondaryColor: '#eefaf8',
    secondaryBorderColor: '#00c2b2',
    secondaryTextColor: '#0e1a26',
    tertiaryColor: '#eef4ff',
    tertiaryBorderColor: '#7aa8f4',
    tertiaryTextColor: '#0e1a26',
    lineColor: '#2563c9',
    clusterBkg: '#f4faff',
    clusterBorder: '#93b9f4',
    defaultLinkColor: '#2563c9',
    edgeLabelBackground: '#ffffff',
    nodeTextColor: '#0e1a26',
    mainBkg: '#ffffff',
    noteBkgColor: '#eefaf8',
    noteBorderColor: '#00a99b',
    fontFamily: '"Avenir Next", "Segoe UI Variable", "IBM Plex Sans", "Helvetica Neue", Arial, sans-serif',
    fontSize: '15px'
  },
  flowchart: {
    useMaxWidth: true,
    htmlLabels: true,
    curve: 'basis',
    padding: 18,
    nodeSpacing: 34,
    rankSpacing: 44
  },
  fontFamily: '"Avenir Next", "Segoe UI Variable", "IBM Plex Sans", "Helvetica Neue", Arial, sans-serif'
});

function _mermaidEscHtml(s) {
  return s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
}

function _mermaidFallback(el) {
  var source = el.textContent;
  var wrapper = el.closest('.mermaid-wrapper');
  var container = wrapper || el;
  var fb = document.createElement('div');
  fb.className = 'mermaid-fallback';
  fb.innerHTML = '<div class="mermaid-fallback-label">mermaid</div>' +
    '<pre><code>' + _mermaidEscHtml(source) + '</code></pre>';
  container.parentNode.replaceChild(fb, container);
}

document.addEventListener('DOMContentLoaded', async function() {
  var elements = document.querySelectorAll('pre.mermaid');
  if (!elements.length) return;

  if (typeof mermaid === 'undefined' || typeof mermaid.render !== 'function') {
    console.warn('mermaid.js not available — displaying raw source');
    elements.forEach(function(el) { _mermaidFallback(el); });
    return;
  }

  for (var i = 0; i < elements.length; i++) {
    var el = elements[i];
    var source = el.textContent;
    var id = 'mmd-' + i;
    try {
      var result = await mermaid.render(id, source);
      el.innerHTML = result.svg;
      el.classList.remove('mermaid');
      el.classList.add('mermaid-rendered');
    } catch (err) {
      console.warn('Mermaid render failed for block ' + i + ':', err.message || err);
      _mermaidFallback(el);
      var orphan = document.getElementById('d' + id);
      if (orphan) orphan.remove();
    }
  }
});"""


# ==============================================================================
# HTML Page Section Generation
# ==============================================================================

def _generate_page_section(
    base: str,
    page_num_str: str,
    img_data_url: str | None,
    html_content: str,
) -> str:
    label = f"{base} — Page {page_num_str}"
    if img_data_url:
        img_html = f'''<img src="{_escape_attr(img_data_url)}"
           alt="{_escape_attr(label)} source image"
           loading="lazy"
           decoding="async" />'''
    else:
        img_html = f'''<div class="image-missing">Missing image for <strong>{_escape_text(label)}</strong></div>'''

    return f'''<section class="page-section" data-base="{_escape_attr(base)}" data-page="{_escape_attr(page_num_str)}" aria-labelledby="page-label-{_escape_attr(base)}-{_escape_attr(page_num_str)}">
  <div class="page-header" id="page-label-{_escape_attr(base)}-{_escape_attr(page_num_str)}">
    <span class="page-indicator" aria-hidden="true"></span>
    <span>{_escape_text(label)}</span>
  </div>
  <div class="page-content">
    <div class="image-panel">
      {img_html}
    </div>
    <div class="markdown-panel">
      <div class="markdown-content tex2jax_process">
        {html_content}
      </div>
    </div>
  </div>
</section>'''


# ==============================================================================
# Full HTML Document Generation (self-contained output)
# ==============================================================================

def _generate_html_document(page_sections: list[str], title: str) -> str:
    css = _get_css_styles()
    mathjax_config = _get_mathjax_config()
    scripts = _get_page_scripts()
    # The company logo is embedded in this script so HTML output stays standalone.
    logo_data_url = _get_company_logo_data_url()

    sections_html = (
        "\n".join(page_sections)
        if page_sections
        else '<div class="loading">No pages found</div>'
    )

    mathjax_js = _load_bundled_js("tex-svg-full.js")
    mermaid_js = _load_bundled_js("mermaid.min.js")
    mathjax_tag = f"<script>{_safe_inline_js(mathjax_js)}</script>"
    mermaid_tag = f"<script>{_safe_inline_js(mermaid_js)}</script>"

    if logo_data_url:
        logo_html = (
            f'<img src="{_escape_attr(logo_data_url)}" alt="RizMoon company logo" '
            'loading="eager" decoding="async">'
        )
    else:
        logo_html = '<span class="logo-fallback" aria-hidden="true"></span>'

    return f'''<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>{_escape_text(title)}</title>
  <meta name="description" content="OCR Extraction Comparison Report - RizMoon">
  <meta name="color-scheme" content="light">
  <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 64 64'%3E%3Ccircle cx='32' cy='32' r='28' fill='%231e6ee8'/%3E%3Ccircle cx='32' cy='32' r='18' fill='%2300c2b2'/%3E%3C/svg%3E">
  <style>
{css}
  </style>
  <script>
{mathjax_config}
  </script>
  {mathjax_tag}
  {mermaid_tag}
  <script>
{_MERMAID_INIT_JS}
  </script>
</head>
<body>
  <div class="blob"></div>
  <div class="blob blob-2"></div>

  <div class="wrap">
    <header aria-label="Report header">
      <div class="header-content">
        <a class="brand" href="https://www.rizmoon.ai" rel="noopener">
          <div class="logo">
            {logo_html}
          </div>
        </a>
        <div class="subtitle-badge">
          <span class="text">{_escape_text(title)}</span>
        </div>
      </div>
    </header>

    <main role="main">
      {sections_html}
    </main>

    <footer>
      <div class="foot">
        <div class="foot-copy">
          &copy; <span id="year">Extracted by</span> <strong>RizMoon</strong>.
          [REDACTED_SECRET].
        </div>
      </div>
    </footer>
  </div>

  <script>
{scripts}
  </script>
</body>
</html>'''


# ==============================================================================
# Main Entry Point
# ==============================================================================

def markdown_to_html(
    image_dir: str | Path,
    markdown_dir: str | Path,
    output_html: str | Path,
) -> None:
    """
    Build an HTML report with side-by-side source images and rendered OCR
    markdown.  Mermaid fenced code blocks are rendered as SVG diagrams.

    Args:
        image_dir:    folder containing scanned page images
        markdown_dir: folder containing OCR-extracted Markdown files
        output_html:  output HTML file path
    """
    img_dir = Path(image_dir)
    md_dir = Path(markdown_dir)
    out_path = Path(output_html)

    if not img_dir.is_dir():
        raise NotADirectoryError(f"Image directory not found: {img_dir}")
    if not md_dir.is_dir():
        raise NotADirectoryError(f"Markdown directory not found: {md_dir}")

    out_path.parent.mkdir(parents=True, exist_ok=True)

    title = f"AI Extracted Content — {out_path.stem}"

    img_map = _scan_pages_by_base_and_num(img_dir, _IMG_EXTS)
    md_map = _scan_pages_by_base_and_num(md_dir, _MD_EXTS)

    keys = _sorted_keys(list(set(img_map.keys()) | set(md_map.keys())))

    if not keys:
        print(f"Warning: No matching pages found in {img_dir} and {md_dir}")
        return

    page_sections: list[str] = []
    for base, num in keys:
        img_path = img_map.get((base, num))
        md_path = md_map.get((base, num))

        img_data_url: str | None = None
        if img_path and img_path.exists():
            img_data_url = _image_to_base64_data_url(img_path)

        if md_path and md_path.exists():
            raw_md = md_path.read_text(encoding="utf-8-sig", errors="replace")
            raw_md = re.sub(r"\nPage Number \d+:\n", "\n", raw_md, flags=re.IGNORECASE)
            rendered = _convert_markdown_to_html(raw_md)
            html_content = (
                rendered
                if rendered.strip()
                else '<div class="loading">Empty extracted content</div>'
            )
        else:
            html_content = '<div class="loading">Missing markdown for this page</div>'

        page_sections.append(
            _generate_page_section(base, num, img_data_url, html_content)
        )

    final_html = _generate_html_document(page_sections, title)
    out_path.write_text(final_html, encoding="utf-8")
    print(f"Report generated: {len(keys)} pages -> {out_path}")


# ==============================================================================
# CLI Entry Point
# ==============================================================================

if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description=(
            "Generate a side-by-side HTML report comparing scanned page "
            "images with OCR-extracted Markdown.  Mermaid diagram blocks "
            "are rendered as interactive SVG graphs."
        ),
    )
    parser.add_argument("image_dir", help="Directory of source page images")
    parser.add_argument("markdown_dir", help="Directory of OCR-extracted Markdown files")
    parser.add_argument("output_html", help="Output HTML file path")
    args = parser.parse_args()

    markdown_to_html(args.image_dir, args.markdown_dir, args.output_html)


path: scripts/ocr_main.py
content: """
OCR Pipeline — end-to-end document extraction.

Converts a source document into page images, extracts content via VLM,
and produces Markdown, HTML comparison, DOCX, and Excel outputs.
"""

from __future__ import annotations

import argparse
import logging
import shutil
import sys
from pathlib import Path

logger = logging.getLogger(__name__)

_SCRIPT_DIR = Path(__file__).resolve().parent
_SKILL_ROOT = _SCRIPT_DIR.parent
DEFAULT_OUTPUT_DIR = Path.cwd() / "ocr_output"

if str(_SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(_SCRIPT_DIR))

from ocr_runtime import NullProgressReporter, ProgressReporter

_preflight_done = False
_IMAGE_EXTS = {".jpeg", ".jpg", ".png", ".webp", ".bmp", ".tiff", ".tif"}


def _preflight() -> None:
    """Validate local dependencies and .env before the pipeline runs.

    Skips silently on subsequent calls within the same process.
    """
    global _preflight_done
    if _preflight_done:
        return

    from ocr_setup import check_packages, check_env, bootstrap_env, python_install_command

    missing_pkgs = check_packages()
    if missing_pkgs:
        sys.exit(
            f"[preflight] Missing packages: {', '.join(missing_pkgs)}\n"
            "[preflight] Automatic installation is disabled for security.\n"
            f"  Run manually: {python_install_command()}"
        )

    missing_env = check_env()
    env_path = _SKILL_ROOT / ".env"
    if missing_env and not env_path.exists():
        bootstrap_env()
        missing_env = check_env()
    if missing_env:
        sys.exit(
            f"[preflight] Missing required config: {', '.join(missing_env)}\n"
            f"  Please edit {env_path} with your VLM credentials.\n"
            f"  Run: python scripts/ocr_setup.py   (for interactive guided setup)"
        )

    _preflight_done = True


def ocr_main(
    source_file: str | Path,
    threads: int = 1,
    reporter: ProgressReporter | None = None,
    project_dir: str | Path | None = None,
    resume: bool = False,
) -> Path:
    """
    Run the full OCR pipeline on *source_file*.

    Args:
        source_file:       Path to the document to process.
        threads:           Number of concurrent OCR threads (default: 1).

    Returns the project directory containing all outputs.
    """
    reporter = reporter or NullProgressReporter()

    reporter.step_started("preflight", "Preflight checks")
    _preflight()
    reporter.step_completed("preflight")

    from docs_to_image import document_to_images
    from image_to_md import image_folder_to_md
    from md_to_html import markdown_to_html
    from md_to_docx import markdown_to_docx
    from md_to_excel import markdown_folder_to_excel

    file_path = Path(source_file).resolve()
    if not file_path.is_file():
        raise FileNotFoundError(f"Source file not found: {file_path}")

    file_name = file_path.stem
    if project_dir is not None:
        project_dir = Path(project_dir).resolve()
    else:
        project_dir = DEFAULT_OUTPUT_DIR / f"OCR_{file_name}"
    image_dir = project_dir / "images"
    markdown_dir = project_dir / "markdowns"
    results_dir = project_dir / "results"
    combined_md_path = results_dir / f"{file_name}_combined.md"
    html_path = results_dir / f"{file_name}.html"
    docx_path = results_dir / f"{file_name}.docx"
    excel_path = results_dir / f"{file_name}.xlsx"

    for folder in (project_dir, image_dir, markdown_dir, results_dir):
        folder.mkdir(parents=True, exist_ok=True)

    print(f"\n{'='*60}")
    print(f"  OCR Pipeline — {file_path.name}")
    print(f"  Project folder: {project_dir}")
    print(f"{'='*60}\n")

    reporter.check_cancelled()
    reporter.step_started("render_pages", "Converting document to images")
    print("[Step 1/5] Converting document to images …")
    existing_images = sorted(
        p for p in image_dir.iterdir()
        if p.is_file() and p.suffix.lower() in _IMAGE_EXTS
    )
    if resume and existing_images:
        print(f"[Resume] Reusing {len(existing_images)} existing page image(s).")
        reporter.log(f"Reusing {len(existing_images)} existing page image(s).")
    else:
        document_to_images(file_path, output_dir=image_dir, reporter=reporter)
        existing_images = sorted(
            p for p in image_dir.iterdir()
            if p.is_file() and p.suffix.lower() in _IMAGE_EXTS
        )
    if not existing_images:
        raise RuntimeError(
            "No page images were generated from the source document. "
            "Check the input file and any required system converters."
        )
    reporter.pages_discovered(len(existing_images))
    reporter.step_completed("render_pages")

    reporter.check_cancelled()
    reporter.step_started("ocr_pages", "Extracting page content via VLM")
    print("\n[Step 2/5] Extracting page content via VLM …")
    succeeded_pages, failed_pages = image_folder_to_md(
        image_dir,
        markdown_dir,
        combined_md_path,
        threads=threads,
        reporter=reporter,
        resume=resume,
    )
    if failed_pages:
        raise RuntimeError(
            f"OCR failed for {failed_pages} of {succeeded_pages + failed_pages} page image(s). "
            "Partial outputs were retained so the job can be resumed."
        )
    reporter.artifact_ready("combined_md", combined_md_path)
    reporter.step_completed("ocr_pages")

    reporter.check_cancelled()
    reporter.step_started("build_html", "Generating HTML comparison report")
    print("\n[Step 3/5] Generating HTML comparison report …")
    markdown_to_html(image_dir, markdown_dir, html_path)
    reporter.artifact_ready("html", html_path)
    reporter.step_completed("build_html")

    reporter.check_cancelled()
    reporter.step_started("build_docx", "Generating DOCX document")
    print("\n[Step 4/5] Generating DOCX document …")
    markdown_to_docx(combined_md_path, docx_path)
    reporter.artifact_ready("docx", docx_path)
    reporter.step_completed("build_docx")

    reporter.check_cancelled()
    reporter.step_started("build_excel", "Generating Excel workbook")
    print("\n[Step 5/5] Generating Excel workbook …")
    markdown_folder_to_excel(markdown_dir, excel_path)
    reporter.artifact_ready("excel", excel_path)
    reporter.step_completed("build_excel")

    reporter.check_cancelled()
    reporter.step_started("finalize", "Finalizing outputs")
    print("\n[Cleanup] Removing intermediate files …")
    freed = 0
    for d in (image_dir, markdown_dir):
        if d.is_dir():
            for f in d.rglob("*"):
                if f.is_file():
                    freed += f.stat().st_size
            shutil.rmtree(d)
    print(f"  Freed {freed / (1024 * 1024):.1f} MB")

    reporter.step_completed("finalize")
    print(f"\nAll done.  Outputs in: {results_dir}")
    return project_dir


if __name__ == "__main__":
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s [%(levelname)s] %(message)s",
    )

    parser = argparse.ArgumentParser(
        description="OCR pipeline: document → images → Markdown → HTML + DOCX + XLSX",
    )
    parser.add_argument("file", help="Path to the source document.")
    parser.add_argument(
        "-t", "--threads",
        type=int,
        default=1,
        help="Number of concurrent OCR threads (default: 1; raise only if the API plan supports parallel OCR requests).",
    )
    args = parser.parse_args()

    ocr_main(
        args.file,
        threads=args.threads,
    )


path: scripts/ocr_prompt.py
content: ocr_to_markdown: str = r"""
ROLE & GOAL
You are an expert OCR and document-layout analyst. Given exactly ONE scanned page image, reconstruct that single page into faithful Markdown for downstream analysis and HTML/DOCX conversion.

SCOPE
- Input: exactly one page image.
- The page may begin or end mid-paragraph, mid-list, mid-table, or mid-figure.
- Use only what is visible on this page.
- Never add content from previous or next pages.

WORKING METHOD (SILENT)
1) Inspect the whole page first.
2) Determine reading order and block types.
3) Reconstruct each block in order.
4) Validate all special blocks before final output.

PRIORITY (resolve conflicts in this order)
1) Output contract
2) Fidelity and uncertainty tokens
3) Block markers and structure rules
4) Reading order and formatting

OUTPUT CONTRACT (STRICT)
1) Output only the reconstructed content of this page as raw Markdown.
2) Do NOT wrap the entire output in any outer code fence such as ```markdown.
3) The only HTML allowed is table markup using these tags only:
   <table>, <thead>, <tbody>, <tr>, <th>, <td>
4) The only allowed HTML attributes are:
   - scope="col" or scope="row" on <th>
   - colspan="N" and rowspan="N" on <th>/<td>, where N is a positive integer
5) The only fenced code block allowed anywhere in the output is an exact Mermaid block as defined below.
6) Do not output explanations, metadata, JSON, comments, or surrounding commentary.
7) Do not translate. Preserve the original language and script.
8) If, after exclusions, the page contains no substantive content, return exactly an empty string.

BLOCK MARKERS (MANDATORY EXACT SYNTAX)
Use these exact forms so downstream programs can reliably detect block types:

1) Ordinary text blocks
- Headings, paragraphs, and lists must be emitted as normal Markdown text.
- Do NOT place ordinary Markdown inside fenced blocks.

2) Mermaid blocks
- If a figure qualifies for Mermaid, you MUST output it as a fenced Mermaid block.
- The opening line must be exactly:
  ```mermaid
- The closing line must be exactly:
  ```
- Use three backticks, not apostrophes or tildes.
- The fence lines must start at column 1 with no indentation.
- Place exactly one blank line before and after each Mermaid block.
- Do not output raw Mermaid text outside a Mermaid fence.
- Do not nest Mermaid blocks inside lists, tables, or other fenced blocks.

3) HTML table blocks
- Every table must begin with <table> on its own line and end with </table> on its own line.
- Place exactly one blank line before and after each table block.
- Do not indent table blocks.
- Never use Markdown pipe tables.

4) Display math blocks
- If display math is needed, use:
  $$
  ...
  $$
- Each $$ marker must be on its own line.
- Place exactly one blank line before and after each display-math block.
- Do not wrap math in fenced code blocks.

5) Inline math
- Use $...$ only inline within a paragraph or list item.

6) Figure placeholders
- Every non-table figure must have a standalone placeholder line:
  ![Concise description <=20 words]

FIDELITY (NON-NEGOTIABLE)
- Transcribe only what is visible: spelling, punctuation, casing, numbers, units, symbols, and diacritics.
- Do not guess, correct, normalize, paraphrase, summarize, or infer missing content.
- Allowed transformations:
  a) Reflow hard line breaks into paragraphs where appropriate.
  b) De-hyphenate obvious end-of-line word breaks.
  c) Replace bullet glyphs such as • or · with Markdown "- " only when they clearly function as list markers.
  d) Apply Markdown escaping where required.
  e) Represent clear sub/superscripts with Unicode where unambiguous, otherwise with LaTeX math.
  f) If a table has no visible header row, create placeholder headers H1, H2, H3, ...

UNCERTAINTY TOKENS (USE ONLY THESE)
- [?]         = unknown single character, digit, symbol, or very short unreadable span
- [unclear]   = visible but ambiguous span
- [illegible] = completely unreadable word, line, cell, or span

Rules:
- Use only these three tokens.
- Never invent other placeholders.
- Keep readable content and insert tokens only where visible content is unreadable.
- Do not use uncertainty tokens for off-page content that is not visible.

READING ORDER
1) Determine the primary reading direction:
   - LTR: columns left to right; within a block, read left to right
   - RTL: columns right to left; preserve printed logical order
2) For multi-column pages, read one full column top to bottom before moving to the next column.
3) Within each column, preserve top-to-bottom block order: headings, paragraphs, lists, tables, figures, captions, notes, footnotes.
4) If a note or footnote is clearly referenced, place it immediately after the referencing text. Otherwise place it near the most relevant content, or at end of page if unclear.

ORIENTATION / ROTATION / VERTICAL TEXT
- Read rotated, tilted, or upside-down content in its correct orientation.
- Convert vertical text into normal horizontal reading order while preserving punctuation.

TEXT FLOW & WHITESPACE
- Merge hard-wrapped lines into one line per paragraph.
- Preserve internal line breaks only when structurally meaningful, such as addresses, verse, or line-based forms.
- Use exactly one blank line between block elements.
- No extra blank lines at the start or end.
- No trailing spaces. No tabs.

HYPHENATION
- If a line ends with a hyphen only because of line wrapping and the continuation is obvious, merge the word and remove the wrap hyphen.
- Otherwise keep the hyphen exactly as printed.

HEADINGS
- Use only:
  - ## for a clear heading
  - ### for a clear subheading
- Never use #.
- Only create headings when typography or layout clearly indicates a true heading.

LISTS
- Use "- " for bulleted lists.
- For ordered lists, preserve visible markers when they are digits followed by "." or ")".
- For other visible markers such as "(a)" or "i.", use a bullet and keep the original marker in the item text.
- Indent nested lists by 2 spaces per level.
- Restart numbering only where it visibly restarts.
- Reproduce checkboxes only if visible, using "- [ ]" or "- [x]".

MARKDOWN ESCAPING (OUTSIDE HTML TABLES AND MERMAID)
Escape leading Markdown control characters only when the line is not intended as that structure:
- \#, \>, \-, \+, \*, \|
- Escape ordered-list-like punctuation for non-lists: "1\. Term", "a\) Term"
- Escape a leading negative number that is not a list item: "\-3.5"
- Prevent accidental horizontal rules by escaping at least one character in a literal line of "---", "***", or "___"

MATH & SCRIPTS (OUTSIDE HTML TABLES AND MERMAID)
- Use $...$ only when the source clearly contains math or scientific notation.
- Use $$...$$ only for clearly displayed formulas.
- Escape literal dollar signs as "\$".
- Preserve symbols exactly.
- Prefer Unicode superscripts/subscripts when unambiguous, otherwise use LaTeX math.
- If math is partially obscured, keep legible parts and use uncertainty tokens for unreadable visible spans.

TABLES (HTML ONLY) - REQUIRED FOR ALL TABULAR CONTENT
- Any tabular or grid-like structure must be output as an HTML table.
- Never use Markdown pipe tables.

Placement:
- Insert exactly one blank line before and after each <table>...</table> block.
- Place each table in reading order.

TABLE GRID DISCOVERY (LOCK THE GRID BEFORE WRITING HTML)
1) Use two passes:
   - Pass A: determine the full grid for the entire table, including column boundaries, row boundaries, and any unambiguous merged cells.
   - Pass B: place cell text into that locked grid.
2) Canonical total column count = the maximum number of distinct columns visible anywhere in the table, unless a clear schema break requires splitting into separate tables.
3) If an extra value appears in its own aligned narrow column for only some rows, treat it as a real column and backfill blanks in rows where it is empty.
4) If a visible header row spans fewer columns than the canonical width:
   - use colspan only when the span is visually explicit
   - otherwise add blank header cells
5) If the grid materially changes within one apparent table, split into separate <table> blocks.

Header rules:
- If a visible header row exists, reproduce it in <thead>.
- If no visible header row exists, create placeholder headers H1, H2, H3, ... matching the canonical width.

Body rules:
- Put data rows in <tbody>.
- Use <th scope="row"> only when the first body cell clearly functions as a row label.
- If multiple leftmost columns exist, only the first stub label may be <th scope="row">; the rest must be <td>.

Merged cells:
- Use rowspan/colspan only when visually unambiguous.
- If merge extents are unclear, do not merge. Put the text in the first cell and fill the rest with empty cells.

RECTANGULAR GRID REQUIREMENT (HARD CONSTRAINT)
- After applying any rowspan/colspan, every row in <thead> and <tbody> must expand to the same canonical total column count.
- If a row is short, add empty cells in the correct visual positions.
- If a row is long, the grid is wrong or there is a schema break. Fix the grid or split the table.
- Never drop values.
- Never shift values sideways.

Cell text:
- Keep cell text plain.
- No emphasis and no nested HTML tags.
- Flatten multi-line cell text to one line with single spaces.
- HTML-escape &, <, and > inside cell text.
- Use uncertainty tokens for unreadable cell text. Never guess.

Multiple tables:
- If the page contains multiple distinct tables, output them as separate <table> blocks in top-to-bottom order.

Canonical table format:
<table>
  <thead>
    <tr>
      <th scope="col">...</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th scope="row">...</th>
      <td>...</td>
    </tr>
  </tbody>
</table>

MERMAID FOR OWNERSHIP / ENTITY STRUCTURE CHARTS
Use Mermaid only for non-table figures that are clearly node-link entity relationship diagrams and can be reconstructed without guessing.

Mermaid-eligible figure types:
- Ownership structure charts
- Shareholding / equity stake charts
- Before-transaction / after-transaction structure charts
- Corporate / legal-entity / subsidiary charts
- SPV / HoldCo / fund / investment holding-chain diagrams
- Org-chart-like box-and-arrow diagrams where edges explicitly show ownership, control, or economic interest

Do NOT use Mermaid for:
- bar, line, area, pie, donut, waterfall, histogram, scatter, bubble, radar, heatmap, map, timeline, photo, or purely illustrative images
- generic process flows unless the links explicitly represent ownership, control, or economic interest
- any figure that would require inferring missing nodes, labels, percentages, directions, or relationships

MANDATORY MERMAID CONVERSION RULE
- If a figure is Mermaid-eligible and structurally reliable, you MUST convert it into a Mermaid fenced block.
- Do not leave a Mermaid-eligible ownership/entity chart as placeholder-only text if it can be represented faithfully.
- Do not describe Mermaid-eligible relationships only as bullet points when Mermaid can represent them reliably.

Mermaid output rules:
1) Keep the required figure placeholder line at the figure position:
   - ![Concise description <=20 words]
2) Immediately after that placeholder, output exactly one Mermaid fenced block for that figure.
3) The Mermaid block must open with exactly:
   ```mermaid
4) The Mermaid block must close with exactly:
   ```
5) If the page contains multiple distinct eligible charts, output separate placeholder + Mermaid blocks in reading order.
6) Use "flowchart TD" by default. Use "flowchart LR" only when the source layout is clearly horizontal and LR materially improves fidelity.
7) Edge direction must follow the visible relationship direction:
   - owner / parent / shareholder / controller --> owned entity / subsidiary / vehicle / target
8) Edge labels must preserve exact printed relationship text, such as:
   - percentages: "35%", "100%", "75.6508%"
   - printed relationship labels: "GP", "LP", "Control"
9) Node labels must preserve the source language exactly.
10) Flatten line breaks inside node labels to single spaces.
11) Use short ASCII node IDs internally, but preserve exact visible text in labels.
12) Use "subgraph" only when a grouping box/container and its label are visibly printed.
13) Use only these Mermaid flowchart features:
   - nodes
   - directed edges
   - edge labels
   - subgraph
14) Do not use style, classDef, linkStyle, click handlers, comments, or decorative syntax.
15) Do not infer indirect holdings, unstated groupings, unstated percentages, or missing intermediate entities.
16) If the chart is only partially legible, emit Mermaid only if the visible structure remains reliable; otherwise omit Mermaid and follow the general figure rules.

FIGURES / CHARTS / DIAGRAMS / NON-TABLE IMAGES
For each non-table figure:
1) Always output a placeholder line at the figure position:
   - ![Concise description <=20 words]
   - Describe only clearly visible content: figure type plus key visible labels. No speculation.

2) If the figure qualifies for Mermaid, follow the Mermaid rules above.

3) Otherwise, if and only if visible structure or data can be captured without estimating, add a bullet list immediately after the placeholder using only applicable bullets:
- Type:
- Title:
- Caption:
- Axes: x=..., y=...
- Legend/Series:
- Data labels:
- Structure: use "A -> B" only for exact visible node or box text

4) If the figure contains clearly legible data that can be represented as a table without guessing, output that data as an HTML table in addition to the placeholder.

5) If reliable reconstruction is not possible, output only the placeholder and any plainly visible caption text.
- Do not add inferred trends, conclusions, or narrative summaries.

FORMS (NON-TABLE)
- If grid-like or aligned by rows and columns, output as an HTML table.
- Otherwise output key-value lines or bullet items in reading order.

HANDWRITING / STAMPS / ANNOTATIONS
- Ignore non-text marks such as arrows, circles, or highlights unless they contain readable text.
- Transcribe substantive handwriting or stamps using the same fidelity and uncertainty rules.

HEADERS / FOOTERS / WATERMARKS (EXCLUSIONS)
- Exclude purely decorative or non-substantive running headers, footers, page numbers, crop marks, and watermarks.
- Include them only if they convey meaningful content needed to understand the page.

CROSS-PAGE CONTINUITY
- Do not add "continued", separators, or invented context.
- If a paragraph, table, or figure continues beyond the page, transcribe only what is visible on this page.

FINAL SELF-CHECK (SILENT)
- Output only this page's content, in correct reading order.
- The response is raw Markdown, not wrapped in ```markdown or any outer fence.
- Ordinary text blocks are plain Markdown, not fenced.
- Every Mermaid-eligible ownership/entity chart has been converted into an exact ```mermaid fenced block when reliable.
- No Mermaid content appears outside a Mermaid fenced block.
- The only HTML tags used are the allowed table tags.
- Every table begins with <table> and ends with </table> on their own lines.
- Display math, if any, uses $$...$$ and is not fenced.
- No commentary, no JSON, and no non-Mermaid code fences.
- Nothing is guessed; uncertainty tokens are used correctly.
- Exactly one blank line separates adjacent block elements.
- Every HTML table is rectangular after accounting for rowspan/colspan.
- Every Mermaid block represents only explicit visible relationships.

FINAL EMISSION
Return only the Markdown content for this page. Do NOT wrap it in outer ``` fences.
""".strip()


path: scripts/ai_service_vlm.py
content: """VLM (Vision-Language Model) service client.

Sends text + image requests to an OpenAI-compatible API and returns
clean Markdown with any outer code fences stripped automatically.

Each call creates a fresh HTTP connection to avoid timeouts during
long-running batch OCR jobs (e.g. hundreds or thousands of pages).

Configuration (environment variables or .env beside this script):
    API_KEY   – API key
    BASE_URL  – API base URL
    VLM_MODEL – Model identifier
"""

from __future__ import annotations

import base64
import logging
import mimetypes
import os
import re
from pathlib import Path

from dotenv import load_dotenv
from openai import APIError, OpenAI

_SCRIPT_DIR = Path(__file__).resolve().parent
_SKILL_ROOT = _SCRIPT_DIR.parent

_dotenv = _SKILL_ROOT / ".env"
if not _dotenv.exists():
    _dotenv = _SCRIPT_DIR / ".env"
load_dotenv(dotenv_path=_dotenv if _dotenv.exists() else None)

logger = logging.getLogger(__name__)

_OUTER_FENCE_RE = re.compile(
    r"\A\s*```(?:markdown|md)?\s*\n(.*)\n\s*```\s*\Z",
    re.DOTALL,
)


def image_encode(image_path: str | Path) -> str:
    """Encode a local image as a ``data:<mime>;base64,…`` URL."""
    path = Path(image_path).expanduser().resolve()
    if not path.is_file():
        raise FileNotFoundError(f"Image not found: {path}")

    mime = mimetypes.guess_type(str(path))[0] or "application/octet-stream"
    encoded = base64.b64encode(path.read_bytes()).decode("ascii")
    return f"data:{mime};base64,{encoded}"


def _resolve_image(source: str | Path | None) -> str | None:
    """Normalise *source* to a URL the API accepts (local path, http(s), or data URL)."""
    if source is None:
        return None
    text = str(source).strip()
    if text.lower().startswith(("http://", "https://", "data:")):
        return text
    return image_encode(text)


def _require_env(name: str) -> str:
    value = os.getenv(name)
    if not value:
        raise OSError(f"Missing required environment variable: {name}")
    return value


def _make_client() -> tuple[OpenAI, str]:
    """Create a fresh OpenAI client and return *(client, model)*.

    A new client (and underlying HTTP connection) is created on every
    call so that long-running batch jobs never hit idle-connection
    timeouts imposed by the API provider or intermediate proxies.
    """
    [REDACTED_SECRET]"API_KEY")
    base_url = _require_env("BASE_URL")
    model    = _require_env("VLM_MODEL")

    client = OpenAI([REDACTED_SECRET], base_url=base_url)
    logger.debug("OpenAI client created for model=%s", model)
    return client, model


def _mock_response() -> str | None:
    """Return a configured mock OCR response for local smoke tests, if any."""
    mock_path = os.getenv("OCR_MOCK_RESPONSE_FILE")
    if mock_path:
        text = Path(mock_path).expanduser().read_text(encoding="utf-8")
        return strip_outer_fence(text)

    mock_text = os.getenv("OCR_MOCK_RESPONSE_TEXT")
    if mock_text:
        return strip_outer_fence(mock_text)

    return None


def strip_outer_fence(text: str) -> str:
    """Remove a single outer ``markdown`` / ``md`` / bare code fence.

    Internal fences (e.g. ``mermaid``) are preserved because the regex
    only matches fences tagged ``markdown``, ``md``, or with no tag.
    A greedy inner group ensures the *last* closing fence at EOF is
    matched, so nested fences are never consumed.
    """
    m = _OUTER_FENCE_RE.match(text)
    return m.group(1) if m else text


def ai_request(prompt: str, image_url: str | Path | None = None) -> str:
    """Send *prompt* (with optional image) to the VLM and return clean Markdown.

    A fresh HTTP connection is established for each call to prevent
    timeout issues during long-running batch OCR operations.

    The response is automatically unwrapped if the model enclosed it in a
    single outer ``markdown`` code fence.
    """
    prompt = (prompt or "").strip()
    if not prompt:
        raise ValueError("prompt must be a non-empty string.")

    mocked = _mock_response()
    if mocked is not None:
        logger.debug("Using mock OCR response from environment.")
        return mocked

    client, model = _make_client()

    content: list[dict] = [{"type": "text", "text": prompt}]
    image = _resolve_image(image_url)
    if image:
        content.insert(0, {"type": "image_url", "image_url": {"url": image}})

    logger.debug("Sending request to %s (image=%s)", model, image is not None)
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": content}],
            temperature=0.3,
        )
    except APIError as exc:
        status_code = getattr(exc, "status_code", None)
        message = getattr(exc, "message", None) or str(exc)
        if status_code is None:
            raise RuntimeError(f"API request failed: {message}") from exc
        raise RuntimeError(f"API request failed ({status_code}): {message}") from exc

    if not response.choices:
        raise RuntimeError("Model returned no choices.")

    raw = response.choices[0].message.content or ""
    return strip_outer_fence(raw)


def _cli() -> None:
    import argparse

    parser = argparse.ArgumentParser(
        description="Send a text+image request to a VLM and print the response.",
    )
    parser.add_argument(
        "-p", "--prompt",
        default="Please extract the contents from the page.",
        help="Prompt to send (default: OCR extraction prompt)",
    )
    parser.add_argument(
        "-i", "--image",
        default=None,
        help="Path or URL of the image to send (default: $VLM_TEST_IMAGE env var)",
    )
    parser.add_argument(
        "-v", "--verbose",
        action="store_true",
        help="Enable debug logging",
    )
    args = parser.parse_args()

    logging.basicConfig(
        level=logging.DEBUG if args.verbose else logging.WARNING,
        format="%(levelname)s: %(message)s",
    )

    image = args.image or os.getenv("VLM_TEST_IMAGE")
    print("Prompt:", args.prompt)
    print("Image: ", image or "(none)")
    print("\n----------- Answer -----------")
    print(ai_request(prompt=args.prompt, image_url=image))


if __name__ == "__main__":
    _cli()


path: scripts/ocr_worker.py
content: """Detached worker entrypoint for disk-backed OCR jobs."""

from __future__ import annotations

import argparse
import logging
import os
import sys
import traceback
from pathlib import Path

from ocr_jobs import DiskProgressReporter, JobStore
from ocr_main import ocr_main
from ocr_runtime import CancellationRequested

logger = logging.getLogger(__name__)


def run_worker(job_dir: str | Path) -> int:
    store = JobStore(job_dir)
    manifest = store.load_manifest()
    params = manifest.get("params", {})
    source_file = manifest["input_path"]
    reporter = DiskProgressReporter(store)

    store.mark_started(pid=os.getpid())
    reporter.job_started()

    try:
        ocr_main(
            source_file,
            threads=int(params.get("threads", 1)),
            reporter=reporter,
            project_dir=store.job_dir,
            resume=True,
        )

        reporter.check_cancelled()

        store.mark_completed()
        return 0
    except CancellationRequested:
        store.mark_cancelled()
        return 0
    except SystemExit as exc:
        message = str(exc) or f"OCR worker exited with status {exc.code!r}."
        store.mark_failed(message)
        return exc.code if isinstance(exc.code, int) else 1
    except Exception as exc:
        traceback.print_exc()
        store.mark_failed(str(exc))
        return 1


def main() -> None:
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s [%(levelname)s] %(message)s",
    )

    parser = argparse.ArgumentParser(description="Run a detached OCR worker for one job.")
    parser.add_argument("--job-dir", required=True, help="Path to the OCR job directory.")
    args = parser.parse_args()

    sys.exit(run_worker(args.job_dir))


if __name__ == "__main__":
    main()


path: scripts/ocrctl.py
content: """CLI controller for durable OCR background jobs."""

from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
from typing import Any

from ocr_jobs import JobStore, create_job_id, list_jobs

_SCRIPT_DIR = Path(__file__).resolve().parent


def _worker_popen_kwargs(*, log_file, env: dict[str, str]) -> dict[str, Any]:
    """Return cross-platform subprocess options for detached OCR workers."""
    kwargs: dict[str, Any] = {
        "cwd": str(_SCRIPT_DIR.parent),
        "stdin": subprocess.DEVNULL,
        "stdout": log_file,
        "stderr": subprocess.STDOUT,
        "env": env,
    }
    if os.name == "nt":
        creationflags = 0
        creationflags |= getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
        creationflags |= getattr(subprocess, "CREATE_NO_WINDOW", 0)
        kwargs["creationflags"] = creationflags
    else:
        kwargs["start_new_session"] = True
    return kwargs


def _emit(payload: dict[str, Any], *, as_json: bool) -> None:
    if as_json:
        print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
        return

    if payload.get("ok") is False:
        print(f"Error: {payload.get('error', 'Unknown error')}")
        return

    for key, value in payload.items():
        print(f"{key}: {value}")


def _load_store(job_id: str) -> JobStore:
    store = JobStore(JobStore.jobs_root() / job_id)
    if not store.exists():
        raise FileNotFoundError(f"OCR job not found: {job_id}")
    return store


def _status_payload(store: JobStore) -> dict[str, Any]:
    snapshot = store.snapshot(refresh_liveness=True)
    total_pages = snapshot.get("total_pages") or 0
    processed_pages = snapshot.get("processed_pages") or 0
    percent = round((processed_pages / total_pages) * 100, 1) if total_pages else None
    return {
        "ok": True,
        "job_id": snapshot["job_id"],
        "status": snapshot["status"],
        "message": snapshot.get("message"),
        "error": snapshot.get("error"),
        "input_path": snapshot.get("input_path"),
        "input_name": snapshot.get("input_name"),
        "job_dir": snapshot.get("job_dir"),
        "current_step": snapshot.get("current_step"),
        "step_label": snapshot.get("step_label"),
        "total_pages": snapshot.get("total_pages"),
        "processed_pages": snapshot.get("processed_pages"),
        "failed_pages": snapshot.get("failed_pages"),
        "percent": percent,
        "created_at": snapshot.get("created_at"),
        "started_at": snapshot.get("started_at"),
        "finished_at": snapshot.get("finished_at"),
        "updated_at": snapshot.get("updated_at"),
        "artifacts_ready": snapshot.get("artifacts_ready", []),
    }


def cmd_start(args: argparse.Namespace) -> dict[str, Any]:
    input_path = Path(args.input_path).expanduser().resolve()
    if not input_path.is_file():
        return {"ok": False, "error": f"Input file not found: {input_path}"}

    job_id = create_job_id()
    store = JobStore(JobStore.jobs_root() / job_id)
    store.create(
        input_path,
        params={
            "threads": args.threads,
        },
    )

    cmd = [sys.executable, str(_SCRIPT_DIR / "ocr_worker.py"), "--job-dir", str(store.job_dir)]
    env = {**os.environ, "PYTHONUNBUFFERED": "1"}
    with store.log_path.open("a", encoding="utf-8") as log_file:
        proc = subprocess.Popen(cmd, **_worker_popen_kwargs(log_file=log_file, env=env))
    store.set_worker_pid(proc.pid)

    return {
        "ok": True,
        "job_id": job_id,
        "status": "queued",
        "input_path": str(input_path),
        "job_dir": str(store.job_dir),
        "worker_pid": proc.pid,
        "message": "OCR job accepted and worker started.",
    }


def cmd_status(args: argparse.Namespace) -> dict[str, Any]:
    store = _load_store(args.job_id)
    return _status_payload(store)


def cmd_list(args: argparse.Namespace) -> dict[str, Any]:
    jobs = list_jobs(state=args.state)
    rows = [
        {
            "job_id": job["job_id"],
            "status": job["status"],
            "input_name": job.get("input_name"),
            "current_step": job.get("current_step"),
            "processed_pages": job.get("processed_pages"),
            "total_pages": job.get("total_pages"),
            "updated_at": job.get("updated_at"),
        }
        for job in jobs
    ]
    return {"ok": True, "jobs": rows}


def cmd_tail(args: argparse.Namespace) -> dict[str, Any]:
    store = _load_store(args.job_id)
    return {
        "ok": True,
        "job_id": args.job_id,
        "lines": store.tail_log(lines=args.lines),
    }


def cmd_artifacts(args: argparse.Namespace) -> dict[str, Any]:
    store = _load_store(args.job_id)
    snapshot = store.snapshot(refresh_liveness=True)
    return {
        "ok": True,
        "job_id": args.job_id,
        "status": snapshot.get("status"),
        "artifacts": snapshot.get("artifacts", {}),
    }


def cmd_cancel(args: argparse.Namespace) -> dict[str, Any]:
    store = _load_store(args.job_id)
    snapshot = store.snapshot(refresh_liveness=True)
    if snapshot.get("status") in {"completed", "failed", "cancelled"}:
        return {
            "ok": True,
            "job_id": args.job_id,
            "status": snapshot.get("status"),
            "message": "OCR job is already finished.",
        }

    store.mark_cancel_requested()
    return {
        "ok": True,
        "job_id": args.job_id,
        "status": "cancelling",
        "message": "Cancellation requested.",
    }


def cmd_resume(args: argparse.Namespace) -> dict[str, Any]:
    store = _load_store(args.job_id)
    snapshot = store.snapshot(refresh_liveness=True)
    if snapshot.get("status") == "running":
        return {"ok": False, "error": "OCR job is already running."}
    if snapshot.get("status") == "completed":
        return {"ok": False, "error": "OCR job is already completed."}

    store.clear_cancel_requested()
    store.prepare_for_resume()

    cmd = [sys.executable, str(_SCRIPT_DIR / "ocr_worker.py"), "--job-dir", str(store.job_dir)]
    env = {**os.environ, "PYTHONUNBUFFERED": "1"}
    with store.log_path.open("a", encoding="utf-8") as log_file:
        proc = subprocess.Popen(cmd, **_worker_popen_kwargs(log_file=log_file, env=env))
    store.set_worker_pid(proc.pid)

    return {
        "ok": True,
        "job_id": args.job_id,
        "status": "queued",
        "worker_pid": proc.pid,
        "message": "OCR job resumed.",
    }


def cmd_doctor(args: argparse.Namespace) -> dict[str, Any]:
    from ocr_setup import find_soffice_binary

    jobs_root = JobStore.jobs_root()
    soffice_path = find_soffice_binary()
    return {
        "ok": True,
        "jobs_root": str(jobs_root),
        "python": sys.executable,
        "soffice_found": soffice_path is not None,
        "soffice_path": str(soffice_path) if soffice_path is not None else None,
        "env_file": str((_SCRIPT_DIR.parent / ".env").resolve()),
    }


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Control durable background OCR jobs.")
    parser.add_argument("--json", action="store_true", help="Emit JSON output.")
    subparsers = parser.add_subparsers(dest="command", required=True)

    start = subparsers.add_parser("start", help="Start a new OCR background job.")
    start.add_argument("input_path", help="Path to the document or image to OCR.")
    start.add_argument("-t", "--threads", type=int, default=1, help="Number of OCR threads (default: 1; raise only if the API plan supports parallel OCR requests).")
    start.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    start.set_defaults(handler=cmd_start)

    status = subparsers.add_parser("status", help="Get the current status of a job.")
    status.add_argument("job_id", help="OCR job id.")
    status.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    status.set_defaults(handler=cmd_status)

    ls = subparsers.add_parser("list", help="List OCR jobs.")
    ls.add_argument("--state", default=None, help="Optional status filter.")
    ls.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    ls.set_defaults(handler=cmd_list)

    tail = subparsers.add_parser("tail", help="Read recent worker log lines.")
    tail.add_argument("job_id", help="OCR job id.")
    tail.add_argument("--lines", type=int, default=100, help="Number of lines to read.")
    tail.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    tail.set_defaults(handler=cmd_tail)

    artifacts = subparsers.add_parser("artifacts", help="List generated files for a job.")
    artifacts.add_argument("job_id", help="OCR job id.")
    artifacts.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    artifacts.set_defaults(handler=cmd_artifacts)

    cancel = subparsers.add_parser("cancel", help="Request cancellation for a running job.")
    cancel.add_argument("job_id", help="OCR job id.")
    cancel.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    cancel.set_defaults(handler=cmd_cancel)

    resume = subparsers.add_parser("resume", help="Resume a failed or cancelled job.")
    resume.add_argument("job_id", help="OCR job id.")
    resume.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    resume.set_defaults(handler=cmd_resume)

    doctor = subparsers.add_parser("doctor", help="Show local OCR control diagnostics.")
    doctor.add_argument("--json", dest="subcommand_json", action="store_true", help=argparse.SUPPRESS)
    doctor.set_defaults(handler=cmd_doctor)
    return parser


def main() -> None:
    parser = build_parser()
    args = parser.parse_args()

    try:
        payload = args.handler(args)
    except Exception as exc:
        payload = {"ok": False, "error": str(exc)}

    _emit(
        payload,
        as_json=bool(getattr(args, "json", False) or getattr(args, "subcommand_json", False)),
    )
    if not payload.get("ok", False):
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/ocr_runtime.py
content: """Shared runtime helpers for the OCR pipeline and background jobs."""

from __future__ import annotations

from pathlib import Path


class CancellationRequested(RuntimeError):
    """Raised when a background OCR job has been asked to stop."""


class ProgressReporter:
    """Interface for reporting progress from the OCR pipeline."""

    def job_started(self) -> None:
        pass

    def step_started(self, step_id: str, label: str) -> None:
        pass

    def step_completed(self, step_id: str) -> None:
        pass

    def pages_discovered(self, total_pages: int) -> None:
        pass

    def page_done(self, page_num: int, total_pages: int, *, skipped: bool = False) -> None:
        pass

    def page_failed(self, page_num: int, total_pages: int, error: str) -> None:
        pass

    def artifact_ready(self, name: str, path: str | Path) -> None:
        pass

    def warning(self, message: str) -> None:
        pass

    def log(self, message: str) -> None:
        pass

    def is_cancelled(self) -> bool:
        return False

    def check_cancelled(self) -> None:
        if self.is_cancelled():
            raise CancellationRequested("Cancellation requested.")


class NullProgressReporter(ProgressReporter):
    """Default reporter that keeps the current synchronous scripts simple."""



path: scripts/ocr_setup.py
content: """
First-run setup for the OCR Document Extraction skill.

Checks Python dependencies and .env configuration, creates local config
templates when needed, and validates that the configured model is likely
a vision-capable (multi-modal) model.

This script never installs packages automatically. When dependencies are
missing it prints the exact `pip install` command for the current Python
interpreter so users can review and run it explicitly.

Usage:
    python scripts/ocr_setup.py           # interactive setup/check
    python scripts/ocr_setup.py --check   # non-interactive check only (exit 0 = ready, 1 = not ready)
"""

from __future__ import annotations

import importlib
import os
import shutil
import sys
from pathlib import Path

_SCRIPT_DIR = Path(__file__).resolve().parent
_SKILL_ROOT = _SCRIPT_DIR.parent
_REQUIREMENTS = _SKILL_ROOT / "requirements.txt"
_ENV_FILE = _SKILL_ROOT / ".env"
_ENV_EXAMPLE = _SKILL_ROOT / ".env.example"

REQUIRED_PACKAGES: list[tuple[str, str]] = [
    ("openai", "openai"),
    ("dotenv", "python-dotenv"),
    ("fitz", "PyMuPDF"),
    ("PIL", "Pillow"),
    ("pdf2image", "pdf2image"),
    ("markdown2", "markdown2"),
    ("docx", "python-docx"),
    ("bs4", "beautifulsoup4"),
    ("lxml", "lxml"),
    ("latex2mathml", "latex2mathml"),
    ("openpyxl", "openpyxl"),
    ("unicodeit", "unicodeit"),
]

REQUIRED_ENV_VARS = ["API_KEY", "BASE_URL", "VLM_MODEL"]


def mock_ocr_enabled() -> bool:
    """Return True when OCR responses are being mocked for local smoke tests."""
    return bool(
        (os.getenv("OCR_MOCK_RESPONSE_FILE") or "").strip()
        or (os.getenv("OCR_MOCK_RESPONSE_TEXT") or "").strip()
    )


def _is_placeholder_env_value(value: str | None) -> bool:
    """Return True when *value* looks like an unedited template placeholder."""
    if not value:
        return True
    lowered = value.strip().lower()
    placeholders = (
        "your-api-key-here",
        "https://your-api-endpoint/v1",
        "your-vision-model-here",
    )
    if lowered.startswith("your-"):
        return True
    return lowered in placeholders


def check_packages() -> list[str]:
    """Return list of missing pip package names."""
    missing = []
    for import_name, pip_name in REQUIRED_PACKAGES:
        try:
            importlib.import_module(import_name)
        except ImportError:
            missing.append(pip_name)
    return missing


def python_install_command() -> str:
    """Return the explicit install command for the current interpreter."""
    return f'"{sys.executable}" -m pip install -r "{_REQUIREMENTS}"'


def check_env() -> list[str]:
    """Return list of missing environment variable names."""
    if mock_ocr_enabled():
        return []

    if _ENV_FILE.exists():
        from dotenv import dotenv_values
        values = dotenv_values(_ENV_FILE)
    else:
        values = {}

    missing = []
    for var in REQUIRED_ENV_VARS:
        val = os.getenv(var) or values.get(var)
        if _is_placeholder_env_value(val):
            missing.append(var)
    return missing


def bootstrap_env() -> None:
    """Create .env from .env.example if it doesn't exist."""
    if _ENV_FILE.exists():
        return
    if _ENV_EXAMPLE.exists():
        shutil.copy2(_ENV_EXAMPLE, _ENV_FILE)
        print(f"[setup] Created {_ENV_FILE} from template.")
    else:
        _ENV_FILE.write_text(
            "[REDACTED_SECRET]"
            "BASE_URL=https://your-api-endpoint/v1\n"
            "VLM_MODEL=your-vision-model-here\n",
            encoding="utf-8",
        )
        print(f"[setup] Created {_ENV_FILE} with placeholder values.")


def find_soffice_binary() -> Path | None:
    """Return the local LibreOffice binary when available."""
    found = shutil.which("soffice")
    if found:
        return Path(found)

    for path in (
        "/Applications/LibreOffice.app/Contents/MacOS/soffice",
        "/usr/bin/soffice",
        "/usr/local/bin/soffice",
        "/snap/bin/soffice",
        r"C:\Program Files\LibreOffice\program\soffice.exe",
        r"C:\Program Files (x86)\LibreOffice\program\soffice.exe",
    ):
        candidate = Path(path)
        if candidate.exists():
            return candidate

    return None


def check_system_deps() -> list[str]:
    """Check for optional but recommended system-level binaries."""
    missing = []
    if find_soffice_binary() is None:
        missing.append("LibreOffice (needed for DOC/DOCX/PPT/PPTX conversion)")
    if not (shutil.which("pdftoppm") or shutil.which("pdftocairo")):
        missing.append("Poppler (recommended for pdf2image fallback on difficult PDF pages)")
    return missing


def run_check() -> bool:
    """Run all checks silently. Returns True if the skill is ready to use."""
    ok = True

    pkg_missing = check_packages()
    if pkg_missing:
        print(f"[check] Missing packages: {', '.join(pkg_missing)}")
        ok = False

    env_missing = check_env()
    if env_missing:
        print(f"[check] Missing env vars: {', '.join(env_missing)}")
        ok = False

    if ok:
        print("[check] Skill is ready to use.")
    return ok


def run_setup() -> None:
    """Interactive first-run setup."""
    print("=" * 56)
    print("  OCR Document Extraction — First-Run Setup")
    print("=" * 56)

    # 1. Python packages
    missing_pkgs = check_packages()
    if missing_pkgs:
        print(f"\n[1/3] Missing Python packages: {', '.join(missing_pkgs)}")
        print("      Automatic installation is intentionally disabled.")
        print(f"      Please review and run manually:\n      {python_install_command()}")
    else:
        print("\n[1/3] Python dependencies — OK")

    # 2. .env file
    bootstrap_env()
    env_missing = check_env()
    if env_missing:
        print(f"\n[2/3] Please configure these in {_ENV_FILE}:")
        for var in env_missing:
            print(f"      - {var}")
        print()
        print("      IMPORTANT: This skill requires a VISION (multi-modal) model.")
        print("      The model must accept image inputs alongside text prompts.")
        print("      Please confirm with your provider that the model is vision-capable.")
        print("      Text-only models will NOT work.")
    else:
        print("\n[2/3] Environment configuration — OK")
        print("      NOTE: This skill requires a vision-capable (multi-modal) model")
        print("      that accepts image inputs alongside text prompts.")
        print("      Text-only models will NOT work.")

    # 3. System dependencies
    sys_missing = check_system_deps()
    if sys_missing:
        print(f"\n[3/3] Optional system dependencies not found:")
        for dep in sys_missing:
            print(f"      - {dep}")
        print("      (LibreOffice is needed for office documents; Poppler improves PDF fallback rendering)")
    else:
        print("\n[3/3] System dependencies — OK")

    print()
    if missing_pkgs:
        print("Setup incomplete. Install the missing Python packages, then re-run the check.")
    elif not env_missing:
        print("Setup complete. Ready to use!")
    else:
        print(f"Edit {_ENV_FILE} with your credentials, then you're ready.")


if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="OCR skill setup & dependency check.")
    parser.add_argument(
        "--check",
        action="store_true",
        help="Non-interactive check only (exit 0 = ready, 1 = not ready).",
    )
    args = parser.parse_args()

    if args.check:
        sys.exit(0 if run_check() else 1)
    else:
        run_setup()


path: scripts/vendor/README.md
content: # Vendored Frontend Assets

This directory contains local copies of JavaScript files used by
`scripts/md_to_html.py`.

- Mermaid `11.14.0` from the `mermaid` npm package
- MathJax `3.2.2` from the `mathjax` npm package

These files are bundled to keep HTML generation self-contained and to avoid
runtime downloads from CDNs.


path: scripts/image_to_md.py
content: """Batch OCR: extract Markdown from a folder of page images via VLM."""

from __future__ import annotations

import argparse
import logging
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path

_SCRIPT_DIR = Path(__file__).resolve().parent
if str(_SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(_SCRIPT_DIR))

from ai_service_vlm import ai_request  # noqa: E402
from ocr_prompt import ocr_to_markdown  # noqa: E402
from ocr_runtime import CancellationRequested, NullProgressReporter, ProgressReporter  # noqa: E402

logger = logging.getLogger(__name__)

IMAGE_EXTS: set[str] = {".jpeg", ".jpg", ".png", ".webp", ".bmp", ".tiff", ".tif"}

DEFAULT_MAX_ATTEMPTS = 2
DEFAULT_RETRY_DELAY = 2.0


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------


def _collect_images(folder: Path) -> list[Path]:
    """Return supported image files in *folder*, sorted by name."""
    return sorted(
        (p for p in folder.iterdir() if p.is_file() and p.suffix.lower() in IMAGE_EXTS),
        key=lambda p: p.name,
    )


def _ocr_single_image(
    image_path: Path,
    *,
    max_attempts: int = DEFAULT_MAX_ATTEMPTS,
    retry_delay: float = DEFAULT_RETRY_DELAY,
) -> str:
    """Send one image to the VLM, retrying on transient failures."""
    last_exc: Exception | None = None
    for attempt in range(1, max_attempts + 1):
        try:
            return ai_request(prompt=ocr_to_markdown, image_url=str(image_path))
        except Exception as exc:
            last_exc = exc
            if attempt < max_attempts:
                wait = retry_delay * attempt
                logger.warning(
                    "Attempt %d/%d failed for %s (%s); retrying in %.1fs …",
                    attempt, max_attempts, image_path.name, exc, wait,
                )
                time.sleep(wait)
    raise RuntimeError(
        f"All {max_attempts} attempts failed for {image_path.name}"
    ) from last_exc


def _build_combined_markdown(
    pages: list[tuple[int, str]],
    *,
    page_marker: bool = True,
) -> str:
    """Assemble per-page markdown texts into one document.

    Each page is separated by a blank line.  When *page_marker* is True a
    ``Page_Order_N`` sentinel line is inserted before each page (consumed
    downstream by md_to_docx for page-break insertion).
    """
    sections: list[str] = []
    for order, text in pages:
        text = text.strip()
        if not text:
            continue
        if page_marker:
            sections.append(f"Page_Order_{order}\n\n{text}")
        else:
            sections.append(text)
    return "\n\n".join(sections) + "\n"


# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------


def _process_one(
    order: int,
    image_path: Path,
    page_folder: Path,
    total: int,
    reporter: ProgressReporter,
) -> tuple[int, str]:
    """OCR a single image and write its page ``.md`` file.

    Returns ``(order, md_text)`` on success; propagates exceptions so the
    caller can record the failure without aborting other workers.
    """
    reporter.check_cancelled()
    print(f"[{order}/{total}] OCR scanning: {image_path.name}")
    md_text = _ocr_single_image(image_path)
    reporter.check_cancelled()
    page_file = page_folder / f"{image_path.stem}.md"
    page_file.write_text(md_text, encoding="utf-8")
    print(f"  [OK]   {image_path.name} → {page_file.name}")
    reporter.page_done(order, total)
    return order, md_text


def image_folder_to_md(
    source_folder: Path,
    page_folder: Path,
    combined_file: Path,
    page_marker: bool = True,
    threads: int = 1,
    reporter: ProgressReporter | None = None,
    resume: bool = False,
) -> tuple[int, int]:
    """OCR every image in *source_folder*, write individual ``.md`` files
    into *page_folder*, and merge them into *combined_file*.

    When *threads* > 1 images are processed concurrently.  Each image is
    independent: a failure in one thread does not affect the others.
    """
    reporter = reporter or NullProgressReporter()
    images = _collect_images(source_folder)
    if not images:
        logger.warning("No supported images found in %s", source_folder)
        print(f"[WARN] No supported images in {source_folder}")
        return 0, 0

    page_folder.mkdir(parents=True, exist_ok=True)
    combined_file.parent.mkdir(parents=True, exist_ok=True)

    total = len(images)
    pages: list[tuple[int, str]] = []
    failed: list[str] = []
    pending: list[tuple[int, Path]] = []

    for order, img in enumerate(images, start=1):
        reporter.check_cancelled()
        page_file = page_folder / f"{img.stem}.md"
        if resume and page_file.exists():
            try:
                pages.append((order, page_file.read_text(encoding="utf-8")))
                print(f"[{order}/{total}] Reusing existing OCR: {page_file.name}")
                reporter.page_done(order, total, skipped=True)
                continue
            except Exception as exc:
                logger.warning("Could not reuse %s (%s); OCR will run again.", page_file.name, exc)
        pending.append((order, img))

    with ThreadPoolExecutor(max_workers=max(1, threads)) as executor:
        future_to_img = {
            executor.submit(_process_one, order, img, page_folder, total, reporter): img
            for order, img in pending
        }
        for future in as_completed(future_to_img):
            img = future_to_img[future]
            try:
                pages.append(future.result())
            except CancellationRequested:
                raise
            except Exception as exc:
                logger.error("Failed: %s — %s", img.name, exc)
                print(f"  [FAIL] {img.name}: {exc}")
                failed.append(img.name)
                try:
                    order = next(order for order, pending_img in pending if pending_img == img)
                except StopIteration:
                    order = 0
                if order:
                    reporter.page_failed(order, total, str(exc))

    pages.sort(key=lambda t: t[0])
    reporter.check_cancelled()

    combined_file.write_text(
        _build_combined_markdown(pages, page_marker=page_marker),
        encoding="utf-8",
    )

    succeeded = total - len(failed)
    summary = f"Done — {succeeded} succeeded, {len(failed)} failed out of {total} images."
    print(f"\n{summary}")
    if failed:
        print("Failed images:")
        for name in failed:
            print(f"  - {name}")

    return succeeded, len(failed)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------


def main() -> None:
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s [%(levelname)s] %(message)s",
    )

    parser = argparse.ArgumentParser(description="Batch OCR images to Markdown via VLM.")
    parser.add_argument("source_folder", type=Path, help="Folder containing source images.")
    parser.add_argument("page_folder", type=Path, help="Output folder for individual page .md files.")
    parser.add_argument("combined_file", type=Path, help="Path for the combined Markdown file.")
    parser.add_argument("--no-page-marker", action="store_true", help="Omit page markers in the combined file.")
    parser.add_argument("-t", "--threads", type=int, default=1, help="Number of concurrent OCR threads (default: 1).")
    args = parser.parse_args()

    image_folder_to_md(
        source_folder=args.source_folder,
        page_folder=args.page_folder,
        combined_file=args.combined_file,
        page_marker=not args.no_page_marker,
        threads=args.threads,
    )


if __name__ == "__main__":
    main()


path: skill.yaml
content: name: finance-ocr-pro
version: 1.0.7
author: RizMoon
description: >
  Extract structured Markdown, HTML, DOCX, and Excel from scanned documents,
  especially financial and complex chart-heavy files. Sends rendered page images
  and OCR prompts to a configured OpenAI-compatible VLM endpoint. HTML report
  assets are bundled locally; no runtime CDN downloads are performed. DOCX
  output includes improved native Word equation restoration for recognized
  LaTeX formulas.
license: MIT

permissions:
  - network
  - filesystem
  - shell

config:
  API_KEY:
    type: string
    required: true
    description: "API key for the OpenAI-compatible VLM endpoint that performs OCR"
    secret: true
  BASE_URL:
    type: string
    required: true
    description: "Base URL of the VLM endpoint (e.g. https://api.openai.com/v1). Page images are transmitted here during OCR"
  VLM_MODEL:
    type: string
    required: true
    description: "Vision-capable model identifier (e.g. gpt-4o). Must support image inputs"

entryPoint:
  type: natural
  prompt: |
    Use this skill when the user asks to OCR, transcribe, extract, or convert
    a document's contents. This skill is especially strong for financial
    documents and other complex files with dense tables, charts, graphs, 
    footnotes, formulas, and multi-part layouts. It produces Markdown, HTML,
    DOCX with improved native Word equation output for recognized LaTeX
    formulas, and Excel. Before starting, announce the default
    mode, model, thread count, result path, and that page images will be
    transmitted to BASE_URL. Also tell the user that this skill supports
    multi-thread OCR, but higher thread counts should only be used when the
    API endpoint, rate limits, and subscription plan support parallel OCR
    requests. First resolve the interpreter path: on macOS/Linux use
    `.venv/bin/python` if present, otherwise `python3`; on Windows use
    `.venv/Scripts/python.exe` if present, otherwise `python`. Then proceed
    unless the user changes defaults. Prefer
    `<python> scripts/ocrctl.py --json start ...` for long jobs and
    `<python> scripts/ocr_main.py ...` only for very small inline jobs.

triggers:
  keywords:
    - "ocr"
    - "transcribe"
    - "extract"
    - "convert to markdown"
    - "convert to docx"
    - "convert to excel"


path: SECURITY.md
content: # Security Notes

Finance OCR Pro is designed to be transparent about the behaviors that could
affect a skill-review risk rating.

## Current Security Posture

- No automatic dependency installation. `scripts/ocr_setup.py` and
  `scripts/ocr_main.py` validate dependencies but never run `pip install`
  automatically.
- No runtime third-party asset downloads. Mermaid and MathJax are vendored
  under `scripts/vendor/` and embedded into generated HTML locally.
- No arbitrary shell execution. The project does not use `shell=True`.
  Subprocess calls are restricted to:
  - the local Python interpreter for detached OCR workers
  - LibreOffice (`soffice`) for office-document conversion
  - `osascript` on macOS for Keynote/Pages export
- Network traffic is purpose-bound:
  - OCR requests send rendered page images and prompts to the user-configured
    `BASE_URL`
  - if users require local-only processing, they must point `BASE_URL` to a
    trusted local service

## Reviewer Checklist

- Verify that `BASE_URL` is clearly disclosed anywhere OCR is described.
- Confirm that `.env.example` is shipped but populated `.env` files are not.
- Confirm that `scripts/vendor/` is included in release artifacts.
- Confirm that runtime packaging does not restore CDN fallbacks or automatic
  dependency installation.


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal:  | SkillSpector: suspicious (CRITICAL)