Skill: baorepo/ee-datasheet-master (version 1.0.0)

--- name: ee-datasheet-master description: "Use when user has/is reading a component datasheet or spec sheet to find chip parameters: pinout, voltage, I2C address, timing, register map, electrical characteristics. Trigger on PDF+chip questions. Also: 规格书, 数据手册, 芯片参数. All IC types." --- # EE Datasheet Master ## Iron Law: PDF Content Only ``` ALL DATA MUST ORIGINATE FROM THE PDF. Allowed: Extract → Calculate from extracted data Forbidden: Use prior knowledge → Fill gaps with guesses ``` ### Allowed Derivations | Type | Example | |------|---------| | Mathematical calculation | P = V × I from voltage and current | | Unit conversion | dBm → mW, binary → hex | | Address calculation | "001000x" → 0x10/0x11 | | Counting | Pin count from Pin Description table | **When deriving: Show source data (page) + calculation steps + result** ### Forbidden Behaviors | Behavior | Correction | |----------|------------| | "I know this chip..." | Find the spec in PDF | | "Typical value is..." | Read the actual value from PDF | | "Similar chips have..." | This one may differ | | Guessing to fill gaps | Output "NOT SPECIFIED IN DATASHEET" + acquisition path (see below) | --- ## When the PDF Cannot Provide the Answer "NOT SPECIFIED IN DATASHEET" is not a dead end. Always follow it with **how to obtain the missing information**. ### Response Template > "[Parameter] is not specified in this datasheet. > To obtain it: [specific method below]." If the datasheet references an application note or supplementary doc by name, cite it: > "Section X references Application Note [AN-xxx] for this topic — search [Manufacturer] website." ### Reasoning Framework for Missing Parameters When a parameter is absent, reason through these questions to give a concrete, actionable path: **1. Why is it missing?** - *Wrong document* — this is a brief/product datasheet; the full reference manual or application note contains it → identify the correct document by name - *Test-condition mismatch* — the spec exists but not at the user's specific conditions (load, frequency, temperature) → explain which conditions differ and how that affects the value - *Application-dependent* — the value depends on external components or PCB layout the user controls → explain what determines it and how to calculate or simulate - *Manufacturer-controlled* — the data is from qualification testing, not released publicly → identify the right contact channel **2. What does the user actually need it for?** - Design margin check → an approximation or worst-case bound may be sufficient - Debugging a failure → direct measurement in the actual circuit is more reliable than a datasheet value - Qualification / compliance → only manufacturer-provided data is acceptable **3. What is the most direct path given the above?** Tailor the recommendation to the specific parameter and context — a thermal resistance question for an LDO in a hot enclosure calls for a different answer than the same question for a signal-path op-amp. Reason about: what equipment would give this measurement, what document would contain this spec, or what formula derives this value from things the user can measure or control. --- ## 6-Phase Workflow ``` ┌─────────────────────────────────────────────────────────────┐ │ Phase 0: Pre-scan → 全文扫描，建结构地图 │ │ Phase 1: Diagnosis → text vs image PDF 决策 │ │ Phase 2: Device ID → 确认器件，推断关键参数 │ │ Phase 2b: Targeted Scan → 推断 patterns，二次精准扫描 │ │ Phase 3: Section Mapping → 定位各功能区页码 │ │ Phase 4: Extraction → 精准提取 + TEMPLATES 结构化输出 │ └─────────────────────────────────────────────────────────────┘ ``` **See [PDF_STRATEGY.md](PDF_STRATEGY.md) for the entry-point decision table and detailed workflow. Read that first — it tells you which phase to start at before running any command.** ### Quick Reference **Most common case — device named, 1–2 specific parameters asked (start here):** ```bash # Phase 3: Search directly for the parameter the user asked about python scripts/pdf_tools.py search_table <pdf_path> "<parameter>" # e.g. "quiescent current", "dropout voltage" python scripts/pdf_tools.py search <pdf_path> "<parameter>" # try alternate phrasings if first is empty # Phase 4: Read the identified page python scripts/pdf_tools.py text <pdf_path> <page_num> python scripts/pdf_tools.py tables <pdf_path> <page_num> ``` **Less common — unknown PDF, open-ended analysis, or complex multi-parameter extraction:** ```bash # Phase 0: Pre-scan (slow — only when you need a structural map) python scripts/pdf_tools.py info <pdf_path> python scripts/pdf_tools.py page_hints <pdf_path> # scan ALL pages → minutes on large docs # Phase 2: Identify Device (only if device is not already known) python scripts/pdf_tools.py text <pdf_path> 1 # Phase 2b: Targeted re-scan (complex ICs only — charger, MCU, CODEC) python scripts/pdf_tools.py dump_patterns > /tmp/custom_patterns.json python scripts/pdf_tools.py page_hints <pdf_path> --patterns /tmp/custom_patterns.json # Phase 3: Caption-based section mapping python scripts/pdf_tools.py search_caption <pdf_path> # find Figure/Table captions python scripts/pdf_tools.py search <pdf_path> "Electrical Characteristics" ``` --- ## Parameter Inference (LLM Decision) ### Universal Parameters (for full-analysis queries only) When the user asks for a complete analysis or overview, extract these 5 baseline parameters. **Skip this for targeted single-parameter queries** — if the user asks "what is the dropout voltage?", go find that, not the package outline. | Parameter | Search Keywords | Notes | |-----------|----------------|-------| | **Manufacturer** | First page header/footer | Company name | | **Part Number** | First page title | Full part number | | **Package** | "Package", "封装" | Must include pin count (e.g., QFN-32) | | **Operating Voltage** | "VDD", "VCC", "Supply Voltage", "电源电压" | Range: min to max | | **Operating Temperature** | "Operating Temperature", "工作温度" | Range: min to max | ### Device-Specific Parameters (Inferred by LLM) After identifying the device, infer what specs matter: ``` 1. Read device description (first 3 pages) 2. Understand: What does this device DO? 3. Infer: What specs matter for this device? 4. Search: Use pdf_tools to locate those specs ``` For the complete device-type → key specs lookup table and per-device extraction shortcuts, see **[PDF_STRATEGY.md → Phase 2 and Device-Type Shortcuts](PDF_STRATEGY.md)**. **Key insight:** Device description tells you what to measure. Don't use predefined lists. --- ## Output Format ```markdown # [Part Number] Datasheet Analysis ## Summary [1-2 sentences] ## Key Specifications | Parameter | Min | Typ | Max | Unit | Source | Notes | |-----------|-----|-----|-----|------|--------|-------| | ... | ... | ... | ... | ... | Page X, "Table Name" | | | [unavailable param] | — | — | — | ... | NOT SPECIFIED | Measure: [method] | ## Pin Configuration - Package: [Type]-[Pin Count] - Power Domains: [List ALL with pin numbers] - Interfaces: [I2C/SPI/UART with addresses] ## Critical Design Considerations 1. [Issue with guidance] ## Common Pitfalls - [Pitfall]: [How to avoid] ``` --- ## Common Mistakes | Mistake | Example | Correction | |---------|---------|------------| | Missing pin count | "QFN package" | "QFN-32 package" | | Partial power domains | "VDD" only | "VDD (pins 1, 13, 32)" | | I2C address wrong | "0x18" | Show calculation from format | | Missing source | "SNR: 93 dB" | "SNR: 93 dB (Page 8, Typ)" | | Hallucinated specs | Any value without source | Always cite page and table | --- ## Red Flags - STOP and Verify If you think: - "I know this chip..." - "Typically this value is..." - "Based on my experience..." - "Similar chips have..." **STOP → Re-read PDF → Extract from source** --- ## Reference Files | File | Purpose | |------|---------| | [PDF_STRATEGY.md](PDF_STRATEGY.md) | 6-phase workflow, device-type extraction shortcuts | | [TEMPLATES.md](TEMPLATES.md) | Structured output templates: device_info, power_domains, I2C, SPI, electrical_specs | | [scripts/pdf_tools.py](scripts/pdf_tools.py) | PDF extraction tools |

path: TEMPLATES.md
content: # Extraction Templates

Extraction templates for datasheet analysis. Each template defines the required structure and validation rules.

---

## Device Info Template

```json
{
    "_template": "device_info",
    "_version": "1.0",
    "manufacturer": "",
    "part_number": "",
    "full_part_number": "",
    "description": "",
    "category": "",
    "package": {
        "type": "",
        "pin_count": 0,
        "dimensions": ""
    },
    "temperature_range": {
        "operating": {"min": "", "max": ""},
        "storage": {"min": "", "max": ""}
    },
    "source": {
        "cover_page": 1,
        "pin_config_page": 0,
        "specs_page": 0
    }
}
```

**Validation Rules:**
- `package.pin_count` MUST match the number in `package.type`
- Example: `QFN-32` → `pin_count: 32`
- All source page numbers must be filled after extraction

---

## Power Domains Template

```json
{
    "_template": "power_domains",
    "_version": "1.0",
    "power_domains": [
        {
            "name": "",
            "pin_names": [],
            "min_voltage": "",
            "typ_voltage": "",
            "max_voltage": "",
            "current_typ": "",
            "purpose": "",
            "decoupling": ""
        }
    ],
    "ground_pins": [],
    "source_page": 0
}
```

**Extraction Rules:**
- MUST extract ALL power pins from Pin Description table
- NOT just examples - list all pins with pin numbers
- WRONG: `"VDD"`
- CORRECT: `"VDD (pins 1, 13, 32, 48)"`
- Check for special power domains: CPVDD, HPVDD, PLLVDD, etc.

---

## I2C Interface Template

```json
{
    "_template": "i2c_interface",
    "_version": "1.0",
    "interface_type": "I2C",
    "pins": {
        "scl": "",
        "sda": ""
    },
    "address": {
        "format_shown": "",
        "format_explanation": "",
        "address_7bit": [],
        "address_8bit_write": [],
        "address_8bit_read": [],
        "calculation_steps": ""
    },
    "max_speed": "",
    "pull_up_required": true,
    "source_page": 0
}
```

**I2C Address Calculation Guide:**

| Format in Datasheet | Example | Calculation | 7-bit Address |
|---------------------|---------|-------------|---------------|
| `001000x` | x=AD0 | 0b0010000=0x10, 0b0010001=0x11 | 0x10/0x11 |
| `11010xx` | xx=A1A0 | 0b1101000=0x68 to 0b1101011=0x6B | 0x68-0x6B |
| `0x20` (8-bit write) | - | 0x20 >> 1 | 0x10 |
| `0xD0` (8-bit write) | - | 0xD0 >> 1 | 0x68 |

**Example Address Calculation:**
```json
{
    "format_shown": "001000x",
    "format_explanation": "where x equals AD0 pin state",
    "address_7bit": ["0x10", "0x11"],
    "calculation_steps": [
        "AD0 = 0: 0010000 binary = 0x10",
        "AD0 = 1: 0010001 binary = 0x11"
    ]
}
```

**Common I2C Device Addresses** *(validation reference — cross-check extracted address against these known values; do not output as fact without verifying against the PDF)*:

| Device Type | Common Addresses |
|-------------|------------------|
| I2C EEPROM (24Cxx) | 0x50-0x57 |
| I2C RTC (DS3231, PCF8563) | 0x51, 0x68 |
| I2C Temp Sensor (LM75) | 0x48-0x4F |
| I2C ADC (ADS1115) | 0x48-0x4B |
| I2C IMU (MPU6050) | 0x68-0x69 |
| I2C Port Expander (MCP23017) | 0x20-0x27 |

---

## SPI Interface Template

```json
{
    "_template": "spi_interface",
    "_version": "1.0",
    "interface_type": "SPI",
    "pins": {
        "mosi": "",
        "miso": "",
        "sck": "",
        "cs": ""
    },
    "modes_supported": [],
    "max_speed": "",
    "source_page": 0
}
```

---

## Electrical Specs Template

```json
{
    "_template": "electrical_specs",
    "_version": "1.0",
    "absolute_maximum": [
        {
            "parameter": "",
            "min": "",
            "max": "",
            "unit": "",
            "notes": "",
            "source": ""
        }
    ],
    "recommended_operating": [
        {
            "parameter": "",
            "min": "",
            "typ": "",
            "max": "",
            "unit": "",
            "conditions": "",
            "source": ""
        }
    ],
    "key_parameters": [
        {
            "parameter": "",
            "min": "",
            "typ": "",
            "max": "",
            "unit": "",
            "conditions": "",
            "footnotes": [],
            "source": ""
        }
    ]
}
```

**Extraction Rules:**
- Extract raw values without transformation
- Include column name (Min/Typ/Max)
- Note test conditions
- Check for footnotes (often contain critical constraints)
- Cite source: page number and table name

---

## Raw Extraction Value Format

Each extracted value MUST include source information:

```json
{
    "parameter": "DAC SNR",
    "value": "93",
    "unit": "dB",
    "column": "Typ",
    "min": "85",
    "max": "95",
    "conditions": "A-weighted, 24-bit",
    "page": 8,
    "table_name": "DAC Performance",
    "confidence": "HIGH"
}
```

**Confidence Levels:**

| Level | Definition | Example |
|-------|------------|---------|
| **HIGH** | Direct read from clear table, verified by re-read | "93 dB" from clean table |
| **MEDIUM** | Some ambiguity (OCR issues, merged cells) | Value from slightly blurry OCR |
| **LOW** | Inferred from context, multiple interpretations | Value estimated from graph |
| **UNVERIFIED** | Extracted but not yet verified | Initial extraction before validation |


path: README.md
content: # EE Datasheet Master

A skill for extracting specifications from electronic component datasheets with full source citations. All data must originate from the PDF — no prior knowledge, no guessing.

## Supported Device Types

Power management · MCU/SoC · ADC/DAC/CODEC · Sensor (IMU, temperature, pressure) · Flash/EEPROM · Interface (USB, Ethernet, CAN, RS-485) · Discretes (MOSFET, diode, gate logic) · Clock/PLL · Port expander · Battery charger/gauge

Optimized for English and Chinese datasheets. Other languages have degraded accuracy.

---

## Workflow Overview

```
Phase 0   → page_hints (all pages, default patterns)  → coarse structural map
Phase 1   → info                                       → text vs image PDF
Phase 2   → text page 1-2                             → device ID + infer params
Phase 2b  → page_hints --patterns (inferred keywords)  → fine-grained section map
Phase 3   → search_caption + search                    → confirm page numbers
Phase 4   → tables / search_table / nearby_text        → extract values
           render_page                                 → timing / diagram pages
```

See [PDF_STRATEGY.md](PDF_STRATEGY.md) for the complete workflow with decision rules.

---

## Dependencies

`scripts/pdf_tools.py` requires the following Python packages:

```bash
# Required — core PDF text/table extraction
pip install pdfplumber

# Required for render_page (page → PNG) — install at least one:
pip install pymupdf        # preferred: MuPDF engine, faster
pip install pypdfium2      # fallback: PDFium engine (Chrome's renderer)
```

| Package | Role | Required? |
|---------|------|-----------|
| `pdfplumber` | Text, table, and page metadata extraction | **Yes** |
| `pymupdf` | Page rendering (`render_page`), primary renderer | One of the two |
| `pypdfium2` | Page rendering (`render_page`), fallback renderer | One of the two |

`render_page` tries `pymupdf` first, falls back to `pypdfium2`. If neither is installed, image/timing/diagram pages cannot be rendered and will fail. All other commands (`text`, `tables`, `search`, etc.) work with `pdfplumber` alone.

---

## Tools (`scripts/pdf_tools.py`)

| Command | Purpose |
|---------|---------|
| `info` | Page count, text vs image detection |
| `page_hints [page] [--patterns json]` | Per-page structural signals + heuristic section labels |
| `page_stats [page]` | Raw metrics: char/word/image/rect/line/curve/table counts |
| `dump_patterns` | Print default patterns JSON (base for `--patterns` customization) |
| `text <page>` | Extract text from a page |
| `search <keyword>` | Find keyword across all pages with context |
| `search_caption [keyword]` | Find Figure/Table captions (en + zh) |
| `nearby_text <page> <regex>` | Regex match with surrounding context lines |
| `tables <page>` | Extract all tables from a page |
| `search_table <keyword>` | Find table rows containing keyword |
| `toc` | Extract table of contents (when available) |
| `render_page <page> [dpi] [out]` | Render page to PNG (pymupdf → pypdfium2 fallback) |

### `--patterns` Usage

`page_hints --patterns` accepts a custom JSON that **replaces** the default pattern set. Use it for a targeted second-pass scan after device identification:

1. `dump_patterns > /tmp/p.json` — get defaults as base
2. Add device-specific labels using proprietary terms from the Features list (e.g., `"VINDPM"`, `"clock tree"`, `"DMA controller"`)
3. Re-run `page_hints --patterns /tmp/p.json` to locate domain-specific sections

Patterns should be **section heading phrases or proprietary feature names**, not generic words. Good: `"charge state machine"`. Bad: `"charge"`.

---

## Known Limitations

### Image-based PDFs
When `is_text_based: false`, text extraction fails entirely. Use `render_page` to render pages as PNG and read visually. **Cross-validate mandatory**: manufacturer + part number from rendered image must match context. If mismatch → output `UNABLE TO VERIFY`. All image-PDF extractions default to `LOW` confidence.

### Complex Table Structures
Merged cells, multi-row headers, and rotated text may mis-extract. Fallback: use `text` to get page text, or `render_page` to read visually. Note: "Extracted from page text / rendered image".

### Graph Data
Numerical values from characteristic curves cannot be extracted precisely. Look for an accompanying data table. If absent: note "Approximate — read from curve".

### Multi-Variant Datasheets
Datasheets covering a product family (e.g., STM32F103C**x**T6) may have variant-specific tables. Always confirm which variant's column applies.

### `page_hints` False Positives
Hints are heuristic. Generic words like `"sequence"` or `"efficiency"` can fire on many pages. Use `score` + `reasons` together — a single low-confidence hit is not reliable. Cross-confirm with `search_caption` or `search`.

---

## What This Skill Cannot Do

| Task | Reason |
|------|--------|
| Guarantee 100% accuracy | LLM extraction has inherent limits |
| Decrypt password-protected PDFs | Not supported |
| Extract 3D models / CAD data | Text and tables only |
| Analyze circuit behavior | Extracts component values, does not simulate |
| Read non-text vector graphics | Render page and read visually |

---

## File Reference

| File | Purpose |
|------|---------|
| `SKILL.md` | Skill trigger rules, iron law, quick reference |
| `PDF_STRATEGY.md` | Full 6-phase workflow with decision trees and device-type shortcuts |
| `TEMPLATES.md` | JSON extraction templates (device info, power domains, I2C, electrical specs) |
| `scripts/pdf_tools.py` | PDF extraction tool — all commands listed above |


path: PDF_STRATEGY.md
content: # PDF Reading Strategy

## Step 0: Choose Your Entry Point (Do This First)

Before running any command, decide where to start. `page_hints` on a 100-page PDF takes minutes — only run it when you genuinely need a structural map of an unknown document.

| Situation | Start at | Why |
|-----------|----------|-----|
| **Device named + 1–2 specific parameters** | **Phase 3** | You already know what and where to look — go directly to `search` / `search_table`. `page_hints` adds no value here. |
| Follow-up question on same document | Phase 3 or 4 | Document already known; rescanning wastes time. |
| Unknown PDF, open-ended analysis ("tell me about this chip") | Phase 0 | Need structural map before anything. |
| Suspected or confirmed image-based PDF | Phase 0 | Must confirm PDF type before choosing read path. |
| Complex IC (charger, MCU, CODEC) + many parameters at once | Phase 0–2b | Proprietary section names need targeted patterns to locate. |

**Fallback rule:** If targeted extraction fails (parameter not found, table malformed), then step back to an earlier phase — run `page_hints` or `search` rather than guessing.

---

## Core Principle

**Navigate first, extract second.** Never read pages blindly. The phases below are a toolkit — pick the right tool for the job.

```
┌─────────────────────────────────────────────────────────────────┐
│  Phase 0: Pre-scan        →  默认 patterns 扫全文，建粗粒度地图  │
│  Phase 1: Diagnosis       →  决定读取路径 (text vs image)        │
│  Phase 2: Device ID       →  确认器件身份，推断要提取的参数      │
│  Phase 2b: Targeted Scan  →  用推断出的关键词二次扫描，细化地图  │
│  Phase 3: Section Mapping →  精确定位各功能区的页码              │
│  Phase 4: Extraction      →  按器件类型执行有针对性的提取        │
└─────────────────────────────────────────────────────────────────┘
```

## Table of Contents

- [Phase 0: Pre-scan](#phase-0-pre-scan-unknown-pdfs-and-open-ended-queries-only)
- [Phase 1: Diagnosis](#phase-1-diagnosis)
- [Phase 2: Device Identification](#phase-2-device-identification)
- [Phase 2b: Targeted Re-scan with Inferred Patterns](#phase-2b-targeted-re-scan-with-inferred-patterns)
- [Phase 3: Section Mapping](#phase-3-section-mapping)
- [Phase 4: Targeted Extraction](#phase-4-targeted-extraction)
- [Device-Type Extraction Shortcuts](#device-type-extraction-shortcuts)
- [Error Recovery](#error-recovery)
- [Image PDF: Full Workflow](#image-pdf-full-workflow)

---

## Phase 0: Pre-scan (unknown PDFs and open-ended queries only)

**Purpose:** Build a structural map of the entire document in one pass. Use this when you don't yet know where to look — if the device and parameter are already known, go to Phase 3 instead. `page_hints` on a large datasheet is the slowest operation available; only run it when the structural map will actually inform your next step.

```bash
python scripts/pdf_tools.py info <pdf_path>
python scripts/pdf_tools.py page_hints <pdf_path>   # scan ALL pages
```

`page_hints` returns per-page signals and heuristic labels. Use this to identify candidate pages for each section:

| Hint Label | Target Section |
|------------|---------------|
| `likely_pinout_page` | Pin configuration, Pin description table |
| `likely_timing_diagram` | Timing waveforms (almost always image, use render_page) |
| `likely_block_diagram` | Functional diagram (image, use render_page) |
| `likely_curve_page` | Electrical characteristics curves (image) |
| `likely_package_page` | Package outline, mechanical dimensions |
| `likely_register_figure` | Register map, bit field descriptions |
| `likely_table_page` | Electrical specs table, absolute maximum ratings |

**Treat hints as candidates, not facts.** A `score > 0.5` with 2+ reasons is reliable. A page that matches `likely_table_page` near a `search` hit for "Electrical Characteristics" is almost certainly correct.

### Pre-scan Output (Record This)

```
PDF: <filename>
Pages: N | Type: text / image
Candidate Pages:
  - Pinout: [p12, p13]
  - Timing diagrams: [p18, p19, p22]
  - Block diagram: [p4]
  - Electrical specs tables: [p8, p9, p10]
  - Package: [p27]
  - Registers: [p30-p45]
```

---

## Phase 1: Diagnosis

Use the `info` result already obtained in Phase 0 — no need to re-run.

| `is_text_based` | Strategy |
|-----------------|----------|
| `true` | Text extraction path. Use `page_hints` scan + targeted reads. |
| `false` | **Image path.** Render key pages with `render_page`, read visually. |

### Image PDF Path

```bash
# Render first 3 pages to identify device
python scripts/pdf_tools.py render_page <pdf_path> 1 180
python scripts/pdf_tools.py render_page <pdf_path> 2 180
python scripts/pdf_tools.py render_page <pdf_path> 3 180
```

Read rendered images visually. Cross-validate: manufacturer + part number from image must match filename/context. If mismatch → output `UNABLE TO VERIFY`.

For image PDFs, use `render_page` for ALL sections. Typical flow:
1. Render cover → identify device
2. Render pre-scan candidate pages → confirm section locations
3. Render target pages → extract specs visually

---

## Phase 2: Device Identification

```bash
python scripts/pdf_tools.py text <pdf_path> 1
# If page 1 is cover only (short text):
python scripts/pdf_tools.py text <pdf_path> 2
```

Extract:
1. **Part number** — first page title, usually `[A-Z]+[0-9]+` pattern
2. **Manufacturer** — first page header/footer
3. **Device function** — Features or Description section

From the device function, **infer which parameters matter:**

| Device Type | Key Specs to Extract |
|-------------|---------------------|
| **LDO / Linear Regulator** | Vout accuracy, dropout voltage, quiescent current, PSRR, load regulation |
| **DC-DC Converter** | Vin/Vout range, efficiency, switching frequency, Iq, feedback voltage |
| **Battery Charger** | Charge voltage, charge current, termination, power path, DPM threshold |
| **MCU / SoC** | Core voltage, I/O voltage, clock speed, flash/RAM, peripherals, startup time |
| **ADC / DAC** | Resolution, INL/DNL, SNR, THD+N, sample rate, reference voltage |
| **Audio CODEC** | ADC/DAC SNR, THD+N, sample rate, I2S format, PLL, analog gain range |
| **Sensor (IMU/Temp/etc.)** | Range, sensitivity/resolution, ODR, noise density, supply current |
| **Flash / EEPROM** | Capacity, read/write speed, erase time, endurance cycles, retention |
| **Interface (USB/CAN/RS485)** | Data rate, voltage levels, bus capacitance, propagation delay |
| **Power Mux / Load Switch** | Ron, Ilim, transition time, Vgs threshold |
| **Port Expander / MUX / Analog Switch** | Ron, bandwidth, leakage, VOH/VOL |

---

## Phase 2b: Targeted Re-scan with Inferred Patterns

**Purpose:** After reading the device description, you now know this chip's proprietary feature names, functional blocks, and internal terminology. Use that knowledge to build a custom `--patterns` JSON and re-run `page_hints` to find sections that default patterns miss.

### When to Trigger

Run Phase 2b when the device has **domain-specific subsystems** not covered by default patterns:
- Complex power ICs (charger, PMU, fuel gauge) — state machines, protection modes
- MCU/SoC — clock tree, DMA, interrupt controller, peripheral registers
- Audio CODEC — signal path, mixer routing, PLL, analog stages
- Memory — command set, erase architecture, status registers
- Interface/Protocol ICs — protocol state machine, enumeration flow, negotiation

Skip Phase 2b for simple devices: LDO, discrete MOSFET, gate logic, op-amp, diode, EEPROM (basic), RTC.

### How to Construct Patterns

**Source material** for inferring patterns — read in this order:

1. **Features list** (page 1-2): Proprietary feature names are the best patterns.
   > "VINDPM tracking" → pattern: `"VINDPM"`
   > "Automatic clock switching between HSE and HSI" → pattern: `"clock switching"`
   > "On-chip FIFO with 32 levels" → pattern: `"FIFO"`

2. **Table of Contents** (if extracted): Section titles are exact patterns.
   > "4.3 Charge State Machine" → pattern: `"charge state machine"`
   > "6.2 DMA Controller" → pattern: `"DMA controller"`

3. **Section headings seen in page text** (from Phase 0 text samples): reuse them verbatim.

### Pattern Quality Rules

| Good Pattern | Bad Pattern | Why |
|---|---|---|
| `"VINDPM"` | `"voltage"` | Specific term, won't false-positive |
| `"charge state machine"` | `"charge"` | Phrase-level, matches section headings |
| `"clock tree"` | `"clock"` | Discriminative for MCU clock config page |
| `"FIFO mode"` | `"mode"` | Too generic otherwise |
| `"DMA controller"` | `"memory"` | Targets a specific section |

**Keep patterns short (2–4 words), prefer exact subsystem names over common words.**

### Execution

```bash
# Step 1: Get default patterns as base (to not lose default coverage)
python scripts/pdf_tools.py dump_patterns > /tmp/custom_patterns.json

# Step 2: Edit /tmp/custom_patterns.json — ADD new labels with inferred patterns.
# Do NOT remove existing default keys. Example additions:
# {
#   ...existing defaults...,
#   "likely_charge_state_machine": ["charge state machine", "charging operation", "VINDPM"],
#   "likely_clock_config":         ["clock tree", "system clock", "PLL configuration"],
#   "likely_dma_section":          ["DMA controller", "DMA channel", "direct memory access"]
# }

# Step 3: Re-scan with custom patterns
python scripts/pdf_tools.py page_hints <pdf_path> --patterns /tmp/custom_patterns.json
```

### Merge Results

After Phase 2b, merge the two scan results:
- Phase 0 result → gives structural pages (pinout, timing, block diagram, package)
- Phase 2b result → gives domain-specific pages (state machine, clock tree, FIFO config)

Together they form the complete section map for Phase 3 targeted extraction.

---

## Phase 3: Section Mapping

Combine pre-scan hints with targeted searches to build a precise section map.

### Step 3a: Use Caption Search (Primary)

```bash
python scripts/pdf_tools.py search_caption <pdf_path>
```

Datasheet figure/table captions are the most reliable section locators. Look for:
- `Table X. Absolute Maximum Ratings`
- `Table X. Recommended Operating Conditions`
- `Table X. Electrical Characteristics`
- `Table X. Pin Functions` / `Pin Description`
- `Figure X. Functional Block Diagram`
- `Figure X. Timing Diagram`

### Step 3b: Keyword Search (Backup)

```bash
python scripts/pdf_tools.py search <pdf_path> "Absolute Maximum"
python scripts/pdf_tools.py search <pdf_path> "Electrical Characteristics"
python scripts/pdf_tools.py search <pdf_path> "Pin Functions"
python scripts/pdf_tools.py search <pdf_path> "Block Diagram"
```

Multi-language keywords:

| Section | English | Chinese |
|---------|---------|---------|
| Absolute Max | "Absolute Maximum Ratings", "Max Ratings" | "绝对最大额定值" |
| Electrical Specs | "Electrical Characteristics", "DC Characteristics" | "电气特性" |
| Pin Config | "Pin Functions", "Pin Description", "Pinout" | "引脚功能", "引脚说明" |
| Timing | "Timing Diagram", "Timing Requirements" | "时序图", "时序要求" |
| Package | "Package Information", "Mechanical Data" | "封装信息", "封装尺寸" |
| Application | "Typical Application", "Application Circuit" | "典型应用" |

### Step 3c: TOC (If Available)

```bash
python scripts/pdf_tools.py toc <pdf_path>
```

Most datasheets don't have a machine-parseable TOC. Use this as a bonus, not a primary strategy. If it returns entries, cross-check page numbers with Phase 0 hints.

### Confirm Pages

For each candidate page, confirm it's the right section:

```bash
python scripts/pdf_tools.py text <pdf_path> <page_num>
```

---

## Phase 4: Targeted Extraction

Before extracting, consider opening **[TEMPLATES.md](TEMPLATES.md)** for structured output formats:
- `device_info` — manufacturer, part number, package, temperature range
- `power_domains` — all VDD/GND pins with pin numbers
- `i2c_interface` — address calculation, speed, pull-up
- `electrical_specs` — absolute max / operating conditions / key parameters

### 4a. Electrical Characteristics Tables

**This is the core extraction.** Most datasheets have 3 mandatory tables:

| Table | Always Present | Search Keyword |
|-------|---------------|----------------|
| Absolute Maximum Ratings | Yes | "Absolute Maximum" |
| Recommended Operating Conditions | Yes | "Recommended Operating" |
| DC Electrical Characteristics | Yes | "Electrical Characteristics", "DC Characteristics" |

```bash
# Extract tables from the identified pages
python scripts/pdf_tools.py tables <pdf_path> <page_num>

# Or search directly in tables
python scripts/pdf_tools.py search_table <pdf_path> "output voltage"
python scripts/pdf_tools.py search_table <pdf_path> "quiescent current"
```

**LLM decision after table extraction:**
1. Is this the correct table? (Check table header row)
2. Which row is the target parameter? (Exact name match, not partial)
3. Which column: Min / Typ / Max? (Use Typ for nominal design, Min/Max for margin analysis)
4. Are there footnotes? (Often contain critical conditions: temperature range, load, frequency)
5. What are the test conditions? (Check rows above table, or separate "Test Conditions" column)

### 4b. Pin Description

Pin description tables are often multi-page. Read ALL pages of the pin section.

```bash
# Find pin section start
python scripts/pdf_tools.py search <pdf_path> "Pin Functions"

# Extract tables from each page of the section
python scripts/pdf_tools.py tables <pdf_path> <page_num>
# Repeat for each subsequent page until pin table ends (increment page number manually)
python scripts/pdf_tools.py tables <pdf_path> <next_page_num>
```

For the pin diagram (image in all PDFs):
```bash
python scripts/pdf_tools.py render_page <pdf_path> <pinout_diagram_page> 200
```

### 4c. Timing Diagrams

**Timing diagrams are almost always images**, even in text-based PDFs. Never try to extract timing from text.

```bash
python scripts/pdf_tools.py render_page <pdf_path> <timing_page> 200
```

For timing parameters (setup/hold/pulse width), these appear as tables alongside the diagram:
```bash
python scripts/pdf_tools.py tables <pdf_path> <timing_page>
python scripts/pdf_tools.py nearby_text <pdf_path> <timing_page> "t[SHDWRCP]" 3
```

### 4d. Inline Parameter Extraction

Some parameters appear inline in text, not in tables (especially in Chinese datasheets):

```bash
python scripts/pdf_tools.py nearby_text <pdf_path> <page_num> "output voltage" 2
python scripts/pdf_tools.py nearby_text <pdf_path> <page_num> "V[oO][uU][tT]\s*=?\s*[\d\.]+" 2
```

### 4e. Application Circuit & Block Diagram

Always image. Render and read visually:

```bash
python scripts/pdf_tools.py render_page <pdf_path> <page_num> 180
```

These pages provide design context: recommended external components, power sequencing, PCB layout hints. Do NOT skip them even if specs are already extracted.

### 4f. Package Information

```bash
python scripts/pdf_tools.py render_page <pdf_path> <package_page> 200
python scripts/pdf_tools.py tables <pdf_path> <package_page>  # for dimension tables
```

Record: package type, pin count, exposed pad (if any), land pattern reference.

---

## Device-Type Extraction Shortcuts

### Power Management ICs (LDO, DC-DC, Charger)

Priority order:
1. `search_table "output voltage"` → Vout range
2. `search_table "input voltage"` → Vin range
3. `search_table "quiescent"` or `"Iq"` → standby current
4. `search_table "efficiency"` → but efficiency is usually a **curve** → `render_page`
5. `search_table "feedback voltage"` → Vfb (for adjustable output calculation)
6. `nearby_text` on application page → Rsense, Rfb values in example circuits

**Trap:** Charger ICs have multiple current specs (precharge, fast charge, termination). Read the full charging state machine diagram — always `render_page`.

### MCU / SoC

Priority order:
1. `text page 1-2` → core architecture, max frequency, flash/RAM size
2. `search "Supply Voltage"` → VDD range
3. `search "I/O"` → GPIO voltage tolerance (3.3V only vs 5V tolerant)
4. `page_hints` to find register section → `render_page` for register bit diagrams
5. `tables` for peripheral electrical specs (I2C, SPI, UART timing)

**Trap:** MCU datasheets often have a separate "Electrical Characteristics" doc. If page count is low, it may be a brief datasheet only — check if there's a full reference manual.

### Sensors (IMU, Temperature, Pressure)

Priority order:
1. `search_table "sensitivity"` → scale factor
2. `search_table "noise"` or `"noise density"` → resolution floor
3. `search_table "output data rate"` or `"ODR"` → sampling speed
4. `search_table "supply current"` → active vs sleep
5. `render_page` on any page with `likely_curve_page` hint → noise vs ODR curves

**Trap:** Sensitivity and full-scale range are linked. A ±2g range at 1mg/LSB ≠ a ±16g range at 1mg/LSB. Always extract the range-sensitivity pair together.

### Memory (Flash, EEPROM, SRAM)

Priority order:
1. `search_table "read"` → read speed (MHz/ns)
2. `search_table "program"` or `"write"` → write time per page/byte
3. `search_table "erase"` → sector/block/chip erase time
4. `search_table "endurance"` → erase cycles
5. `search_table "retention"` → data retention years
6. `render_page` for command table page (often image in legacy datasheets)

### Interface ICs (USB, CAN, RS-485, Ethernet PHY)

Priority order:
1. `search "data rate"` or `"bit rate"` → max speed
2. `search_table "propagation delay"` → driver/receiver delay
3. `render_page` on timing diagram pages
4. `search_table "input threshold"` → VOH/VOL, VIH/VIL
5. `search_table "bus capacitance"` → drive strength requirements

---

## Error Recovery

### Parameter Not Found in Expected Table

```
1. Try search_table with alternative keywords:
   "SNR" → also try "signal to noise", "信噪比"
2. Try nearby_text with regex on the identified page
3. Check if parameter is in a different section (e.g., specs split across pages)
4. Try search across full document
5. If genuinely absent: output "NOT SPECIFIED IN DATASHEET"
```

### Table Extraction Malformed / Empty

```
1. Extract page text: pdf_tools.py text <pdf_path> <page>
2. Use nearby_text with regex to find value in text
3. render_page to visually read the table
4. Note extraction method: "Extracted from page text" or "Read from rendered image"
```

### Multiple Matches for Same Parameter

```
1. Check if one match is in "Absolute Maximum" (stress limit, not operating)
2. Check if one match is in "Recommended Operating Conditions" (normal range)
3. Prefer "Electrical Characteristics" table for typical/min/max specs
4. If still ambiguous: cite ALL sources with different values
```

---

## Image PDF: Full Workflow

```bash
# 1. Confirm image-based
python scripts/pdf_tools.py info <pdf_path>
# → is_text_based: false

# 2. Render cover pages
python scripts/pdf_tools.py render_page <pdf_path> 1 180
python scripts/pdf_tools.py render_page <pdf_path> 2 180

# 3. Identify device from rendered images (visual reading)
# Cross-validate: part number + manufacturer must match context

# 4. Pre-scan with page_hints (works on image PDFs via text layer detection)
python scripts/pdf_tools.py page_hints <pdf_path>

# 5. Render candidate pages for each section
python scripts/pdf_tools.py render_page <pdf_path> <page> 200

# 6. Extract specs visually from rendered images
# All image PDF extractions → confidence: LOW
# Cross-validation passed → confidence: MEDIUM
```

**Confidence for image PDFs:**

| Level | Condition |
|-------|-----------|
| `MEDIUM` | Cross-validation passed, value clearly readable in rendered image |
| `LOW` | Default for all image-based extractions |
| `UNVERIFIED` | Cross-validation failed (part number mismatch) |


path: scripts/pdf_tools.py
content: #!/usr/bin/env python3
"""
通用 PDF 工具 - 不包含业务逻辑，只提供数据提取能力
LLM 负责理解和决策，脚本只负责提取可验证信号

Usage:
    python pdf_tools.py info <pdf_path>
    python pdf_tools.py search <pdf_path> <keyword>
    python pdf_tools.py tables <pdf_path> <page_num>
    python pdf_tools.py text <pdf_path> <page_num>
    python pdf_tools.py search_table <pdf_path> <keyword>
    python pdf_tools.py toc <pdf_path>
    python pdf_tools.py page_stats <pdf_path> [page_num]
    python pdf_tools.py page_hints <pdf_path> [page_num] [--patterns <json_file>]
    python pdf_tools.py search_caption <pdf_path> [keyword]
    python pdf_tools.py nearby_text <pdf_path> <page_num> <pattern> [context_lines]
    python pdf_tools.py render_page <pdf_path> <page_num> [dpi] [out_path]
    python pdf_tools.py dump_patterns  # 输出默认 patterns JSON

Patterns JSON format (for --patterns):
    {
        "likely_block_diagram": ["block diagram", "框图"],
        "likely_timing_diagram": ["timing diagram", "时序图"]
    }
"""

from __future__ import annotations

import json
import io
import re
import sys
import tempfile
from pathlib import Path
from typing import Any

import pdfplumber
from contextlib import contextmanager


@contextmanager
def _open_pdf(pdf_path: str | Path):
    """Open a PDF, retrying with an empty password for owner-protected files.

    Accepts str or Path; normalizes to Path internally so backslash paths on
    Windows are handled correctly regardless of how the caller passes the path.

    Many datasheets are encrypted with an empty user password (owner password
    only) to restrict printing/editing. PDF viewers handle this silently;
    pdfplumber raises an exception. We mirror that behaviour by retrying once
    with password="" before giving up.
    """
    path = Path(pdf_path)  # normalize: handles both \\ and / on Windows
    try:
        with pdfplumber.open(path) as pdf:
            yield pdf
    except Exception as exc:
        err = str(exc).lower()
        if any(kw in err for kw in ("encrypt", "password", "pdfpassword")):
            with pdfplumber.open(path, password="") as pdf:
                yield pdf
        else:
            raise


CAPTION_REGEX = re.compile(
    r"^\s*(Figure|Table|图|表)\s*[A-Za-z0-9\u4e00-\u9fff][A-Za-z0-9\u4e00-\u9fff\-\.]*\s*[:\.：]?\s*(.*)$",
    re.IGNORECASE,
)

# Default patterns for figure/page classification
DEFAULT_LIKELY_PATTERNS: dict[str, tuple[str, ...]] = {
    "likely_block_diagram": (
        "functional block diagram",
        "block diagram",
        "框图",
        "功能框图",
    ),
    "likely_application_figure": (
        "typical application",
        "simplified schematic",
        "reference design",
        "典型应用",
        "参考设计",
    ),
    "likely_timing_diagram": (
        "timing diagram",
        "timing requirements",
        "power-up sequence",
        "power-down sequence",
        "sequence",
        "时序图",
    ),
    "likely_curve_page": (
        "typical characteristics",
        "application curves",
        "efficiency",
        "performance curves",
        "特性曲线",
        "效率曲线",
    ),
    "likely_pinout_page": (
        "pin configuration",
        "pin functions",
        "pinout",
        "pin assignment",
        "pin description",
        "引脚配置",
        "引脚定义",
        "引脚功能",
    ),
    "likely_package_page": (
        "package drawing",
        "mechanical drawing",
        "mechanical dimensions",
        "package dimensions",
        "package outline",
        "封装尺寸",
        "封装图",
    ),
    "likely_register_figure": (
        "register",
        "field descriptions",
        "bit field",
        "寄存器",
        "位域",
    ),
}

# Module-level default (can be overridden)
LIKELY_PATTERNS = DEFAULT_LIKELY_PATTERNS


def pdf_info(pdf_path: str) -> dict[str, Any]:
    path = Path(pdf_path)
    if not path.exists():
        return {"error": f"File not found: {pdf_path}"}

    try:
        with _open_pdf(pdf_path) as pdf:
            total_pages = len(pdf.pages)
            sample_pages = [0, min(5, total_pages - 1), min(10, total_pages - 1)]
            sample_text = ""
            for index in sample_pages:
                if index < len(pdf.pages):
                    sample_text += pdf.pages[index].extract_text() or ""

            is_text_based = len(sample_text.strip()) > 200
            return {
                "pages": total_pages,
                "is_text_based": is_text_based,
                "file_size_kb": path.stat().st_size // 1024,
            }
    except Exception as exc:
        return {"error": str(exc)}



def search_text(pdf_path: str, keyword: str, context_lines: int = 2) -> list[dict[str, Any]]:
    results: list[dict[str, Any]] = []

    try:
        with _open_pdf(pdf_path) as pdf:
            for page_index, page in enumerate(pdf.pages, start=1):
                text = page.extract_text() or ""
                lines = text.split("\n")

                for line_index, line in enumerate(lines):
                    if keyword.lower() in line.lower():
                        start = max(0, line_index - context_lines)
                        end = min(len(lines), line_index + context_lines + 1)
                        results.append(
                            {
                                "page": page_index,
                                "line": line.strip(),
                                "context": "\n".join(lines[start:end]),
                            }
                        )
        return results
    except Exception as exc:
        return [{"error": str(exc)}]



def extract_page_tables(pdf_path: str, page_num: int) -> list[Any]:
    try:
        with _open_pdf(pdf_path) as pdf:
            if page_num < 1 or page_num > len(pdf.pages):
                return [{"error": f"Page {page_num} out of range (1-{len(pdf.pages)})"}]

            page = pdf.pages[page_num - 1]
            tables = page.extract_tables()
            cleaned_tables: list[list[list[str]]] = []
            for table in tables:
                if not table:
                    continue
                cleaned_table: list[list[str]] = []
                for row in table:
                    cleaned_row = ["" if cell is None else str(cell).strip() for cell in row]
                    cleaned_table.append(cleaned_row)
                if cleaned_table:
                    cleaned_tables.append(cleaned_table)
            return cleaned_tables if cleaned_tables else []
    except Exception as exc:
        return [{"error": str(exc)}]



def extract_page_text(pdf_path: str, page_num: int) -> str:
    try:
        with _open_pdf(pdf_path) as pdf:
            if page_num < 1 or page_num > len(pdf.pages):
                return f"Error: Page {page_num} out of range (1-{len(pdf.pages)})"
            return pdf.pages[page_num - 1].extract_text() or ""
    except Exception as exc:
        return f"Error: {str(exc)}"



def search_in_tables(pdf_path: str, keyword: str, pages: list[int] | None = None) -> list[dict[str, Any]]:
    results: list[dict[str, Any]] = []

    try:
        with _open_pdf(pdf_path) as pdf:
            total_pages = len(pdf.pages)
            search_pages = pages if pages else list(range(1, total_pages + 1))

            for page_num in search_pages:
                if page_num < 1 or page_num > total_pages:
                    continue
                page = pdf.pages[page_num - 1]
                tables = page.extract_tables()
                for table_index, table in enumerate(tables):
                    if not table:
                        continue
                    for row_index, row in enumerate(table):
                        row_text = " ".join(str(cell) if cell else "" for cell in row)
                        if keyword.lower() in row_text.lower():
                            results.append(
                                {
                                    "page": page_num,
                                    "table_index": table_index,
                                    "row_index": row_index,
                                    "row": ["" if cell is None else str(cell).strip() for cell in row],
                                    "full_table": [
                                        ["" if cell is None else str(cell).strip() for cell in table_row]
                                        for table_row in table
                                    ],
                                }
                            )
        return results
    except Exception as exc:
        return [{"error": str(exc)}]



def extract_toc(pdf_path: str) -> list[dict[str, Any]]:
    toc: list[dict[str, Any]] = []

    try:
        with _open_pdf(pdf_path) as pdf:
            for page in pdf.pages[: min(10, len(pdf.pages))]:
                text = page.extract_text() or ""
                if not any(keyword in text.lower() for keyword in ("contents", "目录", "table of contents")):
                    continue
                lines = text.split("\n")
                for line in lines:
                    match = re.search(r"(.+?)\s+[\.·\s]+\s*(\d+)\s*$", line.strip())
                    if not match:
                        continue
                    title = match.group(1).strip()
                    page_num = int(match.group(2))
                    if 0 < page_num <= len(pdf.pages):
                        toc.append({"title": title, "page": page_num})
                if toc:
                    return toc
        return []
    except Exception as exc:
        return [{"error": str(exc)}]



def _extract_words(page: pdfplumber.page.Page) -> list[dict[str, Any]]:
    try:
        return page.extract_words() or []
    except Exception:
        return []



def _compact_text(text: str) -> str:
    return re.sub(r"\s+", "", text.lower())


def _caption_hits(text: str) -> dict[str, int]:
    lowered = text.lower()
    figure_caption_hits = len(re.findall(r"figure\s*[a-z0-9]", lowered))
    table_caption_hits = len(re.findall(r"table\s*[a-z0-9]", lowered))
    return {
        "figure_caption_hits": figure_caption_hits,
        "table_caption_hits": table_caption_hits,
    }



def _page_stats(page: pdfplumber.page.Page, page_num: int) -> dict[str, Any]:
    text = page.extract_text() or ""
    words = _extract_words(page)
    caption_hits = _caption_hits(text)
    try:
        table_count = len(page.extract_tables() or [])
    except Exception:
        table_count = 0

    return {
        "page": page_num,
        "width": page.width,
        "height": page.height,
        "text_len": len(text),
        "word_count": len(words),
        "char_count": len(page.chars),
        "image_count": len(page.images),
        "rect_count": len(page.rects),
        "line_count": len(page.lines),
        "curve_count": len(page.curves),
        "table_count": table_count,
        "has_extractable_text": bool(text.strip()),
        **caption_hits,
    }



def collect_page_stats(pdf_path: str, page_num: int | None = None) -> list[dict[str, Any]]:
    try:
        with _open_pdf(pdf_path) as pdf:
            if page_num is not None:
                if page_num < 1 or page_num > len(pdf.pages):
                    return [{"error": f"Page {page_num} out of range (1-{len(pdf.pages)})"}]
                return [_page_stats(pdf.pages[page_num - 1], page_num)]
            return [_page_stats(page, index) for index, page in enumerate(pdf.pages, start=1)]
    except Exception as exc:
        return [{"error": str(exc)}]



def _add_hint(hints: list[dict[str, Any]], label: str, score: float, reasons: list[str]) -> None:
    if score <= 0:
        return
    hints.append(
        {
            "label": label,
            "score": round(min(score, 0.95), 2),
            "reasons": reasons,
        }
    )



def _build_hints(
    page_text: str,
    stats: dict[str, Any],
    patterns_override: dict[str, tuple[str, ...]] | None = None,
) -> list[dict[str, Any]]:
    """Build heuristic hints for page classification.

    Args:
        page_text: Extracted text from the page
        stats: Page statistics from _page_stats()
        patterns_override: Custom patterns dict, uses DEFAULT_LIKELY_PATTERNS if None

    Returns:
        List of hint dicts with 'label', 'score', and 'reasons' keys
    """
    lowered = page_text.lower()
    compact = _compact_text(page_text)
    hints: list[dict[str, Any]] = []
    vector_primitives = stats["rect_count"] + stats["line_count"] + stats["curve_count"]

    # Use override patterns or fall back to default
    patterns_to_use = patterns_override if patterns_override is not None else LIKELY_PATTERNS

    figure_score = 0.0
    figure_reasons: list[str] = []
    if stats["figure_caption_hits"] > 0:
        figure_score += 0.45
        figure_reasons.append("page text contains Figure caption")
    if vector_primitives > 80:
        figure_score += 0.20
        figure_reasons.append("vector primitives are moderate/high")
    if stats["table_caption_hits"] == 0:
        figure_score += 0.05
        figure_reasons.append("table caption absent")
    if stats["image_count"] > 0 and stats["char_count"] < 400:
        figure_score += 0.10
        figure_reasons.append("raster/image objects present with relatively low text density")
    _add_hint(hints, "likely_figure_page", figure_score, figure_reasons)

    table_score = 0.0
    table_reasons: list[str] = []
    if stats["table_caption_hits"] > 0:
        table_score += 0.40
        table_reasons.append("page text contains Table caption")
    if stats["table_count"] > 0:
        table_score += 0.20
        table_reasons.append("table extraction found tabular structure")
    if stats["rect_count"] > 150 and stats["line_count"] < 20:
        table_score += 0.10
        table_reasons.append("many rectangular cells with relatively few line graphics")
    _add_hint(hints, "likely_table_page", table_score, table_reasons)

    for label, patterns in patterns_to_use.items():
        score = 0.0
        reasons: list[str] = []
        for pattern in patterns:
            compact_pattern = _compact_text(pattern)
            # Use substring matching for both normal and compacted text
            if pattern.lower() in lowered or compact_pattern in compact:
                score += 0.55
                reasons.append(f"caption/title matches '{pattern}'")
        if label in {"likely_curve_page", "likely_timing_diagram", "likely_block_diagram"} and vector_primitives > 40:
            score += 0.10
            reasons.append("page has non-trivial vector graphics")
        _add_hint(hints, label, score, reasons)

    if figure_score > 0.35 and table_score > 0.35:
        _add_hint(hints, "likely_mixed_page", 0.60, ["figure and table cues both present"])

    return hints



def collect_page_hints(
    pdf_path: str,
    page_num: int | None = None,
    patterns_override: dict[str, tuple[str, ...]] | None = None,
) -> list[dict[str, Any]]:
    """Collect heuristic hints for page classification.

    Args:
        pdf_path: Path to the PDF file
        page_num: Specific page number, or None for all pages
        patterns_override: Custom patterns dict, uses DEFAULT_LIKELY_PATTERNS if None

    Returns:
        List of page hint results with signals and heuristic_hints
    """
    try:
        with _open_pdf(pdf_path) as pdf:
            target_pages: list[tuple[int, pdfplumber.page.Page]]
            if page_num is not None:
                if page_num < 1 or page_num > len(pdf.pages):
                    return [{"error": f"Page {page_num} out of range (1-{len(pdf.pages)})"}]
                target_pages = [(page_num, pdf.pages[page_num - 1])]
            else:
                target_pages = list(enumerate(pdf.pages, start=1))

            results: list[dict[str, Any]] = []
            for page_index, page in target_pages:
                stats = _page_stats(page, page_index)
                page_text = page.extract_text() or ""
                results.append(
                    {
                        "page": page_index,
                        "signals": stats,
                        "heuristic_hints": _build_hints(page_text, stats, patterns_override),
                        "warning": "Hints are heuristic only; not final evidence.",
                    }
                )
            return results
    except Exception as exc:
        return [{"error": str(exc)}]



def _extract_captions_from_lines(lines: list[str], page_num: int) -> list[dict[str, Any]]:
    """Extract figure/table captions from text lines.

    Supports both English (Figure, Table) and Chinese (图, 表) captions.
    """
    captions: list[dict[str, Any]] = []
    for line_index, line in enumerate(lines):
        match = CAPTION_REGEX.match(line.strip())
        if not match:
            continue
        caption_type = match.group(1).lower()
        captions.append(
            {
                "page": page_num,
                "caption_type": caption_type,
                "caption": line.strip(),
                "line_index": line_index,
            }
        )
    return captions



def search_captions(pdf_path: str, keyword: str | None = None) -> list[dict[str, Any]]:
    results: list[dict[str, Any]] = []
    try:
        with _open_pdf(pdf_path) as pdf:
            for page_num, page in enumerate(pdf.pages, start=1):
                text = page.extract_text() or ""
                lines = text.split("\n")
                page_captions = _extract_captions_from_lines(lines, page_num)
                for caption_info in page_captions:
                    if keyword:
                        keyword_lower = keyword.lower()
                        caption_lower = caption_info["caption"].lower()
                        if keyword_lower not in caption_lower and _compact_text(keyword) not in _compact_text(caption_info["caption"]):
                            continue
                    results.append(caption_info)
        return results
    except Exception as exc:
        return [{"error": str(exc)}]



def extract_nearby_text(pdf_path: str, page_num: int, pattern: str, context_lines: int = 2) -> list[dict[str, Any]]:
    try:
        with _open_pdf(pdf_path) as pdf:
            if page_num < 1 or page_num > len(pdf.pages):
                return [{"error": f"Page {page_num} out of range (1-{len(pdf.pages)})"}]
            text = pdf.pages[page_num - 1].extract_text() or ""
            lines = text.split("\n")
            results: list[dict[str, Any]] = []
            regex = re.compile(pattern, re.IGNORECASE)
            for line_index, line in enumerate(lines):
                if not regex.search(line):
                    continue
                start = max(0, line_index - context_lines)
                end = min(len(lines), line_index + context_lines + 1)
                results.append(
                    {
                        "page": page_num,
                        "pattern": pattern,
                        "matched_line": line.strip(),
                        "line_index": line_index,
                        "before": lines[start:line_index],
                        "after": lines[line_index + 1 : end],
                    }
                )
            return results
    except Exception as exc:
        return [{"error": str(exc)}]



def render_page(pdf_path: str, page_num: int, dpi: int = 180, out_path: str | None = None) -> dict[str, Any]:
    path = Path(pdf_path)
    if not path.exists():
        return {"error": f"File not found: {pdf_path}"}

    output_path = Path(out_path) if out_path else Path(tempfile.gettempdir()) / f"{path.stem}_page_{page_num}.png"
    output_path.parent.mkdir(parents=True, exist_ok=True)

    # Try pymupdf (same MuPDF engine as mutool, no external binary needed)
    try:
        import fitz  # pymupdf
        doc = fitz.open(str(path))
        if doc.is_encrypted:
            doc.authenticate("")  # empty password for owner-protected PDFs
        if page_num < 1 or page_num > len(doc):
            return {"error": f"Page {page_num} out of range (1-{len(doc)})"}
        page = doc[page_num - 1]
        mat = fitz.Matrix(dpi / 72, dpi / 72)
        pix = page.get_pixmap(matrix=mat)
        pix.save(str(output_path))
        return {"page": page_num, "dpi": dpi, "output_path": str(output_path), "renderer": "pymupdf"}
    except ImportError:
        pass
    except Exception as exc:
        return {"error": str(exc), "renderer": "pymupdf"}

    # Fall back to pypdfium2 (PDFium engine, no external binary needed)
    try:
        import pypdfium2 as pdfium
        try:
            pdf = pdfium.PdfDocument(str(path))
        except Exception:
            pdf = pdfium.PdfDocument(str(path), password="")
        if page_num < 1 or page_num > len(pdf):
            return {"error": f"Page {page_num} out of range (1-{len(pdf)})"}
        page = pdf[page_num - 1]
        bitmap = page.render(scale=dpi / 72)
        pil_image = bitmap.to_pil()
        pil_image.save(str(output_path))
        return {"page": page_num, "dpi": dpi, "output_path": str(output_path), "renderer": "pypdfium2"}
    except ImportError:
        pass
    except Exception as exc:
        return {"error": str(exc), "renderer": "pypdfium2"}

    return {"error": "No PDF renderer available. Install pymupdf or pypdfium2."}



def _parse_optional_page(value: str | None) -> int | None:
    if value is None:
        return None
    return int(value)


def _load_patterns_from_file(json_path: str) -> dict[str, tuple[str, ...]] | None:
    """Load custom patterns from a JSON file.

    Args:
        json_path: Path to JSON file with pattern definitions

    Returns:
        Dict mapping labels to tuples of patterns, or None on error
    """
    try:
        path = Path(json_path)
        if not path.exists():
            return None
        with path.open("r", encoding="utf-8") as f:
            data = json.load(f)
        # Convert lists to tuples
        return {k: tuple(v) if isinstance(v, list) else v for k, v in data.items()}
    except (json.JSONDecodeError, OSError):
        return None


def dump_default_patterns() -> dict[str, list[str]]:
    """Return default patterns as JSON-serializable dict (lists instead of tuples)."""
    return {k: list(v) for k, v in DEFAULT_LIKELY_PATTERNS.items()}



def main() -> None:
    # Force UTF-8 stdout so JSON with non-ASCII characters works on Windows,
    # where the default console encoding is often GBK/CP936.
    # sys.stdout.reconfigure() is the clean Python 3.7+ way; the io.TextIOWrapper
    # fallback covers edge cases (e.g. stdout already replaced by a test harness).
    try:
        sys.stdout.reconfigure(encoding="utf-8")
    except AttributeError:
        if hasattr(sys.stdout, "buffer"):
            sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")

    # Handle commands that don't need PDF path
    if len(sys.argv) >= 2 and sys.argv[1] == "dump_patterns":
        print(json.dumps(dump_default_patterns(), indent=2, ensure_ascii=False))
        return

    if len(sys.argv) < 3:
        print(__doc__)
        sys.exit(1)

    command = sys.argv[1]
    pdf_path = Path(sys.argv[2])  # normalize early: handles \\ vs / on Windows

    # Parse --patterns flag if present
    patterns_override: dict[str, tuple[str, ...]] | None = None
    args = sys.argv[3:]

    if "--patterns" in args:
        patterns_idx = args.index("--patterns")
        if patterns_idx + 1 < len(args):
            patterns_file = args[patterns_idx + 1]
            patterns_override = _load_patterns_from_file(patterns_file)
            if patterns_override is None:
                print(f"Error: Failed to load patterns from {patterns_file}")
                sys.exit(1)
            # Remove --patterns and its argument from args
            args = args[:patterns_idx] + args[patterns_idx + 2 :]

    if command == "info":
        result = pdf_info(pdf_path)
    elif command == "search":
        if not args:
            print("Usage: python pdf_tools.py search <pdf_path> <keyword>")
            sys.exit(1)
        result = search_text(pdf_path, args[0])
    elif command == "tables":
        if not args:
            print("Usage: python pdf_tools.py tables <pdf_path> <page_num>")
            sys.exit(1)
        result = extract_page_tables(pdf_path, int(args[0]))
    elif command == "text":
        if not args:
            print("Usage: python pdf_tools.py text <pdf_path> <page_num>")
            sys.exit(1)
        result = extract_page_text(pdf_path, int(args[0]))
    elif command == "search_table":
        if not args:
            print("Usage: python pdf_tools.py search_table <pdf_path> <keyword>")
            sys.exit(1)
        result = search_in_tables(pdf_path, args[0])
    elif command == "toc":
        result = extract_toc(pdf_path)
    elif command == "page_stats":
        page_num = int(args[0]) if args else None
        result = collect_page_stats(pdf_path, page_num)
    elif command == "page_hints":
        page_num = int(args[0]) if args else None
        result = collect_page_hints(pdf_path, page_num, patterns_override)
    elif command == "search_caption":
        keyword = args[0] if args else None
        result = search_captions(pdf_path, keyword)
    elif command == "nearby_text":
        if len(args) < 2:
            print("Usage: python pdf_tools.py nearby_text <pdf_path> <page_num> <pattern> [context_lines]")
            sys.exit(1)
        context_lines = int(args[2]) if len(args) >= 3 else 2
        result = extract_nearby_text(pdf_path, int(args[0]), args[1], context_lines)
    elif command == "render_page":
        if not args:
            print("Usage: python pdf_tools.py render_page <pdf_path> <page_num> [dpi] [out_path]")
            sys.exit(1)
        dpi = int(args[1]) if len(args) >= 2 else 180
        out_path = args[2] if len(args) >= 3 else None
        result = render_page(pdf_path, int(args[0]), dpi, out_path)
    else:
        print(f"Unknown command: {command}")
        print(__doc__)
        sys.exit(1)

    if isinstance(result, str):
        print(result)
    else:
        print(json.dumps(result, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)