Skill: zmy1006-sudo/opendataloader-pdf-zmy (version 1.0.0)

--- name: opendataloader-pdf description: OpenDataLoader PDF — AI-ready PDF parser. Parse PDFs into Markdown/JSON/HTML for RAG pipelines, extract tables with bounding boxes, OCR scanned PDFs, and enrich charts/formulas with AI descriptions. Use when: (1) parsing PDFs for knowledge bases or RAG systems; (2) extracting structured data from medical reports, academic papers, invoices; (3) building AI knowledge bases from PDF documents; (4) converting PDF documents to Markdown/JSON for further processing; (5) any PDF-to-LLM data extraction task. --- # OpenDataLoader PDF Skill ## Quick Install ```bash # Basic (CPU, ~20 pages/sec) pip install -U opendataloader-pdf # Hybrid mode (AI-enhanced, for complex docs, ~2 pages/sec) pip install -U "opendataloader-pdf[hybrid]" # LangChain integration pip install langchain-opendataloader-pdf ``` **Requirements:** Java 11+ (for hybrid mode), Python 3.10+ --- ## Core Usage Patterns ### 1. Parse PDF → Markdown (best for RAG chunking) ```python from opendataloader_pdf import convert convert( input_path=["file1.pdf", "folder/"], output_dir="output/", format="markdown" # clean text, LLM-ready ) ``` ### 2. Parse PDF → JSON (with bounding boxes for citations) ```python convert( input_path=["report.pdf"], output_dir="output/", format="json", # structured data + coordinates image_output="embedded" # "off" | "embedded" | "external" ) ``` ### 3. LangChain + RAG Pipeline ```python from langchain_opendataloader_pdf import OpenDataLoaderPDFLoader from langchain.text_splitter import RecursiveCharacterTextSplitter loader = OpenDataLoaderPDFLoader(file_path="document.pdf", format="text") docs = loader.load() splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) chunks = splitter.split_documents(docs) # → embed → vector store → RAG ``` --- ## CLI Commands ```bash # Basic: single file or folder opendataloader-pdf file1.pdf file2.pdf folder/ # Complex tables / nested structure (hybrid mode) opendataloader-pdf --hybrid docling-fast file1.pdf # Start hybrid backend first, then: opendataloader-pdf-hybrid --port 5002 # (in another terminal) opendataloader-pdf --hybrid docling-fast file1.pdf # OCR for scanned PDFs opendataloader-pdf-hybrid --port 5002 --force-ocr file1.pdf # Math formula extraction (LaTeX) opendataloader-pdf-hybrid --enrich-formula opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf # Chart/image AI description opendataloader-pdf-hybrid --enrich-picture-description opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf # Security: sanitize prompt injection opendataloader-pdf file1.pdf --sanitize ``` --- ## Output Format Selection Guide | Document Type | Recommended Format | Mode | |--------------|-------------------|------| | Standard digital PDF | `markdown` | Basic | | Complex/nested tables | `json` | Hybrid | | Scanned PDFs | any + `--force-ocr` | Hybrid | | Math formulas | `markdown` + `--enrich-formula` | Hybrid | | Charts needing description | `markdown` + `--enrich-picture-description` | Hybrid | | Medical reports (cite-able) | `json` | Hybrid | | RAG knowledge base | `markdown` | Basic or Hybrid | --- ## Key Reference Files - [API Reference](references/api-reference.md) — Full Python API, all parameters - [CLI Reference](references/cli-reference.md) — All CLI flags and hybrid mode - [Examples](references/examples.md) — RAG pipeline, table extraction, batch processing --- ## Benchmark Results (v2.0) | Metric | Score | |--------|-------| | Overall Accuracy | **0.90** | | Reading Order | 0.94 | | Table Accuracy | 0.93 | | Heading Accuracy | 0.83 | **License:** Apache 2.0 | GitHub: `opendataloader-project/opendataloader-pdf`

path: references/api-reference.md
content: # OpenDataLoader PDF — API Reference

## Python API

### `convert()` — Main Entry Point

```python
from opendataloader_pdf import convert

convert(
    input_path: list[str],       # Files and/or folders
    output_dir: str,              # Output directory
    format: str = "markdown",     # "markdown" | "json" | "html" | "text" | combined e.g. "markdown,json"
    hybrid: str = None,          # "docling-fast" | "docling" | "unstructured"; omit for basic mode
    hybrid_mode: str = None,      # "fast" | "full"; only with hybrid
    image_output: str = "off",    # "off" | "embedded" | "external"
    image_format: str = "jpeg",   # "png" | "jpeg"
    use_struct_tree: bool = False,  # Use PDF structure tags
    sanitize: bool = False,       # Remove prompt injection risks
    **kwargs
)
```

### `OpenDataLoaderPDF` Class

```python
from opendataloader_pdf import OpenDataLoaderPDF

pdf = OpenDataLoaderPDF("document.pdf")

# Extract as different formats
markdown = pdf.extract(format="markdown")
json_data = pdf.extract(format="json")
html_data = pdf.extract(format="html")
text_data = pdf.extract(format="text")

# Hybrid mode
pdf_hybrid = OpenDataLoaderPDF("complex_doc.pdf", hybrid="docling-fast")
markdown = pdf_hybrid.extract(format="markdown", hybrid_mode="full")
```

### LangChain Loader

```python
from langchain_opendataloader_pdf import OpenDataLoaderPDFLoader

loader = OpenDataLoaderPDFLoader(
    file_path="document.pdf",   # str, list[str], or Path
    format="text"               # "text" | "markdown"
)
documents = loader.load()       # Returns list[Document]
```

## JSON Output Schema

Each element in the JSON output:

```json
{
  "type": "heading | paragraph | table | list | image | caption | formula",
  "id": 42,
  "page number": 1,
  "bounding box": [left, bottom, right, top],  // PDF points (72pt = 1 inch)
  "heading level": 1,
  "font": "Helvetica-Bold",
  "font size": 24.0,
  "content": "Extracted text content"
}
```

## Key Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input_path` | list[str] | required | PDF files/folders |
| `output_dir` | str | required | Output directory |
| `format` | str | `"markdown"` | Output format(s) |
| `hybrid` | str | None | AI backend: `"docling-fast"` recommended |
| `hybrid_mode` | str | `"fast"` | `"fast"` or `"full"` (with formula/chart enrichment) |
| `image_output` | str | `"off"` | Image embedding mode |
| `image_format` | str | `"jpeg"` | `"png"` or `"jpeg"` |
| `use_struct_tree` | bool | False | Use native PDF structure |
| `sanitize` | bool | False | Remove prompt injection |

## Performance

| Mode | Speed | GPU Required |
|------|-------|-------------|
| Basic | ~20 pages/sec | No |
| Hybrid Fast | ~2 pages/sec | No |
| Hybrid Full | ~0.5 pages/sec | No |
| Batch (8+ cores) | 100+ pages/sec | No |


path: references/cli-reference.md
content: # OpenDataLoader PDF — CLI Reference

## Basic CLI

```bash
# Single file
opendataloader-pdf document.pdf

# Multiple files + folder
opendataloader-pdf file1.pdf file2.pdf folder/

# Output to specific directory
opendataloader-pdf document.pdf -o output/
```

## Hybrid Mode (AI-Enhanced Parsing)

Hybrid mode requires a backend server running first:

```bash
# Terminal 1: Start backend
opendataloader-pdf-hybrid --port 5002

# Terminal 2: Process PDFs
opendataloader-pdf --hybrid docling-fast document.pdf
```

### Backend Options

```bash
# Standard hybrid backend
opendataloader-pdf-hybrid --port 5002

# Force OCR on all pages (scanned PDFs)
opendataloader-pdf-hybrid --port 5002 --force-ocr

# Specific OCR languages (comma-separated)
opendataloader-pdf-hybrid --port 5002 --force-ocr --ocr-lang "chi_sim,eng"

# Extract math formulas as LaTeX
opendataloader-pdf-hybrid --enrich-formula

# AI description for charts/images
opendataloader-pdf-hybrid --enrich-picture-description

# Combine multiple enrichments
opendataloader-pdf-hybrid --port 5002 --force-ocr --enrich-formula --enrich-picture-description
```

### Hybrid Backend Flags

| Flag | Description |
|------|-------------|
| `--port` | Backend port (default: 5002) |
| `--force-ocr` | Force OCR on all pages |
| `--ocr-lang` | OCR languages, comma-separated |
| `--enrich-formula` | Extract formulas as LaTeX |
| `--enrich-picture-description` | AI describe charts/images |

### Processing Flags

```bash
# Fast hybrid (default)
opendataloader-pdf --hybrid docling-fast document.pdf

# Full hybrid (with formula + chart enrichment)
opendataloader-pdf --hybrid docling-fast --hybrid-mode full document.pdf

# Sanitize output (remove prompt injection)
opendataloader-pdf document.pdf --sanitize

# Use PDF structure tree (if available)
opendataloader-pdf document.pdf --use-struct-tree

# Custom output directory
opendataloader-pdf document.pdf -o ./output/

# Multiple output formats
opendataloader-pdf document.pdf -o ./output/ --format json --format markdown
```

## Output Formats

```bash
# Markdown only
opendataloader-pdf document.pdf --format markdown

# JSON only (with bounding boxes)
opendataloader-pdf document.pdf --format json

# Multiple formats at once
opendataloader-pdf document.pdf --format json --format markdown --format html

# Image output modes
opendataloader-pdf document.pdf --image-output embedded   # Base64 in JSON
opendataloader-pdf document.pdf --image-output external    # Separate image files
opendataloader-pdf document.pdf --image-output off       # No images (default)
```

## Environment Variables

```bash
# Java path (if not on PATH)
export JAVA_HOME=/path/to/java11

# Hybrid backend URL (for remote processing)
export OPENDATALOADER_HYBRID_URL=http://localhost:5002
```


path: references/examples.md
content: # OpenDataLoader PDF — Examples

## 1. RAG Knowledge Base Pipeline

```python
import os
from langchain_opendataloader_pdf import OpenDataLoaderPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

def build_rag_pipeline(pdf_folder: str, persist_dir: str = "./chroma_db"):
    """Build RAG pipeline from PDF folder → vector store."""
    pdf_files = [os.path.join(pdf_folder, f) for f in os.listdir(pdf_folder) if f.endswith(".pdf")]

    # Load all PDFs
    loader = OpenDataLoaderPDFLoader(file_path=pdf_files, format="text")
    documents = loader.load()

    # Split into chunks
    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
    chunks = splitter.split_documents(documents)

    # Embed and store
    embeddings = OpenAIEmbeddings()
    vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory=persist_dir)
    vectorstore.persist()

    return vectorstore

# Usage
vs = build_rag_pipeline("./fmt_guides/")
retriever = vs.as_retriever()
```

## 2. Extract Tables to CSV

```python
import json
import csv
from opendataloader_pdf import convert

convert(
    input_path=["report.pdf"],
    output_dir="./output/",
    format="json"
)

# Parse JSON and extract table rows
with open("./output/report.json") as f:
    data = json.load(f)

tables = [item for item in data if item["type"] == "table"]
for table in tables:
    rows = table["content"]["rows"]
    headers = rows[0]
    data_rows = rows[1:]
    print(headers)
```

## 3. Batch Process Folder

```python
from opendataloader_pdf import convert
import os

pdf_dir = "./pdfs/"
output_dir = "./parsed/"

pdf_files = [
    os.path.join(pdf_dir, f)
    for f in os.listdir(pdf_dir)
    if f.endswith(".pdf")
]

# Batch: markdown for RAG
convert(
    input_path=pdf_files,
    output_dir=output_dir,
    format="markdown"
)
print(f"Parsed {len(pdf_files)} files → {output_dir}")
```

## 4. Medical Report with Citation Coordinates

```python
from opendataloader_pdf import convert

# Extract with bounding boxes for precise citation
convert(
    input_path=["lab_report.pdf"],
    output_dir="./output/",
    format="json",
    hybrid="docling-fast"  # Better table accuracy
)

# JSON output includes bounding boxes like:
# {
#   "type": "table",
#   "id": 7,
#   "page number": 2,
#   "bounding box": [72.0, 500.0, 540.0, 700.0],
#   "content": { "rows": [["指标", "数值"], ...] }
# }
```

## 5. Scanned Document OCR + RAG

```python
from opendataloader_pdf import convert

# Start hybrid backend first:
# opendataloader-pdf-hybrid --port 5002 --force-ocr

convert(
    input_path=["scanned_document.pdf"],
    output_dir="./output/",
    format="markdown",
    hybrid="docling-fast",
    hybrid_mode="full"
)

# Then feed to RAG pipeline
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)