Skill: baokui/pdf-ocr-layout (version 1.0.2)

--- name: pdf-ocr-layout description: Multimodal document deep analysis tool based on Zhipu GLM-OCR, GLM-4.7, and GLM-4.6V. Use when: - Need to extract tables from documents (PDF/images) with high precision and convert to Markdown format - Need to automatically crop and extract illustrations and charts from document pages as independent files - Need to perform deep semantic understanding on extracted charts (based on GLM-4.6V visual analysis) - Need to perform logical analysis on extracted table data (based on GLM-4.7 text analysis) Core Architecture: 1. Visual Extraction: GLM-OCR 2. Semantic Understanding: GLM-4.7 (text/tables) + GLM-4.6V (multimodal/images) --- # GLM-OCR Multimodal Deep Analysis This tool builds a high-precision document parsing pipeline: using **GLM-OCR** for layout element extraction, calling **GLM-4.7** for logical interpretation of table data, and calling **GLM-4.6V** for multimodal visual interpretation of images and charts. ## Pipeline Implementation Architecture This Skill consists of two core script stages, orchestrated through `glm_ocr_pipeline.py`: ### 1. Extraction Stage (`scripts/glm_ocr_extract.py`) - **Core Model**: GLM-OCR - **Function**: Responsible for physical layout analysis of documents - **Output**: Extract table HTML and clean to Markdown, automatically crop independent chart image files based on Bbox coordinates, and generate intermediate JSON containing full page reading order ### 2. Understanding Stage (`scripts/glm_understanding.py`) - **Core Model**: GLM-4.7 (text) / GLM-4.6V (visual) - **Function**: Responsible for deep semantic reasoning of content - **Logic**: - **Tables**: Combine full text context, use GLM-4.7 to analyze business meaning of Markdown table data - **Charts**: Combine full text context + cropped images, use GLM-4.6V for multimodal visual analysis ## Invocation Methods ### Command Line Invocation ```bash # Run complete pipeline: extraction -> cropping -> understanding analysis, supports input in .pdf, .jpg, .png and other formats python scripts/glm_ocr_pipeline.py \ --file_path "/data/report_page.jpg" \ --output_dir "/data/output" ``` ## API Parameter Description | Parameter | Type | Required | Description | | --- | --- | --- | --- | | file_path | string | ✅ | Absolute path to input file (supports .pdf, .png, .jpg) | | output_dir | string | ✅ | Result output directory (used to save cropped images and JSON reports) | ## Return Result Structure (JSON) The tool returns a list containing layout elements and their deep understanding: ```json [ { "type": "table", "bbox": [100, 200, 500, 600], "content_info": "| Revenue | Q1 |\n|---|---|\n| 100M | ... |", "deep_understanding": "(Generated by GLM-4.7) This table shows Q1 2024 revenue data. Combined with the 'market expansion strategy' mentioned in paragraph 3 of the body text, it can be seen that..." }, { "type": "image", "bbox": [100, 700, 500, 900], "content_info": "/data/output/images/report_page_img_2.png", "deep_understanding": "(Generated by GLM-4.6V) This is a system architecture diagram. Visually, it shows the flow of clients connecting to servers through a Load Balancer. Combined with the title 'Fig 3' and context, this diagram is mainly used to illustrate..." } ] ``` ## Environment Requirements - Environment variable `ZHIPU_API_KEY` must be configured - Python 3.8+ - Dependencies: `zhipuai`, `pillow`, `beautifulsoup4` ## Notes ### 1. Model Routing Strategy - **Table (表格)**: Content passed to **GLM-4.7**, combined with full text Markdown context for logical reasoning - **Image (图片)**: Image Base64 encoded and passed to **GLM-4.6V**, combined with OCR-extracted titles and full text context for multimodal understanding ### 2. Context Association All understanding is based on the complete layout logic of the document (Markdown Context), not isolated fragment analysis. ### 3. PDF Processing Multi-page PDFs default to processing the first page. For batch processing, please extend the loop logic at the script level.

path: SKILL_zh.md
content: ---
name: pdf-ocr-layout
description: 基于智谱 GLM-OCR、GLM-4.7 及 GLM-4.6V 的多模态文档深度解析工具。

  Use when:
  - 需要高精度提取文档（PDF/图片）中的表格并转换为 Markdown 格式
  - 需要从文档页面中自动裁剪并提取插图、图表为独立文件
  - 需要对提取的图表进行深度语义理解（基于 GLM-4.6V 视觉分析）
  - 需要对提取的表格数据进行逻辑分析（基于 GLM-4.7 文本分析）

  核心架构：
  1. 视觉提取：GLM-OCR
  2. 语义理解：GLM-4.7 (纯文本/表格) + GLM-4.6V (多模态/图像)
---

# GLM-OCR 多模态深度解析 (Multimodal Layout Analysis)

本工具构建了一个高精度的文档解析流水线：利用 **GLM-OCR** 进行版面要素提取，针对表格数据调用 **GLM-4.7** 进行逻辑解读，针对图像图表调用 **GLM-4.6V** 进行多模态视觉解读。

## 流水线实现架构 (Pipeline Architecture)

本 Skill 由两个核心脚本阶段组成，通过 `glm_ocr_pipeline.py` 进行统筹调用：

### 1. 提取阶段 (`scripts/glm_ocr_extract.py`)

- **核心模型**: GLM-OCR
- **功能**: 负责文档的物理版面分析
- **产出**: 提取表格 HTML 并清洗为 Markdown，根据 Bbox 坐标自动裁剪出独立的图表图片文件，并生成包含整页阅读顺序的中间态 JSON

### 2. 理解阶段 (`scripts/glm_understanding.py`)

- **核心模型**: GLM-4.7 (文本) / GLM-4.6V (视觉)
- **功能**: 负责内容的深度语义推理
- **逻辑**:
  - **表格**: 结合全文上下文，使用 GLM-4.7 分析 Markdown 表格数据的业务含义
  - **图表**: 结合全文上下文 + 裁剪后的图片，使用 GLM-4.6V 进行多模态视觉分析

## 调用方式

### 命令行调用

```bash
# 运行完整流水线：提取 -> 裁剪 -> 理解分析，支持输入 .pdf, .jpg, .png 等格式
python scripts/glm_ocr_pipeline.py \
  --file_path "/data/report_page.jpg" \
  --output_dir "/data/output"
```

## API 参数说明

| 参数 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| file_path | string | ✅ | 输入文件绝对路径 (支持 .pdf, .png, .jpg) |
| output_dir | string | ✅ | 结果输出目录 (用于保存裁剪图片和 JSON 报告) |

## 返回结果结构 (JSON)

工具返回包含版面元素及其深度理解的列表：

```json
[
  {
    "type": "table",
    "bbox": [100, 200, 500, 600],
    "content_info": "| Revenue | Q1 |\n|---|---|\n| 100M | ... |",
    "deep_understanding": "（由 GLM-4.7 生成）该表格展示了2024年Q1的营收数据。结合正文第3段提到的'市场扩张策略'，可以看出..."
  },
  {
    "type": "image",
    "bbox": [100, 700, 500, 900],
    "content_info": "/data/output/images/report_page_img_2.png",
    "deep_understanding": "（由 GLM-4.6V 生成）这是一张系统架构图。视觉上展示了客户端通过 Load Balancer 连接服务器的流程。结合标题 'Fig 3' 和上下文，这张图主要用于说明..."
  }
]
```

## 环境要求

- 环境变量 `ZHIPU_API_KEY` 必须已配置
- Python 3.8+
- 依赖库：`zhipuai`, `pillow`, `beautifulsoup4`

## 注意事项

### 1. 模型分流策略

- **Table (表格)**: 内容传入 **GLM-4.7**，结合全文 Markdown 上下文进行逻辑推理
- **Image (图片)**: 图片 Base64 编码传入 **GLM-4.6V**，结合 OCR 提取的标题和全文上下文进行多模态理解

### 2. 上下文关联

所有理解均基于文档的完整版面逻辑（Markdown Context），而非孤立的片段分析。

### 3. PDF 处理

多页 PDF 默认处理首页，如需批量处理请在脚本层扩展循环逻辑。


path: script/glm_ocr_pipeline.py
content: import argparse
import sys
import json
from pathlib import Path

# 导入刚才写的两个脚本模块
# 假设它们分别命名为 glm_ocr_extract.py 和 glm_understanding.py
# 并且放在同一目录下
from glm_ocr_extract import process_file as run_extraction
from glm_understanding import process_understanding as run_analysis

def main():
    parser = argparse.ArgumentParser(description="GLM-OCR Layout & Understanding Pipeline")
    parser.add_argument("--file_path", required=True, help="Input file path (PDF/Image)")
    parser.add_argument("--output_dir", required=True, help="Directory to save results")
    args = parser.parse_args()

    input_path = Path(args.file_path)
    output_dir = Path(args.output_dir)

    if not input_path.exists():
        print(json.dumps([{"error": f"File not found: {input_path}"}]))
        sys.exit(1)

    try:
        # Step 1: 提取 (OCR + Crop + Markdown)
        # 返回中间态 json 路径
        intermediate_json = run_extraction(str(input_path), str(output_dir))
        
        # Step 2: 理解 (LLM + VLM)
        # 读取中间态 json，生成最终 json，并打印结果
        run_analysis(intermediate_json)

    except Exception as e:
        # 捕获顶层异常，确保输出 JSON 格式错误信息
        print(json.dumps([{"error": f"Pipeline failed: {str(e)}"}], ensure_ascii=False))
        sys.exit(1)

if __name__ == "__main__":
    main()
    

path: script/glm_understanding.py
content: import json
import os
import base64
import argparse
from pathlib import Path
from zai import ZhipuAiClient

# 配置模型
TEXT_MODEL = "glm-4.7"     # 用于分析表格和纯文本逻辑
VISION_MODEL = "glm-4.6v"  # 用于分析图片/图表 

client = ZhipuAiClient([REDACTED_SECRET]"ZHIPU_API_KEY"))

def encode_image(image_path):
    """将图片文件编码为 Base64，供 VLM 使用"""
    with open(image_path, "rb") as image_file:
        encoded = base64.b64encode(image_file.read()).decode('utf-8')
    return f"data:image/png;base64,{encoded}"

def analyze_table_with_text_model(full_context, table_element):
    """
    针对表格：使用文本模型 (GLM-4-Plus)
    输入：全文上下文 + 表格 Markdown
    """
    markdown_content = table_element.get('content', '')
    title = table_element.get('detected_title', '未识别到标题')
    
    prompt = f"""
你是一位专业的学术文档分析专家。
请结合提供的【全文上下文】和【目标表格数据】，对表格进行深度解读。

### 1. 全文上下文 (Context)
---
{full_context[:6000]} 
...(上下文截断)
---

### 2. 目标表格 (Target Table)
**识别到的表格标题**: {title}
**表格内容**:
{markdown_content}

### 3. 分析任务
请分析：
1. **核心数据**：表格展示了哪些关键指标？有哪些显著的最大值/最小值或异常值？
2. **论证作用**：结合正文，作者列出这个表格是为了证明什么结论？

直接给出分析，不用输出额外内容。
"""
    try:
        response = client.chat.completions.create(
            model=TEXT_MODEL,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.6,
            thinking={
            "type": "diabled",    # 启用深度思考模式
        },
        top_p=0.8,
        )
        return response.choices[0].message.content
    except Exception as e:
        return f"[Text Analysis Failed]: {str(e)}"

def analyze_image_with_vision_model(full_context, image_element):
    """
    针对图片：使用多模态模型 (GLM-4V-Plus)
    输入：全文上下文 + 图片 Base64 + 标题
    """
    img_path = image_element.get('file_path')
    title = image_element.get('detected_title', '未识别到标题')
    
    if not img_path or not os.path.exists(img_path):
        return "无法分析：找不到图片文件路径。"

    # 读取并编码图片
    base64_img = encode_image(img_path)
    
    # 构建 VLM Prompt
    # 注意：GLM-4V 支持在文本中穿插图片，这里我们将上下文作为背景知识
    prompt_text = f"""
你是一位多模态文档分析专家。
请参考【全文上下文】中的背景信息，观察这张【图片】，并结合它的【标题】进行深度解读。

### 1. 图片元数据
**识别到的图像标题/Caption**: {title}

### 2. 全文上下文 (Context)
(仅供参考，用于理解图片在文中的作用)
---
{full_context[:6000]} 
---

### 3. 分析任务
请结合图片视觉内容和文字上下文回答：
1. **视觉描述**：这张图展示了什么？(例如：系统架构图、折线趋势图、实验对比图等)。如果是数据图，请描述具体的趋势（如上升、下降、收敛）。
2. **语义关联**：结合上下文，这张图对应文中的哪个 Figure？
3. **核心结论**：这张图得出了什么结论？它如何支持文章的核心观点？
"""

    try:
        response = client.chat.completions.create(
            model=VISION_MODEL,
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": prompt_text
                        },
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": base64_img
                            }
                        }
                    ]
                }
            ],
            temperature=0.6, # 视觉模型通常需要较低温度以减少幻觉
            thinking={
            "type": "diabled",   
            },
            top_p=0.8,
            
        )
        return response.choices[0].message.content
    except Exception as e:
        return f"[Vision Analysis Failed]: {str(e)}"

def process_understanding(json_path):
    print(f"[Analysis] Loading data from {json_path}...")
    json_path = Path(json_path).resolve()
    
    with open(json_path, 'r', encoding='utf-8') as f:
        data = json.load(f)

    full_context = data.get('full_markdown_context', '')
    elements = data.get('elements', [])
    
    results = []

    for item in elements:
        element_type = item['type']
        item_id = item['id']
        print(f"  > Analyzing {item_id} ({element_type})...")
        
        analysis_result = ""
        
        # === 分流逻辑 ===
        if element_type == 'table':
            # 表格 -> 文本模型
            analysis_result = analyze_table_with_text_model(full_context, item)
            
        elif element_type == 'image':
            # 图片 -> 多模态视觉模型
            analysis_result = analyze_image_with_vision_model(full_context, item)
            
        else:
            analysis_result = "未知类型，跳过分析。"

        # 回写结果
        result_item = item.copy()
        result_item['deep_understanding'] = analysis_result
        
        # 为了输出整洁，可以把 content (表格md) 或 file_path 也就是原始输入保留，
        # 但如果 base64 这种大字段在内存里，输出时不要带上
        results.append(result_item)

    # 保存最终结果
    final_output_path = json_path.parent / f"{json_path.stem}_analysis.json"
    with open(final_output_path, 'w', encoding='utf-8') as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    
    print(f"[Analysis] Done. Report saved to {final_output_path}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--data", required=True, help="Path to the JSON generated by extracted script")
    args = parser.parse_args()
    
    if not os.getenv("ZHIPU_API_KEY"):
        print("Error: Please set ZHIPU_API_KEY environment variable.")
    else:
        process_understanding(args.data)

path: script/glm_ocr_extract.py
content: import os
import json
import base64
import argparse
from pathlib import Path
from bs4 import BeautifulSoup
from PIL import Image, ImageOps
from zai import ZhipuAiClient

# 初始化客户端
client = ZhipuAiClient([REDACTED_SECRET]"ZHIPU_API_KEY"))



def image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        encoded = base64.b64encode(image_file.read()).decode('utf-8')
    return f"data:image/png;base64,{encoded}"

def html_to_markdown(html_content):
    """将HTML表格转为Markdown，方便LLM阅读"""
    try:
        soup = BeautifulSoup(html_content, 'html.parser')
        rows = soup.find_all('tr')
        md_lines = []
        for i, row in enumerate(rows):
            cols = row.find_all(['td', 'th'])
            col_texts = [ele.get_text(strip=True).replace('\n', ' ') for ele in cols]
            if not col_texts: continue
            md_lines.append("| " + " | ".join(col_texts) + " |")
            if i == 0: # 假设第一行是表头
                md_lines.append("| " + " | ".join(["---"] * len(col_texts)) + " |")
        return "\n".join(md_lines)
    except:
        return html_content

def get_nearest_title(target_bbox, all_details, page_h):
    """几何算法：寻找距离最近的 figure_title 或 table_title"""
    target_cy = (target_bbox[1] + target_bbox[3]) / 2
    nearest_title = None
    min_dist = page_h * 0.15 # 搜索范围阈值

    for item in all_details:
        # 确保 item 是对象或字典，兼容 LayoutDetail 对象访问
        label = item.native_label if hasattr(item, 'native_label') else item.get('native_label')
        bbox = item.bbox_2d if hasattr(item, 'bbox_2d') else item.get('bbox_2d')
        content = item.content if hasattr(item, 'content') else item.get('content')

        if label in ['figure_title', 'table_title']:
            item_cy = (bbox[1] + bbox[3]) / 2
            dist = abs(target_cy - item_cy)
            if dist < min_dist:
                min_dist = dist
                # 清洗 HTML 标签
                clean_content = BeautifulSoup(content, 'html.parser').get_text(strip=True)
                nearest_title = clean_content
    return nearest_title

def process_file(file_path, output_dir):
    file_path = Path(file_path).resolve()
    output_dir = Path(output_dir).resolve()
    img_dir = output_dir / "images"
    img_dir.mkdir(parents=True, exist_ok=True)

    print(f"[OCR] Processing {file_path.name}...")

    print(file_path)
    # 1. 调用 GLM-OCR
    response = client.layout_parsing.create(
        model="glm-ocr",
        file=image_to_base64(file_path)
    )

    # 2. 准备图片裁剪
    try:
        original_img = Image.open(file_path)
        original_img = ImageOps.exif_transpose(original_img)
        # 计算缩放比例 (API返回的坐标基于 data_info.pages[0] 的尺寸)
        page_w = response.data_info.pages[0].width
        page_h = response.data_info.pages[0].height
        scale_x = original_img.width / page_w
        scale_y = original_img.height / page_h
    except Exception as e:
        print(f"[Warn] Cannot load image for cropping: {e}")
        original_img = None
        scale_x, scale_y = 1, 1

    extracted_elements = []
    
    # 3. 提取关键信息
    # md_results 是理解上下文的核心，必须保存
    full_context_md = response.md_results 

    print(response)
    for idx, item in enumerate(response.layout_details[0]):
        if item.label not in ['image', 'table']:
            continue

        element_data = {
            "id": f"{item.label}_{idx}",
            "type": item.label,
            "bbox": item.bbox_2d,
            "detected_title": get_nearest_title(item.bbox_2d, response.layout_details[0], page_h)
        }

        # 处理表格
        if item.label == 'table':
            element_data["content"] = html_to_markdown(item.content)
            element_data["file_path"] = None # 表格不需要保存图片，除非你想

        # 处理图片
        elif item.label == 'image':
            element_data["content"] = "[Image File]"
            if original_img:
                x1, y1, x2, y2 = item.bbox_2d
                crop = original_img.crop((x1*scale_x, y1*scale_y, x2*scale_x, y2*scale_y))
                save_name = f"{file_path.stem}_{item.label}_{idx}.png"
                save_path = img_dir / save_name
                crop.save(save_path)
                element_data["file_path"] = str(save_path)
            else:
                element_data["file_path"] = None

        extracted_elements.append(element_data)

    # 4. 生成中间态数据包 (Payload for Script 2)
    payload = {
        "source_file": str(file_path),
        "full_markdown_context": full_context_md, # 核心：整页的阅读逻辑文本
        "elements": extracted_elements
    }

    json_path = output_dir / f"{file_path.stem}_data.json"
    with open(json_path, "w", encoding='utf-8') as f:
        json.dump(payload, f, ensure_ascii=False, indent=2)
    
    print(f"[OCR] Done. Data saved to {json_path}")
    return str(json_path)

if __name__ == "__main__":
    # 示例调用
    parser = argparse.ArgumentParser(description="Zhipu GLM-OCR Layout Parser")
    parser.add_argument("--file", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    process_file(args.file, args.out)

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)