Skill: aiyoubug/yindenganalyse (version 1.0.0)

# CoPAW 银登公告与结果爬取 Skill 这是一个用于 CoPAW (Copilot Automation Workflow) 的标准 Skill，支持自动爬取银登网不良贷款转让公告及转让结果，并利用大模型（LLM）提取关键数据。 ## 目录结构 ``` yindeng_skill/ ├── skill.py # 标准 Skill 接口 (Agent/Tool Use) ├── skill.json # Skill 元数据定义 ├── main.py # CLI 入口脚本 ├── crawler.py # 核心爬虫逻辑 ├── ocr_utils.py # OCR 与数据汇总工具 ├── requirements.txt # 依赖清单 └── SKILL.md # 说明文档 ``` ## Agent 集成 (CoPaw / Tool Use) 本 Skill 提供了标准的 Python 函数接口和 JSON Schema 定义，可供 AI Agent 直接调用。 ### Python 接口 ```python from skill import run_yindeng_crawler # 爬取并分析 result = run_yindeng_crawler(source="result", analyze=True) print(result) ``` ### Skill 定义 (skill.json) 包含完整的工具描述和参数定义，符合 OpenAI/Anthropics Tool Schema 标准。 ## 功能特性 1. **多任务支持**：可同时爬取“转让公告”和“转让结果公告”。 2. **自动爬取**：根据日期（自动计算或指定）爬取银登网公告并下载 PDF。 3. **智能分析**：支持多模型（DeepSeek、千问、硅基流动等）提取 PDF 中的关键金融数据。 4. **数据导出**： - 爬取记录 Excel - 债权转让证明信息汇总（仅结果公告，含 OCR 数据） - LLM 分析结果报表 (`analysis_result.xlsx`) ## 部署与使用 ### 1. 安装依赖 在运行环境或 Docker 容器中安装 Python 依赖（需提前安装 Tesseract-OCR 和 Poppler）： ```bash pip install -r requirements.txt ``` ### 2. 环境变量配置 (LLM) 设置以下环境变量以启用 LLM 分析功能。本 Skill 支持多种 LLM 提供商。 #### 通用配置 - `LLM_API_KEY`: **(必须)** 您的 API Key。 - `LLM_PROVIDER`: (可选) 指定服务商，如 `deepseek`, `siliconflow`, `qwen`。默认为 `deepseek`。 - `LLM_API_BASE`: (可选) API 基础地址。 - `LLM_MODEL`: (可选) 模型名称。 #### 常用服务商配置示例 **DeepSeek (默认)** ```bash export LLM_API_KEY="sk-..." export LLM_PROVIDER="deepseek" # 默认 BASE: https://api.deepseek.com/v1 # 默认 MODEL: deepseek-chat ``` **硅基流动 (SiliconFlow)** ```bash export LLM_API_KEY="sk-..." export LLM_PROVIDER="siliconflow" # 默认 BASE: https://api.siliconflow.cn/v1 # 默认 MODEL: deepseek-ai/DeepSeek-V3 ``` **通义千问 (DashScope/Aliyun)** ```bash export LLM_API_KEY="sk-..." export LLM_PROVIDER="qwen" # 默认 BASE: https://dashscope.aliyuncs.com/compatible-mode/v1 # 默认 MODEL: qwen-plus ``` ### 3. 调用方式 **默认运行（爬取公告 + 结果）：** ```bash python main.py ``` **仅爬取转让结果：** ```bash python main.py --source result ``` **爬取并启用 LLM 分析：** ```bash python main.py --analyze ``` **指定日期运行：** ```bash python main.py 2026-03-03 ``` ## 输出结果 - **转让公告**: `YYYY-MM-DD银登公告/` - **转让结果**: `YYYY-MM-DD银登公告结果/` - `*债权转让证明信息汇总.xlsx`: 包含 OCR 识别后的结构化数据。 - `analysis_result.xlsx`: 包含 LLM 提取的深度分析数据。 ## 退出码说明 - `0`: 成功且有新数据。 - `2`: 成功但**暂无更新**（所有任务均无新增数据）。 - `1`: 运行出错。

path: skill.py
content: import os
import sys
from datetime import datetime
from typing import Optional

# Ensure local modules can be imported
sys.path.append(os.path.dirname(os.path.abspath(__file__)))

from crawler import run_crawl_task, get_default_date_str

# Try to import analysis function from main.py
try:
    from main import process_downloaded_files
except ImportError:
    def process_downloaded_files(*args, **kwargs):
        print("Warning: Analysis function could not be imported from main.py")

def log_callback(msg):
    """Standard logging callback."""
    print(f"[YindengSkill] {msg}")

class YindengSkill:
    """
    Standard Skill for CoPAW to crawl Yindeng announcements and results.
    """
    
    def run(self, date: Optional[str] = None, source: str = "all", analyze: bool = False) -> str:
        """
        Execute the crawling task.

        Args:
            date: Start date in 'YYYY-MM-DD' format. If not provided, defaults to auto-calculation (1-3 days ago).
            source: Source to crawl. Options: 'result', 'announcement', 'all'. Default is 'all'.
            analyze: Whether to perform LLM analysis on the downloaded documents. Default is False.
            
        Returns:
            A summary string of the execution result.
        """
        try:
            # Determine date
            if date:
                try:
                    datetime.strptime(date, "%Y-%m-%d")
                    start_date_str = date
                except ValueError:
                    return f"Error: Invalid date format '{date}'. Please use YYYY-MM-DD."
            else:
                start_date_str = get_default_date_str()
            
            # Determine tasks
            tasks = []
            if source == "all":
                tasks = ["announcement", "result"]
            elif source in ["announcement", "result"]:
                tasks = [source]
            else:
                return f"Error: Invalid source '{source}'. Must be 'result', 'announcement', or 'all'."

            results = []
            total_added = 0
            
            for task_name in tasks:
                log_callback(f"Starting task: {task_name} from {start_date_str}")
                pdf_dir, excel_path, added, no_update = run_crawl_task(task_name, start_date_str=start_date_str, log_cb=log_callback)
                
                if added is not None:
                    total_added += added
                
                task_result = f"Task '{task_name}': "
                if no_update:
                    task_result += "No new updates."
                else:
                    task_result += f"Added {added} records."
                
                if analyze and pdf_dir and excel_path:
                    log_callback(f"Analyzing {task_name}...")
                    process_downloaded_files(pdf_dir, excel_path, log_callback)
                    task_result += " Analysis completed."
                
                results.append(task_result)

            final_summary = "\n".join(results)
            return f"Execution completed.\n{final_summary}\nTotal new records: {total_added}"

        except Exception as e:
            import traceback
            traceback.print_exc()
            return f"Error executing skill: {str(e)}"

# Expose a standalone function for Agent integration
def run_yindeng_crawler(date: Optional[str] = None, source: str = "all", analyze: bool = False) -> str:
    """
    Crawl Yindeng (银登网) announcements and results.
    
    Args:
        date: Start date (YYYY-MM-DD). Defaults to auto (1-3 days ago).
        source: 'result', 'announcement', or 'all'.
        analyze: Enable LLM analysis of PDFs.
    """
    skill = YindengSkill()
    return skill.run(date, source, analyze)

if __name__ == "__main__":
    # Test run
    print(run_yindeng_crawler(source="result", analyze=False))


path: requirements.txt
content: requests
beautifulsoup4
lxml
openpyxl
pandas
pytesseract
pdf2image
Pillow
opencv-python
numpy
PyPDF2
xlsxwriter
pdfminer.six


path: main.py
content: import os
import sys
import time
import argparse
import json
import requests
import pandas as pd
import re
from datetime import datetime, timedelta
from pdfminer.high_level import extract_text
import crawler  # Import the module to access globals
from crawler import run_crawl_task

def get_target_date():
    """
    Calculate the target start date for crawling based on the current weekday.
    Logic:
    - If today is Monday (weekday=0), start from last Friday (3 days ago).
    - Otherwise, start from yesterday (1 day ago).
    This logic matches the default behavior in the GUI version.
    """
    now = datetime.now()
    if now.weekday() == 0:  # Monday
        target_date = now - timedelta(days=3)
    else:
        target_date = now - timedelta(days=1)
    return target_date.strftime("%Y-%m-%d")

def log_callback(msg):
    """
    Simple logging callback that prints to stdout with a timestamp.
    CoPAW or other schedulers can capture this output.
    """
    timestamp = datetime.now().strftime("[%Y-%m-%d %H:%M:%S]")
    print(f"{timestamp} {msg}")
    sys.stdout.flush()  # Ensure logs are flushed immediately

def extract_content_from_pdf(pdf_path):
    """
    Extract text from a PDF file using pdfminer.six.
    """
    try:
        text = extract_text(pdf_path)
        return text
    except Exception as e:
        log_callback(f"Error extracting text from {pdf_path}: {e}")
        return ""

def call_llm_analysis(text, file_name, log_cb=log_callback):
    """
    Call LLM to extract structured data from the text.
    """
    # Define the columns as requested by the user
    columns = [
        '文件名', '标题', '交易基准日', '未偿本金总额', '资产笔数', '未偿利息总额', '借款人户数', '未偿本息总额', 
        '加权平均逾期天数', '其他费用', '单一借款人最高未偿本息余额', '借款人平均未偿本息余额', '借款人加权平均年龄', 
        '借款人加权平均授信额度', '五级分类情况', '核销情况', '担保情况概述', '诉讼情况概述', '备注', 
        '竞价报名截止时间', '竞价日', '转让方式', '竞价方式', '自由竞价开始时间', '自由竞价结束时间', 
        '延时周期（分钟）', '起始价（元）', '加价幅度（元）', '发布日期'
    ]
    
    prompt = f"""
    你是一个金融数据提取助手。请从以下不良贷款转让公告的文本中提取关键信息。
    请提取以下字段的值，并返回一个合法的 JSON 对象。
    
    需要提取的字段:
    {', '.join(columns)}
    
    注意:
    - '文件名' 字段请填写: '{file_name}'
    - 如果某个字段在文本中没有找到，请使用空字符串 ""。
    - 日期字段请统一格式为 YYYYMMDD 或 YYYY-MM-DD。
    - 金额字段请尽量保留数值，如果包含单位请保留单位。
    - 只返回 JSON 对象，不要包含 markdown 格式标记（如 ```json ... ```）。
    
    文本内容:
    {text[:15000]} 
    """
    
    # Get API configuration from environment variables
    [REDACTED_SECRET]"LLM_API_KEY")
    api_base = os.getenv("LLM_API_BASE", "https://api.deepseek.com/v1")
    model_name = os.getenv("LLM_MODEL", "deepseek-chat")
    provider = os.getenv("LLM_PROVIDER", "deepseek").lower()

    # Provider specific default configurations
    if provider == "siliconflow" or provider == "硅基流动":
        # SiliconFlow default
        if not os.getenv("LLM_API_BASE"):
            api_base = "https://api.siliconflow.cn/v1"
        if not os.getenv("LLM_MODEL"):
            model_name = "deepseek-ai/DeepSeek-V3" # Common model on SiliconFlow
    elif provider == "qwen" or provider == "dashscope" or provider == "通义千问":
        # DashScope (Aliyun) default - usually requires compatible OpenAI client or specific URL
        # Here we assume using OpenAI-compatible endpoint if available, or user sets BASE_URL
        if not os.getenv("LLM_API_BASE"):
            api_base = "https://dashscope.aliyuncs.com/compatible-mode/v1"
        if not os.getenv("LLM_MODEL"):
            model_name = "qwen-plus"
    
    if not [REDACTED_SECRET]"Skipping LLM analysis: LLM_API_KEY environment variable not set.")
        return None

    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    
    data = {
        "model": model_name,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant that extracts data into JSON."},
            {"role": "user", "content": prompt}
        ],
        "temperature": 0.1,
        "response_format": {"type": "json_object"}
    }
    
    try:
        response = requests.post(f"{api_base}/chat/completions", headers=headers, json=data, timeout=120)
        response.raise_for_status()
        result = response.json()
        content = result['choices'][0]['message']['content']
        
        # Clean up potential markdown code blocks
        content = re.sub(r'^```json\s*', '', content)
        content = re.sub(r'^```\s*', '', content)
        content = re.sub(r'\s*```$', '', content)
        
        return json.loads(content)
    except Exception as e:
        log_cb(f"LLM Call failed for {file_name}: {e}")
        if 'response' in locals() and hasattr(response, 'text'):
            log_cb(f"Response text: {response.text}")
        return None

def process_downloaded_files(pdf_dir, excel_path, log_cb=log_callback):
    """
    Iterate through downloaded files and perform LLM analysis.
    """
    log_cb(f"Starting analysis for files in directory: {pdf_dir}")
    
    if not os.path.exists(pdf_dir):
        log_cb(f"Directory not found: {pdf_dir}")
        return

    # Path for the analysis result Excel
    analysis_excel_path = os.path.join(pdf_dir, "analysis_result.xlsx")
    
    # Load existing analysis to avoid re-processing
    processed_files = set()
    df_existing = pd.DataFrame()
    
    if os.path.exists(analysis_excel_path):
        try:
            df_existing = pd.read_excel(analysis_excel_path)
            if '文件名' in df_existing.columns:
                processed_files = set(df_existing['文件名'].astype(str))
            log_cb(f"Loaded existing analysis records: {len(df_existing)}")
        except Exception as e:
            log_cb(f"Error loading existing analysis file: {e}")
            df_existing = pd.DataFrame()
    
    # Load the crawl log to find files
    if not os.path.exists(excel_path):
        log_cb(f"Crawl log not found: {excel_path}")
        return
        
    try:
        wb = pd.read_excel(excel_path)
    except Exception as e:
        log_cb(f"Failed to read crawl log: {e}")
        return

    new_results = []
    
    # Check for API Key presence
    if not os.getenv("LLM_API_KEY"):
        log_cb("Warning: LLM_API_KEY is not set. Only text extraction will be attempted (but not saved). Please set LLM_API_KEY to enable analysis.")
        return

    for index, row in wb.iterrows():
        pdf_name = str(row.get('PDF文件名', ''))
        pdf_path = str(row.get('保存路径', ''))
        
        if not pdf_name or not pdf_path or pdf_path == "nan" or pdf_name == "nan":
            continue
            
        if pdf_name in processed_files:
            continue
            
        # Handle relative paths or verify existence
        if not os.path.isabs(pdf_path):
            pdf_path = os.path.join(pdf_dir, pdf_path)
            
        if not os.path.exists(pdf_path):
            # Try to find it in the directory by name
            potential_path = os.path.join(pdf_dir, pdf_name)
            if os.path.exists(potential_path):
                pdf_path = potential_path
            else:
                # log_cb(f"File not found: {pdf_path}")
                continue
        
        # Skip non-PDF files (e.g. .txt) for now unless logic supports it
        if not pdf_path.lower().endswith('.pdf'):
            continue
            
        log_cb(f"Analyzing new file: {pdf_name}")
        
        # Extract text
        text = extract_content_from_pdf(pdf_path)
        if not text or len(text.strip()) < 10:
            log_cb(f"Text extraction failed or empty for {pdf_name}")
            continue
            
        # Call LLM
        data = call_llm_analysis(text, pdf_name, log_cb)
        
        if data:
            # Ensure critical fields are present
            data['文件名'] = pdf_name
            if not data.get('发布日期'):
                data['发布日期'] = row.get('发布日期', '')
            if not data.get('标题'):
                data['标题'] = row.get('标题', '')
                
            new_results.append(data)
            log_cb(f"Analysis successful for {pdf_name}")
            
            # Save incrementally (optional, but safer)
            # For now, we collect and save at the end of batch
        
    if new_results:
        df_new = pd.DataFrame(new_results)
        
        # Ensure all target columns exist
        target_columns = [
            '文件名', '标题', '交易基准日', '未偿本金总额', '资产笔数', '未偿利息总额', '借款人户数', '未偿本息总额', 
            '加权平均逾期天数', '其他费用', '单一借款人最高未偿本息余额', '借款人平均未偿本息余额', '借款人加权平均年龄', 
            '借款人加权平均授信额度', '五级分类情况', '核销情况', '担保情况概述', '诉讼情况概述', '备注', 
            '竞价报名截止时间', '竞价日', '转让方式', '竞价方式', '自由竞价开始时间', '自由竞价结束时间', 
            '延时周期（分钟）', '起始价（元）', '加价幅度（元）', '发布日期'
        ]
        
        # Add missing columns with empty strings
        for col in target_columns:
            if col not in df_new.columns:
                df_new[col] = ""
                
        # Reorder columns
        df_new = df_new[target_columns]
        
        # Combine with existing
        if not df_existing.empty:
            # Ensure existing df has same columns
            for col in target_columns:
                if col not in df_existing.columns:
                    df_existing[col] = ""
            df_final = pd.concat([df_existing[target_columns], df_new], ignore_index=True)
        else:
            df_final = df_new
            
        try:
            df_final.to_excel(analysis_excel_path, index=False)
            log_cb(f"Successfully saved analysis results to {analysis_excel_path}")
        except Exception as e:
            log_cb(f"Error saving Excel: {e}")
    else:
        log_cb("No new files to analyze or analysis failed.")

def main():
    parser = argparse.ArgumentParser(description="Run scheduled crawl for Yindeng announcements and optionally analyze with LLM.")
    parser.add_argument("date", nargs="?", help="Start date in YYYY-MM-DD format (e.g. 2025-01-01). Defaults to auto-calculation.")
    parser.add_argument("--analyze", action="store_true", help="Enable LLM analysis of downloaded documents.")
    parser.add_argument("--source", choices=["result", "announcement", "all"], default="all", help="Source to crawl: result (default), announcement, or all.")
    args = parser.parse_args()

    log_callback("Starting scheduled crawl task...")
    
    try:
        # 1. Determine the date range
        if args.date:
            try:
                datetime.strptime(args.date, "%Y-%m-%d")
                start_date_str = args.date
                log_callback(f"Using provided start date: {start_date_str}")
            except ValueError:
                log_callback(f"Invalid date format: {args.date}. Please use YYYY-MM-DD.")
                sys.exit(1)
        else:
            start_date_str = get_target_date()
            log_callback(f"Auto-calculated target start date: {start_date_str}")
        
        # Determine tasks
        tasks = []
        if args.source == "all":
            tasks = ["announcement", "result"]
        else:
            tasks = [args.source]
            
        total_added = 0
        all_no_update = True
        
        for task_name in tasks:
            log_callback(f"=== Starting task: {task_name} ===")
            pdf_dir, excel_path, added, no_update = run_crawl_task(task_name, start_date_str=start_date_str, log_cb=log_callback)
            
            total_added += (added or 0)
            if not no_update:
                all_no_update = False
                
            log_callback(f"Task {task_name} completed, added rows: {added}")
            
            # 3. Run Analysis if requested (Only for 'result' or if generic analysis is desired)
            # Currently analysis logic is generic for downloaded PDFs, so we can run it for both if needed.
            if args.analyze and pdf_dir and excel_path:
                log_callback(f"Starting LLM analysis phase for {task_name}...")
                process_downloaded_files(pdf_dir, excel_path, log_callback)
            elif args.analyze:
                log_callback(f"Skipping analysis for {task_name} due to missing output paths.")
        
        # Check for no updates (exit code 2 logic)
        # If ALL tasks had no updates, we exit with 2.
        if all_no_update:
            log_callback(f"{start_date_str}暂无更新 (All tasks)")
            sys.exit(2)
        
        log_callback(f"Scheduled task completed successfully. Total added rows: {total_added}")
        sys.exit(0)
        
    except SystemExit as se:
        raise se
    except Exception as e:
        log_callback(f"Error during scheduled crawl: {str(e)}")
        # Print full traceback for debugging
        import traceback
        traceback.print_exc()
        sys.exit(1)

if __name__ == "__main__":
    main()


path: ocr_utils.py
content: # 在导入部分添加PyPDF2库
import os
import re
import pandas as pd
import pytesseract
from pdf2image import convert_from_path
from PIL import Image
import cv2
import numpy as np
from openpyxl import Workbook
from openpyxl.styles import Alignment, Border, Side, Font
import PyPDF2  # 添加PyPDF2库用于文本型PDF处理

def clean_text_for_excel(text):
    """清理文本中Excel不支持的字符"""
    if not isinstance(text, str):
        return text
    
    # 替换已知的非法字符
    replacements = {
        '\x00': '', '\x01': '', '\x02': '', '\x03': '', '\x04': '', '\x05': '', '\x06': '', '\x07': '',
        '\x08': '', '\x0b': '', '\x0c': '', '\x0e': '', '\x0f': '', '\x10': '', '\x11': '', '\x12': '',
        '\x13': '', '\x14': '', '\x15': '', '\x16': '', '\x17': '', '\x18': '', '\x19': '', '\x1a': '',
        '\x1b': '', '\x1c': '', '\x1d': '', '\x1e': '', '\x1f': '',
        '×': 'x',  # 替换乘号
        '÷': '/',  # 替换除号
    }
    
    for old, new in replacements.items():
        text = text.replace(old, new)
    
    # 移除其他可能的控制字符
    text = ''.join(ch for ch in text if ord(ch) >= 32 or ch in '\r\n\t')
    
    # 截断过长的文本
    if len(text) > 32000:  # Excel单元格字符限制
        text = text[:32000] + "..."
    
    return text

def extract_date_from_filename(filename):
    """从文件名中提取日期时间信息"""
    # 尝试匹配常见的日期格式，如20230101、2023-01-01、2023.01.01等
    date_patterns = [
        r'(\d{4}[-_\.年]\d{1,2}[-_\.月]\d{1,2}[日]?)',  # 2023-01-01、2023.01.01、2023年01月01日
        r'(\d{8})',  # 20230101
        r'(\d{4}[-_\.]\d{2})',  # 2023-01、2023.01
    ]
    
    for pattern in date_patterns:
        match = re.search(pattern, filename)
        if match:
            return match.group(1)
    
    return "未识别"

def extract_text_from_pdf_image(pdf_path):
    """从PDF图片中提取文本"""
    try:
        # 将PDF转换为图像
        images = convert_from_path(pdf_path)
        
        # 处理第一页（假设表格在第一页）
        if images:
            image = images[0]
            
            # 转换为OpenCV格式进行处理
            img_np = np.array(image)
            img_cv = cv2.cvtColor(img_np, cv2.COLOR_RGB2BGR)
            
            # 转为灰度图
            gray = cv2.cvtColor(img_cv, cv2.COLOR_BGR2GRAY)
            
            # 使用pytesseract进行OCR识别
            text = pytesseract.image_to_string(gray, lang='chi_sim')
            return text
        else:
            print(f"无法从PDF提取图像: {pdf_path}")
            return ""
    except Exception as e:
        print(f"从PDF图像提取文本时出错: {e}")
        return ""

def parse_table_data_from_image(text, pdf_path):
    """从图像提取的文本中解析表格数据"""
    data = {}
    
    # 注意：已移除姓名、身份证号和借据号的识别
    
    # 项目名称 - 保留原有逻辑，但可能不适用于债权转让证明
    project_name_pattern = r"项目名称\s*(.*?)(?=\s*项目编码|\s*项目编号|\s*出让方全称|\n\s*\n)"
    project_name_match = re.search(project_name_pattern, text, re.DOTALL)
    if project_name_match:
        data["项目名称"] = clean_text_for_excel(project_name_match.group(1).strip().replace('\n', ' '))
    else:
        # 尝试从整个文本中提取项目名称
        full_name_pattern = r"关于\s*(.*?)不良贷款转让"
        full_name_match = re.search(full_name_pattern, text, re.DOTALL)
        if full_name_match:
            data["项目名称"] = clean_text_for_excel(full_name_match.group(1).strip().replace('\n', ' '))
        else:
            data["项目名称"] = "未识别"
    
    # 项目编码/项目编号 - 确保在项目名称之后单独匹配
    project_code_pattern = r"(项目编号|项目编码)\s*([A-Za-z0-9\-_]+)"
    project_code_match = re.search(project_code_pattern, text, re.DOTALL)
    if project_code_match:
        data["项目编码"] = project_code_match.group(2).strip()
    else:
        data["项目编码"] = "未识别"
    
    # 出让方全称
    seller_pattern = r"出让方全称\s*(.*?)(?=\n|$)"
    seller_match = re.search(seller_pattern, text, re.DOTALL)
    if seller_match:
        data["出让方全称"] = clean_text_for_excel(seller_match.group(1).strip())
    else:
        # 尝试匹配债权转让证明中的银行名称
        bank_pattern = r"与([\u4e00-\u9fa5]+银行[\u4e00-\u9fa5]*公司).*?签订"
        bank_match = re.search(bank_pattern, text, re.DOTALL)
        if bank_match:
            data["出让方全称"] = clean_text_for_excel(bank_match.group(1).strip())
        else:
            data["出让方全称"] = "未识别"
    
    # 受让方全称
    buyer_pattern = r"受让方全称\s*(.*?)(?=\n|$)"
    buyer_match = re.search(buyer_pattern, text, re.DOTALL)
    if buyer_match:
        data["受让方全称"] = clean_text_for_excel(buyer_match.group(1).strip())
    else:
        # 尝试匹配债权转让证明中的受让方
        new_buyer_pattern = r"转让至([\u4e00-\u9fa5]+(?:有限公司|有限责任公司|企业))"
        new_buyer_match = re.search(new_buyer_pattern, text, re.DOTALL)
        if new_buyer_match:
            data["受让方全称"] = clean_text_for_excel(new_buyer_match.group(1).strip())
        else:
            data["受让方全称"] = "未识别"
    
    # 转让协议签署日期
    date_pattern = r"转让协议签署日期\s*(\d{4}年\d{1,2}月\d{1,2}日)"
    date_match = re.search(date_pattern, text, re.DOTALL)
    if date_match:
        data["转让协议签署日期"] = date_match.group(1).strip()
    else:
        # 尝试匹配债权转让证明中的日期格式
        new_date_pattern = r"(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日"
        new_date_match = re.search(new_date_pattern, text)
        if new_date_match:
            year = new_date_match.group(1)
            month = new_date_match.group(2)
            day = new_date_match.group(3)
            data["转让协议签署日期"] = f"{year}年{month}月{day}日"
        else:
            data["转让协议签署日期"] = "未识别"
    
    # 添加文件名
    filename = os.path.basename(pdf_path)
    data["文件名"] = filename
    
    # 获取PDF所在文件夹的名称
    parent_folder = os.path.basename(os.path.dirname(pdf_path))
    data["文件夹"] = parent_folder
    # 从PDF右下角提取时间
    # 匹配PDF中的日期格式（如：2024年9月23日），优先选择靠近文本末尾的日期
    footer_date_pattern = r"(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日"
    # 查找所有匹配项
    footer_date_matches = list(re.finditer(footer_date_pattern, text))
    
    if footer_date_matches:
        # 选择最后一个匹配项（通常是PDF右下角的日期）
        last_match = footer_date_matches[-1]
        year = last_match.group(1)
        month = last_match.group(2)
        day = last_match.group(3)
        data["文件时间"] = f"{year}年{month}月{day}日"
        print(f"从PDF提取到日期: {data['文件时间']}")
    else:
        # 如果在PDF中没有找到日期，则尝试从文件名提取
        file_date = extract_date_from_filename(parent_folder)
        data["文件时间"] = file_date
        print(f"未从PDF提取到日期，使用文件名日期: {file_date}")
    
    # 提取欠款金额信息
    amount_pattern = r"([0-9]+\.[0-9]+)\s*元\s*\(大写[^)]+\)"
    amount_match = re.search(amount_pattern, text)
    if amount_match:
        data["欠款金额"] = amount_match.group(1).strip()
    else:
        data["欠款金额"] = "未识别"
    
    return data

def is_text_pdf(pdf_path):
    """检测PDF是否为文本型PDF"""
    try:
        with open(pdf_path, 'rb') as file:
            reader = PyPDF2.PdfReader(file)
            page = reader.pages[0]  # 检查第一页
            text = page.extract_text()
            
            # 如果提取的文本长度超过一定阈值，认为是文本型PDF
            # 这里设置100个字符作为阈值，可以根据实际情况调整
            if len(text) > 100:
                return True, text
            return False, ""
    except Exception as e:
        print(f"检测PDF类型时出错: {e}")
        return False, ""

def extract_text_from_text_pdf(pdf_path):
    """从文本型PDF中提取文本"""
    try:
        all_text = ""
        with open(pdf_path, 'rb') as file:
            reader = PyPDF2.PdfReader(file)
            # 提取所有页面的文本
            for page_num in range(len(reader.pages)):
                page = reader.pages[page_num]
                text = page.extract_text()
                all_text += text + "\n"
        return all_text
    except Exception as e:
        print(f"从文本型PDF提取文本时出错: {e}")
        return ""

def process_pdf(pdf_path):
    """处理PDF文件并提取表格信息，自动区分文本型和图像型PDF"""
    # 首先检测PDF类型
    is_text, text = is_text_pdf(pdf_path)
    
    if is_text and text:
        print(f"检测到文本型PDF: {pdf_path}")
        # 对于文本型PDF，直接使用提取的文本
        full_text = extract_text_from_text_pdf(pdf_path)
    else:
        print(f"检测到图像型PDF或无法确定类型: {pdf_path}")
        # 对于图像型PDF或无法确定类型的PDF，使用OCR方法
        full_text = extract_text_from_pdf_image(pdf_path)
    
    if not full_text:
        print(f"无法从PDF提取文本: {pdf_path}")
        return None
    
    # 解析表格数据
    data = parse_table_data_from_image(full_text, pdf_path)
    
    return data

def find_all_pdfs(directory):
    """递归查找目录及其子目录中的所有PDF文件"""
    pdf_files = []
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file.lower().endswith('.pdf'):
                pdf_files.append(os.path.join(root, file))
    return pdf_files

def save_to_combined_excel(all_data, output_path):
    """将所有数据保存到一个Excel文件中"""
    # 确保输出目录存在
    output_dir = os.path.dirname(output_path)
    if output_dir and not os.path.exists(output_dir):
        os.makedirs(output_dir)
    
    # 创建DataFrame
    columns = ["文件名", "文件时间", "文件夹", "项目名称", "项目编码", "出让方全称", "受让方全称", "转让协议签署日期"]
    
    # 清理所有数据中的非法字符
    cleaned_data = []
    for item in all_data:
        if item:  # 确保数据不为None
            cleaned_item = {}
            for key, value in item.items():
                if key in columns:  # 只保留需要的列
                    cleaned_item[key] = clean_text_for_excel(value)
            cleaned_data.append(cleaned_item)
    
    df = pd.DataFrame(cleaned_data, columns=columns)
    
    # 尝试使用xlsxwriter引擎保存，这可能有助于处理特殊字符
    try:
        df.to_excel(output_path, index=False, sheet_name="不良贷款转让信息", engine='xlsxwriter')
    except Exception as e:
        print(f"使用xlsxwriter保存Excel失败: {e}")
        # 如果xlsxwriter失败，尝试使用默认引擎
        try:
            df.to_excel(output_path, index=False, sheet_name="不良贷款转让信息")
        except Exception as e2:
            print(f"使用默认引擎保存Excel也失败: {e2}")
            # 最后尝试保存为CSV
            csv_path = output_path.replace('.xlsx', '.csv')
            df.to_csv(csv_path, index=False, encoding='utf-8-sig')
            print(f"已将数据保存为CSV格式: {csv_path}")
            return
    
    print(f"所有数据已保存到: {output_path}")

def main():
    # 设置固定的输入和输出路径
    pdf_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "银登结果爬取记录")
    output_dir = os.path.dirname(os.path.abspath(__file__))
    output_file = os.path.join(output_dir, "债权转让证明信息汇总.xlsx")
    
    print(f"正在查找 {pdf_dir} 目录下的所有PDF文件...")
    
    # 查找所有PDF文件
    pdf_files = find_all_pdfs(pdf_dir)
    print(f"找到 {len(pdf_files)} 个PDF文件")
    
    # 存储所有PDF的处理结果
    all_data = []
    
    # 处理所有找到的PDF文件
    for pdf_path in pdf_files:
        print(f"正在处理: {pdf_path}")
        
        try:
            # 提取数据
            data = process_pdf(pdf_path)
            if data:
                all_data.append(data)
                print(f"成功处理: {os.path.basename(pdf_path)}")
            else:
                print(f"处理失败，无法提取数据: {os.path.basename(pdf_path)}")
        except Exception as e:
            print(f"处理 {pdf_path} 时出错: {e}")
    
    # 将所有数据保存到一个Excel文件
    if all_data:
        save_to_combined_excel(all_data, output_file)
        print(f"所有数据已合并保存到: {output_file}")
    else:
        print("没有成功处理任何PDF文件，未生成Excel文件")

if __name__ == "__main__":
    main()

path: skill.json
content: {
  "name": "yindeng_crawler",
  "version": "1.0.0",
  "description": "Crawl Yindeng announcements and results, optionally analyzing them with LLM.",
  "entry_point": "skill.py:run_yindeng_crawler",
  "parameters": {
    "type": "object",
    "properties": {
      "date": {
        "type": "string",
        "description": "Start date in YYYY-MM-DD format. Defaults to auto-calculation based on weekday.",
        "pattern": "^\\d{4}-\\d{2}-\\d{2}$"
      },
      "source": {
        "type": "string",
        "enum": ["result", "announcement", "all"],
        "default": "all",
        "description": "The source type to crawl."
      },
      "analyze": {
        "type": "boolean",
        "default": false,
        "description": "Whether to perform LLM analysis on downloaded documents."
      }
    },
    "required": []
  }
}


path: crawler.py
content: import time
import os
import requests
from urllib.parse import urljoin
import sys
import re
from datetime import datetime, timedelta
import json
from bs4 import BeautifulSoup
import openpyxl as op
from openpyxl import Workbook, load_workbook
from ocr_utils import process_pdf, save_to_combined_excel

# ==========================================
#              爬取速度设置
# ==========================================
REQUEST_DELAY = 1.5
# ==========================================

# 文件名清洗
def sanitize_filename(name, default_ext='.pdf'):
    name = (name or '').strip()
    invalid = '\\/:*?"<>|'
    for ch in invalid:
        name = name.replace(ch, '_')
    if default_ext and not name.lower().endswith(default_ext):
        name = name + default_ext
    return name

def normalize_title(s):
    s = (s or '').strip().lower()
    s = re.sub(r'\.pdf$', '', s)
    s = re.sub(r'[\s\-_]+', '', s)
    return s

def calculate_default_date():
    today = datetime.now().date()
    if today.weekday() == 0:
        return today - timedelta(days=3)
    return today - timedelta(days=1)

def get_default_date_str():
    return calculate_default_date().strftime("%Y-%m-%d")

def contains_chinese(s):
    return bool(re.search(r'[\u4e00-\u9fa5]', s or ''))

def is_english_title(s):
    s = s or ''
    return (not contains_chinese(s)) and bool(re.search(r'[a-zA-Z]', s))

def is_garbled(s):
    s = s or ''
    if '�' in s:
        return True
    if re.search(r'(锟斤拷|Ã|Â)', s):
        return True
    return False

def is_generic_result_announcement(s):
    return normalize_title(s) == normalize_title('转让结果公告')

def should_use_attachment_only(title, raw_title):
    if normalize_title(raw_title) == normalize_title(title):
        return True
    if is_garbled(title) or is_english_title(title) or is_generic_result_announcement(title):
        return True
    return False

_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
if getattr(sys, 'frozen', False):
    # 如果是打包后的exe运行，使用exe所在的目录
    _SCRIPT_DIR = os.path.dirname(sys.executable)

# 全局Session，保持会话
session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
})

# 定义爬取源配置
CRAWL_SOURCES = {
    "announcement": {
        "name": "银登公告",
        "url": "https://www.yindeng.com.cn/xxpl/xxpl_bldkzr/bldkzr_zrgg/index.html",
        "folder_suffix": "银登公告"
    },
    "result": {
        "name": "银登结果",
        "url": "https://www.yindeng.com.cn/xxpl/xxpl_bldkzr/bldkzr_zrjggg/index.html",
        "folder_suffix": "银登公告结果"
    }
}

time_start = None

# 全局变量定义，用于 run_crawl 内部
wb = None
ws = None
row_count = 1
all_recognition_data = []
last_run_added_rows = 0
last_run_no_update = False

def download_pdf(pdf_url, file_name, save_dir, referer=None, base_url=None, log_func=print):
    try:
        headers = {
            "Referer": referer or base_url,
            "Accept": "application/pdf,application/octet-stream;q=0.9,*/*;q=0.8",
        }
        
        safe_name = sanitize_filename(file_name)
        file_path = os.path.join(save_dir, safe_name)
        if os.path.exists(file_path):
            base, ext = os.path.splitext(safe_name)
            idx = 2
            while True:
                candidate = f"{base}({idx}){ext}"
                candidate_path = os.path.join(save_dir, candidate)
                if not os.path.exists(candidate_path):
                    file_path = candidate_path
                    break
                idx += 1
        
        response = session.get(pdf_url, headers=headers, stream=True, allow_redirects=True, timeout=30)
        if response.status_code != 200:
            log_func(f"下载失败，状态码: {response.status_code}")
            return None
            
        content_type = (response.headers.get("Content-Type") or "").lower()
        first_bytes = b""
        with open(file_path, "wb") as f:
            for chunk in response.iter_content(chunk_size=8192):
                if chunk:
                    if not first_bytes:
                        first_bytes = chunk[:5]
                    f.write(chunk)
                    
        if ("pdf" not in content_type) and not first_bytes.startswith(b"%PDF"):
            log_func("响应内容不是PDF，可能需要会话或权限")
            try:
                os.remove(file_path)
            except Exception:
                pass
            return None
            
        log_func(f"成功下载: {safe_name}")
        return file_path
    except Exception as e:
        log_func(f"下载出错: {e}")
        return None

# 获取当前页面的所有结果链接
def get_announcement_links(page_url, log_func=print):
    try:
        resp = session.get(page_url, timeout=15)
        if resp.status_code != 200:
            log_func(f"页面请求失败: {resp.status_code}")
            return [], None
        
        # 尝试检测编码，防止乱码
        resp.encoding = resp.apparent_encoding or 'utf-8'
        
        soup = BeautifulSoup(resp.text, 'lxml')
        
        # 获取总条数/总页数信息
        total_items = 0
        try:
            total_span = soup.find('span', class_='pageTotal', id='pageTotal')
            if total_span:
                match = re.search(r'\d+', total_span.get_text())
                if match:
                    total_items = int(match.group())
        except Exception:
            pass
            
        # 解析列表
        links = []
        list_box = soup.find('ul', class_='list-box')
        if list_box:
            rows = list_box.find_all('div', class_='rightListContent list-item')
            log_func(f"找到 {len(rows)} 个结果行")
            for row in rows:
                try:
                    # 标题和链接
                    link_element = row.find('span', class_='rightListImport unlock list-content').find('a')
                    title = link_element.get_text(strip=True)
                    href = link_element.get('href')
                    if href and not href.startswith('http'):
                        href = urljoin(page_url, href)
                    
                    # 日期
                    date_element = row.find('span', class_='rightListTime data-time')
                    date = date_element.get_text(strip=True)
                    
                    if href and title:
                        links.append({"title": title, "date": date, "href": href})
                except Exception as e:
                    log_func(f"处理结果行时出错: {e}")
                    continue
                    
        return links, total_items
    except Exception as e:
        log_func(f"获取结果链接出错: {e}")
        return [], None

# 检查页面是否包含PDF链接或内容
def process_announcement_page(page_url, title, log_func=print):
    try:
        resp = session.get(page_url, timeout=15)
        if resp.status_code != 200:
            return []
        resp.encoding = resp.apparent_encoding or 'utf-8'
        soup = BeautifulSoup(resp.text, 'lxml')
        
        pdf_links = []
        
        # 1. 查找页面中的PDF链接
        # 优先查找附件区域
        attach_area = soup.select('div.attach_main.attachment div.attach_file a[href$=".pdf"]')
        if not attach_area:
             attach_area = soup.select('div.attach_file a[href$=".pdf"]')
        
        # 如果还没找到，查找所有PDF链接
        if not attach_area:
             attach_area = soup.select('a[href$=".pdf"]')
             
        for element in attach_area:
            pdf_url = element.get('href')
            if pdf_url:
                if not pdf_url.startswith('http'):
                    pdf_url = urljoin(page_url, pdf_url)
                
                raw_title = element.get('title') or element.get_text(strip=True) or pdf_url.split('/')[-1]
                # 简单过滤，避免获取到无关的PDF
                # 统一使用公告标题作为文件名
                pdf_name = sanitize_filename(title)
                pdf_links.append({"url": pdf_url, "name": pdf_name})
        
        # 2. 查找iframe
        if not pdf_links:
            iframes = soup.find_all('iframe')
            for iframe in iframes:
                src = iframe.get('src')
                if src and ".pdf" in src.lower():
                    pdf_url = src
                    if not pdf_url.startswith('http'):
                        pdf_url = urljoin(page_url, pdf_url)
                    # 统一使用公告标题作为文件名
                    pdf_name = sanitize_filename(title)
                    pdf_links.append({"url": pdf_url, "name": pdf_name})
                    
        # 3. 如果仍然没有找到PDF，标记为文本
        if not pdf_links:
            # 简单检查是否有内容区域
            content_div = soup.find('div', class_='article-box')
            if content_div:
                 pdf_links.append({"url": page_url, "name": f"{title}.txt", "is_text": True})
                 
        return pdf_links
    except Exception as e:
        log_func(f"处理结果页面出错: {e}")
        return []

def parse_date(s):
    try:
        return datetime.strptime(s.strip(), "%Y-%m-%d").date()
    except Exception:
        return None

# 这里我们把 run_crawl 改造成一个通用函数，可以指定 source_type
def run_crawl_task(source_type, start_date_str=None, log_cb=None):
    # global wb, ws, row_count, time_start, last_run_added_rows, last_run_no_update, all_recognition_data, base_url
    
    config = CRAWL_SOURCES.get(source_type)
    if not config:
        if log_cb: log_cb(f"未知的爬取源: {source_type}")
        return None, None
        
    base_url = config['url']
    source_name = config['name']
    
    def log(msg):
        if log_cb:
            log_cb(f"[{source_name}] {str(msg)}")
        else:
            print(f"[{source_name}] {str(msg)}")
            
    time_start = time.time()
    all_recognition_data = [] # Reset for each run
    
    # 确定保存目录和文件路径
    date_prefix = start_date_str if start_date_str else get_default_date_str()
    date_prefix = date_prefix.replace(":", "-").replace("/", "-")
    folder_name = f"{date_prefix}{config['folder_suffix']}"
    
    current_pdf_dir = os.path.join(_SCRIPT_DIR, folder_name)
    if not os.path.exists(current_pdf_dir):
        os.makedirs(current_pdf_dir)
        log(f"创建下载目录: {current_pdf_dir}")
    else:
        log(f"使用下载目录: {current_pdf_dir}")
        
    # Excel路径也放在该目录下
    current_excel_path = os.path.join(current_pdf_dir, f"{date_prefix}{source_name}爬取记录.xlsx")
    current_combined_output_path = os.path.join(current_pdf_dir, f"{date_prefix}债权转让证明信息汇总.xlsx")
    
    # 初始化Excel
    if os.path.exists(current_excel_path):
        try:
            wb_loaded = load_workbook(current_excel_path)
            wb = wb_loaded
            ws = wb.active
            row_count = ws.max_row
            log(f"继续写入现有Excel，当前 {row_count} 行")
        except Exception:
            wb = Workbook()
            ws = wb.active
            ws.append(["标题", "发布日期", "详情链接", "PDF文件名", "PDF原始链接", "识别结果", "保存路径"])
            row_count = 1
    else:
        wb = Workbook()
        ws = wb.active
        ws.append(["标题", "发布日期", "详情链接", "PDF文件名", "PDF原始链接", "识别结果", "保存路径"])
        row_count = 1
        
    # 获取第一页数据并计算总页数
    log("正在获取首页数据并计算总页数...")
    first_page_links, total_items = get_announcement_links(base_url, log_func=log)
    
    total_pages = 1
    if total_items:
        items_per_page = 10
        total_pages = (total_items + items_per_page - 1) // items_per_page
        log(f"根据总条目数({total_items})计算得到总页数: {total_pages}")
    else:
        log("未能获取总条目数，将默认爬取直到无数据")
        total_pages = 1000 # 临时最大值
        
    pages_to_crawl = total_pages
        
    log(f"计划爬取 {pages_to_crawl} 页")
    
    start_date = None
    if start_date_str:
        try:
            start_date = datetime.strptime(start_date_str.strip(), "%Y-%m-%d").date()
        except Exception:
            start_date = None
            
    crawled_pages = 0
    starting_rows = row_count
    
    # 循环爬取
    for page in range(1, pages_to_crawl + 1):
        # 银登网翻页逻辑
        current_page_index = total_pages - page + 1 if page > 1 else 1
        page_url = base_url if page == 1 else f"{base_url.replace('/index.html', '')}/index_{current_page_index}.html"
        
        log(f"正在处理第 {page} 页: {page_url}")
        
        if page == 1:
            announcements = first_page_links
        else:
            # 翻页前等待，控制速度
            time.sleep(REQUEST_DELAY)
            announcements, _ = get_announcement_links(page_url, log_func=log)
            
        if not announcements:
            log(f"第 {page} 页未获取到数据，跳过")
            continue
            
        log(f"找到 {len(announcements)} 个结果")
        
        # 时间过滤检查
        if start_date:
            parsed = [parse_date(a['date']) for a in announcements]
            has_target = any((d is not None) and (d >= start_date) for d in parsed)
            if not has_target:
                log(f"第 {page} 页全部早于 {start_date}，提前结束")
                break
                
        for announcement in announcements:
            if start_date:
                d = parse_date(announcement['date'])
                if d and d < start_date:
                    continue
            
            # 处理详情页前等待，控制速度
            time.sleep(REQUEST_DELAY)
            
            log(f"处理结果: {announcement['title']} ({announcement['date']})")
            pdf_links = process_announcement_page(announcement['href'], announcement['title'], log_func=log)
            
            if pdf_links:
                for pdf_link in pdf_links:
                    rec_json = ""
                    saved_file_path = ""
                    if not pdf_link.get('is_text', False):
                        saved_file_path = download_pdf(pdf_link['url'], pdf_link['name'], current_pdf_dir, referer=announcement['href'], base_url=base_url, log_func=log)
                        try:
                            # 只有结果公告需要进行OCR识别并汇总
                            if source_type == "result" and saved_file_path:
                                data = process_pdf(saved_file_path)
                                if data:
                                    rec_json = json.dumps(data, ensure_ascii=False)
                                    all_recognition_data.append(data)
                        except Exception as e:
                            log(f"PDF处理异常: {e}")
                            rec_json = ""
                    
                    row_count += 1
                    ws.cell(row_count, 1).value = announcement['title']
                    ws.cell(row_count, 2).value = announcement['date']
                    ws.cell(row_count, 3).value = announcement['href']
                    ws.cell(row_count, 4).value = pdf_link['name']
                    ws.cell(row_count, 5).value = pdf_link['url']
                    ws.cell(row_count, 6).value = rec_json
                    ws.cell(row_count, 7).value = saved_file_path
            else:
                log(f"未找到PDF或内容: {announcement['title']}")
                row_count += 1
                ws.cell(row_count, 1).value = announcement['title']
                ws.cell(row_count, 2).value = announcement['date']
                ws.cell(row_count, 3).value = announcement['href']
                ws.cell(row_count, 4).value = "未找到PDF或内容"
                ws.cell(row_count, 5).value = ""
                ws.cell(row_count, 6).value = ""
                ws.cell(row_count, 7).value = ""
                
        try:
            wb.save(current_excel_path)
            log(f"Excel记录已保存: {current_excel_path}")
        except Exception as e:
            log(f"保存Excel出错: {e}")

        # 只有结果公告需要生成汇总表
        if source_type == "result":
            try:
                save_to_combined_excel(all_recognition_data, current_combined_output_path)
            except Exception:
                pass
        log(f"第 {page} 页处理完成")
        crawled_pages += 1
        
    time_end = time.time()
    log(f"爬取完成，总用时: {time_end - time_start:.2f} 秒")
    last_run_added_rows = row_count - starting_rows
    last_run_no_update = (start_date is not None and last_run_added_rows <= 0)
    if last_run_no_update:
        log(f"{start_date}暂无更新")
        
    return current_pdf_dir, current_excel_path, last_run_added_rows, last_run_no_update

# 为了兼容旧接口，提供 run_crawl 默认调用 result
def run_crawl(start_date_str=None, on_complete=None, log_cb=None):
    pdf_dir, excel_path, added, no_update = run_crawl_task("result", start_date_str, log_cb)
    # 更新全局变量状态以供 main.py 检查
    global last_run_added_rows, last_run_no_update
    last_run_added_rows = added
    last_run_no_update = no_update
    
    if on_complete:
        on_complete()
    return pdf_dir, excel_path


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)