Skill: dashiming/panscrapling-web-scraper (version 1.0.0)

--- name: panscrapling-web-scraper version: 1.0.0 description: | 强大的网页抓取技能。基于 Scrapling，自动绕过 Cloudflare/反爬系统。 触发词：抓取网页、爬取、scrape、fetch、抓取内容、提取网页、获取页面。 使用场景： (1) 抓取被 Cloudflare 保护的网页 (2) 提取页面内容 (3) 网页数据采集 (4) 动态渲染页面抓取 自动安装：首次使用时自动检测并安装 Python 3.10+、Scrapling 及浏览器依赖。 嵌入分发：包含所有依赖的 wheel 包，支持离线安装。 --- # Panscrapling Web Scraper 基于 [Scrapling](https://github.com/D4Vinci/Scrapling) 的网页抓取技能。 **自动绕过 Cloudflare Turnstile**，无需手动处理验证码。 ## 特点 ✅ **完全嵌入分发** - 包含所有 Python 依赖，支持离线安装 ✅ **自动安装 Python** - 自动检测并安装 Python 3.10+ ✅ **自动绕过 Cloudflare** - 无需手动处理验证码 ✅ **多种抓取模式** - Fast / Stealthy / Dynamic ## 使用方式 直接说： - "抓取 https://example.com 的内容" - "用 Scrapling 抓取这个页面" - "获取 https://xxx.com 的 .product 元素" ## 抓取模式 | 模式 | 用途 | 特点 | |------|------|------| | `auto` | 自动选择 | 默认模式，适合大多数情况 | | `stealthy` | 隐身抓取 | 绕过 Cloudflare，解验证码 | | `dynamic` | 动态渲染 | 完整浏览器，JS 执行 | | `fast` | 快速 HTTP | 轻量级，适合简单页面 | ## CLI 使用 ```bash # 基础抓取 python3 scripts/fetch.py "https://example.com" # 绕过 Cloudflare python3 scripts/fetch.py "https://protected-site.com" --mode stealthy # 提取特定元素 python3 scripts/fetch.py "https://shop.com" --selector ".product-title" # 输出 Markdown python3 scripts/fetch.py "https://blog.com/article" --markdown # 提取链接、图片、元数据 python3 scripts/fetch.py "https://example.com" --links --images --meta # 仅运行安装 python3 scripts/fetch.py --setup ``` ## 自动安装流程 首次使用时自动执行： 1. **检测 Python 3.10+** - 查找系统已安装的 Python 2. **安装 Python** - 如果未找到，通过 Homebrew 自动安装 3. **安装依赖** - 从嵌入的 wheel 包安装 Scrapling 4. **安装浏览器** - 下载 Playwright/Patchright Chromium ## 与 pansxng-websearch 配合 - **pansxng-websearch**: 搜索引擎查询 → 获取链接列表 - **panscrapling-web-scraper**: 深入抓取页面 → 提取具体内容 典型工作流： 1. 用 pansxng-websearch 搜索关键词 2. 用 panscrapling-web-scraper 抓取目标页面 ## 文件结构 ``` panscrapling-web-scraper/ ├── SKILL.md # 本文档 ├── scripts/ │ ├── setup.py # 自动安装脚本 │ └── fetch.py # 主抓取脚本 ├── wheels/ # 嵌入的 Python 依赖包 │ ├── scrapling-*.whl │ ├── playwright-*.whl │ ├── patchright-*.whl │ └── ... (其他依赖) └── browsers/ # 浏览器（首次运行时下载） ``` ## 隐私说明 - 所有请求通过本地浏览器发出 - 无第三方 API 调用 - 不泄露搜索/抓取记录 ## 依赖 - Python 3.10+（自动安装） - Homebrew（macOS，自动安装） - Playwright Chromium（自动下载）

path: scripts/fetch.py
content: #!/usr/bin/env python3
"""
Panscrapling Web Scraper - 主抓取脚本
自动绕过 Cloudflare，支持动态渲染
"""
import sys
import json
import argparse
import subprocess
import os
import re
from pathlib import Path
from urllib.parse import urlparse

# ═══════════════════════════════════════════════════════════════
#  配置
# ═══════════════════════════════════════════════════════════════

SKILL_DIR = Path(__file__).parent.parent
WHEELS_DIR = SKILL_DIR / "wheels"
SETUP_SCRIPT = SKILL_DIR / "scripts" / "setup.py"

SCRAPLING_INSTALLED = False
PYTHON_PATH = None

# ═══════════════════════════════════════════════════════════════
#  Python 检测
# ═══════════════════════════════════════════════════════════════

PYTHON_PATHS = [
    "/opt/homebrew/bin/python3.11",
    "/opt/homebrew/bin/python3.12",
    "/opt/homebrew/bin/python3.13",
    "/usr/local/bin/python3.11",
    "/usr/local/bin/python3.12",
    "/usr/local/bin/python3.13",
]


def find_python():
    """查找 Python 3.10+"""
    global PYTHON_PATH
    
    if PYTHON_PATH:
        return PYTHON_PATH
    
    # 检查预定义路径
    for py in PYTHON_PATHS:
        if os.path.isfile(py):
            try:
                r = subprocess.run([py, "--version"], capture_output=True, text=True, timeout=5)
                m = re.search(r"Python (\d+)\.(\d+)", r.stdout)
                if m and int(m.group(2)) >= 10:
                    PYTHON_PATH = py
                    return py
            except:
                pass
    
    # 检查当前 Python
    v = sys.version_info
    if v.major == 3 and v.minor >= 10:
        PYTHON_PATH = sys.executable
        return sys.executable
    
    return None


# ═══════════════════════════════════════════════════════════════
#  自动安装
# ═══════════════════════════════════════════════════════════════

def run_setup():
    """运行自动安装脚本"""
    python_path = find_python()
    
    if not python_path:
        print("📦 Python 3.10+ not found, running setup...")
    else:
        # 检查 scrapling 是否已安装
        try:
            r = subprocess.run(
                [python_path, "-c", "import scrapling"],
                capture_output=True, timeout=5
            )
            if r.returncode == 0:
                global SCRAPLING_INSTALLED
                SCRAPLING_INSTALLED = True
                return python_path
        except:
            pass
    
    # 运行 setup.py
    if SETUP_SCRIPT.exists():
        print("📦 Running setup script...")
        subprocess.run([sys.executable, str(SETUP_SCRIPT)], check=False)
        
        # 重新检测
        python_path = find_python()
        if python_path:
            SCRAPLING_INSTALLED = True
        return python_path
    
    return None


# ═══════════════════════════════════════════════════════════════
#  Fetcher
# ═══════════════════════════════════════════════════════════════

def get_fetcher(mode="auto", headless=True, solve_cloudflare=True):
    """获取 Fetcher"""
    from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
    
    if mode == "fast":
        return lambda url: Fetcher.get(url, stealthy_headers=True)
    elif mode == "stealthy":
        return lambda url: StealthyFetcher.fetch(url, headless=headless, solve_cloudflare=solve_cloudflare)
    elif mode == "dynamic":
        return lambda url: DynamicFetcher.fetch(url, headless=headless, network_idle=True)
    else:  # auto
        return lambda url: StealthyFetcher.fetch(url, headless=headless, solve_cloudflare=solve_cloudflare)


# ═══════════════════════════════════════════════════════════════
#  内容提取
# ═══════════════════════════════════════════════════════════════

def extract_content(page, selector=None, xpath=None, output="text"):
    """从页面提取内容"""
    result = {"url": getattr(page, 'url', None), "status": "success"}
    
    if selector:
        elements = page.css(selector)
        if elements:
            result["data"] = [e.css("::text").get() for e in elements] if output != "html" else [e.get() for e in elements]
        else:
            result["data"] = []
            result["warning"] = f"No elements found for selector: {selector}"
    elif xpath:
        elements = page.xpath(xpath)
        if elements:
            result["data"] = [e.css("::text").get() for e in elements] if output != "html" else [e.get() for e in elements]
        else:
            result["data"] = []
            result["warning"] = f"No elements found for xpath: {xpath}"
    else:
        if output == "text":
            texts = page.css("body ::text").getall()
            result["data"] = " ".join(texts).strip()
        elif output == "markdown":
            result["data"] = to_markdown(page)
        else:
            result["data"] = page.css("body").get()
    
    return result


def extract_links(page):
    """提取链接"""
    return [{"text": a.css("::text").get(default="").strip(), "href": a.attrib.get("href", "")} 
            for a in page.css("a")]


def extract_images(page):
    """提取图片"""
    return [{"src": img.attrib.get("src", ""), "alt": img.attrib.get("alt", "")} 
            for img in page.css("img")]


def extract_meta(page):
    """提取元数据"""
    meta = {"title": page.css("title::text").get(default="")}
    for m in page.css("meta"):
        name = m.attrib.get("name", m.attrib.get("property", ""))
        if name in ["description", "keywords", "og:title", "og:description", "og:image"]:
            meta[name] = m.attrib.get("content", "")
    return meta


def to_markdown(page):
    """转 Markdown"""
    lines = []
    for i in range(1, 7):
        for h in page.css(f"h{i}"):
            text = h.css("::text").get(default="").strip()
            if text:
                lines.append("#" * i + " " + text)
    for p in page.css("p"):
        text = p.css("::text").get(default="").strip()
        if text:
            lines.append(text + "\n")
    return "\n".join(lines)


# ═══════════════════════════════════════════════════════════════
#  主函数
# ═══════════════════════════════════════════════════════════════

def fetch(url, mode="auto", selector=None, xpath=None, output="text", 
          headless=True, solve_cloudflare=True):
    """抓取网页"""
    global SCRAPLING_INSTALLED
    
    if not SCRAPLING_INSTALLED:
        python_path = run_setup()
        if not python_path:
            return {"error": "Failed to setup Python/Scrapling", "url": url}
    
    try:
        fetcher = get_fetcher(mode, headless, solve_cloudflare)
        page = fetcher(url)
        return extract_content(page, selector, xpath, output)
    except Exception as e:
        return {"error": str(e), "url": url, "hint": "Try --mode stealthy or --mode dynamic"}


def main():
    parser = argparse.ArgumentParser(description="Panscrapling Web Scraper")
    parser.add_argument("url", nargs="?", help="URL to scrape")
    parser.add_argument("--mode", "-m", default="auto", 
                       choices=["auto", "stealthy", "dynamic", "fast"])
    parser.add_argument("--selector", "-s", help="CSS selector")
    parser.add_argument("--xpath", "-x", help="XPath")
    parser.add_argument("--output", "-o", default="text", 
                       choices=["text", "html", "json", "markdown"])
    parser.add_argument("--text", action="store_true")
    parser.add_argument("--links", action="store_true")
    parser.add_argument("--images", action="store_true")
    parser.add_argument("--meta", action="store_true")
    parser.add_argument("--markdown", action="store_true")
    parser.add_argument("--setup", action="store_true", help="Run setup only")
    
    args = parser.parse_args()
    
    if args.setup:
        run_setup()
        return
    
    if not args.url:
        parser.print_help()
        return
    
    output = "markdown" if args.markdown else args.output
    result = fetch(args.url, args.mode, args.selector, args.xpath, output)
    
    # 额外提取
    if "error" not in result:
        if args.text or args.links or args.images or args.meta:
            try:
                page = get_fetcher(args.mode)(args.url)
                if args.text:
                    result["text"] = " ".join(page.css("body ::text").getall()).strip()
                if args.links:
                    result["links"] = extract_links(page)
                if args.images:
                    result["images"] = extract_images(page)
                if args.meta:
                    result["meta"] = extract_meta(page)
            except:
                pass
    
    print(json.dumps(result, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()


path: scripts/setup.py
content: #!/usr/bin/env python3
"""
Panscrapling Web Scraper - 自动安装脚本
自动检测并安装 Python 3.10+、Scrapling 及浏览器依赖
"""
import sys
import os
import subprocess
import platform
import shutil
from pathlib import Path

# ═══════════════════════════════════════════════════════════════
#  配置
# ═══════════════════════════════════════════════════════════════

SKILL_DIR = Path(__file__).parent.parent
WHEELS_DIR = SKILL_DIR / "wheels"
BROWSERS_DIR = SKILL_DIR / "browsers"

# Homebrew 安装路径
BREW_PATHS = [
    "/opt/homebrew/bin/brew",  # Apple Silicon
    "/usr/local/bin/brew",     # Intel Mac
]

# Python 路径优先级
PYTHON_PATHS = [
    "/opt/homebrew/bin/python3.11",
    "/opt/homebrew/bin/python3.12",
    "/opt/homebrew/bin/python3.13",
    "/usr/local/bin/python3.11",
    "/usr/local/bin/python3.12",
    "/usr/local/bin/python3.13",
]

# ═══════════════════════════════════════════════════════════════
#  工具函数
# ═══════════════════════════════════════════════════════════════

def run(cmd, timeout=300, check=True, capture=True):
    """运行命令"""
    print(f"🔧 Running: {' '.join(cmd) if isinstance(cmd, list) else cmd}")
    try:
        result = subprocess.run(
            cmd,
            capture_output=capture,
            text=True,
            timeout=timeout,
            check=False
        )
        if check and result.returncode != 0:
            print(f"❌ Error: {result.stderr}")
            return None
        return result
    except subprocess.TimeoutExpired:
        print(f"❌ Timeout after {timeout}s")
        return None
    except Exception as e:
        print(f"❌ Exception: {e}")
        return None


def check_python_version(python_path):
    """检查 Python 版本是否 >= 3.10"""
    if not os.path.isfile(python_path):
        return None
    try:
        r = subprocess.run(
            [python_path, "--version"],
            capture_output=True, text=True, timeout=5
        )
        if r.returncode == 0:
            import re
            m = re.search(r"Python (\d+)\.(\d+)", r.stdout)
            if m:
                major, minor = int(m.group(1)), int(m.group(2))
                if major == 3 and minor >= 10:
                    return (major, minor, python_path)
    except:
        pass
    return None


# ═══════════════════════════════════════════════════════════════
#  步骤 1: 检测/安装 Python 3.10+
# ═══════════════════════════════════════════════════════════════

def find_python():
    """查找 Python 3.10+"""
    # 1. 检查预定义路径
    for py in PYTHON_PATHS:
        result = check_python_version(py)
        if result:
            print(f"✅ Found Python {result[0]}.{result[1]} at {result[2]}")
            return result[2]
    
    # 2. 检查系统 PATH 中的 python3
    for py in ["python3", "python3.11", "python3.12", "python3.13"]:
        py_path = shutil.which(py)
        if py_path:
            result = check_python_version(py_path)
            if result:
                print(f"✅ Found Python {result[0]}.{result[1]} at {result[2]}")
                return result[2]
    
    return None


def find_homebrew():
    """查找 Homebrew"""
    for brew in BREW_PATHS:
        if os.path.isfile(brew):
            print(f"✅ Found Homebrew at {brew}")
            return brew
    return None


def install_python_via_homebrew():
    """通过 Homebrew 安装 Python 3.11"""
    print("\n" + "="*60)
    print("📦 Installing Python 3.11 via Homebrew...")
    print("="*60 + "\n")
    
    # 检查 Homebrew
    brew = find_homebrew()
    if not brew:
        print("❌ Homebrew not found. Installing Homebrew first...")
        print("⏳ This may take a few minutes...\n")
        
        # 安装 Homebrew
        install_cmd = '/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"'
        r = run(["/bin/bash", "-c", 
                 'NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"'],
                timeout=600, check=False)
        
        if not r or r.returncode != 0:
            print("❌ Failed to install Homebrew")
            return None
        
        brew = find_homebrew()
        if not brew:
            print("❌ Homebrew still not found after installation")
            return None
    
    # 安装 Python 3.11
    print(f"🔧 Installing python@3.11...")
    r = run([brew, "install", "python@3.11"], timeout=600, check=False)
    
    if not r:
        print("❌ Failed to install Python 3.11")
        return None
    
    # 验证安装
    py_path = "/opt/homebrew/bin/python3.11" if "/opt/homebrew" in brew else "/usr/local/bin/python3.11"
    result = check_python_version(py_path)
    if result:
        print(f"✅ Python {result[0]}.{result[1]} installed at {result[2]}")
        return result[2]
    
    print("❌ Python installed but version check failed")
    return None


def ensure_python():
    """确保有 Python 3.10+"""
    py = find_python()
    if py:
        return py
    
    print("\n⚠️  Python 3.10+ not found!")
    print("🔄 Attempting to install Python 3.11 via Homebrew...\n")
    
    return install_python_via_homebrew()


# ═══════════════════════════════════════════════════════════════
#  步骤 2: 安装 Scrapling 及依赖
# ═══════════════════════════════════════════════════════════════

def install_wheels(python_path):
    """从本地 wheels 目录安装依赖"""
    print("\n" + "="*60)
    print("📦 Installing Scrapling and dependencies...")
    print("="*60 + "\n")
    
    if not WHEELS_DIR.exists():
        print(f"❌ Wheels directory not found: {WHEELS_DIR}")
        return False
    
    wheels = list(WHEELS_DIR.glob("*.whl"))
    if not wheels:
        print(f"❌ No wheel files found in {WHEELS_DIR}")
        return False
    
    print(f"📦 Found {len(wheels)} wheel files")
    
    # 安装所有 wheels
    r = run(
        [python_path, "-m", "pip", "install", "--force-reinstall", 
         "--no-index", "--find-links", str(WHEELS_DIR),
         "scrapling[fetchers]", "playwright", "patchright"],
        timeout=300
    )
    
    if not r:
        print("❌ Failed to install from wheels, trying online...")
        # 降级到在线安装
        r = run(
            [python_path, "-m", "pip", "install", 
             "-i", "https://pypi.tuna.tsinghua.edu.cn/simple",
             "scrapling[fetchers]", "playwright", "patchright"],
            timeout=300
        )
        if not r:
            return False
    
    # 验证安装
    r = run([python_path, "-c", "import scrapling; print(scrapling.__version__)"], 
            check=False, capture=False)
    if r and r.returncode == 0:
        print("✅ Scrapling installed successfully")
        return True
    
    print("❌ Scrapling installation verification failed")
    return False


# ═══════════════════════════════════════════════════════════════
#  步骤 3: 安装浏览器
# ═══════════════════════════════════════════════════════════════

def install_browsers(python_path):
    """安装 Playwright 和 Patchright 浏览器"""
    print("\n" + "="*60)
    print("🌐 Installing browsers (this may take a few minutes)...")
    print("="*60 + "\n")
    
    # 检查是否已有浏览器
    browser_cache = Path.home() / "Library" / "Caches" / "ms-playwright"
    if browser_cache.exists():
        chromium_dirs = list(browser_cache.glob("chromium*"))
        if chromium_dirs:
            print(f"✅ Found existing Playwright browser at {browser_cache}")
            return True
    
    # 安装 Playwright Chromium
    print("📦 Installing Playwright Chromium...")
    r = run([python_path, "-m", "playwright", "install", "chromium"], 
            timeout=600, capture=False)
    
    if not r:
        print("⚠️  Playwright browser install failed, trying patchright...")
    
    # 安装 Patchright Chromium
    print("📦 Installing Patchright Chromium...")
    r = run([python_path, "-m", "patchright", "install", "chromium"], 
            timeout=600, capture=False)
    
    if not r:
        print("⚠️  Patchright browser install warning (may still work)")
    
    print("✅ Browser installation complete")
    return True


# ═══════════════════════════════════════════════════════════════
#  步骤 4: 验证
# ═══════════════════════════════════════════════════════════════

def verify_installation(python_path):
    """验证完整安装"""
    print("\n" + "="*60)
    print("✅ Verifying installation...")
    print("="*60 + "\n")
    
    checks = [
        ("Python version", [python_path, "--version"]),
        ("Scrapling", [python_path, "-c", "import scrapling; print(f'Scrapling {scrapling.__version__}')"]),
        ("Playwright", [python_path, "-c", "from playwright.sync_api import sync_playwright; print('Playwright OK')"]),
        ("Patchright", [python_path, "-c", "from patchright.sync_api import sync_playwright; print('Patchright OK')"]),
    ]
    
    all_ok = True
    for name, cmd in checks:
        r = run(cmd, check=False, capture=False)
        if r and r.returncode == 0:
            print(f"  ✅ {name}")
        else:
            print(f"  ❌ {name}")
            all_ok = False
    
    return all_ok


# ═══════════════════════════════════════════════════════════════
#  主入口
# ═══════════════════════════════════════════════════════════════

def main():
    print("""
╔═══════════════════════════════════════════════════════════════╗
║     Panscrapling Web Scraper - 自动安装脚本                   ║
║                                                               ║
║  此脚本将自动安装：                                           ║
║  1. Python 3.10+ (通过 Homebrew)                              ║
║  2. Scrapling 及所有依赖                                      ║
║  3. Playwright/Patchright 浏览器                              ║
╚═══════════════════════════════════════════════════════════════╝
""")
    
    # 步骤 1: Python
    python_path = ensure_python()
    if not python_path:
        print("\n❌ Failed to install Python 3.10+")
        print("💡 Please install manually: brew install python@3.11")
        return 1
    
    # 步骤 2: Scrapling
    if not install_wheels(python_path):
        print("\n❌ Failed to install Scrapling")
        return 1
    
    # 步骤 3: Browsers
    install_browsers(python_path)
    
    # 步骤 4: 验证
    if verify_installation(python_path):
        print("\n" + "="*60)
        print("🎉 Installation complete!")
        print("="*60)
        print(f"\n📁 Python: {python_path}")
        print(f"📁 Wheels: {WHEELS_DIR}")
        print("\n🚀 You can now use the scraper!")
        return 0
    else:
        print("\n⚠️  Installation completed with warnings")
        return 1


if __name__ == "__main__":
    sys.exit(main())


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)