Skill: yuzhihui886/novel-scraper-spa (version 1.0.0)

--- name: novel-scraper-spa description: 支持 JavaScript 加载的小说抓取工具，使用 Playwright 渲染 SPA 网站。自动检测网站类型，静态网站使用 curl，SPA 网站使用浏览器渲染。 --- # Novel Scraper SPA ## 功能 支持抓取 JavaScript 加载的小说网站（SPA 架构），使用 Playwright 浏览器渲染。 ## 使用场景 - 抓取笔趣阁等 SPA 架构小说网站 - 需要 JavaScript 渲染才能获取内容的网站 - 普通静态网站（自动使用 curl，更快速） ## 命令 ```bash python3 scripts/scraper.py --url <URL> --book <书名> --chapter <章节号> ``` ## 参数 - `--url`: 目标 URL - `--book`: 书名（用于输出文件名） - `--chapter`: 章节号 - `--force-spa`: 强制使用浏览器渲染 - `-v`: 详细日志 ## 输出 保存到 `~/.openclaw/workspace/novels/` 目录 **2. Task-Based** (best for tool collections) - Works well when the skill offers different operations/capabilities - Example: PDF skill with "Quick Start" → "Merge PDFs" → "Split PDFs" → "Extract Text" - Structure: ## Overview → ## Quick Start → ## Task Category 1 → ## Task Category 2... **3. Reference/Guidelines** (best for standards or specifications) - Works well for brand guidelines, coding standards, or requirements - Example: Brand styling with "Brand Guidelines" → "Colors" → "Typography" → "Features" - Structure: ## Overview → ## Guidelines → ## Specifications → ## Usage... **4. Capabilities-Based** (best for integrated systems) - Works well when the skill provides multiple interrelated features - Example: Product Management with "Core Capabilities" → numbered capability list - Structure: ## Overview → ## Core Capabilities → ### 1. Feature → ### 2. Feature... Patterns can be mixed and matched as needed. Most skills combine patterns (e.g., start with task-based, add workflow for complex operations). Delete this entire "Structuring This Skill" section when done - it's just guidance.] ## [TODO: Replace with the first main section based on chosen structure] [TODO: Add content here. See examples in existing skills: - Code samples for technical skills - Decision trees for complex workflows - Concrete examples with realistic user requests - References to scripts/templates/references as needed] ## Resources This skill includes example resource directories that demonstrate how to organize different types of bundled resources: ### scripts/ Executable code (Python/Bash/etc.) that can be run directly to perform specific operations. **Examples from other skills:** - PDF skill: `fill_fillable_fields.py`, `extract_form_field_info.py` - utilities for PDF manipulation - DOCX skill: `document.py`, `utilities.py` - Python modules for document processing **Appropriate for:** Python scripts, shell scripts, or any executable code that performs automation, data processing, or specific operations. **Note:** Scripts may be executed without loading into context, but can still be read by Claude for patching or environment adjustments. ### references/ Documentation and reference material intended to be loaded into context to inform Claude's process and thinking. **Examples from other skills:** - Product management: `communication.md`, `context_building.md` - detailed workflow guides - BigQuery: API reference documentation and query examples - Finance: Schema documentation, company policies **Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Claude should reference while working. ### assets/ Files not intended to be loaded into context, but rather used within the output Claude produces. **Examples from other skills:** - Brand styling: PowerPoint template files (.pptx), logo files - Frontend builder: HTML/React boilerplate project directories - Typography: Font files (.ttf, .woff2) **Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output. --- **Any unneeded directories can be deleted.** Not every skill requires all three types of resources.

path: scripts/browser.py
content: #!/usr/bin/env python3.11
"""
Playwright browser module for novel-scraper-spa

Browser instance singleton with content loading support.
"""

import logging
from typing import Optional
from playwright.sync_api import (
    sync_playwright,
    Page,
    TimeoutError as PlaywrightTimeout,
    Playwright,
)

logger = logging.getLogger(__name__)


class BrowserManager:
    """Singleton browser manager using Playwright."""

    _instance: Optional["BrowserManager"] = None
    _playwright: Optional[Playwright] = None
    _browser: Optional[object] = None
    _context: Optional[object] = None
    _page: Optional[Page] = None

    def __new__(cls) -> "BrowserManager":
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __init__(self):
        if self._browser is None:
            pw = sync_playwright().start()
            assert pw is not None
            self._playwright = pw
            self._browser = pw.chromium.launch(headless=True)
            assert self._browser is not None
            self._context = self._browser.new_context(
                viewport={"width": 1920, "height": 1080}
            )
            self._page = self._context.new_page()
            logger.info("Browser instance created (singleton)")

    def get_page(self) -> Page:
        """Get the current page instance."""
        assert self._page is not None
        return self._page

    def goto(self, url: str, timeout: int = 30000) -> bool:
        """Navigate to URL and wait for content."""
        assert self._page is not None
        assert self._browser is not None
        try:
            logger.info(f"Navigating to: {url}")
            self._page.goto(url, timeout=timeout, wait_until="networkidle")
            return True
        except PlaywrightTimeout:
            logger.error(f"Timeout loading: {url}")
            return False

    def wait_for_selector(self, selector: str, timeout: int = 10000) -> bool:
        """Wait for element to appear."""
        assert self._page is not None
        try:
            self._page.wait_for_selector(selector, timeout=timeout)
            return True
        except PlaywrightTimeout:
            logger.warning(f"Element not found: {selector}")
            return False

    def get_content(self) -> str:
        """Get page HTML content."""
        assert self._page is not None
        return self._page.content()

    def close(self):
        """Close browser and cleanup."""
        if self._browser and self._playwright:
            self._browser.close()
            self._playwright.stop()
            logger.info("Browser instance closed")


def get_html(url: str, wait_selector: Optional[str] = None) -> Optional[str]:
    """
    Get HTML content from URL using Playwright.

    Args:
        url: Target URL
        wait_selector: Optional selector to wait for before extracting content

    Returns:
        HTML content or None if failed
    """
    browser = BrowserManager()

    if not browser.goto(url):
        return None

    if wait_selector:
        if not browser.wait_for_selector(wait_selector):
            logger.warning(f"Content selector not found: {wait_selector}")

    content = browser.get_content()
    logger.debug(f"Retrieved {len(content)} bytes from {url}")

    return content


def cleanup():
    """Cleanup browser resources."""
    browser = BrowserManager()
    browser.close()


path: scripts/scraper.py
content: #!/usr/bin/env python3.11
"""
Main scraper script for novel-scraper-spa

CLI tool for scraping novels from websites, supporting both static and SPA sites.
"""

import argparse
import logging
import re
import sys
from datetime import datetime
from pathlib import Path
from typing import Optional

import requests
from bs4 import BeautifulSoup

from browser import get_html, cleanup

logging.basicConfig(
    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)

NOVELS_DIR = Path.home() / ".openclaw" / "workspace" / "novels"


def detect_spa(url: str, html: Optional[str] = None) -> bool:
    """
    Detect if a website is SPA by checking for common patterns.

    Args:
        url: Target URL
        html: Optional HTML content to analyze

    Returns:
        True if likely SPA, False otherwise
    """
    spa_indicators = [
        r'<div[^>]*id=["\']app["\']',
        r'<div[^>]*id=["\']root["\']',
        r"data-reactroot",
        r"<script[^>]*window\.__REACT__",
        r"<script[^>]*window\.__STATE__",
        r"window\.__INITIAL_STATE__",
        r"v-app=",
        r"data-v-",
    ]

    if html is None:
        try:
            response = requests.get(url, timeout=10)
            html = response.text
        except requests.RequestException as e:
            logger.error(f"Failed to fetch {url}: {e}")
            return False

    for pattern in spa_indicators:
        if re.search(pattern, html):
            logger.info(f"SPA indicator found: {pattern}")
            return True

    return False


def extract_content(html: str, url: str) -> Optional[str]:
    """
    Extract novel content from HTML using BeautifulSoup.

    Args:
        html: HTML content
        url: Source URL for context

    Returns:
        Extracted text content or None
    """
    soup = BeautifulSoup(html, "html.parser")

    article = soup.find("article") or soup.find("div", class_="content")

    if article:
        for selector in ["script", "style", "nav", "footer", "header", "aside"]:
            for tag in article.find_all(selector):
                tag.decompose()

        content = article.get_text("\n", strip=True)

        title = soup.find("title")
        if title:
            content = f"# {title.get_text(strip=True)}\n\n{content}"

        return content

    return None


def save_content(content: str, url: str, output_path: Optional[str] = None) -> str:
    """
    Save content to file.

    Args:
        content: Text content to save
        url: Source URL
        output_path: Optional custom output path

    Returns:
        Path to saved file
    """
    if output_path:
        output_file = Path(output_path)
    else:
        output_file = (
            NOVELS_DIR / f"scraped_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"
        )

    output_file.parent.mkdir(parents=True, exist_ok=True)

    with open(output_file, "w", encoding="utf-8") as f:
        f.write(content)

    logger.info(f"Content saved to: {output_file}")
    return str(output_file)


def scrape_url(
    url: str, output_path: Optional[str] = None, force_spa: bool = False
) -> Optional[str]:
    """
    Scrape content from URL, auto-detecting SPA vs static.

    Args:
        url: Target URL
        output_path: Optional custom output path
        force_spa: Force SPA mode (use browser)

    Returns:
        Path to saved file or None if failed
    """
    logger.info(f"Processing: {url}")

    html = None
    use_browser = force_spa

    if not use_browser:
        use_browser = detect_spa(url)

    if use_browser:
        logger.info("Using browser (SPA detected)")
        html = get_html(url)
    else:
        logger.info("Using requests (static site)")
        try:
            response = requests.get(url, timeout=30)
            response.raise_for_status()
            html = response.text
        except requests.RequestException as e:
            logger.error(f"Failed to fetch {url}: {e}")
            if detect_spa(url):
                logger.info("Retrying with browser...")
                html = get_html(url)

    if not html:
        logger.error("Failed to retrieve content")
        return None

    content = extract_content(html, url)

    if not content:
        logger.error("Failed to extract content")
        return None

    return save_content(content, url, output_path)


def main():
    """CLI entry point."""
    parser = argparse.ArgumentParser(
        description="Scrape novel content from websites (static or SPA)"
    )
    parser.add_argument("--url", help="Target URL to scrape")
    parser.add_argument("--book", help="Book name for output filename")
    parser.add_argument("--chapter", help="Chapter number/title")
    parser.add_argument(
        "-o", "--output", help="Output file path (default: ~/novels/scraped_*.txt)"
    )
    parser.add_argument(
        "--force-spa", action="store_true", help="Force using browser (SPA mode)"
    )
    parser.add_argument(
        "-v", "--verbose", action="store_true", help="Enable debug logging"
    )

    args = parser.parse_args()

    if not args.url:
        parser.error("--url is required")

    if args.verbose:
        logging.getLogger().setLevel(logging.DEBUG)

    try:
        # Build output path from book and chapter if provided
        output_path = args.output
        if not output_path and args.book:
            chapter_str = args.chapter or "unknown"
            safe_book = re.sub(r'[^\w\u4e00-\u9fff-]', '_', args.book)
            safe_chapter = re.sub(r'[^\w\u4e00-\u9fff-]', '_', chapter_str)
            output_path = str(NOVELS_DIR / f"{safe_book}_{safe_chapter}.txt")

        result = scrape_url(args.url, output_path, args.force_spa)
        if result:
            print(f"Success: {result}")
            sys.exit(0)
        else:
            print("Failed to scrape content")
            sys.exit(1)
    except KeyboardInterrupt:
        print("\nInterrupted")
        sys.exit(130)
    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        sys.exit(1)
    finally:
        cleanup()


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)