Skill: ichendong/dcard (version 0.1.1)

--- name: dcard description: 擷取 Dcard 文章完整內文。Dcard 有 Cloudflare 保護，需用 patchright (Playwright) headless browser 繞過。支援文章標題、全文、圖片連結輸出。 metadata: openclaw: emoji: 🎴 requires: bins: - python3.14 - camoufox --- # Dcard 🎴 用 Camoufox（Firefox-based stealth browser）+ patchright 繞過 Dcard 的 Cloudflare 保護。 Camoufox 內建 uBlock Origin + 反指紋偵測，可以直接 bypass Cloudflare challenge， 且 Firefox 不會有 WSL2 + Tailscale DNS 的問題。 ## 使用方式 ```bash # 單篇文章 dcard_fetch.py https://www.dcard.tw/f/relationship/p/261529038 # 純 ID（預設看板 relationship） dcard_fetch.py --id 261529038 # 指定看板 dcard_fetch.py --forum makeup --id 123456 # JSON 輸出 dcard_fetch.py https://www.dcard.tw/f/relationship/p/261529038 --format json # 看板熱門 dcard_fetch.py list relationship # 看板最新 dcard_fetch.py list relationship --sort latest --limit 10 # 全站熱門看板 dcard_fetch.py list --all-forums ``` ## 腳本 - `scripts/dcard_fetch.py` — 主腳本（Python3.14） - `scripts/_list_posts.js` — 看板列表用 JS extractor ## 安裝依賴 需要 **Camoufox** + **patchright** + **lxml**。 ```bash # 1. 安裝 Camoufox + patchright + lxml pip3.14 install camoufox lxml patchright # 2. 下載 Camoufox 瀏覽器（~280MB） camoufox fetch # 檢查 camoufox version # → Camoufox: v135.0.1-beta.xx (Up to date!) ``` ## 技術細節 **為什麼用 Camoufox 而不是 Chromium？** - Firefox 在 WSL2 下 DNS 正常（Tailscale DNS 不會炸） - 內建 uBlock Origin 擋廣告與 tracker - 自訂指紋不會被 Cloudflare 標記為 headless - 不需要 `--host-resolver-rules` 繞 DNS **JSON-LD 解析：** 內文從 `<script type="application/ld+json">` 的 SocialMediaPosting schema 提取， 圖片抓 `megapx-assets` domain 的 orig/1280 圖床連結。 ## 限制 - 短時間大量 request 仍可能觸發 Cloudflare rate-limit - 首次使用需下載 Camoufox（`camoufox fetch`） - 不支援搜尋功能（v0.1.0）

path: scripts/_list_posts.js
content: // Dcard forum post list extractor
// Used by dcard_fetch.py
(forumSlug) => {
    const items = [];
    const seen = new Set();
    const links = document.querySelectorAll('a[href*="/f/"][href*="/p/"]');
    links.forEach(a => {
        const href = a.href;
        if (seen.has(href)) return;
        seen.add(href);

        const m = href.match(/\/p\/(\d+)/);
        if (!m) return;
        const postId = m[1];

        const card = a.closest('[class*="PostList"], article, [class*="post_"], [class*="PostEntry"]');
        const fullText = card ? card.textContent : (a.textContent || '');

        // Extract title: longest meaningful line
        const allText = a.textContent.trim();
        const lines = allText.split('\n').map(s => s.trim()).filter(Boolean);
        let title = '';
        for (const line of lines) {
            if (/^\d+\s*(小時|分鐘|天|週|月|[hdwm])$/i.test(line.trim())) continue;
            if (/^\d+$/.test(line.trim())) continue;
            if (line.length < 4) continue;
            title = line;
            break;
        }
        if (!title) title = lines.find(l => l.length > 4) || '';
        if (!title || title.length < 3) return;

        // Extract time
        const timeMatch = fullText.match(/(\d+[hdwm])\s*/);
        const timeAgo = timeMatch ? timeMatch[1] : '';

        // Preview: text after title
        let preview = '';
        const titleIdx = fullText.indexOf(title);
        if (titleIdx >= 0) {
            preview = fullText.substring(titleIdx + title.length, titleIdx + title.length + 100)
                .replace(/\s+/g, ' ').trim();
        }

        items.push({
            title: title,
            post_id: postId,
            forum: forumSlug,
            time_ago: timeAgo,
            preview: preview.substring(0, 120),
        });
    });
    return items;
}


path: scripts/dcard_fetch.py
content: #!/home/linuxbrew/.linuxbrew/opt/python@3.14/bin/python3.14
"""
Dcard 文章工具 — 單篇抓取 + 看板列表

Usage:
  dcard_fetch.py <post-url-or-id>
  dcard_fetch.py --forum relationship --id 261529038
  dcard_fetch.py list relationship
  dcard_fetch.py list relationship --sort popular --limit 10
  dcard_fetch.py list --all-forums
"""

import argparse
import json
import re
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import List

_CAMOUFOX = "/home/ichen/.cache/camoufox/camoufox"
_JS_DIR = Path(__file__).parent


# ── browser helpers ─────────────────────────────


def _browser():
    from patchright.sync_api import sync_playwright

    ps = sync_playwright().start()
    browser = ps.firefox.launch(headless=True, executable_path=_CAMOUFOX, args=["--no-sandbox"])
    context = browser.new_context(
        viewport={"width": 1280, "height": 720},
        locale="zh-TW",
        timezone_id="Asia/Taipei",
    )
    page = context.new_page()
    return ps, browser, page


# ── fetch single post ────────────────────────────


def fetch_post(forum: str | None, post_id: str) -> dict:
    url = f"https://www.dcard.tw/f/{forum or 'relationship'}/p/{post_id}"
    ps, browser, page = _browser()

    try:
        page.goto(url, wait_until="domcontentloaded", timeout=60000)
        for _ in range(4):
            page.wait_for_timeout(5000)
            title = page.title()
            if title and title != "Dcard" and "Dcard" in title:
                break

        html = page.content()
        post_data = _extract_post(html)
        if not post_data:
            post_data = {"title": page.title(), "text": page.inner_text("body"), "images": []}
            if post_data["title"] == "Dcard":
                post_data["text"] = "（無法載入文章，可能不存在或需登入）"
        post_data["url"] = url
        post_data["post_id"] = post_id
        return post_data
    finally:
        browser.close()
        ps.stop()


def _extract_post(html: str) -> dict | None:
    import lxml.html

    try:
        root = lxml.html.fromstring(html)
        for script in root.cssselect('script[type="application/ld+json"]'):
            text = script.text_content().strip()
            if '"SocialMediaPosting"' in text:
                data = json.loads(text)
                images = [
                    img.get("src")
                    for img in root.cssselect("img[src*='megapx-assets']")
                    if "orig" in (img.get("src") or "") or "1280" in (img.get("src") or "")
                ]
                return {"title": data.get("headline", "").strip(), "text": data.get("text", "").strip(), "images": images}
    except Exception:
        pass
    return None


# ── list forum posts ─────────────────────────────


@dataclass
class PostItem:
    title: str
    post_id: str
    forum: str
    time_ago: str = ""
    preview: str = ""
    like_count: int = 0
    comment_count: int = 0


def list_posts(forum: str, sort: str = "popular", limit: int = 15) -> List[PostItem]:
    if sort == "latest":
        url = f"https://www.dcard.tw/f/{forum}?latest=true"
    else:
        url = f"https://www.dcard.tw/f/{forum}"

    ps, browser, page = _browser()

    try:
        page.goto(url, wait_until="domcontentloaded", timeout=60000)
        page.wait_for_timeout(8000)

        js_file = _JS_DIR / "_list_posts.js"
        with open(js_file) as f:
            js_code = f.read()

        raw = page.evaluate(js_code, forum)
        return [PostItem(**p) for p in raw[:limit]]
    finally:
        browser.close()
        ps.stop()


def list_all_forums() -> dict:
    """抓熱門看板列表"""
    ps, browser, page = _browser()
    try:
        page.goto("https://www.dcard.tw/f/relationship", wait_until="domcontentloaded", timeout=30000)
        page.wait_for_timeout(5000)
        forums = page.evaluate("""
            () => {
                const items = [];
                const seen = new Set();
                document.querySelectorAll('a[href*="/f/"]:not([href*="/p/"])').forEach(a => {
                    const m = a.href.match(/\\/f\\/([\\w-]+)/);
                    if (!m || m[1] === 'popular' || seen.has(m[1])) return;
                    seen.add(m[1]);
                    items.push({name: a.textContent.trim() || m[1], slug: m[1]});
                });
                return items.slice(0, 30);
            }
        """)
        return {"forums": forums}
    finally:
        browser.close()
        ps.stop()


# ── output formatting ────────────────────────────


def fmt_post(data: dict, fmt: str) -> str:
    if fmt == "json":
        return json.dumps(data, ensure_ascii=False, indent=2)
    lines = [f"標題：{data.get('title', '')}", f"網址：{data.get('url', '')}", "", data.get("text", "")]
    imgs = data.get("images", [])
    if imgs:
        lines.extend(["", "— 圖片 —"] + imgs)
    return "\n".join(lines)


def fmt_list(items: List[PostItem], forum: str, sort: str, fmt: str) -> str:
    if fmt == "json":
        return json.dumps({"forum": forum, "sort": sort, "count": len(items), "posts": [asdict(p) for p in items]},
                          ensure_ascii=False, indent=2)
    label = "🔥 熱門" if sort == "popular" else "🕐 最新"
    lines = [f"=== Dcard /{forum} {label} ===", ""]
    for i, p in enumerate(items, 1):
        prev = (p.preview[:80] + "…") if len(p.preview) > 80 else p.preview
        lines.append(f"{i:2d}. {p.title}")
        if p.time_ago: lines.append(f"    ⏱ {p.time_ago}")
        if prev: lines.append(f"    💬 {prev}")
        lines.append(f"    🔗 https://www.dcard.tw/f/{p.forum}/p/{p.post_id}")
        lines.append("")
    lines.append(f"共 {len(items)} 篇文章")
    return "\n".join(lines)


# ── CLI ──────────────────────────────────────────


def _parse_id(s: str) -> tuple:
    m = re.search(r'/f/(\w+)/p/(\d+)', s)
    if m: return m.group(1), m.group(2)
    m = re.search(r'/p/(\d+)', s)
    if m: return None, m.group(1)
    if s.isdigit(): return None, s
    return None, None


def cmd_fetch(args):
    pid = args.id
    forum = args.forum
    if not pid and args.url_or_id:
        f, _id = _parse_id(args.url_or_id)
        forum = forum or f
        pid = _id
    if not pid:
        print("錯誤：請提供文章 ID 或 URL", file=sys.stderr)
        sys.exit(1)
    print(fmt_post(fetch_post(forum, pid), args.format))


def cmd_list(args):
    if args.all_forums:
        data = list_all_forums()
        if args.format == "json":
            print(json.dumps(data, ensure_ascii=False, indent=2))
        else:
            print("=== Dcard 熱門看板 ===")
            for f in data.get("forums", []):
                print(f"  {f['name']:20s} /{f['slug']}")
        return
    if not args.forum:
        print("錯誤：請提供看板名稱（如 relationship）或使用 --all-forums", file=sys.stderr)
        sys.exit(1)
    items = list_posts(args.forum, sort=args.sort, limit=args.limit)
    print(fmt_list(items, args.forum, args.sort, args.format))


def main():
    parser = argparse.ArgumentParser(description="Dcard 文章工具")
    sub = parser.add_subparsers(dest="command")
    # fetch
    pf = sub.add_parser("fetch", help="抓取單篇文章")
    pf.add_argument("url_or_id", nargs="?")
    pf.add_argument("--forum", "-f", default=None)
    pf.add_argument("--id", "-i", default=None)
    pf.add_argument("--format", "-o", choices=["text", "json"], default="text")
    # list
    pl = sub.add_parser("list", help="看板文章列表")
    pl.add_argument("forum", nargs="?", default=None)
    pl.add_argument("--sort", "-s", choices=["popular", "latest"], default="popular")
    pl.add_argument("--limit", "-l", type=int, default=15)
    pl.add_argument("--format", "-o", choices=["text", "json"], default="text")
    pl.add_argument("--all-forums", "-A", action="store_true", help="顯示熱門看板")
    args = parser.parse_args()
    if args.command == "list":
        cmd_list(args)
    elif args.command == "fetch":
        cmd_fetch(args)
    else:
        # default = fetch
        args.command = "fetch"
        cmd_fetch(args)


if __name__ == "__main__":
    main()


path: skill.json
content: {
  "name": "dcard",
  "version": "0.1.0",
  "description": "Fetch and extract full content from Dcard articles (Taiwan's largest anonymous forum). Handles Cloudflare protection using Camoufox stealth browser.",
  "tags": ["dcard", "taiwan", "forum", "scraping", "social"],
  "author": "ichendong"
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)