Skill: 7487/hottrender-dingtalk-bot (version 2.0.0)

--- name: hottrender-dingtalk-bot description: Use when users need a lightweight HotTrender crawler for four-region daily hotspot trends or custom keyword/vertical hotspot discovery. Prefer the bundled basic crawler runtime and existing provider scripts before writing code. This skill intentionally excludes DingTalk push, OSS publishing, ActionCard pages, lp-ads workspace, worker queues, databases, and LLM summaries. --- # HotTrender Basic Crawler ## Core Rule For four-region daily hotspot trends or vertical/custom-keyword hotspot discovery, use the bundled crawler runtime first. Do not reimplement platform crawling until the existing providers and scripts are checked. Before changing code, answer these questions: 1. Is there already a script, API, provider, doc, or test covering this need? 2. Can the user goal be satisfied by running or configuring that capability? 3. If not, what exact gap remains, and where is the smallest extension point? Only edit code after that evaluation. ## Repository Layout This skill bundles a sanitized basic crawler runtime under `assets/hottrender-runtime/`. It does not bundle DingTalk, OSS, ActionCard, lp-ads, worker queues, databases, logs, LLM, or any secrets. First resolve the runtime path from environment variables, the current workspace, or the bundled runtime: ```text HOTTRENDER_APP_DIR # directory containing scripts/fetch_daily_trends.py ``` If `HOTTRENDER_APP_DIR` is missing, install the bundled runtime: ```bash python assets/install_hottrender_runtime.py --target ./HotTrenderRuntime export HOTTRENDER_APP_DIR="$PWD/HotTrenderRuntime" ``` If variables are missing but a local checkout may exist, discover it safely: ```bash find "$PWD" "$HOME" -maxdepth 5 -path '*/scripts/fetch_daily_trends.py' 2>/dev/null ``` ## Fast Path Use these references only when needed: - Setup requirements and portability constraints: [setup.md](references/setup.md) - Daily trend and custom keyword commands: [commands.md](references/commands.md) - Existing provider capabilities and extension points: [capabilities.md](references/capabilities.md) - How to decide whether to configure, run, or modify code: [extension-policy.md](references/extension-policy.md) ## Operating Workflow 1. For "四地区热点", "每日热点", "daily trends", or "jp/us/tw/kr", start from `scripts/fetch_daily_trends.py`. 2. For "垂类热点", "关键词热点", "自定义关键词", or "custom keyword", start from `scripts/fetch_keyword_hotspots.py`. 3. For "抓取是否有效", "平台数据不对", or "为什么没结果", inspect `configs/providers.yaml`, run offline mode first, then real mode. 4. For code changes, keep the runtime basic. Do not add DingTalk, OSS, lp-ads, database, worker, or LLM features back into this skill. ## Guardrails - Never print API keys, cookies, tokens, msToken, proxy credentials, or other secrets. - Do not fabricate live platform data. Offline/sample mode must be called out as offline/sample. - Do not introduce push, publishing, database, queue, or workspace features into this basic crawler. - If the user has no HotTrender checkout, use the bundled runtime installer before proposing code rewrites. - Keep changes scoped: provider logic in `src/providers`, orchestration in `src/crawler.py`, CLI entrypoints in `scripts/`. ## Verification Prefer focused verification: ```bash cd "$HOTTRENDER_APP_DIR" python -m pytest tests/test_basic_crawler.py -q python scripts/fetch_daily_trends.py --config configs/providers.yaml --output out/daily_trends.md ```

path: agents/openai.yaml
content: interface:
  display_name: "HotTrender Basic Crawler"
  short_description: "四地区每日热点与自定义关键词爬取"
  default_prompt: "Use $hottrender-dingtalk-bot to fetch four-region daily trends or custom keyword hotspots with the bundled basic crawler runtime."
policy:
  allow_implicit_invocation: true


path: assets/hottrender-runtime/README_RUNTIME.md
content: # HotTrender Basic Crawler Runtime

This runtime is intentionally small. It only supports:

- Four-region daily hotspot trend crawling: `jp`, `us`, `tw`, `kr`
- Custom keyword / vertical hotspot crawling
- Markdown and JSON outputs for local inspection

It does not include DingTalk push, OSS publishing, ActionCard, lp-ads workspace, worker queues, databases, or LLM summaries.

## Install

```bash
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```

## Daily Trends

```bash
python scripts/fetch_daily_trends.py --output out/daily_trends.md
```

Default platforms are `google,youtube,x`. `x` uses Trends24 and may not support every region; unsupported regions are reported in the `抓取失败` section instead of being faked.

## Custom Keyword Hotspots

```bash
python scripts/fetch_keyword_hotspots.py \
  --keywords "乙游,短剧,AI男友" \
  --regions jp,us,tw,kr \
  --platforms google,youtube \
  --output out/keyword_hotspots.md
```

## Optional TikTok

TikTok is not a default dependency because it requires Playwright and a usable network/session environment.

```bash
pip install -r requirements-tiktok.txt
python -m playwright install
export TIKTOK_MS_TOKEN=...
python scripts/fetch_daily_trends.py --platforms tiktok --output out/tiktok_trends.md
```

## Offline Test Mode

Set provider `mode: offline` in `configs/providers.yaml` for deterministic local tests without network access.


path: assets/hottrender-runtime/configs/providers.yaml
content: google:
  mode: real
  timeout_seconds: 15
  source_engine: google-news-rss

youtube:
  mode: real
  timeout_seconds: 20
  source_engine: youtube-public-search

x:
  mode: real
  timeout_seconds: 20
  source_engine: trends24
  encoding: utf-8

# Optional. TikTok requires extra runtime setup:
#   pip install -r requirements-tiktok.txt
#   python -m playwright install
#   export TIKTOK_MS_TOKEN=...
tiktok:
  mode: real
  timeout_seconds: 30
  source_engine: tiktok-api
  trend_candidate_count: 30
  trend_limit: 10
  proxy: ""


path: assets/hottrender-runtime/requirements-tiktok.txt
content: TikTokApi>=6.0
playwright>=1.40


path: assets/hottrender-runtime/requirements.txt
content: PyYAML>=6.0
requests>=2.31
beautifulsoup4>=4.12
pytest>=8.0


path: assets/hottrender-runtime/scripts/fetch_daily_trends.py
content: #!/usr/bin/env python3
"""Fetch four-region daily hotspot trends."""

from __future__ import annotations

import argparse
import sys
from pathlib import Path

ROOT_DIR = Path(__file__).resolve().parents[1]
if str(ROOT_DIR) not in sys.path:
    sys.path.insert(0, str(ROOT_DIR))

from src.crawler import DEFAULT_DAILY_PLATFORMS, DEFAULT_REGIONS, fetch_daily_trends, parse_csv, write_result


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Fetch daily hotspot trends for jp/us/tw/kr.")
    parser.add_argument("--regions", default=",".join(DEFAULT_REGIONS), help="Comma-separated regions: jp,us,tw,kr")
    parser.add_argument("--platforms", default=",".join(DEFAULT_DAILY_PLATFORMS), help="Comma-separated platforms")
    parser.add_argument("--limit", type=int, default=10)
    parser.add_argument("--config", default="", help="Provider config YAML path")
    parser.add_argument("--output", default="out/daily_trends.md", help="Output .md or .json path")
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    payload = fetch_daily_trends(
        regions=parse_csv(args.regions, DEFAULT_REGIONS),
        platforms=parse_csv(args.platforms, DEFAULT_DAILY_PLATFORMS),
        limit=max(args.limit, 1),
        config_path=args.config or None,
    )
    write_result(payload, args.output)
    print(f"wrote {args.output}")


if __name__ == "__main__":
    main()


path: assets/hottrender-runtime/scripts/fetch_keyword_hotspots.py
content: #!/usr/bin/env python3
"""Fetch vertical/custom-keyword hotspot search results."""

from __future__ import annotations

import argparse
import sys
from pathlib import Path

ROOT_DIR = Path(__file__).resolve().parents[1]
if str(ROOT_DIR) not in sys.path:
    sys.path.insert(0, str(ROOT_DIR))

from src.crawler import (
    DEFAULT_KEYWORD_PLATFORMS,
    DEFAULT_REGIONS,
    fetch_keyword_hotspots,
    parse_csv,
    write_result,
)


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Fetch custom keyword hotspots.")
    parser.add_argument("--keywords", required=True, help="Comma-separated custom keywords, e.g. 乙游,短剧,AI男友")
    parser.add_argument("--regions", default=",".join(DEFAULT_REGIONS), help="Comma-separated regions: jp,us,tw,kr")
    parser.add_argument("--platforms", default=",".join(DEFAULT_KEYWORD_PLATFORMS), help="Comma-separated platforms")
    parser.add_argument("--time-range", default="7d")
    parser.add_argument("--limit", type=int, default=10)
    parser.add_argument("--config", default="", help="Provider config YAML path")
    parser.add_argument("--output", default="out/keyword_hotspots.md", help="Output .md or .json path")
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    payload = fetch_keyword_hotspots(
        keywords=parse_csv(args.keywords, []),
        regions=parse_csv(args.regions, DEFAULT_REGIONS),
        platforms=parse_csv(args.platforms, DEFAULT_KEYWORD_PLATFORMS),
        time_range=args.time_range,
        limit=max(args.limit, 1),
        config_path=args.config or None,
    )
    write_result(payload, args.output)
    print(f"wrote {args.output}")


if __name__ == "__main__":
    main()


path: assets/hottrender-runtime/src/config.py
content: """Minimal configuration loader for the bundled crawler runtime."""

from __future__ import annotations

from pathlib import Path
from typing import Any

import yaml


BASE_DIR = Path(__file__).resolve().parents[1]
DEFAULT_PROVIDERS_PATH = BASE_DIR / "configs" / "providers.yaml"


def load_provider_config(path: str | Path | None = None) -> dict[str, dict[str, Any]]:
    """Load provider configuration from YAML."""

    config_path = Path(path) if path else DEFAULT_PROVIDERS_PATH
    if not config_path.exists():
        return {}
    payload = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
    if not isinstance(payload, dict):
        return {}
    return {str(key): value if isinstance(value, dict) else {} for key, value in payload.items()}


path: assets/hottrender-runtime/src/crawler.py
content: """Basic crawling orchestration for daily trends and keyword hotspots."""

from __future__ import annotations

import json
from pathlib import Path
from typing import Any

from src.config import load_provider_config
from src.errors import user_error_message
from src.models import TrendItem, now_shanghai_iso
from src.providers import build_provider


DEFAULT_REGIONS = ["jp", "us", "tw", "kr"]
DEFAULT_DAILY_PLATFORMS = ["google", "youtube", "x"]
DEFAULT_KEYWORD_PLATFORMS = ["google", "youtube"]

REGION_LABELS = {
    "jp": "日本",
    "us": "美国",
    "tw": "台湾",
    "kr": "韩国",
}


def parse_csv(value: str | list[str] | tuple[str, ...] | None, default: list[str]) -> list[str]:
    """Parse a comma-separated option."""

    if isinstance(value, (list, tuple)):
        items = [str(item).strip() for item in value]
    else:
        items = [item.strip() for item in str(value or "").split(",")]
    parsed = [item for item in items if item]
    return parsed or list(default)


def fetch_daily_trends(
    *,
    regions: list[str] | None = None,
    platforms: list[str] | None = None,
    limit: int = 10,
    config_path: str | Path | None = None,
) -> dict[str, Any]:
    """Fetch daily trend lists across regions and platforms."""

    return _collect(
        mode="daily",
        regions=regions or DEFAULT_REGIONS,
        platforms=platforms or DEFAULT_DAILY_PLATFORMS,
        limit=limit,
        config_path=config_path,
        keywords=[],
        time_range="24h",
    )


def fetch_keyword_hotspots(
    *,
    keywords: list[str],
    regions: list[str] | None = None,
    platforms: list[str] | None = None,
    time_range: str = "7d",
    limit: int = 10,
    config_path: str | Path | None = None,
) -> dict[str, Any]:
    """Fetch custom keyword hotspot search results."""

    if not keywords:
        raise ValueError("at least one keyword is required")
    return _collect(
        mode="keyword",
        regions=regions or DEFAULT_REGIONS,
        platforms=platforms or DEFAULT_KEYWORD_PLATFORMS,
        limit=limit,
        config_path=config_path,
        keywords=keywords,
        time_range=time_range,
    )


def write_result(payload: dict[str, Any], output: str | Path | None, markdown: bool = True) -> None:
    """Write JSON and optional Markdown output."""

    if not output:
        print(json.dumps(payload, ensure_ascii=False, indent=2))
        return
    output_path = Path(output)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    suffix = output_path.suffix.lower()
    if suffix == ".json":
        output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    elif suffix == ".md" or markdown:
        output_path.write_text(render_markdown(payload), encoding="utf-8")
        json_path = output_path.with_suffix(".json")
        json_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    else:
        output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")


def render_markdown(payload: dict[str, Any]) -> str:
    """Render a compact Markdown report for human inspection."""

    mode = payload.get("mode", "")
    title = "四地区每日热点趋势" if mode == "daily" else "垂类关键词热点"
    lines = [
        f"# {title}",
        "",
        f"- 生成时间: {payload.get('generated_at', '')}",
        f"- 地区: {', '.join(_region_label(region) for region in payload.get('regions', []))}",
        f"- 平台: {', '.join(payload.get('platforms', []))}",
    ]
    if payload.get("keywords"):
        lines.append(f"- 关键词: {', '.join(payload['keywords'])}")
    lines.append("")

    grouped: dict[str, dict[str, list[dict[str, Any]]]] = {}
    for item in payload.get("items", []):
        grouped.setdefault(item["region"], {}).setdefault(item["platform"], []).append(item)
    if not grouped:
        lines.extend(["暂无热点数据。", ""])
    for region in payload.get("regions", []):
        region_items = grouped.get(region, {})
        if not region_items:
            continue
        lines.extend([f"## {_region_label(region)}", ""])
        for platform in payload.get("platforms", []):
            items = region_items.get(platform, [])
            if not items:
                continue
            lines.extend([f"### {platform}", ""])
            for idx, item in enumerate(items, start=1):
                rank = item.get("rank") or idx
                keyword = f" [{item['keyword']}]" if item.get("keyword") else ""
                lines.append(f"{rank}. {item['title_original']}{keyword}")
                lines.append(f"   - 链接: {item['raw_url']}")
            lines.append("")

    errors = payload.get("errors", {})
    if errors:
        lines.extend(["## 抓取失败", ""])
        for key, message in errors.items():
            lines.append(f"- {key}: {message}")
        lines.append("")
    return "\n".join(lines).rstrip() + "\n"


def _collect(
    *,
    mode: str,
    regions: list[str],
    platforms: list[str],
    limit: int,
    config_path: str | Path | None,
    keywords: list[str],
    time_range: str,
) -> dict[str, Any]:
    provider_config = load_provider_config(config_path)
    items: list[TrendItem] = []
    errors: dict[str, str] = {}
    for platform in platforms:
        provider = build_provider(platform, provider_config.get(platform, {}))
        for region in regions:
            try:
                if mode == "daily":
                    fetched = provider.fetch_trends(region)[:limit]
                else:
                    fetched = []
                    for keyword in keywords:
                        fetched.extend(provider.search(region, keyword, time_range=time_range, limit=limit))
                    fetched = _dedupe(fetched)[: max(limit * max(len(keywords), 1), limit)]
                items.extend(fetched)
            except Exception as exc:
                errors[f"{platform}:{region}"] = user_error_message(exc)
    return {
        "mode": mode,
        "generated_at": now_shanghai_iso(),
        "regions": regions,
        "platforms": platforms,
        "keywords": keywords,
        "time_range": time_range,
        "items": [item.to_dict() for item in _dedupe(items)],
        "errors": errors,
    }


def _dedupe(items: list[TrendItem]) -> list[TrendItem]:
    seen = set()
    result = []
    for item in items:
        key = item.dedup_key()
        if key in seen:
            continue
        seen.add(key)
        result.append(item)
    return result


def _region_label(region: str) -> str:
    return REGION_LABELS.get(region, region)


path: assets/hottrender-runtime/src/errors.py
content: """Application error taxonomy."""

from __future__ import annotations

from enum import StrEnum


class ErrorCode(StrEnum):
    PROXY_UNREACHABLE = "PROXY_UNREACHABLE"
    TARGET_TIMEOUT = "TARGET_TIMEOUT"
    RATE_LIMITED = "RATE_LIMITED"
    COOKIE_EXPIRED = "COOKIE_EXPIRED"
    PARSING_FAILED = "PARSING_FAILED"
    NO_RESULT = "NO_RESULT"
    PLATFORM_BLOCKED = "PLATFORM_BLOCKED"


class AppError(Exception):
    def __init__(self, code: ErrorCode, message: str):
        self.code = code
        self.message = message
        super().__init__(f"{code.value}: {message}")


class ProviderError(AppError):
    pass


def user_error_message(exc: Exception) -> str:
    if isinstance(exc, AppError):
        return f"{exc.code.value}: {exc.message}"
    return f"{ErrorCode.PARSING_FAILED.value}: {exc}"


path: assets/hottrender-runtime/src/models.py
content: """Shared data models."""

from __future__ import annotations

from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone, timedelta
from typing import Any


CN_TZ = timezone(timedelta(hours=8))


def now_shanghai_iso() -> str:
    return datetime.now(CN_TZ).replace(microsecond=0).isoformat()


@dataclass(frozen=True)
class TrendItem:
    platform: str
    region: str
    title_original: str
    summary_zh: str
    raw_url: str
    source_type: str
    source_engine: str
    trust_level: str
    content_type: str
    published_at: str
    fetched_at: str
    rank: int | None = None
    keyword: str | None = None
    title_zh: str | None = None
    meta_json: str | None = None

    def dedup_key(self) -> str:
        return f"{self.platform}:{self.region}:{self.raw_url}"

    def to_dict(self) -> dict[str, Any]:
        return asdict(self)


@dataclass(frozen=True)
class QueryRequest:
    user_id: str
    region: str
    platforms: list[str]
    keyword: str
    time_range: str = "7d"
    limit: int = 10
    filters: dict[str, Any] = field(default_factory=dict)


@dataclass(frozen=True)
class QueryResult:
    items: list[TrendItem]
    errors: dict[str, str]
    async_refetch_started: bool = False

    @property
    def status(self) -> str:
        if self.items and self.errors:
            return "partial"
        if self.errors and not self.items:
            return "failed"
        return "success"


path: assets/hottrender-runtime/src/providers/__init__.py
content: """Basic provider registry."""

from __future__ import annotations

from .base import BaseProvider
from .google import GoogleProvider
from .x import XProvider
from .youtube import YouTubeProvider

__all__ = [
    "BaseProvider",
    "GoogleProvider",
    "TikTokProvider",
    "XProvider",
    "YouTubeProvider",
    "build_provider",
    "available_providers",
]


def available_providers() -> list[str]:
    """Return provider names supported by the basic runtime."""

    return ["google", "youtube", "x", "tiktok"]


def build_provider(name: str, config: dict | None = None) -> BaseProvider:
    """Build a provider by name."""

    normalized = name.strip().lower()
    if normalized == "google":
        return GoogleProvider(config)
    if normalized == "youtube":
        return YouTubeProvider(config)
    if normalized == "x":
        return XProvider(config)
    if normalized == "tiktok":
        from .tiktok import TikTokProvider

        return TikTokProvider(config)
    raise ValueError(f"unsupported provider: {name}")


path: assets/hottrender-runtime/src/providers/base.py
content: """Provider interface and shared helpers."""

from __future__ import annotations

from abc import ABC, abstractmethod
from html import unescape
from urllib.parse import quote_plus

import requests

from src.errors import ErrorCode, ProviderError
from src.models import TrendItem, now_shanghai_iso


SUPPORTED_REGIONS = {"jp", "us", "tw", "kr"}


class BaseProvider(ABC):
    name: str
    source_type: str
    source_engine: str
    trust_level: str
    content_type: str

    def __init__(self, config: dict | None = None):
        self.config = config or {}

    @property
    def offline_mode(self) -> bool:
        return str(self.config.get("mode", "real")).lower() == "offline"

    @property
    def timeout_seconds(self) -> float:
        try:
            return float(self.config.get("timeout_seconds", 20))
        except (TypeError, ValueError):
            return 20.0

    @abstractmethod
    def fetch_trends(self, region: str) -> list[TrendItem]:
        """Fetch trend inventory for one region."""

    @abstractmethod
    def search(
        self,
        region: str,
        keyword: str,
        time_range: str = "7d",
        limit: int = 10,
        filters: dict | None = None,
    ) -> list[TrendItem]:
        """Search platform content by keyword."""

    def healthcheck(self) -> dict:
        return {"status": "ok", "message": f"{self.name} provider ready"}

    def _validate_region(self, region: str) -> None:
        if region not in SUPPORTED_REGIONS:
            raise ProviderError(ErrorCode.NO_RESULT, f"unsupported region: {region}")

    def _build_item(
        self,
        region: str,
        title: str,
        url: str,
        rank: int | None = None,
        keyword: str | None = None,
        content_type: str | None = None,
    ) -> TrendItem:
        timestamp = now_shanghai_iso()
        return TrendItem(
            platform=self.name,
            region=region,
            title_original=title,
            title_zh=None,
            summary_zh=title,
            raw_url=url,
            source_type=self.source_type,
            source_engine=self.source_engine,
            trust_level=self.trust_level,
            content_type=content_type or self.content_type,
            published_at=timestamp,
            fetched_at=timestamp,
            rank=rank,
            keyword=keyword,
        )

    def _get_text(self, url: str, params: dict | None = None) -> str:
        try:
            response = requests.get(
                url,
                params=params,
                headers={
                    "User-Agent": self.config.get(
                        "user_agent",
                        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                        "(KHTML, like Gecko) Chrome/122.0 Safari/537.36",
                    ),
                    "Accept-Language": self.config.get("accept_language", "ja,en-US;q=0.9,en;q=0.8"),
                },
                timeout=self.timeout_seconds,
            )
        except requests.Timeout as exc:
            raise ProviderError(ErrorCode.TARGET_TIMEOUT, f"{self.name} request timeout") from exc
        except requests.RequestException as exc:
            raise ProviderError(ErrorCode.PROXY_UNREACHABLE, str(exc)) from exc
        if response.status_code in {401, 403}:
            raise ProviderError(ErrorCode.PLATFORM_BLOCKED, f"{self.name} returned {response.status_code}")
        if response.status_code == 429:
            raise ProviderError(ErrorCode.RATE_LIMITED, f"{self.name} returned 429")
        if not 200 <= response.status_code < 300:
            raise ProviderError(ErrorCode.PARSING_FAILED, f"{self.name} returned {response.status_code}")
        configured_encoding = str(self.config.get("encoding", "")).strip()
        if configured_encoding:
            response.encoding = configured_encoding
        elif not response.encoding or response.encoding.lower() in {"iso-8859-1", "windows-1252"}:
            response.encoding = "utf-8"
        return response.text

    @staticmethod
    def _clean_text(value: str) -> str:
        return " ".join(unescape(value).replace("\\u0026", "&").split())

    def _offline_trends(self, region: str, topics: list[str], base_url: str) -> list[TrendItem]:
        self._validate_region(region)
        return [
            self._build_item(region, f"{region.upper()} {topic}", f"{base_url}/{region}/{quote_plus(topic)}", idx)
            for idx, topic in enumerate(topics, start=1)
        ]

    def _offline_search(self, region: str, keyword: str, limit: int, base_url: str) -> list[TrendItem]:
        self._validate_region(region)
        if not keyword.strip():
            raise ProviderError(ErrorCode.NO_RESULT, "keyword is required")
        return [
            self._build_item(
                region,
                f"{keyword} related {self.name} result {idx}",
                f"{base_url}/search/{region}/{quote_plus(keyword)}/{idx}",
                idx,
                keyword=keyword,
            )
            for idx in range(1, limit + 1)
        ]


path: assets/hottrender-runtime/src/providers/google.py
content: """Google Trends / News provider."""

from __future__ import annotations

from email.utils import parsedate_to_datetime
from urllib.parse import quote_plus
import xml.etree.ElementTree as ET

from src.errors import ErrorCode, ProviderError
from src.models import TrendItem, now_shanghai_iso
from src.providers.base import BaseProvider


class GoogleProvider(BaseProvider):
    name = "google"
    source_type = "official_rss"
    source_engine = "google-news-rss"
    trust_level = "A"
    content_type = "text"

    def fetch_trends(self, region: str):
        if not self.offline_mode:
            return self.search(region, "trending", "24h", 10)
        topics = ["breaking news", "entertainment", "game industry", "ai products", "culture"]
        return self._offline_trends(region, topics, "https://news.google.com/rss")

    def search(self, region: str, keyword: str, time_range: str = "7d", limit: int = 10, filters: dict | None = None):
        if self.offline_mode:
            return self._offline_search(region, keyword, limit, "https://news.google.com")
        self._validate_region(region)
        if not keyword.strip():
            raise ProviderError(ErrorCode.NO_RESULT, "keyword is required")
        locale = {
            "jp": ("ja", "JP"),
            "us": ("en-US", "US"),
            "tw": ("zh-TW", "TW"),
            "kr": ("ko", "KR"),
        }[region]
        url = "https://news.google.com/rss/search"
        query = keyword if keyword != "trending" else "ニュース OR trending"
        xml_text = self._get_text(url, {"q": query, "hl": locale[0], "gl": locale[1], "ceid": f"{locale[1]}:{locale[0].split('-')[0]}"})
        root = ET.fromstring(xml_text)
        items = []
        fetched_at = now_shanghai_iso()
        for idx, node in enumerate(root.findall("./channel/item")[:limit], start=1):
            title = self._clean_text(node.findtext("title") or "")
            link = self._clean_text(node.findtext("link") or f"https://news.google.com/search?q={quote_plus(keyword)}")
            published_raw = node.findtext("pubDate") or ""
            try:
                published_at = parsedate_to_datetime(published_raw).astimezone().replace(microsecond=0).isoformat()
            except Exception:
                published_at = fetched_at
            if not title:
                continue
            items.append(
                TrendItem(
                    platform=self.name,
                    region=region,
                    title_original=title,
                    title_zh=None,
                    summary_zh=title,
                    raw_url=link,
                    source_type=self.source_type,
                    source_engine=self.source_engine,
                    trust_level=self.trust_level,
                    content_type=self.content_type,
                    published_at=published_at,
                    fetched_at=fetched_at,
                    rank=idx,
                    keyword=None if keyword == "trending" else keyword,
                )
            )
        if not items:
            raise ProviderError(ErrorCode.NO_RESULT, f"no google news result for {keyword}")
        return items


path: assets/hottrender-runtime/src/providers/x.py
content: """X/Twitter provider."""

from __future__ import annotations

from urllib.parse import quote_plus

from bs4 import BeautifulSoup

from src.errors import ErrorCode, ProviderError
from src.providers.base import BaseProvider


class XProvider(BaseProvider):
    name = "x"
    source_type = "unofficial_scraper"
    source_engine = "trends24"
    trust_level = "B"
    content_type = "text"

    _REGION_PATHS = {
        "jp": "japan",
        "us": "united-states",
        "kr": "korea",
    }

    def fetch_trends(self, region: str):
        if not self.offline_mode:
            self._validate_region(region)
            region_path = self._REGION_PATHS.get(region)
            if region_path is None:
                raise ProviderError(ErrorCode.NO_RESULT, f"X trends via trends24 not available for region '{region}'; use a different platform or region")
            html = self._get_text(f"https://trends24.in/{region_path}/")
            soup = BeautifulSoup(html, "html.parser")
            topics = [item.get_text(strip=True) for item in soup.select("ol.trend-card__list li a")]
            topics = [topic for topic in dict.fromkeys(topics) if topic]
            if not topics:
                raise ProviderError(ErrorCode.PARSING_FAILED, f"no X trends parsed for {region}")
            return [
                self._build_item(region, topic, f"https://x.com/search?q={quote_plus(topic)}&src=trend_click&f=live", idx)
                for idx, topic in enumerate(topics[:20], start=1)
            ]
        topics = ["anime trend", "game release", "idol news", "ai boyfriend", "drama talk"]
        return self._offline_trends(region, topics, "https://x.com/i/trends")

    def search(self, region: str, keyword: str, time_range: str = "7d", limit: int = 10, filters: dict | None = None):
        if self.offline_mode:
            return self._offline_search(region, keyword, limit, "https://x.com")
        self._validate_region(region)
        # Public X search is client-rendered and often unavailable without auth. Return a hard error
        # instead of fabricating results so channel validation can fail honestly.
        raise ProviderError(ErrorCode.PLATFORM_BLOCKED, "X search requires an authenticated scraper or snscrape-compatible backend")


if __name__ == "__main__":
    for item in XProvider().fetch_trends("jp"):
        print(item.to_dict())


path: assets/hottrender-runtime/src/providers/youtube.py
content: """YouTube search provider."""

from __future__ import annotations

import json
import re
from urllib.parse import quote_plus

from src.errors import ErrorCode, ProviderError
from src.models import TrendItem, now_shanghai_iso
from src.providers.base import BaseProvider


class YouTubeProvider(BaseProvider):
    name = "youtube"
    source_type = "official_open_page"
    source_engine = "youtube-search-python"
    trust_level = "B"
    content_type = "video"

    def fetch_trends(self, region: str):
        if not self.offline_mode:
            return self.search(region, "trending", "24h", 10)
        topics = ["music video", "vtuber clip", "game trailer", "variety cut", "anime review"]
        return self._offline_trends(region, topics, "https://www.youtube.com/feed/trending")

    def search(self, region: str, keyword: str, time_range: str = "7d", limit: int = 10, filters: dict | None = None):
        if self.offline_mode:
            return self._offline_search(region, keyword, limit, "https://www.youtube.com")
        self._validate_region(region)
        html = self._get_text("https://www.youtube.com/results", {"search_query": keyword})
        video_ids = re.findall(r'"videoId":"([^"]+)"', html)
        titles = re.findall(r'"title":\{"runs":\[\{"text":"((?:[^"\\]|\\.)+)"', html)
        seen = set()
        items = []
        fetched_at = now_shanghai_iso()
        title_idx = 0
        for video_id in video_ids:
            if video_id in seen:
                continue
            seen.add(video_id)
            title = ""
            while title_idx < len(titles) and not title:
                try:
                    title = json.loads(f'"{titles[title_idx]}"')
                except json.JSONDecodeError:
                    title = titles[title_idx]
                title_idx += 1
            if not title:
                title = f"YouTube video {video_id}"
            items.append(
                TrendItem(
                    platform=self.name,
                    region=region,
                    title_original=self._clean_text(title),
                    title_zh=None,
                    summary_zh=title,
                    raw_url=f"https://www.youtube.com/watch?v={video_id}",
                    source_type=self.source_type,
                    source_engine=self.source_engine,
                    trust_level=self.trust_level,
                    content_type=self.content_type,
                    published_at=fetched_at,
                    fetched_at=fetched_at,
                    rank=len(items) + 1,
                    keyword=None if keyword == "trending" else keyword,
                )
            )
            if len(items) >= limit:
                break
        if not items:
            raise ProviderError(ErrorCode.PARSING_FAILED, f"no youtube result parsed for {quote_plus(keyword)}")
        return items


path: assets/hottrender-runtime/tests/test_basic_crawler.py
content: from src.crawler import fetch_daily_trends, fetch_keyword_hotspots, render_markdown


def test_fetch_daily_trends_offline(tmp_path):
    config = tmp_path / "providers.yaml"
    config.write_text(
        """
google:
  mode: offline
youtube:
  mode: offline
""",
        encoding="utf-8",
    )

    payload = fetch_daily_trends(
        regions=["jp", "us"],
        platforms=["google", "youtube"],
        limit=2,
        config_path=config,
    )

    assert payload["mode"] == "daily"
    assert len(payload["items"]) == 8
    assert payload["errors"] == {}
    assert "四地区每日热点趋势" in render_markdown(payload)


def test_fetch_keyword_hotspots_offline(tmp_path):
    config = tmp_path / "providers.yaml"
    config.write_text(
        """
google:
  mode: offline
""",
        encoding="utf-8",
    )

    payload = fetch_keyword_hotspots(
        keywords=["乙游", "短剧"],
        regions=["jp"],
        platforms=["google"],
        limit=2,
        config_path=config,
    )

    assert payload["mode"] == "keyword"
    assert len(payload["items"]) == 4
    assert {item["keyword"] for item in payload["items"]} == {"乙游", "短剧"}
    assert "垂类关键词热点" in render_markdown(payload)


path: assets/install_hottrender_runtime.py
content: #!/usr/bin/env python3
"""Install the bundled HotTrender runtime into a target directory."""

from __future__ import annotations

import argparse
import shutil
from pathlib import Path


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Install bundled HotTrender runtime.")
    parser.add_argument("--target", required=True, help="Destination directory for the runtime copy.")
    parser.add_argument("--force", action="store_true", help="Overwrite an existing target directory.")
    return parser.parse_args()


def copy_runtime(source: Path, target: Path, force: bool) -> None:
    if target.exists():
        if not force:
            raise SystemExit(f"target already exists: {target}; pass --force to overwrite")
        shutil.rmtree(target)
    shutil.copytree(source, target)
    (target / "out").mkdir(parents=True, exist_ok=True)
    env_example = target / ".env.example"
    env_file = target / ".env"
    if env_example.exists() and not env_file.exists():
        shutil.copy2(env_example, env_file)


def main() -> None:
    args = parse_args()
    skill_dir = Path(__file__).resolve().parents[1]
    source = skill_dir / "assets" / "hottrender-runtime"
    if not source.exists():
        raise SystemExit(f"runtime bundle not found: {source}")
    target = Path(args.target).expanduser().resolve()
    copy_runtime(source, target, args.force)
    print(f"installed HotTrender runtime -> {target}")
    print("next:")
    print(f"  cd {target}")
    print("  python -m venv .venv")
    print("  . .venv/bin/activate")
    print("  pip install -r requirements.txt")
    print("  python scripts/fetch_daily_trends.py --output out/daily_trends.md")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)