Skill: gcdd1993/digest-builder (version 1.0.0)

--- name: digest-builder description: "将聚合池中的原始候选内容转换为可判断的 Digest：优先从 FreshRSS 的未读列表经 Google Reader API 拉取候选，再执行 URL 去重、相似事件聚类、正文抓取检查、噪音过滤、摘要生成与初步排序。用于需要把一批原始 feed 条目或 FreshRSS 未读条目整理成结构化候选文档的场景。" --- # Digest Builder 以简体中文工作。目标是把原始候选整理成干净、结构化、适合快速判断的候选池，而不是直接产出日报终稿。若工作区已经配置 FreshRSS，则默认先从 FreshRSS API 拉取未读列表，再进入 Digest 预处理。 ## 核心任务 - 通过 FreshRSS Google Reader API 拉取未读列表 - 读取配置中的 `api_password` - URL 精确去重 - 相似事件聚类 - 正文抓取可用性检查 - 噪音过滤 - 摘要生成 - 标签补全 - 初步排序 ## 处理顺序 1. 若已配置 FreshRSS，先通过 Google Reader API 完成登录并拉取未读列表。 2. 默认按“昨天”这个自然日窗口切片未读条目；只有用户明确指定其他日期时才覆盖。 3. 先做 URL 归一和精确去重。 4. 再识别同一事件的相似条目，进行聚类或合并。 5. 检查正文是否可抓取、是否存在明显质量问题。 6. 剔除标题党、无正文、广告型、明显重复价值不足的内容。 7. 为保留条目生成 2 到 5 行摘要。 8. 补充主题标签、风险标签、行动价值标签。 9. 按公共重要性、个人相关性、来源可靠度做初排序。 ## 去重硬约束 - Digest 必须严格去重，禁止同一事件、同一结论、同一新闻变体重复出现。 - 去重分三层执行： - URL 层：相同链接、归一化后等价链接、追踪参数不同但正文相同的链接只保留一条。 - 事件层：同一事件的多来源只保留信息增量最高的一条为主条目，其余仅作为参考来源存在，不再生成独立摘要。 - 摘要层：若两条候选的摘要核心信息高度重合，必须重写或合并，确保每段摘要独立且有辨识度。 - 不能依赖标题不同就视为不同条目；必须按事件事实和信息增量判断。 ## 英文文章处理 - 所有英文文章必须生成中文摘要。 - 中文摘要必须准确覆盖原文核心信息、关键观点和重要结论，不能只翻译标题。 - 语言要求简洁、逻辑清晰，不遗漏影响判断的关键信息。 - 若原文是英文但正文无法完整访问，必须明确标记“英文原文未完整获取”，不能假装已完整总结。 ## 链接格式 - 所有链接统一使用 Markdown 格式：`[标题文本](链接)` - 标题文本必须简洁且能精准对应目标内容。 - 禁止直接裸露 URL 作为最终输出主格式，除非是调试或原始抓取产物。 ## FreshRSS 配置约定 优先读取配置文件，其次读取环境变量。 推荐配置项： - `base_url` - `username` - `api_password` - `output_json` - `output_md` - `limit` - `include_read` 若既没有配置文件，也没有环境变量，不要假装已读取 FreshRSS，必须明确指出缺失项。 ## FreshRSS API 约定 - API 地址：`<base_url>/api/greader.php` - 登录接口：`accounts/ClientLogin` - 未读列表接口：`reader/api/0/stream/contents/reading-list?xt=user/-/state/com.google/read&output=json` - 已读标记接口：`reader/api/0/edit-tag` - 认证方式：`Authorization: GoogleLogin auth=<Auth>` 默认优先输出两份原始产物： - `digests/raw-freshrss.json` - `digests/raw-freshrss.md` 然后默认继续输出“昨天窗口”切片： - `digests/freshrss-yesterday.json` - `digests/freshrss-yesterday.md` 再基于昨天窗口产物进入去重、聚类和摘要阶段。 ## 已读回写状态 - 当前默认流程不回写 FreshRSS 已读状态。 - 生成 `Daily Review`、上传知识库、写入笔记后，也不要自动把源条目标记为已读。 - 只有用户明确要求回写已读时，才允许进入该分支。 - 若未来恢复该能力，也必须按“处理时间窗口”精确回写，不能清空整个原始未读池。 - 需要人工触发时可使用 `scripts/mark_freshrss_read.py`，但默认流程禁止调用。 ## 输出格式 每条候选至少包含： - 条目 ID - 标题 - 来源 - 时间 - 链接（Markdown 格式） - 原始摘要或内容片段 - 摘要 - 主题标签 - 聚类标记 - 排序理由 ## 排序原则 - 公共重要性优先于个人偏好。 - 个人相关性只作为弱信号。 - 同一事件的多来源不应占满前排。 ## 输出风格 - 摘要尽量具体，避免空话。 - 每段摘要必须与其他条目明显区分，禁止模板化重复表述。 - 排序理由要可解释。 - 若正文不可用，明确标记，不要装作已读过全文。 ## 反例禁区 - 只整理标题，不检查正文质量。 - 摘要变成泛泛改写标题。 - 多条摘要重复表达同一结论，却未合并或改写。 - 用个人偏好压过公共重要性事件。 - 还没做聚类就直接排序，导致同一事件反复出现。 ## 参考文件 - Digest 字段和过滤规则：`references/digest-spec.md` - FreshRSS Google Reader API 流程：`references/freshrss-google-reader.md` - FreshRSS 抓取脚本：`scripts/fetch_freshrss_unread.py` - FreshRSS 日期切片脚本：`scripts/slice_freshrss_by_date.py` - Digest 构建脚本：`scripts/build_digest.py` - FreshRSS 已读回写脚本：`scripts/mark_freshrss_read.py` - FreshRSS 配置样板：`scripts/freshrss.config.example.json`

path: references/digest-spec.md
content: # Digest Spec

## 建议字段

- `item_id`
- `cluster_id`
- `title`
- `source`
- `published_at`
- `raw_url`
- `normalized_url`
- `summary`
- `summary_zh`
- `topic_tags`
- `risk_tags`
- `actionability`
- `public_importance`
- `personal_relevance`
- `ranking_reason`
- `origin_system`
- `origin_feed`
- `freshrss_categories`

## 噪音过滤规则

优先过滤：

- 无法访问正文且无可靠替代来源
- 标题与正文明显不符
- 纯营销宣传
- 重复转载但无新增信息
- 与当前主题体系长期无关的低价值内容

## 严格去重规则

### URL 去重

- 去除常见追踪参数后的归一化 URL 相同，视为同一条目。
- 同一正文被多个短链接或跳转链接指向时，只保留最终正文源。

### 事件去重

- 不同来源报道同一事件时，只保留信息增量最高的一条作为主条目。
- 其他来源只记录在参考来源列表中，不再生成独立摘要段。

### 摘要去重

- 若两条摘要的核心事实、结论、价值判断高度重合，必须合并或重写。
- Digest 中每条摘要都必须能回答“它和上一条有什么本质不同”。

## 摘要要求

- 说明发生了什么。
- 说明为什么值得关注。
- 说明是否有动作价值或风险信号。
- 用简洁中文表达，避免空泛形容。
- 英文文章也必须输出中文摘要。
- 摘要不能只是标题改写，必须包含正文关键信息。

## 链接规范

- 最终输出中的链接统一使用 Markdown：`[标题文本](链接)`
- 标题文本应优先使用人工可识别的短标题，而不是完整 URL

## 聚类要求

- 同一事件多来源时，保留信息增量高的来源为主。
- 其他来源作为补充引用，不重复占位。

## FreshRSS 原始映射建议

若上游来自 FreshRSS：

- `item_id` 对应 FreshRSS / Google Reader item id
- `origin_system` 固定为 `freshrss`
- `origin_feed` 保存订阅源标题或 feed id
- `summary` 可先用 API 返回的 `summary.content` 或正文片段兜底
- `summary_zh` 作为最终对外摘要字段，尤其用于英文文章的中文摘要



path: references/freshrss-google-reader.md
content: # FreshRSS Google Reader API

## 推荐前提

- 已在 FreshRSS 中启用 API access
- 当前用户已设置 API password
- 配置里保存的是 API password，而不是网页登录密码

## 基本流程

1. 请求 `accounts/ClientLogin`
2. 从返回内容中提取 `Auth=...`
3. 通过 `Authorization: GoogleLogin auth=<Auth>` 访问阅读列表接口
4. 默认读取未读阅读列表

## 关键接口

### 登录

`<base_url>/api/greader.php/accounts/ClientLogin`

参数：

- `Email`
- `Passwd`

### 未读阅读列表

`<base_url>/api/greader.php/reader/api/0/stream/contents/reading-list?xt=user/-/state/com.google/read&output=json`

## 处理要求

- 不在文档中回显明文 `api_password`
- 若登录失败，优先检查：
  - API access 是否开启
  - 用户名是否正确
  - 使用的是 API password 而不是网页登录密码
  - `base_url` 是否包含多余路径
- 若返回成功但无条目，要明确说明“当前无未读项”

## 推荐输出

- 原始 JSON：便于下游脚本消费
- Markdown 预览：便于人工快速检查本次抓取结果


path: scripts/build_digest.py
content: #!/usr/bin/env python3
"""Build a structured digest from FreshRSS items.

Pipeline stages:
1. URL exact deduplication
2. Similar-content clustering and deduplication
3. Article body fetch
4. Quality checks
5. Noise filtering
6. Summary generation
7. Initial ranking and document output
"""

from __future__ import annotations

import argparse
import html
import json
import math
import re
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime, timezone
from difflib import SequenceMatcher
from html.parser import HTMLParser
from pathlib import Path
from typing import Any


TRACKING_PARAMS = {
    "cmpid",
    "fbclid",
    "gclid",
    "igshid",
    "mkt_tok",
    "mc_cid",
    "mc_eid",
    "ref",
    "ref_src",
    "rss",
    "spm",
    "src",
    "source",
    "tag",
    "trk",
}

MARKETING_PATTERNS = [
    "扫码",
    "加群",
    "进群",
    "优惠",
    "折扣",
    "抽奖",
    "报名",
    "立即报名",
    "课程",
    "社群",
    "会员",
    "推广",
    "广告",
    "sponsored",
    "sign up",
    "subscribe",
    "newsletter",
    "promo",
    "discount",
    "webinar",
]

ACCESS_PATTERNS = [
    "enable javascript",
    "please enable javascript",
    "access denied",
    "forbidden",
    "登录后查看",
    "请登录",
    "subscribe to continue",
    "cookie policy",
]

LOW_SIGNAL_PATTERNS = [
    "comments",
    "求推荐",
    "求助",
    "bro-mode",
    "纯娱乐向",
    "摸鱼",
]

SOURCE_WEIGHTS = {
    "readhub": 1.40,
    "infoq": 1.32,
    "the new stack": 1.24,
    "techcrunch": 1.20,
    "stanford": 1.18,
    "slack engineering": 1.16,
    "cloudflare": 1.14,
    "hacker news": 1.02,
    "36kr": 1.05,
    "量子位": 1.05,
    "v2ex": 0.76,
}

TOPIC_RULES = {
    "AI": ["ai", "llm", "agent", "模型", "大模型", "openai", "anthropic", "claude", "gemini", "siri"],
    "安全": ["安全", "漏洞", "exploit", "breach", "supply chain", "secret", "风险", "scanner"],
    "开源": ["open source", "开源", "github", "release", "repo"],
    "基础设施": ["cloud", "kubernetes", "vault", "tracing", "otel", "infra", "platform", "数据湖"],
    "公司动态": ["融资", "ipo", "投资", "离职", "挖角", "软银", "meta", "apple", "华为", "openai"],
    "开发工具": ["skill", "copilot", "codex", "devops", "tool", "workflow", "ci/cd", "sdk"],
    "研究治理": ["research", "论文", "neurips", "治理", "study", "同行评审"],
}

RISK_RULES = {
    "security": ["安全", "漏洞", "attack", "exploit", "风险", "secret", "supply chain"],
    "governance": ["治理", "policy", "合规", "监管", "审查"],
    "availability": ["宕机", "incident", "latency", "故障", "稳定性"],
}


class BlockTextExtractor(HTMLParser):
    """Convert HTML into text blocks while ignoring obvious boilerplate tags."""

    BLOCK_TAGS = {
        "article",
        "blockquote",
        "br",
        "div",
        "h1",
        "h2",
        "h3",
        "h4",
        "h5",
        "h6",
        "li",
        "ol",
        "p",
        "pre",
        "section",
        "table",
        "tr",
        "ul",
    }
    SKIP_TAGS = {"head", "noscript", "script", "style", "svg"}

    def __init__(self) -> None:
        super().__init__()
        self.parts: list[str] = []
        self.skip_depth = 0
        self.in_title = False
        self.title_parts: list[str] = []
        self.meta: dict[str, str] = {}

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        attrs_map = {key.lower(): value or "" for key, value in attrs}
        lowered = tag.lower()
        if lowered in self.SKIP_TAGS:
            self.skip_depth += 1
            return
        if lowered in self.BLOCK_TAGS:
            self.parts.append("\n")
        if lowered == "title":
            self.in_title = True
        if lowered == "meta":
            name = attrs_map.get("name") or attrs_map.get("property") or ""
            content = attrs_map.get("content") or ""
            if name and content:
                self.meta[name.lower()] = content.strip()

    def handle_endtag(self, tag: str) -> None:
        lowered = tag.lower()
        if lowered in self.SKIP_TAGS and self.skip_depth > 0:
            self.skip_depth -= 1
            return
        if lowered in self.BLOCK_TAGS:
            self.parts.append("\n")
        if lowered == "title":
            self.in_title = False

    def handle_data(self, data: str) -> None:
        if self.skip_depth > 0:
            return
        cleaned = collapse_ws(html.unescape(data))
        if not cleaned:
            return
        if self.in_title:
            self.title_parts.append(cleaned)
        self.parts.append(cleaned)


@dataclass
class Candidate:
    item_id: str
    title: str
    raw_url: str
    normalized_url: str
    source: str
    origin_feed: str
    origin_system: str
    published_at: str
    published_ts: int
    freshrss_categories: list[str]
    summary_html: str
    summary_text: str
    source_weight: float
    title_tokens: set[str]
    body_text: str = ""
    fetched_url: str = ""
    body_fetch_status: str = "pending"
    body_fetch_error: str = ""
    body_length: int = 0
    content_language: str = "unknown"
    content_title: str = ""
    meta_description: str = ""
    quality_score: int = 0
    quality_flags: list[str] | None = None
    noise_flags: list[str] | None = None
    filtered_reason: str = ""
    cluster_id: str = ""
    duplicate_item_ids: list[str] | None = None
    related_item_ids: list[str] | None = None
    topic_tags: list[str] | None = None
    risk_tags: list[str] | None = None
    actionability: str = "low"
    public_importance: int = 1
    personal_relevance: int = 1
    ranking_reason: str = ""
    summary: str = ""
    summary_zh: str = ""
    raw_excerpt: str = ""
    quality_passed: bool = True
    rank_score: float = 0.0

    def as_output(self) -> dict[str, Any]:
        return {
            "item_id": self.item_id,
            "cluster_id": self.cluster_id,
            "title": self.title,
            "source": self.source,
            "published_at": self.published_at,
            "raw_url": self.raw_url,
            "normalized_url": self.normalized_url,
            "summary": self.summary,
            "summary_zh": self.summary_zh,
            "topic_tags": self.topic_tags or [],
            "risk_tags": self.risk_tags or [],
            "actionability": self.actionability,
            "public_importance": self.public_importance,
            "personal_relevance": self.personal_relevance,
            "ranking_reason": self.ranking_reason,
            "origin_system": self.origin_system,
            "origin_feed": self.origin_feed,
            "freshrss_categories": self.freshrss_categories,
            "quality_score": self.quality_score,
            "quality_flags": self.quality_flags or [],
            "noise_flags": self.noise_flags or [],
            "body_fetch_status": self.body_fetch_status,
            "fetched_url": self.fetched_url,
            "body_length": self.body_length,
            "duplicate_item_ids": self.duplicate_item_ids or [],
            "related_item_ids": self.related_item_ids or [],
            "content_language": self.content_language,
            "raw_excerpt": self.raw_excerpt,
            "rank_score": round(self.rank_score, 2),
        }


def collapse_ws(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


def markdown_escape(text: str) -> str:
    return text.replace("[", "\\[").replace("]", "\\]")


def canonical_link(item: dict[str, Any]) -> str:
    for key in ("canonical", "alternate"):
        values = item.get(key) or []
        if values and isinstance(values, list):
            first = values[0]
            if isinstance(first, dict) and first.get("href"):
                return str(first["href"])
    return ""


def parse_published(ts: int) -> str:
    return datetime.fromtimestamp(ts, tz=timezone.utc).astimezone().isoformat()


def normalize_url(url: str) -> str:
    if not url:
        return ""
    parsed = urllib.parse.urlsplit(url.strip())
    scheme = (parsed.scheme or "https").lower()
    host = parsed.hostname.lower() if parsed.hostname else ""
    if not host:
        return url.strip()
    port = parsed.port
    if port and not ((scheme == "http" and port == 80) or (scheme == "https" and port == 443)):
        netloc = f"{host}:{port}"
    else:
        netloc = host
    path = parsed.path or "/"
    path = re.sub(r"/{2,}", "/", path)
    if path != "/" and path.endswith("/"):
        path = path[:-1]
    query_pairs = urllib.parse.parse_qsl(parsed.query, keep_blank_values=False)
    filtered_pairs = []
    for key, value in query_pairs:
        lowered = key.lower()
        if lowered.startswith("utm_") or lowered in TRACKING_PARAMS:
            continue
        filtered_pairs.append((key, value))
    filtered_pairs.sort()
    query = urllib.parse.urlencode(filtered_pairs, doseq=True)
    return urllib.parse.urlunsplit((scheme, netloc, path, query, ""))


def normalized_title(text: str) -> str:
    lowered = text.lower()
    lowered = re.sub(r"https?://\S+", "", lowered)
    lowered = re.sub(r"[^0-9a-z\u4e00-\u9fff]+", " ", lowered)
    return collapse_ws(lowered)


def build_tokens(text: str) -> set[str]:
    tokens: set[str] = set()
    for token in re.findall(r"[a-z0-9]{2,}|[\u4e00-\u9fff]{2,}", text.lower()):
        if re.search(r"[\u4e00-\u9fff]", token):
            tokens.add(token)
            if len(token) <= 8:
                for index in range(len(token) - 1):
                    tokens.add(token[index : index + 2])
        else:
            if token.endswith("s") and len(token) > 4:
                [REDACTED_SECRET]
            tokens.add(token)
    return {token for token in tokens if token}


def detect_language(text: str) -> str:
    zh_count = len(re.findall(r"[\u4e00-\u9fff]", text))
    latin_count = len(re.findall(r"[A-Za-z]", text))
    if zh_count == 0 and latin_count == 0:
        return "unknown"
    if zh_count * 2 >= max(1, latin_count):
        return "zh"
    if latin_count * 2 >= max(1, zh_count):
        return "en"
    return "mixed"


def keyword_in_text(text: str, keyword: str) -> bool:
    if re.search(r"[\u4e00-\u9fff]", keyword):
        return keyword in text
    return re.search(rf"\b{re.escape(keyword.lower())}\b", text) is not None


def html_to_text(text: str) -> tuple[str, str, str]:
    parser = BlockTextExtractor()
    parser.feed(text)
    parser.close()
    joined = "\n".join(line.strip() for line in "".join(parser.parts).splitlines())
    lines: list[str] = []
    for line in joined.splitlines():
        line = collapse_ws(line)
        if not line:
            continue
        lowered = line.lower()
        if any(token in lowered for token in ("cookie", "privacy policy", "accept all", "advertisement")):
            continue
        lines.append(line)
    deduped: list[str] = []
    seen: set[str] = set()
    for line in lines:
        if line in seen:
            continue
        seen.add(line)
        deduped.append(line)
    body = "\n".join(deduped)
    title = collapse_ws(" ".join(parser.title_parts)) or collapse_ws(parser.meta.get("og:title", ""))
    description = collapse_ws(
        parser.meta.get("description", "")
        or parser.meta.get("og:description", "")
        or parser.meta.get("twitter:description", "")
    )
    return body, title, description


def clean_excerpt(text: str, limit: int = 280) -> str:
    cleaned = collapse_ws(text)
    if len(cleaned) <= limit:
        return cleaned
    return cleaned[: limit - 3].rstrip() + "..."


def item_source(item: dict[str, Any]) -> str:
    origin = item.get("origin") or {}
    return str(origin.get("title") or origin.get("streamId") or "Unknown")


def source_weight(source: str, url: str) -> float:
    lowered = source.lower()
    domain = urllib.parse.urlsplit(url).hostname or ""
    haystack = f"{lowered} {domain.lower()}"
    for key, value in SOURCE_WEIGHTS.items():
        if keyword_in_text(haystack, key):
            return value
    return 1.0


def parse_candidates(items: list[dict[str, Any]]) -> list[Candidate]:
    parsed: list[Candidate] = []
    for item in items:
        raw_url = canonical_link(item)
        normalized = normalize_url(raw_url)
        raw_summary = item.get("summary") or {}
        summary_html = raw_summary.get("content", "") if isinstance(raw_summary, dict) else ""
        summary_text = clean_excerpt(html_to_text(summary_html)[0], limit=1200)
        title = str(item.get("title") or "(Untitled)")
        published_ts = int(item.get("published") or 0)
        source = item_source(item)
        categories = [str(value) for value in item.get("categories") or []]
        parsed.append(
            Candidate(
                item_id=str(item.get("id") or ""),
                title=title,
                raw_url=raw_url,
                normalized_url=normalized or normalized_title(title),
                source=source,
                origin_feed=source,
                origin_system="freshrss",
                published_at=parse_published(published_ts) if published_ts else "",
                published_ts=published_ts,
                freshrss_categories=categories,
                summary_html=summary_html,
                summary_text=summary_text,
                source_weight=source_weight(source, raw_url),
                title_tokens=build_tokens(normalized_title(title)),
                raw_excerpt=clean_excerpt(summary_text or title, limit=220),
                quality_flags=[],
                noise_flags=[],
                duplicate_item_ids=[],
                related_item_ids=[],
                topic_tags=[],
                risk_tags=[],
            )
        )
    return parsed


def dedupe_exact_urls(candidates: list[Candidate]) -> tuple[list[Candidate], list[dict[str, Any]]]:
    buckets: dict[str, list[Candidate]] = {}
    for candidate in candidates:
        key = candidate.normalized_url or normalized_title(candidate.title)
        buckets.setdefault(key, []).append(candidate)

    kept: list[Candidate] = []
    removed: list[dict[str, Any]] = []
    for key, grouped in buckets.items():
        grouped.sort(
            key=lambda item: (
                item.source_weight,
                len(item.summary_text),
                item.published_ts,
            ),
            reverse=True,
        )
        primary = grouped[0]
        primary.duplicate_item_ids = [item.item_id for item in grouped[1:]]
        kept.append(primary)
        for duplicate in grouped[1:]:
            removed.append(
                {
                    "item_id": duplicate.item_id,
                    "title": duplicate.title,
                    "source": duplicate.source,
                    "raw_url": duplicate.raw_url,
                    "normalized_url": key,
                    "reason": "exact_url_duplicate",
                    "kept_item_id": primary.item_id,
                }
            )
    return kept, removed


def fetch_url(url: str, timeout: int) -> tuple[str, str, str]:
    request = urllib.request.Request(
        url,
        headers={
            "User-Agent": (
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0 Safari/537.36"
            )
        },
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        content_type = response.headers.get("Content-Type", "")
        if "text/html" not in content_type and "application/xhtml+xml" not in content_type:
            raise ValueError(f"Unsupported content type: {content_type}")
        payload = response.read(1_500_000).decode("utf-8", errors="replace")
        return payload, response.geturl(), content_type


def enrich_candidate(candidate: Candidate, timeout: int) -> Candidate:
    if not candidate.raw_url:
        candidate.body_fetch_status = "missing_url"
        candidate.body_fetch_error = "missing canonical url"
        candidate.body_text = candidate.summary_text
        candidate.body_length = len(candidate.body_text)
        candidate.content_language = detect_language(candidate.summary_text)
        return candidate

    try:
        html_text, final_url, _ = fetch_url(candidate.raw_url, timeout=timeout)
        body_text, content_title, description = html_to_text(html_text)
        host = urllib.parse.urlsplit(final_url or candidate.raw_url).hostname or ""
        if "github.com" in host and description:
            generic_markers = ("appearance settings", "github copilot", "sign in", "navigation menu")
            if not body_text or any(marker in body_text.lower() for marker in generic_markers):
                body_text = description
        candidate.body_text = body_text
        candidate.body_length = len(body_text)
        candidate.fetched_url = final_url
        candidate.content_title = content_title
        candidate.meta_description = description
        candidate.body_fetch_status = "fetched" if body_text else "empty_body"
        candidate.content_language = detect_language(
            "\n".join(part for part in (body_text[:4000], candidate.summary_text, candidate.title) if part)
        )
    except urllib.error.HTTPError as exc:
        candidate.body_fetch_status = "http_error"
        candidate.body_fetch_error = f"http_{exc.code}"
        candidate.body_text = candidate.summary_text
        candidate.body_length = len(candidate.body_text)
        candidate.content_language = detect_language(candidate.summary_text)
    except urllib.error.URLError as exc:
        candidate.body_fetch_status = "network_error"
        candidate.body_fetch_error = str(exc.reason)
        candidate.body_text = candidate.summary_text
        candidate.body_length = len(candidate.body_text)
        candidate.content_language = detect_language(candidate.summary_text)
    except Exception as exc:  # noqa: BLE001
        candidate.body_fetch_status = "fetch_error"
        candidate.body_fetch_error = str(exc)
        candidate.body_text = candidate.summary_text
        candidate.body_length = len(candidate.body_text)
        candidate.content_language = detect_language(candidate.summary_text)

    if not candidate.body_text:
        candidate.body_text = candidate.summary_text
        candidate.body_length = len(candidate.body_text)

    return candidate


def fetch_bodies(candidates: list[Candidate], timeout: int, concurrency: int) -> None:
    if not candidates:
        return
    workers = max(1, min(concurrency, len(candidates)))
    with ThreadPoolExecutor(max_workers=workers) as executor:
        futures = {executor.submit(enrich_candidate, candidate, timeout): candidate for candidate in candidates}
        for future in as_completed(futures):
            future.result()


def marketing_hits(text: str) -> int:
    lowered = text.lower()
    return sum(1 for pattern in MARKETING_PATTERNS if pattern.lower() in lowered)


def accessibility_hits(text: str) -> int:
    lowered = text.lower()
    return sum(1 for pattern in ACCESS_PATTERNS if pattern in lowered)


def quality_check(candidate: Candidate) -> None:
    score = 40.0
    flags: list[str] = []
    noise_flags: list[str] = []
    text = candidate.body_text or candidate.summary_text
    excerpt = text[:2400]
    body_length = len(text)
    url_host = urllib.parse.urlsplit(candidate.raw_url or candidate.fetched_url).hostname or ""

    if candidate.body_fetch_status == "fetched":
        score += 18
    elif candidate.body_fetch_status in {"network_error", "http_error", "fetch_error", "empty_body"}:
        flags.append("body_fetch_failed")
        score -= 14

    if body_length >= 4000:
        score += 20
    elif body_length >= 1800:
        score += 14
    elif body_length >= 800:
        score += 8
    elif body_length >= 240:
        score += 2
    else:
        flags.append("short_body")
        score -= 18

    if candidate.summary_text and len(candidate.summary_text) >= 80:
        score += 6
    else:
        flags.append("weak_feed_summary")
        score -= 4

    if candidate.content_language == "en":
        flags.append("needs_manual_translation")

    marketing = marketing_hits("\n".join([candidate.title, candidate.summary_text, excerpt[:1200]]))
    if marketing >= 4:
        noise_flags.append("marketing_heavy")
        score -= 30
    elif marketing >= 2:
        noise_flags.append("marketing_signal")
        score -= 12

    access_issues = accessibility_hits(excerpt)
    if access_issues >= 1 and body_length < 500:
        flags.append("body_access_limited")
        score -= 12

    title_lowered = candidate.title.lower()
    if any(pattern in title_lowered for pattern in LOW_SIGNAL_PATTERNS):
        noise_flags.append("low_signal_title")
        score -= 20

    if re.search(r"招聘|hiring|job[s]?\b|remote work|远程工作|兼职", title_lowered):
        noise_flags.append("recruitment_post")
        score -= 26

    if candidate.summary_text.lower() == "comments" and (body_length < 500 or "news.ycombinator.com" in url_host):
        noise_flags.append("comments_only")
        score -= 22

    if "v2ex" in candidate.source.lower() and body_length < 180:
        noise_flags.append("short_forum_post")
        score -= 16

    unique_ratio = len(set(excerpt.split())) / max(1, len(excerpt.split()))
    if unique_ratio < 0.35 and len(excerpt.split()) > 40:
        flags.append("repetitive_content")
        score -= 8

    candidate.quality_score = max(0, min(100, int(round(score))))
    candidate.quality_flags = flags
    candidate.noise_flags = noise_flags
    candidate.quality_passed = candidate.quality_score >= 35 and "marketing_heavy" not in noise_flags


def filter_noise(candidates: list[Candidate]) -> tuple[list[Candidate], list[dict[str, Any]]]:
    kept: list[Candidate] = []
    removed: list[dict[str, Any]] = []
    for candidate in candidates:
        reason = ""
        if not candidate.quality_passed:
            reason = "quality_below_threshold"
        elif "comments_only" in (candidate.noise_flags or []):
            reason = "comments_only"
        elif "marketing_heavy" in (candidate.noise_flags or []):
            reason = "marketing_heavy"
        elif "recruitment_post" in (candidate.noise_flags or []):
            reason = "recruitment_post"
        elif "short_forum_post" in (candidate.noise_flags or []) and candidate.quality_score < 48:
            reason = "short_forum_noise"

        if reason:
            candidate.filtered_reason = reason
            removed.append(
                {
                    "item_id": candidate.item_id,
                    "title": candidate.title,
                    "source": candidate.source,
                    "raw_url": candidate.raw_url,
                    "reason": reason,
                    "quality_score": candidate.quality_score,
                    "quality_flags": candidate.quality_flags or [],
                    "noise_flags": candidate.noise_flags or [],
                }
            )
            continue
        kept.append(candidate)
    return kept, removed


def jaccard_similarity(left: set[str], right: set[str]) -> float:
    if not left or not right:
        return 0.0
    intersection = len(left & right)
    union = len(left | right)
    return intersection / union if union else 0.0


def build_similarity_tokens(candidate: Candidate) -> set[str]:
    combined = "\n".join(
        part
        for part in (
            normalized_title(candidate.title),
            candidate.summary_text[:500],
            candidate.body_text[:1200],
        )
        if part
    )
    return build_tokens(combined)


def similarity_score(left: Candidate, right: Candidate, left_tokens: set[str], right_tokens: set[str]) -> float:
    title_left = normalized_title(left.title)
    title_right = normalized_title(right.title)
    title_ratio = SequenceMatcher(None, title_left, title_right).ratio()
    token_ratio = jaccard_similarity(left_tokens, right_tokens)
    url_ratio = 1.0 if left.normalized_url == right.normalized_url else 0.0
    same_source_window = abs(left.published_ts - right.published_ts) <= 3 * 24 * 3600

    if url_ratio == 1.0:
        return 1.0
    if title_left and title_left == title_right:
        return 0.98
    if same_source_window and title_ratio >= 0.90:
        return 0.92
    if same_source_window and title_ratio >= 0.82 and token_ratio >= 0.45:
        return 0.86
    if same_source_window and title_ratio >= 0.60 and token_ratio >= 0.70:
        return token_ratio
    if same_source_window and title_ratio >= 0.72 and token_ratio >= 0.32:
        return 0.84
    return title_ratio * 0.6 + token_ratio * 0.4


def cluster_similar_candidates(candidates: list[Candidate]) -> tuple[list[Candidate], list[dict[str, Any]]]:
    if not candidates:
        return [], []

    tokens_map = {candidate.item_id: build_similarity_tokens(candidate) for candidate in candidates}
    parent = {candidate.item_id: candidate.item_id for candidate in candidates}

    def find(item_id: str) -> str:
        while parent[item_id] != item_id:
            parent[item_id] = parent[parent[item_id]]
            item_id = parent[item_id]
        return item_id

    def union(left_id: str, right_id: str) -> None:
        left_root = find(left_id)
        right_root = find(right_id)
        if left_root != right_root:
            parent[right_root] = left_root

    for index, left in enumerate(candidates):
        for right in candidates[index + 1 :]:
            similarity = similarity_score(left, right, tokens_map[left.item_id], tokens_map[right.item_id])
            if similarity >= 0.84:
                union(left.item_id, right.item_id)

    grouped: dict[str, list[Candidate]] = {}
    for candidate in candidates:
        grouped.setdefault(find(candidate.item_id), []).append(candidate)

    kept: list[Candidate] = []
    removed: list[dict[str, Any]] = []
    cluster_number = 0
    for grouped_candidates in grouped.values():
        cluster_number += 1
        grouped_candidates.sort(
            key=lambda item: (
                item.quality_score,
                item.source_weight,
                item.body_length,
                len(item.summary_text),
            ),
            reverse=True,
        )
        primary = grouped_candidates[0]
        primary.cluster_id = f"cluster-{cluster_number:03d}"
        primary.related_item_ids = [item.item_id for item in grouped_candidates[1:]]
        kept.append(primary)
        for duplicate in grouped_candidates[1:]:
            removed.append(
                {
                    "item_id": duplicate.item_id,
                    "title": duplicate.title,
                    "source": duplicate.source,
                    "raw_url": duplicate.raw_url,
                    "reason": "similar_content_duplicate",
                    "kept_item_id": primary.item_id,
                    "cluster_id": primary.cluster_id,
                }
            )
    return kept, removed


def split_sentences(text: str) -> list[str]:
    normalized = re.sub(r"\n+", "\n", text)
    pieces = re.split(r"(?<=[。！？!?；;])\s+|(?<=[.?!])\s+(?=[A-Z0-9])|\n", normalized)
    sentences: list[str] = []
    for piece in pieces:
        cleaned = collapse_ws(piece)
        if len(cleaned) < 18:
            continue
        if len(cleaned) > 420:
            continue
        sentences.append(cleaned)
    return sentences


def score_sentence(sentence: str, title_tokens: set[str], index: int) -> float:
    sentence_tokens = build_tokens(sentence)
    overlap = len(sentence_tokens & title_tokens)
    numeric_bonus = 1.5 if re.search(r"\d", sentence) else 0.0
    keyword_bonus = 1.2 if re.search(r"风险|发布|推出|融资|漏洞|升级|supports|introduces|announced", sentence, re.I) else 0.0
    position_bonus = max(0.0, 3.0 - index * 0.35)
    length_penalty = 0.0 if len(sentence) <= 180 else -1.5
    return overlap * 2.4 + numeric_bonus + keyword_bonus + position_bonus + length_penalty


def choose_summary_sentences(candidate: Candidate, limit: int = 3) -> list[str]:
    source_text = candidate.body_text if len(candidate.body_text) >= 240 else candidate.summary_text
    sentences = split_sentences(source_text)
    if not sentences and candidate.summary_text:
        sentences = split_sentences(candidate.summary_text)
    if not sentences:
        fallback = collapse_ws(candidate.summary_text or candidate.title)
        return [fallback] if fallback else []

    scored = [
        (score_sentence(sentence, candidate.title_tokens, index), index, sentence)
        for index, sentence in enumerate(sentences[:40])
    ]
    scored.sort(reverse=True)
    chosen: list[str] = []
    chosen_tokens: list[set[str]] = []
    title_norm = normalized_title(candidate.title)
    for _, _, sentence in scored:
        if normalized_title(sentence) == title_norm and len(sentences) > 1:
            continue
        sentence_tokens = build_tokens(sentence)
        if any(jaccard_similarity(sentence_tokens, existing) >= 0.78 for existing in chosen_tokens):
            continue
        chosen.append(sentence)
        chosen_tokens.append(sentence_tokens)
        if len(chosen) >= limit:
            break
    return chosen or [sentences[0]]


def finalize_summary_text(title: str, sentences: list[str], max_chars: int = 220) -> str:
    cleaned_sentences: list[str] = []
    normalized = normalized_title(title)
    for sentence in sentences:
        current = collapse_ws(sentence)
        if not current:
            continue
        if normalized and normalized_title(current).startswith(normalized):
            current = current[len(title) :].strip(" ：:,-")
        if current and current not in cleaned_sentences:
            cleaned_sentences.append(current)

    parts: list[str] = []
    current_length = 0
    for sentence in cleaned_sentences:
        extra = len(sentence) + (1 if parts else 0)
        if current_length + extra > max_chars:
            break
        parts.append(sentence)
        current_length += extra

    summary = " ".join(parts).strip()
    return clean_excerpt(summary or " ".join(cleaned_sentences) or title, limit=max_chars)


def infer_topic_tags(candidate: Candidate) -> list[str]:
    haystack = "\n".join([candidate.title, candidate.summary_text, candidate.body_text[:1500]]).lower()
    tags = [tag for tag, keywords in TOPIC_RULES.items() if any(keyword_in_text(haystack, keyword.lower()) for keyword in keywords)]
    return tags or ["其他"]


def infer_risk_tags(candidate: Candidate) -> list[str]:
    haystack = "\n".join([candidate.title, candidate.summary_text, candidate.body_text[:1200]]).lower()
    return [tag for tag, keywords in RISK_RULES.items() if any(keyword_in_text(haystack, keyword.lower()) for keyword in keywords)]


def infer_actionability(candidate: Candidate) -> str:
    text = "\n".join([candidate.title, candidate.summary_text, candidate.body_text[:800]]).lower()
    if re.search(r"发布|上线|教程|实战|how to|guide|skill|workflow|release|sdk|api", text):
        return "high"
    if re.search(r"融资|离职|ipo|research|study|政策|趋势|治理", text):
        return "medium"
    return "low"


def infer_public_importance(candidate: Candidate) -> int:
    text = "\n".join([candidate.title, candidate.summary_text]).lower()
    score = 1
    if re.search(r"openai|anthropic|apple|meta|华为|软银|microsoft|google", text):
        score += 2
    if re.search(r"安全|漏洞|风险|attack|exploit|监管|policy|融资|ipo", text):
        score += 1
    if "Readhub" in candidate.source or "InfoQ" in candidate.source:
        score += 1
    return min(5, score)


def infer_personal_relevance(candidate: Candidate) -> int:
    text = "\n".join([candidate.title, candidate.summary_text, candidate.body_text[:800]]).lower()
    score = 1
    if re.search(r"agent|skill|workflow|devops|open source|开源|自动化|infra|tool|安全", text):
        score += 2
    if re.search(r"codex|claude|copilot|freshrss|知识库|笔记", text):
        score += 1
    return min(5, score)


def generate_summaries(candidate: Candidate) -> None:
    sentences = choose_summary_sentences(candidate)
    max_chars = 220 if candidate.content_language in {"zh", "mixed"} else 260
    candidate.summary = finalize_summary_text(candidate.title, sentences, max_chars=max_chars)
    if normalized_title(candidate.summary) == normalized_title(candidate.title) and candidate.summary_text:
        candidate.summary = clean_excerpt(candidate.summary_text, limit=max_chars)

    summary_language = detect_language(candidate.summary or candidate.summary_text)
    if summary_language in {"zh", "mixed"}:
        candidate.summary_zh = candidate.summary
        return

    if detect_language(candidate.summary_text) in {"zh", "mixed"}:
        candidate.summary_zh = finalize_summary_text(candidate.title, choose_summary_sentences(candidate, limit=2), max_chars=220)
        return

    wrapped = finalize_summary_text(candidate.title, sentences[:2], max_chars=160)
    candidate.summary_zh = (
        f"英文原文，主题是《{candidate.title}》。"
        f"核心信息主要集中在：{wrapped}"
    ).strip()
    if "needs_manual_translation" not in (candidate.quality_flags or []):
        (candidate.quality_flags or []).append("needs_manual_translation")


def rank_candidate(candidate: Candidate) -> None:
    candidate.topic_tags = infer_topic_tags(candidate)
    candidate.risk_tags = infer_risk_tags(candidate)
    candidate.actionability = infer_actionability(candidate)
    candidate.public_importance = infer_public_importance(candidate)
    candidate.personal_relevance = infer_personal_relevance(candidate)

    actionability_bonus = {"high": 8.0, "medium": 4.0, "low": 1.0}[candidate.actionability]
    risk_bonus = 4.0 if candidate.risk_tags else 0.0
    cluster_bonus = min(6.0, len(candidate.related_item_ids or []) * 1.2)
    source_bonus = candidate.source_weight * 10.0
    importance_bonus = candidate.public_importance * 5.0
    relevance_bonus = candidate.personal_relevance * 3.0
    length_bonus = min(6.0, math.log(max(candidate.body_length, 120), 10) * 2.5)
    candidate.rank_score = (
        candidate.quality_score
        + source_bonus
        + actionability_bonus
        + risk_bonus
        + cluster_bonus
        + importance_bonus
        + relevance_bonus
        + length_bonus
    )
    reason_parts = [
        f"quality={candidate.quality_score}",
        f"source_weight={candidate.source_weight:.2f}",
        f"public_importance={candidate.public_importance}",
        f"personal_relevance={candidate.personal_relevance}",
        f"actionability={candidate.actionability}",
    ]
    if candidate.related_item_ids:
        reason_parts.append(f"cluster_refs={len(candidate.related_item_ids)}")
    if candidate.risk_tags:
        reason_parts.append(f"risk={','.join(candidate.risk_tags)}")
    candidate.ranking_reason = "; ".join(reason_parts)


def build_markdown(result: dict[str, Any]) -> str:
    lines = [
        f"# Digest | {result['window']['label']}",
        "",
        "## Overview",
        "",
        f"- Input items: `{result['stats']['input_items']}`",
        f"- After URL dedup: `{result['stats']['after_url_dedup']}`",
        f"- After noise filter: `{result['stats']['after_noise_filter']}`",
        f"- Final digest items: `{result['stats']['final_items']}`",
        "",
        "## Ranked Items",
        "",
    ]

    for index, item in enumerate(result["items"], start=1):
        lines.extend(
            [
                f"### {index}. {markdown_escape(item['title'])}",
                "",
                f"- Source: **{item['source']}**",
                f"- Published: `{item['published_at']}`",
                f"- Link: [{markdown_escape(item['title'])}]({item['raw_url']})" if item["raw_url"] else "- Link: N/A",
                f"- Score: **{item['rank_score']}**",
                f"- Topics: {', '.join(item['topic_tags']) or 'N/A'}",
                f"- Risks: {', '.join(item['risk_tags']) if item['risk_tags'] else 'None'}",
                f"- Actionability: **{item['actionability']}**",
                f"- Summary: {item['summary_zh'] or item['summary'] or 'N/A'}",
                f"- Ranking reason: `{item['ranking_reason']}`",
                "",
            ]
        )
        if item["duplicate_item_ids"] or item["related_item_ids"]:
            refs = item["duplicate_item_ids"] + item["related_item_ids"]
            lines.append(f"- Cluster refs: `{len(refs)}` related items")
            lines.append("")

    if result["filtered"]:
        lines.extend(["## Filtered Items", ""])
        for entry in result["filtered"][:30]:
            lines.append(f"- `{entry['reason']}` | {entry['source']} | {entry['title']}")
        lines.append("")
    return "\n".join(lines)


def build_result(
    input_payload: dict[str, Any],
    input_items: list[Candidate],
    url_dedup_removed: list[dict[str, Any]],
    filtered_removed: list[dict[str, Any]],
    similar_removed: list[dict[str, Any]],
    final_items: list[Candidate],
) -> dict[str, Any]:
    label = input_payload.get("target_date") or input_payload.get("label") or "unknown-window"
    return {
        "generated_at": datetime.now().astimezone().isoformat(),
        "window": {
            "label": label,
            "target_date": input_payload.get("target_date"),
            "timezone": input_payload.get("timezone"),
        },
        "stats": {
            "input_items": len(input_items),
            "after_url_dedup": len(input_items) - len(url_dedup_removed),
            "after_noise_filter": len(input_items) - len(url_dedup_removed) - len(filtered_removed),
            "final_items": len(final_items),
            "url_duplicates_removed": len(url_dedup_removed),
            "filtered_removed": len(filtered_removed),
            "similar_duplicates_removed": len(similar_removed),
        },
        "items": [candidate.as_output() for candidate in final_items],
        "filtered": filtered_removed + url_dedup_removed + similar_removed,
    }


def load_source_payload(path: Path) -> dict[str, Any]:
    payload = json.loads(path.read_text(encoding="utf-8"))
    items = payload.get("items")
    if not isinstance(items, list):
        raise SystemExit("Source JSON missing items list")
    return payload


def main() -> int:
    parser = argparse.ArgumentParser(description="Build a structured digest from FreshRSS items.")
    parser.add_argument("--source-json", required=True, help="Path to sliced or raw FreshRSS JSON.")
    parser.add_argument("--output-json", required=True, help="Structured digest JSON output.")
    parser.add_argument("--output-md", required=True, help="Structured digest Markdown output.")
    parser.add_argument("--timeout", type=int, default=8, help="Per-page fetch timeout in seconds.")
    parser.add_argument("--concurrency", type=int, default=8, help="Concurrent fetch workers.")
    args = parser.parse_args()

    source_json = Path(args.source_json)
    output_json = Path(args.output_json)
    output_md = Path(args.output_md)

    if not source_json.exists():
        raise SystemExit(f"Source JSON not found: {source_json}")

    input_payload = load_source_payload(source_json)
    raw_items = parse_candidates(input_payload["items"])

    url_deduped, url_removed = dedupe_exact_urls(raw_items)
    fetch_bodies(url_deduped, timeout=args.timeout, concurrency=args.concurrency)

    for candidate in url_deduped:
        quality_check(candidate)

    filtered_candidates, filtered_removed = filter_noise(url_deduped)
    clustered_candidates, similar_removed = cluster_similar_candidates(filtered_candidates)

    for candidate in clustered_candidates:
        generate_summaries(candidate)
        rank_candidate(candidate)

    clustered_candidates.sort(key=lambda item: item.rank_score, reverse=True)

    result = build_result(
        input_payload=input_payload,
        input_items=raw_items,
        url_dedup_removed=url_removed,
        filtered_removed=filtered_removed,
        similar_removed=similar_removed,
        final_items=clustered_candidates,
    )

    output_json.parent.mkdir(parents=True, exist_ok=True)
    output_md.parent.mkdir(parents=True, exist_ok=True)
    output_json.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
    output_md.write_text(build_markdown(result), encoding="utf-8")

    print(f"input_items={result['stats']['input_items']}")
    print(f"after_url_dedup={result['stats']['after_url_dedup']}")
    print(f"after_noise_filter={result['stats']['after_noise_filter']}")
    print(f"final_items={result['stats']['final_items']}")
    print(f"json={output_json}")
    print(f"md={output_md}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/fetch_freshrss_unread.py
content: #!/usr/bin/env python3
"""Fetch unread items from FreshRSS Google Reader API."""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
from typing import Any


def load_config(config_path: str | None) -> dict[str, Any]:
    config: dict[str, Any] = {}
    if config_path:
        with open(config_path, "r", encoding="utf-8") as fh:
            config = json.load(fh)

    env_map = {
        "base_url": "FRESHRSS_BASE_URL",
        "username": "FRESHRSS_USERNAME",
        "api_password": "FRESHRSS_API_PASSWORD",
        "output_json": "FRESHRSS_OUTPUT_JSON",
        "output_md": "FRESHRSS_OUTPUT_MD",
        "limit": "FRESHRSS_LIMIT",
        "page_size": "FRESHRSS_PAGE_SIZE",
        "include_read": "FRESHRSS_INCLUDE_READ",
    }

    for key, env_name in env_map.items():
        value = os.getenv(env_name)
        if value not in (None, ""):
            config[key] = value

    if "limit" in config:
        config["limit"] = int(config["limit"])
    if "page_size" in config:
        config["page_size"] = int(config["page_size"])
    if "include_read" in config:
        include_read = config["include_read"]
        if not isinstance(include_read, bool):
            config["include_read"] = str(include_read).lower() in {"1", "true", "yes", "on"}

    required = ["base_url", "username", "api_password"]
    missing = [key for key in required if not config.get(key)]
    if missing:
        missing_text = ", ".join(missing)
        raise SystemExit(f"Missing FreshRSS config: {missing_text}")

    config.setdefault("output_json", "digests/raw-freshrss.json")
    config.setdefault("output_md", "digests/raw-freshrss.md")
    config.setdefault("limit", 0)
    config.setdefault("page_size", 1000)
    config.setdefault("include_read", False)

    return config


def normalize_base_url(base_url: str) -> str:
    return base_url.rstrip("/")


def client_login(base_url: str, username: str, api_password: str) -> str:
    login_url = f"{normalize_base_url(base_url)}/api/greader.php/accounts/ClientLogin"
    payload = urllib.parse.urlencode({"Email": username, "Passwd": api_password}).encode("utf-8")
    request = urllib.request.Request(login_url, data=payload, method="POST")

    try:
        with urllib.request.urlopen(request, timeout=30) as response:
            content = response.read().decode("utf-8", errors="replace")
    except urllib.error.HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise SystemExit(f"FreshRSS login failed: HTTP {exc.code} {body}") from exc
    except urllib.error.URLError as exc:
        raise SystemExit(f"FreshRSS login failed: {exc}") from exc

    for line in content.splitlines():
        if line.startswith("Auth="):
            return line.split("=", 1)[1].strip()

    raise SystemExit("FreshRSS login failed: missing Auth token in response")


def fetch_stream_page(
    base_url: str,
    auth_token: str,
    include_read: bool,
    page_size: int,
    continuation: str | None,
) -> dict[str, Any]:
    params = {"output": "json", "n": str(page_size)}
    if not include_read:
        params["xt"] = "user/-/state/com.google/read"
    if continuation:
        params["c"] = continuation

    query = urllib.parse.urlencode(params)
    stream_url = (
        f"{normalize_base_url(base_url)}/api/greader.php/reader/api/0/stream/contents/"
        f"reading-list?{query}"
    )

    request = urllib.request.Request(
        stream_url,
        headers={"Authorization": f"GoogleLogin auth={auth_token}"},
        method="GET",
    )

    try:
        with urllib.request.urlopen(request, timeout=30) as response:
            return json.loads(response.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise SystemExit(f"FreshRSS unread fetch failed: HTTP {exc.code} {body}") from exc
    except urllib.error.URLError as exc:
        raise SystemExit(f"FreshRSS unread fetch failed: {exc}") from exc


def fetch_unread_items(
    base_url: str,
    auth_token: str,
    include_read: bool,
    page_size: int,
    limit: int,
) -> dict[str, Any]:
    all_items: list[dict[str, Any]] = []
    continuation: str | None = None
    first_payload: dict[str, Any] | None = None

    while True:
        payload = fetch_stream_page(base_url, auth_token, include_read, page_size, continuation)
        if first_payload is None:
            first_payload = payload

        items = payload.get("items", [])
        if not isinstance(items, list):
            raise SystemExit("FreshRSS unread fetch failed: response missing items list")

        all_items.extend(items)
        if limit > 0 and len(all_items) >= limit:
            all_items = all_items[:limit]
            break

        continuation = payload.get("continuation")
        if not continuation or not items:
            break

    if first_payload is None:
        first_payload = {"items": []}

    first_payload["items"] = all_items
    if continuation:
        first_payload["continuation"] = continuation
    return first_payload


def build_markdown(items: list[dict[str, Any]], source_url: str) -> str:
    lines = [
        "# FreshRSS 未读列表",
        "",
        f"- 来源：`{source_url}`",
        f"- 条目数：`{len(items)}`",
        "",
    ]

    if not items:
        lines.append("当前没有未读条目。")
        return "\n".join(lines) + "\n"

    for index, item in enumerate(items, start=1):
        canonical = item.get("canonical", [])
        link = ""
        if canonical and isinstance(canonical, list):
            first = canonical[0]
            if isinstance(first, dict):
                link = first.get("href", "")

        title = item.get("title") or "(无标题)"
        origin = item.get("origin", {}) or {}
        origin_title = origin.get("title") or origin.get("streamId") or "未知来源"
        published = item.get("published") or ""
        summary = ""
        raw_summary = item.get("summary", {}) or {}
        if isinstance(raw_summary, dict):
            summary = raw_summary.get("content", "") or ""
        summary = " ".join(summary.split())
        if len(summary) > 280:
            summary = summary[:277] + "..."

        lines.extend(
            [
                f"## {index}. {title}",
                "",
                f"- 来源：{origin_title}",
                f"- 发布时间：{published}",
                f"- 链接：{link}",
                f"- 条目 ID：{item.get('id', '')}",
                "",
            ]
        )
        if summary:
            lines.append(summary)
            lines.append("")

    return "\n".join(lines)


def ensure_parent(path_str: str) -> Path:
    path = Path(path_str)
    path.parent.mkdir(parents=True, exist_ok=True)
    return path


def main() -> int:
    parser = argparse.ArgumentParser(description="Fetch unread FreshRSS items.")
    parser.add_argument("--config", help="Path to JSON config file.", default=None)
    args = parser.parse_args()

    config = load_config(args.config)
    auth_token = client_login(config["base_url"], config["username"], config["api_password"])
    payload = fetch_unread_items(
        config["base_url"],
        auth_token,
        bool(config["include_read"]),
        int(config["page_size"]),
        int(config["limit"]),
    )

    items = payload.get("items", [])
    if not isinstance(items, list):
        raise SystemExit("FreshRSS unread fetch failed: response missing items list")

    output_json = ensure_parent(config["output_json"])
    output_md = ensure_parent(config["output_md"])

    output_json.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    output_md.write_text(
        build_markdown(items, f"{normalize_base_url(config['base_url'])}/api/greader.php"),
        encoding="utf-8",
    )

    print(f"Fetched {len(items)} FreshRSS items")
    print(f"JSON written to: {output_json}")
    print(f"Markdown written to: {output_md}")
    return 0


if __name__ == "__main__":
    sys.exit(main())


path: scripts/freshrss.config.example.json
content: base_url: https://your-freshrss.example.com
username: your-freshrss-username
api_password: replace-with-api-password
output_json: digests/raw-freshrss.json
output_md: digests/raw-freshrss.md
limit: 0
page_size: 1000
include_read: no

path: scripts/mark_freshrss_read.py
content: #!/usr/bin/env python3
"""Mark processed FreshRSS items as read via Google Reader API."""

from __future__ import annotations

import argparse
import json
import os
import socket
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any


def load_config() -> dict[str, Any]:
    config = {
        "base_url": os.getenv("FRESHRSS_BASE_URL", ""),
        "username": os.getenv("FRESHRSS_USERNAME", ""),
        "api_password": os.getenv("FRESHRSS_API_PASSWORD", ""),
    }
    missing = [key for key, value in config.items() if not value]
    if missing:
        raise SystemExit(f"Missing FreshRSS config: {', '.join(missing)}")
    return config


def normalize_base_url(base_url: str) -> str:
    return base_url.rstrip("/")


def client_login(base_url: str, username: str, api_password: str) -> str:
    login_url = f"{normalize_base_url(base_url)}/api/greader.php/accounts/ClientLogin"
    payload = urllib.parse.urlencode({"Email": username, "Passwd": api_password}).encode("utf-8")
    request = urllib.request.Request(login_url, data=payload, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=30) as response:
            content = response.read().decode("utf-8", errors="replace")
    except urllib.error.HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise SystemExit(f"FreshRSS login failed: HTTP {exc.code} {body}") from exc
    except urllib.error.URLError as exc:
        raise SystemExit(f"FreshRSS login failed: {exc}") from exc

    for line in content.splitlines():
        if line.startswith("Auth="):
            return line.split("=", 1)[1].strip()
    raise SystemExit("FreshRSS login failed: missing Auth token in response")


def parse_timezone(value: str) -> timezone:
    if value in {"Asia/Shanghai", "UTC+08:00", "+08:00"}:
        return timezone(timedelta(hours=8))
    if value in {"UTC", "+00:00"}:
        return timezone.utc
    raise SystemExit(f"Unsupported timezone value: {value}")


def collect_item_ids(source_json: Path, date_str: str | None, tz: timezone) -> list[str]:
    data = json.loads(source_json.read_text(encoding="utf-8"))
    items = data.get("items", [])
    if not isinstance(items, list):
        raise SystemExit("Source JSON is missing items list")

    selected: list[str] = []
    for item in items:
        item_id = item.get("id")
        if not item_id:
            continue
        if date_str:
            published = item.get("published")
            if not published:
                continue
            dt = datetime.fromtimestamp(int(published), tz=timezone.utc).astimezone(tz)
            if dt.strftime("%Y-%m-%d") != date_str:
                continue
        selected.append(item_id)
    return selected


def batched(values: list[str], size: int) -> list[list[str]]:
    return [values[index : index + size] for index in range(0, len(values), size)]


def mark_read_batch(base_url: str, auth_token: str, item_ids: list[str], timeout: int, retries: int) -> None:
    pairs: list[tuple[str, str]] = [("a", "user/-/state/com.google/read"), ("async", "true")]
    for item_id in item_ids:
        pairs.append(("i", item_id))
    payload = urllib.parse.urlencode(pairs).encode("utf-8")
    url = f"{normalize_base_url(base_url)}/api/greader.php/reader/api/0/edit-tag"
    last_error: Exception | None = None
    for attempt in range(1, retries + 1):
        request = urllib.request.Request(
            url,
            data=payload,
            headers={
                "Authorization": f"GoogleLogin auth={auth_token}",
                "Content-Type": "application/x-www-form-urlencoded",
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=timeout) as response:
                body = response.read().decode("utf-8", errors="replace").strip()
            if body != "OK":
                raise SystemExit(f"FreshRSS mark read failed: unexpected response {body!r}")
            return
        except urllib.error.HTTPError as exc:
            body = exc.read().decode("utf-8", errors="replace")
            raise SystemExit(f"FreshRSS mark read failed: HTTP {exc.code} {body}") from exc
        except (urllib.error.URLError, TimeoutError, socket.timeout) as exc:
            last_error = exc
            if attempt == retries:
                break
            time.sleep(min(attempt, 3))
    raise SystemExit(f"FreshRSS mark read failed after {retries} attempts: {last_error}")


def main() -> int:
    parser = argparse.ArgumentParser(description="Mark FreshRSS items as read.")
    parser.add_argument("--source-json", required=True, help="Path to fetched FreshRSS JSON.")
    parser.add_argument("--date", default=None, help="Only mark items from YYYY-MM-DD as read.")
    parser.add_argument("--timezone", default="Asia/Shanghai", help="Timezone for --date filtering.")
    parser.add_argument("--batch-size", type=int, default=1, help="Items per edit-tag request.")
    parser.add_argument("--timeout", type=int, default=60, help="Per-request timeout in seconds.")
    parser.add_argument("--retries", type=int, default=3, help="Retries per batch on transient failures.")
    parser.add_argument("--dry-run", action="store_true", help="Only print selected count.")
    args = parser.parse_args()

    source_json = Path(args.source_json)
    if not source_json.exists():
        raise SystemExit(f"Source JSON not found: {source_json}")

    tz = parse_timezone(args.timezone)
    item_ids = collect_item_ids(source_json, args.date, tz)
    item_ids = list(dict.fromkeys(item_ids))

    print(f"Selected {len(item_ids)} items to mark as read")
    if args.dry_run or not item_ids:
        return 0

    config = load_config()
    auth_token = client_login(config["base_url"], config["username"], config["api_password"])
    for batch in batched(item_ids, args.batch_size):
        mark_read_batch(config["base_url"], auth_token, batch, args.timeout, args.retries)

    print(f"Marked {len(item_ids)} items as read")
    return 0


if __name__ == "__main__":
    sys.exit(main())


path: scripts/slice_freshrss_by_date.py
content: #!/usr/bin/env python3
"""Slice FreshRSS unread items by calendar date."""

from __future__ import annotations

import argparse
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any


def parse_timezone(value: str) -> timezone:
    if value in {"Asia/Shanghai", "UTC+08:00", "+08:00"}:
        return timezone(timedelta(hours=8))
    if value in {"UTC", "+00:00"}:
        return timezone.utc
    raise SystemExit(f"Unsupported timezone value: {value}")


def resolve_target_date(date_str: str | None, offset_days: int, tz: timezone) -> str:
    if date_str:
        return date_str
    now = datetime.now(tz)
    return (now + timedelta(days=offset_days)).strftime("%Y-%m-%d")


def canonical_link(item: dict[str, Any]) -> str:
    canonical = item.get("canonical") or []
    if canonical and isinstance(canonical, list) and isinstance(canonical[0], dict):
        return canonical[0].get("href", "")
    return ""


def item_source(item: dict[str, Any]) -> str:
    origin = item.get("origin") or {}
    return origin.get("title") or origin.get("streamId") or "Unknown"


def build_markdown(items: list[dict[str, Any]], target_date: str, source_json: str) -> str:
    lines = [
        f"# FreshRSS Slice | {target_date}",
        "",
        f"- source_json: `{source_json}`",
        f"- target_date: `{target_date}`",
        f"- count: `{len(items)}`",
        "",
    ]
    for index, item in enumerate(items, start=1):
        title = item.get("title") or "(Untitled)"
        source = item_source(item)
        link = canonical_link(item)
        published = item.get("published") or ""
        lines.extend(
            [
                f"## {index}. {title}",
                "",
                f"- source: {source}",
                f"- published: {published}",
                f"- link: {link}",
                f"- item_id: {item.get('id', '')}",
                "",
            ]
        )
    return "\n".join(lines) + "\n"


def main() -> int:
    parser = argparse.ArgumentParser(description="Slice FreshRSS unread items by date.")
    parser.add_argument("--source-json", required=True, help="Path to raw FreshRSS JSON.")
    parser.add_argument("--output-json", required=True, help="Path to sliced JSON output.")
    parser.add_argument("--output-md", required=True, help="Path to sliced Markdown output.")
    parser.add_argument("--date", default=None, help="Target date in YYYY-MM-DD.")
    parser.add_argument(
        "--offset-days",
        type=int,
        default=-1,
        help="Offset days from now when --date is omitted. Default: -1 (yesterday).",
    )
    parser.add_argument("--timezone", default="Asia/Shanghai", help="Timezone for date slicing.")
    args = parser.parse_args()

    source_json = Path(args.source_json)
    if not source_json.exists():
        raise SystemExit(f"Source JSON not found: {source_json}")

    tz = parse_timezone(args.timezone)
    target_date = resolve_target_date(args.date, args.offset_days, tz)

    raw = json.loads(source_json.read_text(encoding="utf-8"))
    items = raw.get("items", [])
    if not isinstance(items, list):
        raise SystemExit("Source JSON is missing items list")

    sliced: list[dict[str, Any]] = []
    for item in items:
        ts = item.get("published")
        if not ts:
            continue
        dt = datetime.fromtimestamp(int(ts), tz=timezone.utc).astimezone(tz)
        if dt.strftime("%Y-%m-%d") == target_date:
            sliced.append(item)

    result = {
        "target_date": target_date,
        "timezone": args.timezone,
        "count": len(sliced),
        "items": sliced,
    }

    output_json = Path(args.output_json)
    output_md = Path(args.output_md)
    output_json.parent.mkdir(parents=True, exist_ok=True)
    output_md.parent.mkdir(parents=True, exist_ok=True)
    output_json.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
    output_md.write_text(build_markdown(sliced, target_date, str(source_json)), encoding="utf-8")

    print(f"target_date={target_date}")
    print(f"count={len(sliced)}")
    print(f"json={output_json}")
    print(f"md={output_md}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)