Skill: leogoat2004/paper-lark-report (version 1.1.1)

# paper-lark-report 全自动科研论文日报/周报生成 Skill。基于 arXiv API 精准检索 + LLM 语义评分 + 飞书 Wiki 推送。 --- ## 安装 ```bash # 方式一：通过 ClawHub 安装 npx clawhub@latest install leogoat2004/paper-lark-report # 方式二：通过 OpenClaw CLI 安装 openclaw skills install leogoat2004/paper-lark-report ``` --- ## 核心流程 ``` cron --(isolated session)--> run_daily() ├─ build_arxiv_query(research_direction) ├─ fetch_arxiv_papers(query, max_search_results=20) ├─ 去重（processed_ids.json） ├─ fetch_arxiv_details(filtered[:20]) └─ 保存 data/daily_papers.json │ ▼ LLM isolated session ├─ 评分（0-10） ├─ 精选 Top max_daily_papers ├─ 从 full_abstract 提取 motivation + core_innovation（中文） ├─ 写入 data/selected_papers.json ├─ --save-selected ├─ feishu-create-doc skill 创建 Wiki 文档（子节点） └─ --register-doc ``` --- ## 目录结构 ``` paper-lark-report/ ├── SKILL.md ├── config.yaml ├── data/ │ ├── doc_registry.json │ ├── processed_ids.json │ ├── daily_papers.json # 候选论文（LLM 输入） │ ├── selected_papers.json # 精选结果（含中文分析） │ └── doc_result.json # 最近创建文档的 token/url ├── processed_log/ │ └── YYYY-MM-DD.json # 每日归档（供周报聚合） ├── scripts/ │ ├── arxiv_search.py # arXiv API │ ├── paper_lark_report.py # 主入口 │ └── create_feishu_doc.py # 飞书 Wiki 创建（直接 API） └── templates/ ├── daily_report.md # 日报模板（含 Instructions） └── weekly_report.md # 周报模板（含 Instructions） ``` --- ## 配置（config.yaml） | 字段 | 说明 | |------|------| | `feishu_space_id` | Wiki 空间 ID（整数，URL 中提取） | | `feishu_parent_node` | 父节点 token，创建在 paper-lark-report 节点下 | | `research_direction` | 自由文本研究方向描述 | | `max_search_results` | 每日 arXiv 最多获取篇数（默认 20） | | `max_daily_papers` | 日报最多精选篇数（默认 3） | | `arxiv_paper_max_days` | 论文最大天数（默认 7） | | `daily_cron` / `weekly_cron` | cron 表达式（UTC+8） | --- ## arXiv 查询策略 Query 构建规则：`abs:core_term AND (abs:term1 OR abs:term2 OR ...)` - 第一个词作为 AND 核心，其余 OR 扩展 - 识别复合词（multi-agent 等）作为原子单元 - 过滤泛化词（towards/safe/efficient 等） - 最多 8 个词 --- ## 飞书 Wiki API 验证过的要点 ### 节点创建 ``` POST /wiki/v2/spaces/{space_id}/nodes body: { obj_type: "docx", parent_node_token, node_type: "origin", title } 返回: { node_token, obj_token } ``` **注意**：Wiki API 忽略传入的 obj_token，始终创建自己的空文档，必须用返回的 obj_token 写入。 ### 写入可用 block type | block_type | 类型 | 可用 | |-----------|------|------| | 2 | text/paragraph | ✅ | | 3 | heading1 | ✅ | | 4 | heading2 | ✅ | | 5 | heading3 | ✅ | | 25 | divider | ❌ 1770029 | | 27 | callout | ❌ 字段校验严 | | 31 | table | ❌ 参数结构不对 | ### 关键参数 - `space_id`：必须是**整数**，不是字符串 - `parent_node`：父节点 token，文档创建在其下级 - Token 获取：从 `openclaw.json` 的 `channels.feishu.appId/appSecret` 换取 tenant_access_token --- ## CLI ```bash # 日报（cron 触发） python3 scripts/paper_lark_report.py # 周报 python3 scripts/paper_lark_report.py --weekly # LLM 选完论文后 python3 scripts/paper_lark_report.py --save-selected "YYYY-MM-DD" "data/selected_papers.json" python3 scripts/paper_lark_report.py --register-doc "<node_token>" "<obj_token>" "<doc_url>" ```

path: config.yaml
content: # paper-lark-report configuration
# Fill in your values before running.

# ============================================================================
# FEISHU WIKI
# ============================================================================
feishu_space_id: ""          # Wiki space ID (integer)
feishu_parent_node: ""       # Parent node token
feishu_root: ""              # Root node token

# ============================================================================
# RESEARCH DIRECTION
# ============================================================================
# The LLM scorer uses this to evaluate paper relevance.
# Keep it specific and concise for best retrieval precision.
research_direction: ""       # e.g. "multi-agent LLM orchestration"

# ============================================================================
# LIMITS
# ============================================================================
max_search_results: 20      # Max papers to fetch from arXiv per run
max_daily_papers: 3           # Max selected papers per daily report
arxiv_paper_max_days: 7       # Max age of papers (days)

# ============================================================================
# SCHEDULE
# ============================================================================
daily_cron: "0 6 * * 1-5"    # Mon-Fri (UTC 6:00 = BJT 14:00)
weekly_cron: "0 6 * * 0"     # Sunday (UTC 6:00 = BJT 14:00)


path: templates/daily_report.md
content: # INSTRUCTIONS (fill in all fields, do not skip any)

## Research Direction
> {{research_direction}}

## Scoring Criteria
Score each paper 0–10 on relevance to the research direction above:
- 9–10: directly solves a core problem in the direction
- 7–8: highly relevant
- 5–6: tangentially related
- <5: not relevant

## Analysis Requirements (IMPORTANT: write in Chinese)
For each selected paper, extract ONLY from `full_abstract`:
- motivation: 论文解决了什么问题？为什么重要？（2-3句话，用中文）
- core_innovation: 核心贡献是什么？与现有工作有何不同？（2-3句话，用中文）
- Do NOT fabricate details. Only use what is in the abstract.

## Output Steps
1. Write selected papers to `data/selected_papers.json`:
   ```json
   {
     "date": "{{date}}",
     "papers": [
       {
         "paper_id": "2603.12345",
         "title": "...",
         "authors": ["..."],
         "arxiv_url": "https://arxiv.org/abs/2603.12345",
         "pdf_url": "https://arxiv.org/pdf/2603.12345",
         "relevance_score": 9,
         "posted_date": "2026-03-30",
         "full_abstract": "...",
         "motivation": "...",
         "core_innovation": "..."
       }
     ]
   }
   ```
2. Run: `python3 scripts/paper_lark_report.py --save-selected "{{date}}" "data/selected_papers.json"`
3. Run: `python3 scripts/paper_lark_report.py --register-doc "<doc_id>" "<doc_url>" "<title>"`
4. Create Feishu wiki doc at feishu_root using `feishu-create-doc` skill, title: `📰 Research Daily {{date}}`
5. Fill the document content using the template below.

---

# 科研日报 | {{date}}

<callout emoji="📡" background-color="light-blue">
**paper-lark-report** | 研究方向：{{research_direction}}
</callout>

---

## 今日概览

| 统计项 | 数值 |
|--------|------|
| 📅 日期 | {{date}} |
| 📚 候选论文 | {{total_papers}} 篇 |
| ✨ 精选论文 | {{selected_papers}} 篇 |

---

{% for paper in papers %}

## {{ forloop.counter }}. {{ paper.title }}

{% if paper.relevance_score >= 8 %}
<callout emoji="⭐" background-color="light-green">
**相关性：{{paper.relevance_score}}/10** — 高度相关
</callout>
{% elif paper.relevance_score >= 6 %}
<callout emoji="📎" background-color="light-yellow">
**相关性：{{paper.relevance_score}}/10** — 相关
</callout>
{% endif %}

| 字段 | 内容 |
|------|------|
| **arXiv ID** | `{{ paper.paper_id }}` |
| **发布日期** | {{ paper.posted_date }} |
| **作者** | {{ paper.authors }} |
| **链接** | [arXiv]({{ paper.arxiv_url }}) · [PDF]({{ paper.pdf_url }}) |

### 【摘要】
{{ paper.full_abstract }}

### 【研究动机】
{{ paper.motivation }}

### 【核心创新】
{{ paper.core_innovation }}

---
{% endfor %}

<callout emoji="📌" background-color="pale-gray">
**数据来源**：arXiv

*本报告由 AI 自动生成，分析基于论文摘要，未补充额外信息。*
</callout>


path: templates/weekly_report.md
content: # INSTRUCTIONS

## Input Data
Papers are saved in `data/weekly_papers.json` (aggregated from Mon–Fri daily logs).

## Analysis Requirements
- Group papers by research theme or technical approach.
- Identify complementary, contrasting, and extending relationships across papers.
- Extract shared themes or trends from all papers' motivation + core_innovation.
- Do NOT fabricate details. Only use content from the papers.

## Output Steps
1. Create Feishu wiki doc at feishu_root using `feishu-create-doc` skill.
   Title: `📊 Research Weekly {{year}}-W{{week_num}}`
2. Fill the document using the template below.

---

# 科研周报 | {{year}}-W{{week_num}}

<callout emoji="📊" background-color="light-blue">
**周期**：{{start_date}} 至 {{end_date}} | 研究方向：{{research_direction}}
</callout>

---

## 本周期概览

| 统计项 | 数值 |
|--------|------|
| 📅 覆盖天数 | {{days_covered}} 天 |
| 📚 论文总数 | {{total_papers}} 篇 |
| 📈 日均论文 | {{daily_avg}} 篇 |

---

{% if daily_stats %}
### 各日论文分布

| 日期 | 论文数 |
|------|--------|
{% for day in daily_stats %}
| {{ day.date }} | {{ day.count }} |
{% endfor %}
{% endif %}

---

## 本周期论文

{% for paper in all_papers %}

### {{ forloop.counter }}. {{ paper.title }}

<callout emoji="⭐" background-color="light-green">
**相关性：{{paper.relevance_score}}/10**
</callout>

| 字段 | 内容 |
|------|------|
| **日期** | {{ paper.posted_date }} |
| **作者** | {{ paper.authors }} |
| **链接** | [arXiv]({{ paper.arxiv_url }}) |

**研究动机**：{{ paper.motivation }}

**核心创新**：{{ paper.core_innovation }}

---
{% endfor %}

## 跨论文横向比较

### 研究方向关联
> 本周期论文中，哪些在解决相似问题？哪些采用了相似的技术路线？

{{ cross_paper_analysis }}

### 共同主题
> 从所有论文的摘要和核心创新中，提炼出的共同主题或趋势。

{{ shared_themes }}

---

<callout emoji="📌" background-color="pale-gray">
**数据来源**：本周每日日报归档

*本报告由 AI 自动生成，分析基于论文摘要和每日日报，未补充额外信息。*
</callout>


path: scripts/arxiv_search.py
content: """
arxiv_search.py - arXiv API client for paper fetching and query building.
"""
import json
import re
import ssl
import time
import urllib.error
import urllib.request
import urllib.parse
import xml.etree.ElementTree as ET
from datetime import datetime
from pathlib import Path
from dataclasses import dataclass, asdict
from typing import List, Optional

# ============================================================================
# CONSTANTS
# ============================================================================
SKILL_DIR = Path(__file__).parent.parent
ARXIV_ATOM_NS = "{http://www.w3.org/2005/Atom}"
ARXIV_API = "https://export.arxiv.org/api/query"

# ============================================================================
# DATA CLASS
# ============================================================================
@dataclass
class Paper:
    title: str
    authors: List[str]
    venue: str
    year: int
    paper_id: str
    url: str
    abstract: str            # short abstract (from initial fetch)
    full_abstract: str = ""  # complete abstract (fetched separately)
    posted_date: Optional[str] = None
    relevance_score: int = 0

# ============================================================================
# QUERY BUILDER
# ============================================================================
_GENERIC_TERMS = {
    "towards", "toward", "systems", "system", "approach",
    "method", "methods", "model", "models", "paper", "work",
    "research", "study", "based", "using", "used", "new",
    "safe", "safety", "efficient", "efficiency", "effective",
    "real", "world", "constraint", "constraints", "problem",
    "提出", "研究", "实现", "方法", "系统",
}

_COMPOUND_TERMS = [
    "multi-agent", "multi agent", "single-agent", "single agent",
    "multiagent", "llm-based", "llm based",
    "harness-based", "runtime assurance", "trace-based",
    "role-aware", "context routing", "least-privilege",
    "tool access", "orchestration", "collaboration",
]


def build_arxiv_query(research_direction: str) -> str:
    """
    Convert free-text research_direction into a precise arXiv abs: query.
    Extracts compound terms as atomic units, filters generic/academic
    stopwords, deduplicates, caps at 8 terms.
    """
    text = research_direction.lower()
    text_spaced = re.sub(r"-", " ", text)

    # Capture compound terms that appear in the text
    compound_terms: List[str] = []
    for comp in _COMPOUND_TERMS:
        pattern = r"\b" + re.escape(comp) + r"\b"
        if re.search(pattern, text_spaced):
            compound_terms.append(comp.replace(" ", "-"))

    # Capture remaining significant single tokens
    captured_substrings = set()
    for comp in compound_terms:
        captured_substrings.update(comp.split("-"))

    tokens = re.findall(r"[a-z]+", text_spaced)
    seen = set()
    for tok in tokens:
        if tok in _GENERIC_TERMS or tok in captured_substrings or len(tok) < 5:
            continue
        if tok not in seen:
            seen.add(tok)
            compound_terms.append(tok)
            if len(compound_terms) >= 8:
                break

    # AND the single most core term + OR the remaining terms for recall
    # The LLM scorer does fine-grained filtering; this just ensures baseline relevance.
    if len(compound_terms) == 0:
        return "abs:agent"
    if len(compound_terms) == 1:
        return f"abs:{compound_terms[0]}"
    if len(compound_terms) == 2:
        return f"abs:{compound_terms[0]}"
    core = compound_terms[0]
    rest = " OR ".join(f"abs:{t}" for t in compound_terms[1:])
    return f"abs:{core} AND ({rest})"


# ============================================================================
# FETCH HELPERS
# ============================================================================
def _get_text(entry, tag: str) -> str:
    el = entry.find(f"{ARXIV_ATOM_NS}{tag}")
    return el.text.strip() if el is not None and el.text else ""


def _build_paper(entry) -> Optional[Paper]:
    paper_id = _get_text(entry, "id").split("/")[-1]
    if not paper_id:
        return None

    title = _get_text(entry, "title").replace("\n", " ")
    summary = _get_text(entry, "summary").replace("\n", " ")
    published = _get_text(entry, "published")[:10]

    authors = [
        a.find(f"{ARXIV_ATOM_NS}name").text
        for a in entry.findall(f"{ARXIV_ATOM_NS}author")
        if a.find(f"{ARXIV_ATOM_NS}name") is not None
    ]

    category = ""
    for cat in entry.findall(f"{ARXIV_ATOM_NS}category"):
        if cat.get("term"):
            category = cat.get("term")
            break

    return Paper(
        title=title,
        authors=authors,
        venue=f"arXiv {category}" if category else "arXiv",
        year=int(published[:4]) if published else datetime.now().year,
        paper_id=paper_id,
        url=f"https://arxiv.org/abs/{paper_id}",
        abstract=summary[:500],
        full_abstract=summary,
        posted_date=published,
        relevance_score=0,
    )


# ============================================================================
# PUBLIC API
# ============================================================================
def fetch_arxiv_papers(query: str, max_results: int = 50) -> List[Paper]:
    """
    Search arXiv by query string and return a list of Paper objects.
    Uses abs: field queries for abstract-level matching.
    Retries on 429/503 with exponential backoff.
    """
    url = (
        f"{ARXIV_API}?"
        f"search_query={urllib.parse.quote(query)}"
        f"&start=0&max_results={max_results}"
        f"&sortBy=submittedDate&sortOrder=descending"
    )

    papers = []
    for attempt in range(5):
        try:
            req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
            ctx = ssl.create_default_context()
            with urllib.request.urlopen(req, timeout=60, context=ctx) as resp:
                raw = resp.read().decode("utf-8")

            root = ET.fromstring(raw)
            for entry in root:
                if entry.tag != f"{ARXIV_ATOM_NS}entry":
                    continue
                paper = _build_paper(entry)
                if paper:
                    papers.append(paper)

            print(f"  Fetched {len(papers)} papers for query: {query}")
            break  # success

        except urllib.error.HTTPError as e:
            if e.code in (429, 503) and attempt < 4:
                wait = (attempt + 1) * 30
                print(f"  HTTP {e.code}, retrying in {wait}s (attempt {attempt+1}/5)...")
                time.sleep(wait)
            else:
                print(f"  arXiv fetch failed: HTTP {e.code}")
                break
        except Exception as e:
            print(f"  arXiv fetch failed: {e}")
            break

    return papers


def fetch_arxiv_details(papers: List[Paper]) -> List[Paper]:
    """
    Batch-fetch full abstracts for a list of Paper objects.
    Updates full_abstract, authors, and posted_date in-place.
    Max 5 IDs per request; 3-second delay between batches.
    Retries on 429/503 with exponential backoff.
    """
    if not papers:
        return papers

    paper_map = {p.paper_id: p for p in papers}
    batch_size = 5

    print(f"Fetching full abstracts for {len(papers)} papers...")

    for i in range(0, len(papers), batch_size):
        batch = papers[i : i + batch_size]
        ids = "+".join(p.paper_id for p in batch)
        batch_num = i // batch_size + 1

        for attempt in range(5):
            try:
                url = f"{ARXIV_API}?id_list={ids}"
                req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
                ctx = ssl.create_default_context()
                with urllib.request.urlopen(req, timeout=60, context=ctx) as resp:
                    raw = resp.read().decode("utf-8")

                root = ET.fromstring(raw)
                for entry in root:
                    if entry.tag != f"{ARXIV_ATOM_NS}entry":
                        continue

                    pid_elem = entry.find(f"{ARXIV_ATOM_NS}id")
                    if pid_elem is None or not pid_elem.text:
                        continue

                    pid = pid_elem.text.split("/")[-1]
                    if pid not in paper_map:
                        continue

                    p = paper_map[pid]

                    summary_el = entry.find(f"{ARXIV_ATOM_NS}summary")
                    if summary_el is not None and summary_el.text:
                        p.full_abstract = summary_el.text.strip()

                    authors = [
                        a.find(f"{ARXIV_ATOM_NS}name").text
                        for a in entry.findall(f"{ARXIV_ATOM_NS}author")
                        if a.find(f"{ARXIV_ATOM_NS}name") is not None
                    ]
                    if authors:
                        p.authors = authors

                    pub_el = entry.find(f"{ARXIV_ATOM_NS}published")
                    if pub_el is not None and pub_el.text:
                        p.posted_date = pub_el.text[:10]

                print(f"  Batch {batch_num}: {len(batch)} papers done")
                break  # success

            except urllib.error.HTTPError as e:
                if e.code in (429, 503) and attempt < 4:
                    wait = (attempt + 1) * 15
                    print(f"  Batch {batch_num} HTTP {e.code}, retrying in {wait}s (attempt {attempt+1}/5)...")
                    time.sleep(wait)
                else:
                    print(f"  Batch {batch_num} failed: HTTP {e.code}")
                    break
            except Exception as e:
                print(f"  Batch {batch_num} failed: {e}")
                break

        time.sleep(5)  # arXiv requires >= 3 s between requests

    return list(paper_map.values())


path: scripts/create_feishu_doc.py
content: #!/usr/bin/env python3
"""
create_feishu_doc.py - Create a Feishu wiki document for the daily paper digest.
Uses Feishu Open API directly (no MCP).
"""
import json
import ssl
import urllib.request
from datetime import datetime
from pathlib import Path

SKILL_DIR = Path(__file__).parent.parent
FEISHU_BASE = "https://open.feishu.cn/open-apis"


def load_token() -> str:
    """Get a fresh tenant_access_token from appId + appSecret via Feishu API."""
    import json
    config_path = Path.home() / ".openclaw" / "openclaw.json"
    with open(config_path, "r", encoding="utf-8") as f:
        cfg = json.load(f)
    feishu_cfg = cfg.get("channels", {}).get("feishu", {})
    app_id = feishu_cfg.get("appId", "")
    app_secret = feishu_cfg.get("appSecret", "")
    if not app_id or not app_secret:
        raise RuntimeError("Feishu appId/appSecret not found in openclaw.json")

    # Exchange for tenant_access_token
    body = {"app_id": app_id, "app_secret": app_secret}
    req = urllib.request.Request(
        "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal",
        data=json.dumps(body).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    ctx = ssl.create_default_context()
    with urllib.request.urlopen(req, timeout=30, context=ctx) as resp:
        result = json.loads(resp.read().decode("utf-8"))
    if result.get("code") != 0:
        raise RuntimeError(f"Token exchange failed: {result}")
    [REDACTED_SECRET]"tenant_access_token", "")
    if not token:
        raise RuntimeError("No tenant_access_token in response")
    return token


def api_call(method: str, path: str, token: str, body=None) -> dict:
    url = f"{FEISHU_BASE}{path}"
    data = json.dumps(body).encode("utf-8") if body is not None else None
    req = urllib.request.Request(
        url, method=method, data=data,
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
    )
    ctx = ssl.create_default_context()
    with urllib.request.urlopen(req, timeout=30, context=ctx) as resp:
        result = json.loads(resp.read().decode("utf-8"))
    if result.get("code") != 0:
        raise RuntimeError(f"Feishu API error {result.get('code')}: {result.get('msg')}")
    return result.get("data", {})


def get_token_info(token: str) -> dict:
    """Check who this token belongs to."""
    return api_call("GET", "/authen/v1/user_info", token)



def create_wiki_node(space_id: str, parent_node_token: str, token: str, title: str) -> dict:
    """Create a wiki node (without obj_token - wiki API creates its own doc)."""
    body = {
        "obj_type": "docx",
        "parent_node_token": parent_node_token,
        "node_type": "origin",
        "title": title,
    }
    data = api_call("POST", f"/wiki/v2/spaces/{space_id}/nodes", token, body)
    node = data["node"]
    return {"node_token": node["node_token"], "obj_token": node["obj_token"]}


def write_blocks(document_id: str, blocks: list, token: str):
    """Write blocks to a Feishu document. Batches of 15."""
    for i in range(0, len(blocks), 15):
        batch = blocks[i:i+15]
        body = {"children": batch, "index": -1}
        api_call("POST", f"/docx/v1/documents/{document_id}/blocks/{document_id}/children", token, body)
        print(f"  Wrote blocks {i+1}-{i+len(batch)}")


def make_heading(level: int, content: str) -> dict:
    bt = level + 2
    key = f"heading{level}"
    return {
        "block_type": bt,
        key: {
            "elements": [{"text_run": {"content": content, "text_element_style": {}}}],
            "style": {},
        },
    }


def make_paragraph(content: str, bold: bool = False) -> dict:
    style = {"bold": True} if bold else {}
    return {
        "block_type": 2,
        "text": {
            "elements": [{"text_run": {"content": content, "text_element_style": style}}],
            "style": {},
        },
    }




def build_blocks(papers: list, date: str, total: int, research_direction: str = "") -> list:
    blocks = []

    # Header
    blocks.append(make_heading(1, "📰 Research Daily " + date))
    if research_direction:
        blocks.append(make_paragraph("研究方向：" + research_direction))
    blocks.append(make_paragraph(f"日期：{date} | 候选论文：{total} 篇 | 精选论文：{len(papers)} 篇"))

    for i, p in enumerate(papers, 1):
        blocks.append(make_heading(2, f"{i}. {p['title']}"))
        blocks.append(make_paragraph(f"相关性：{p['relevance_score']}/10"))
        blocks.append(make_paragraph(f"arXiv ID：{p['paper_id']} | 发布日期：{p['posted_date']}"))
        blocks.append(make_paragraph("作者：" + ", ".join(p['authors'][:3]) + (" et al." if len(p['authors']) > 3 else "")))
        blocks.append(make_paragraph(f"arXiv：{p['arxiv_url']}"))
        blocks.append(make_heading(3, "【研究动机】"))
        blocks.append(make_paragraph(p.get('motivation', '')))
        blocks.append(make_heading(3, "【核心创新】"))
        blocks.append(make_paragraph(p.get('core_innovation', '')))

    blocks.append(make_paragraph("数据来源：arXiv | 本报告由 AI 自动生成，分析基于论文摘要，未补充额外信息。"))

    return [b for b in blocks if b is not None]


def main():
    import yaml

    # Load selected papers
    with open(SKILL_DIR / "data" / "selected_papers.json", "r", encoding="utf-8") as f:
        data = json.load(f)

    date = data["date"]
    papers = data["papers"]

    print(f"Creating Feishu wiki doc for {date} with {len(papers)} papers...")

    # Get token
    [REDACTED_SECRET])
    print(f"Token loaded: {token[:20]}...")

    # Load wiki config
    with open(SKILL_DIR / "config.yaml", "r", encoding="utf-8") as f:
        config = yaml.safe_load(f)
    space_id = config.get("feishu_space_id")
    parent_node = config.get("feishu_parent_node")
    if not space_id or not parent_node:
        raise RuntimeError("feishu_space_id and feishu_parent_node must be set in config.yaml")
    print(f"Space: {space_id}, Parent node: {parent_node}")

    # Create wiki node
    title = f"📰 Research Daily {date}"
    result = create_wiki_node(space_id, parent_node, token, title)
    node_token = result["node_token"]
    obj_token = result["obj_token"]
    print(f"Wiki node created: {node_token}")
    print(f"Document ID (obj_token): {obj_token}")

    # Build blocks
    research_direction = config.get("research_direction", "")
    blocks = build_blocks(papers, date, 20, research_direction)

    # Write blocks
    write_blocks(obj_token, blocks, token)

    # Build doc URL
    doc_url = f"https://my.feishu.cn/wiki/{space_id}?nodeToken={node_token}"
    print(f"Document URL: {doc_url}")

    # Save result
    result_file = SKILL_DIR / "data" / "doc_result.json"
    with open(result_file, "w", encoding="utf-8") as f:
        json.dump({
            "node_token": node_token,
            "obj_token": obj_token,
            "doc_url": doc_url,
            "date": date,
        }, f, ensure_ascii=False, indent=2)
    print(f"Result saved to {result_file}")

    # Register in doc registry
    doc_registry_path = SKILL_DIR / "data" / "doc_registry.json"
    registry = {}
    if doc_registry_path.exists():
        with open(doc_registry_path, "r", encoding="utf-8") as f:
            registry = json.load(f)
    registry[date] = {
        "node_token": node_token,
        "obj_token": obj_token,
        "url": doc_url,
        "title": title,
        "registered_at": datetime.now().isoformat(),
    }
    with open(doc_registry_path, "w", encoding="utf-8") as f:
        json.dump(registry, f, ensure_ascii=False, indent=2)
    print(f"Registered in doc_registry.json")
    print("Done!")


if __name__ == "__main__":
    main()


path: scripts/paper_lark_report.py
content: #!/usr/bin/env python3
"""
paper_lark_report.py - Entry point for paper-lark-report daily/weekly digest.

Imports:
    arxiv_search - arXiv API: query building, paper fetch, detail fetch
"""
import json
from datetime import datetime, timedelta
from pathlib import Path

from arxiv_search import Paper, build_arxiv_query, fetch_arxiv_papers, fetch_arxiv_details

# ============================================================================
# PATHS
# ============================================================================
SKILL_DIR = Path(__file__).parent.parent
PROCESSED_IDS_PATH = SKILL_DIR / "data" / "processed_ids.json"
DOC_REGISTRY_PATH = SKILL_DIR / "data" / "doc_registry.json"
PROCESSED_LOG_DIR = SKILL_DIR / "processed_log"
OUTPUT_PATH = SKILL_DIR / "data" / "daily_papers.json"

# ============================================================================
# CONFIG
# ============================================================================
def load_config() -> dict:
    import yaml
    with open(SKILL_DIR / "config.yaml", "r", encoding="utf-8") as f:
        return yaml.safe_load(f)

# ============================================================================
# PROCESSED IDS (cross-day dedup)
# ============================================================================
def load_processed_ids() -> dict:
    if not PROCESSED_IDS_PATH.exists():
        return {"processed": [], "last_updated": ""}
    with open(PROCESSED_IDS_PATH, "r", encoding="utf-8") as f:
        return json.load(f)


def save_processed_ids(data: dict):
    PROCESSED_IDS_PATH.parent.mkdir(parents=True, exist_ok=True)
    with open(PROCESSED_IDS_PATH, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)


def is_processed(paper_id: str, processed_data: dict) -> bool:
    return paper_id in processed_data.get("processed", [])


def add_processed(paper_id: str, processed_data: dict):
    processed_data.setdefault("processed", []).append(paper_id)
    processed_data["last_updated"] = datetime.now().isoformat()

# ============================================================================
# DOC REGISTRY (daily dedup)
# ============================================================================
def load_doc_registry() -> dict:
    if not DOC_REGISTRY_PATH.exists():
        return {}
    with open(DOC_REGISTRY_PATH, "r", encoding="utf-8") as f:
        return json.load(f)


def save_doc_registry(registry: dict):
    with open(DOC_REGISTRY_PATH, "w", encoding="utf-8") as f:
        json.dump(registry, f, ensure_ascii=False, indent=2)


def get_today_doc_info() -> dict | None:
    return load_doc_registry().get(datetime.now().strftime("%Y-%m-%d"))


def register_doc(node_token: str, obj_token: str, doc_url: str, title: str = ""):
    today = datetime.now().strftime("%Y-%m-%d")
    registry = load_doc_registry()
    registry[today] = {
        "node_token": node_token,
        "obj_token": obj_token,
        "url": doc_url,
        "title": title or f"Research Daily {today}",
        "registered_at": datetime.now().isoformat(),
    }
    save_doc_registry(registry)
    print(f"Registered: {today} -> {doc_url}")

# ============================================================================
# DAILY RUN
# ============================================================================
def run_daily():
    print("=" * 60)
    print("paper-lark-report  |  daily run")
    print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
    print("=" * 60)

    config = load_config()
    processed_data = load_processed_ids()
    today_str = datetime.now().strftime("%Y-%m-%d")

    feishu_root = config.get("feishu_root", "")
    research_direction = config.get("research_direction", "")
    max_search = config.get("max_search_results", 20)
    max_daily = config.get("max_daily_papers", 3)

    if not feishu_root:
        print("ERROR: feishu_root not configured")
        return
    if not research_direction:
        print("ERROR: research_direction not configured")
        return

    if get_today_doc_info() is not None:
        print("Today's document already exists. Skip.")
        return

    query = build_arxiv_query(research_direction)
    print(f"arXiv query: {query}")

    papers = fetch_arxiv_papers(query, max_results=max_search)
    filtered = [p for p in papers if not is_processed(p.paper_id, processed_data)]
    print(f"After dedup: {len(filtered)} papers")

    if not filtered:
        print("No new papers. Skip.")
        return

    filtered = fetch_arxiv_details(filtered[:20])

    output = {
        "date": today_str,
        "research_direction": research_direction,
        "arxiv_query": query,
        "papers": [p.__dict__ for p in filtered],
        "max_daily_papers": max_daily,
        "skip": False,
        "config": {"feishu_root": feishu_root},
    }

    with open(OUTPUT_PATH, "w", encoding="utf-8") as f:
        json.dump(output, f, ensure_ascii=False, indent=2)

    print(f"Output saved to {OUTPUT_PATH}")
    print(f"{len(filtered)} papers ready for LLM scoring")

# ============================================================================
# WEEKLY RUN
# ============================================================================
def run_weekly():
    print("=" * 60)
    print("paper-lark-report  |  weekly run")
    print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
    print("=" * 60)

    config = load_config()
    today = datetime.now()
    week_papers = []
    dates_included = []

    for i in range(1, 6):
        date = (today - timedelta(days=i)).strftime("%Y-%m-%d")
        log_file = PROCESSED_LOG_DIR / f"{date}.json"
        if log_file.exists():
            with open(log_file, "r", encoding="utf-8") as f:
                data = json.load(f)
                week_papers.extend(data.get("papers", []))
                dates_included.append(date)

    if not week_papers:
        print("No papers this week. Skip.")
        return

    output = {
        "week": f"{today.year}-W{today.isocalendar()[1]:02d}",
        "start_date": dates_included[0],
        "end_date": dates_included[-1],
        "research_direction": config.get("research_direction", ""),
        "dates_included": dates_included,
        "papers": week_papers,
        "config": {"feishu_root": config.get("feishu_root", "")},
    }

    output_path = SKILL_DIR / "data" / "weekly_papers.json"
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(output, f, ensure_ascii=False, indent=2)

    print(f"Week: {', '.join(dates_included)} | {len(week_papers)} papers")
    print(f"Weekly data saved to {output_path}")

# ============================================================================
# SAVE SELECTED PAPERS (called by LLM after scoring)
# ============================================================================
def save_selected_papers(date: str, papers: list):
    PROCESSED_LOG_DIR.mkdir(parents=True, exist_ok=True)
    log_file = PROCESSED_LOG_DIR / f"{date}.json"
    with open(log_file, "w", encoding="utf-8") as f:
        json.dump({
            "date": date,
            "papers": papers,
            "generated_at": datetime.now().isoformat(),
        }, f, ensure_ascii=False, indent=2)

    processed_data = load_processed_ids()
    for paper in papers:
        pid = paper.get("paper_id", "")
        if pid:
            add_processed(pid, processed_data)
    save_processed_ids(processed_data)

    print(f"Saved {len(papers)} selected papers for {date}")

# ============================================================================
# CLI
# ============================================================================
if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="paper-lark-report")
    parser.add_argument("--weekly", action="store_true", help="Generate weekly report")
    parser.add_argument("--register-doc", nargs=3,
                        metavar=("NODE_TOKEN", "OBJ_TOKEN", "DOC_URL"),
                        help="Register a created Feishu wiki doc (NODE_TOKEN OBJ_TOKEN DOC_URL)")
    parser.add_argument("--save-selected", nargs=2,
                        metavar=("DATE", "PAPERS_JSON"),
                        help="Save LLM-selected papers (DATE PAPERS_JSON_FILE)")
    args = parser.parse_args()

    if args.save_selected:
        date_str, papers_json = args.save_selected
        with open(papers_json, "r", encoding="utf-8") as f:
            data = json.load(f)
        # Support both a bare list and a {"date":..., "papers":[...]} wrapper
        papers = data if isinstance(data, list) else data.get("papers", [])
        save_selected_papers(date_str, papers)

    elif args.register_doc:
        node_token, obj_token, doc_url = args.register_doc
        register_doc(node_token, obj_token, doc_url)

    elif args.weekly:
        run_weekly()

    else:
        run_daily()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)