Skill: lococaeco/paper-notion-summarizer (version 1.0.0)

--- name: paper-notion-summarizer description: Fetch paper metadata by title or arXiv/DOI link, create a deep structured summary, and post it as a Notion page. The agent reads the paper and writes a seminar-quality summary adapted to the user's language. --- # Paper Notion Summarizer ## Purpose Given a paper title or arXiv link, create a **seminar-quality, deeply structured summary** and upload it to Notion. > ⚠️ This is NOT an extractive summary. The **agent reads the full paper and writes an original analysis**. ## Language Adaptation **Write the summary in the same language as the user's request.** - If the user writes in Korean → write in Korean (technical terms in English) - If the user writes in English → write in English - If the user writes in Japanese → write in Japanese - And so on for any language. Section headings in the summary JSON should match the user's language. The English template below is the canonical structure — adapt headings to the user's language. ## Workflow (3 phases) ### Phase 1: Extract paper content ```bash python3 scripts/extract_paper.py \ --output /tmp/paper_extract.json \ "https://arxiv.org/abs/2301.12345" ``` Or by title: ```bash python3 scripts/extract_paper.py \ --output /tmp/paper_extract.json \ --title "Attention Is All You Need" ``` Options: - `--output`, `-o`: Output file path (defaults to stdout) - `--skip-fulltext`: Extract abstract only (fast mode, skip PDF) - `--doi`: Explicit DOI - `--arxiv-id`: Explicit arXiv ID ### Phase 2: Agent reads and writes the summary Read the extracted JSON section by section (`read` tool with `offset`/`limit` for large files), then write a structured summary JSON to `/tmp/paper_summary.json`. #### Reading strategy (context management) - Read Abstract → Introduction → Method → Experiments → Conclusion in order - For long papers, read in chunks and accumulate understanding - Focus on: core idea, key equations, experimental setup, main results, ablations #### Summary JSON template ```json { "title": "Paper Title (original language)", "metadata": { "authors": "Author list", "year": "2024", "venue": "NeurIPS 2024", "doi": "10.xxxx/xxxxx", "url": "https://arxiv.org/abs/xxxx.xxxxx", "source": "arXiv" }, "sections": [ { "heading": "0. Metadata", "content": "- Authors: ...\n- Year: ...\n- Venue: ...\n- Code: ..." }, { "heading": "1. One-line Summary", "content": "What this paper does in one sentence." }, { "heading": "2. Problem & Motivation", "content": "- What problem does it solve?\n- Why are existing methods insufficient?\n- Why is this research needed?" }, { "heading": "3. Key Contributions", "content": "1. First contribution\n2. Second contribution\n3. Third contribution" }, { "heading": "4. Method", "content": "Detailed pipeline/architecture description.\nCore ideas, key equations included.\n\n### Core Idea\n...\n\n### Architecture\n...\n\n### Training\n...\n\n### Key Equations\n$$equation$$" }, { "heading": "5. Experiments", "content": "### Setup\n- Datasets: ...\n- Baselines: ...\n- Metrics: ...\n\n### Main Results\n- Key numbers and comparisons\n- Where it works and where it doesn't" }, { "heading": "6. Ablation & Analysis", "content": "- Per-component contributions\n- Interesting analysis results\n- Hyperparameter sensitivity" }, { "heading": "7. Limitations & Future Work", "content": "- Author-acknowledged limitations\n- Additional limitations you identify\n- Future research directions" }, { "heading": "8. Overall Assessment", "content": "- Research significance\n- Strengths and weaknesses\n- Connections to related work\n- Ideas applicable to user's research" } ] } ``` #### Quality guidelines 1. **Terminology**: Keep technical terms in their original language; explanations in the user's language. 2. **Equations**: Include key equations in LaTeX (`$$ ... $$`). 3. **Depth**: Seminar-presentation level understanding. - Method: Not just "they did X" but "why they designed it this way, what each component does" - Experiments: Not just "it worked" but "X% improvement over Y baseline under Z conditions" 4. **Critical perspective**: Record limitations and open questions, not just strengths. 5. **Connections**: If you know the user's research interests, connect the paper to them. 6. **No programming code blocks**: Do NOT use fenced code blocks (``` ```) in `sections[*].content`. Math expressions (`$$ ... $$`, `` ```latex ``) are allowed. 7. **No emoji in headings**: Use numbered prefixes: `0. Metadata`, `1. One-line Summary`, etc. ### Phase 3: Push to Notion ```bash python3 scripts/push_to_notion.py \ /tmp/paper_summary.json \ --parent-page-id YOUR_PAGE_ID ``` Options: - `--parent-page-id`: Notion page ID to create the summary under - `--force-update`: Overwrite existing page with same title - `--dry-run`: Preview without uploading - `--notion-key`: Explicit Notion API token ## Quick Start (full agent flow) ``` 1. python3 scripts/extract_paper.py -o /tmp/paper_extract.json "https://arxiv.org/abs/..." 2. read /tmp/paper_extract.json (section by section) 3. Write summary → /tmp/paper_summary.json 4. python3 scripts/push_to_notion.py /tmp/paper_summary.json --parent-page-id PAGE_ID ``` ## Configuration | Config | Source | Description | |--------|--------|-------------| | Notion API key | `NOTION_API_KEY` env or `~/.config/notion/api_key` | Required for Notion upload | | Parent page | `NOTION_PARENT_PAGE_ID` env or `--parent-page-id` | Notion page to create summaries under | ## Notes - arXiv papers use PDF extraction (requires `pypdf`). Install: `pip install pypdf` - For very long papers (>100 pages), use `--skip-fulltext` and read HTML via `web_fetch`. - Notion API version: `2025-09-03` - The `extract_paper.py` script does NOT require a Notion API key — it only fetches and extracts.

path: _meta.json
content: {
  "name": "paper-notion-summarizer",
  "description": "Fetch paper metadata by title or arXiv/DOI link, create a deep structured summary, and post it as a Notion page. The agent reads the paper and writes a seminar-quality summary adapted to the user's language.",
  "version": "1.0.0",
  "author": "openclaw-community",
  "tags": ["research", "papers", "notion", "summarizer", "arxiv"],
  "requires": ["notion"]
}


path: references/notion_api.md
content: # Paper Lookup & Notion API Reference

## Sources
- arXiv API: `https://export.arxiv.org/api/query?search_query=all:{query}`
- Crossref API: `https://api.crossref.org/works?query.title={query}`

## Notion API
- Version: `2025-09-03`
- Parent page creation: `POST /v1/pages` with `{"parent":{"page_id":"..."}}`
- Auth: `Authorization: Bearer ${NOTION_API_KEY}`
- Query existing pages: `POST /v1/search` with query title string
- Append blocks: `PATCH /v1/blocks/{page_id}/children` with `{"children": [...]}`
- Delete blocks: `DELETE /v1/blocks/{block_id}`

## Required env
- `NOTION_API_KEY` (or `~/.config/notion/api_key`)
- `NOTION_PARENT_PAGE_ID` (or `--parent-page-id` CLI arg)

## Notion block types used
- `heading_1`, `heading_2`, `heading_3`
- `paragraph`
- `bulleted_list_item`, `numbered_list_item`
- `equation` (KaTeX)
- `divider`
- `callout`

## Block limits
- Rich text content: max ~2000 chars per block (we use 1800 for safety)
- Children append: max 100 blocks per request (we use 80)


path: scripts/extract_paper.py
content: #!/usr/bin/env python3
"""Extract paper metadata and full text by title, DOI, or arXiv URL/ID.

Outputs a structured JSON with metadata, sections, and detected equations.
This script does NOT upload to Notion — use push_to_notion.py for that.
"""

from __future__ import annotations

import argparse
import json
import re
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from html import unescape
from io import BytesIO
from pathlib import Path
from typing import Dict, List, Optional
from urllib.parse import quote_plus
from xml.etree import ElementTree as ET
from difflib import SequenceMatcher

import requests

try:
    from pypdf import PdfReader
except Exception:
    PdfReader = None


SECTION_PATTERNS = [
    (r"^(abstract)\b", "Abstract"),
    (r"^((1\.?\s*)?(introduction|intro)\b)", "Introduction"),
    (r"^((1\.?\s*)?(background|motivation)\b)", "Background"),
    (r"^((2\.?\s*)?(related\s*work|related\s*research|prior\s*work)\b)", "Related Work"),
    (r"^((2\.?\s*)?(preliminaries|preliminary\s*knowledge)\b)", "Preliminaries"),
    (r"^((3\.?\s*)?(method|methods|approach|methodology)\b)", "Method"),
    (r"^((4\.?\s*)?(framework|model|proposed\s*method|proposed\s*model|architecture)\b)", "Method"),
    (r"^((5\.?\s*)?(experimental\s*results|experiments|experiments\s*and\s*analysis|evaluation)\b)", "Experiments"),
    (r"^((6\.?\s*)?(ablation|analysis|case\s*study|study)\b)", "Experiments"),
    (r"^((7\.?\s*)?(discussion|discussion\s*and\s*analysis)\b)", "Discussion"),
    (r"^((8\.?\s*)?(results|conclusion|conclusions|summary)\b)", "Conclusion"),
    (r"^((9\.?\s*)?(limitation|limitations|future\s*work)\b)", "Limitations"),
    (r"^((appendix|supplementary)\b)", "Appendix"),
]

SECTION_ORDER = [
    "Abstract", "Introduction", "Background", "Related Work", "Preliminaries",
    "Method", "Experiments", "Discussion", "Conclusion", "Limitations", "Appendix",
]


@dataclass
class Section:
    title: str
    content: str


@dataclass
class Paper:
    title: str
    authors: str
    year: str
    venue: str
    abstract: str
    doi: str
    url: str
    source: str
    sections: List[Section]
    equations: List[str]
    extra: Dict[str, str]

    def to_dict(self):
        return asdict(self)


def _now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


def _normalize_title(t: str) -> str:
    return re.sub(r"\s+", " ", t).strip().lower()


def _title_tokens(text: str) -> List[str]:
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s]", " ", text)
    return [t for t in re.split(r"\s+", text) if t and len(t) > 1]


def _title_match_score(a: str, b: str) -> float:
    at = set(_title_tokens(a))
    bt = set(_title_tokens(b))
    if not at or not bt:
        return 0.0
    overlap = len(at & bt) / max(1, len(at | bt))
    seq = SequenceMatcher(None, a.lower(), b.lower()).ratio()
    return round(0.65 * seq + 0.35 * overlap, 4)


def _pick_first(x):
    return x[0] if isinstance(x, list) and x else ""


def _authors_from_arxiv(entry: ET.Element) -> str:
    names = []
    ns = {"a": "http://www.w3.org/2005/Atom"}
    for author in entry.findall("a:author", ns):
        name = _pick_first(author.findall("a:name", ns))
        if name is not None and name.text:
            names.append(name.text.strip())
    return ", ".join(names)


# ---------- arXiv ----------

def fetch_from_arxiv(query: str, arxiv_id: Optional[str] = None) -> Optional[Paper]:
    if arxiv_id:
        q = quote_plus(f"id:{arxiv_id}")
        url = f"https://export.arxiv.org/api/query?search_query={q}&start=0&max_results=20"
    else:
        q = quote_plus(query)
        url = f"https://export.arxiv.org/api/query?search_query=all:{q}&start=0&max_results=20"

    r = requests.get(url, headers={"User-Agent": "paper-notion-summarizer/2.0"}, timeout=40)
    if r.status_code != 200:
        return None

    try:
        root = ET.fromstring(r.text)
    except Exception:
        return None

    entries = root.findall("{http://www.w3.org/2005/Atom}entry")
    if not entries:
        return None

    selected = None
    if arxiv_id:
        for e in entries:
            entry_id = e.findtext("{http://www.w3.org/2005/Atom}id", "") or ""
            if arxiv_id in entry_id:
                selected = e
                break

    if selected is None:
        best_score = 0.0
        for e in entries:
            title_candidate = e.findtext("{http://www.w3.org/2005/Atom}title", "") or ""
            score = _title_match_score(query, title_candidate)
            if score > best_score:
                best_score = score
                selected = e
        if not selected or best_score < 0.18:
            return None

    entry = selected
    ns = "{http://www.w3.org/2005/Atom}"
    title = entry.findtext(f"{ns}title", "").strip().replace("\n", " ")
    summary = entry.findtext(f"{ns}summary", "").strip().replace("\n", " ")
    published = entry.findtext(f"{ns}published", "")
    year = published[:4] if published else ""

    arxiv_abs_url = entry.findtext(f"{ns}id", "")
    found_pdf = ""
    arxiv_raw_id = ""

    for d in entry.findall(f"{ns}link"):
        href = d.attrib.get("href", "") or ""
        if d.attrib.get("title") == "pdf" or href.endswith(".pdf"):
            found_pdf = href

    m = re.search(r"arxiv\.org/abs/([\w\.\-]+)", arxiv_abs_url)
    if m:
        arxiv_raw_id = m.group(1)

    if not found_pdf and arxiv_raw_id:
        found_pdf = f"https://arxiv.org/pdf/{arxiv_raw_id}.pdf"

    return Paper(
        title=title, authors=_authors_from_arxiv(entry), year=year, venue="arXiv",
        abstract=summary, doi="", url=arxiv_abs_url, source="arXiv",
        sections=[], equations=[],
        extra={"queried_at": _now_iso(), "raw_id": arxiv_raw_id or (arxiv_id or ""), "pdf_url": found_pdf},
    )


# ---------- Crossref ----------

def fetch_from_crossref(query: str, doi: Optional[str] = None) -> Optional[Paper]:
    if doi:
        url = f"https://api.crossref.org/works/{quote_plus(doi)}"
        r = requests.get(url, headers={"User-Agent": "paper-notion-summarizer/2.0"}, timeout=40)
        if r.status_code != 200:
            return None
        items = [r.json().get("message", {})]
    else:
        url = f"https://api.crossref.org/works?query.title={quote_plus(query)}&rows=20"
        r = requests.get(url, headers={"User-Agent": "paper-notion-summarizer/2.0"}, timeout=40)
        if r.status_code != 200:
            return None
        items = r.json().get("message", {}).get("items", [])
        if not items:
            return None

    item = None
    if doi:
        item = items[0]
    else:
        best_score = 0.0
        for candidate in items:
            cand_title = _pick_first(candidate.get("title", [])) or ""
            score = _title_match_score(query, cand_title)
            if score > best_score:
                best_score = score
                item = candidate
        if not item or best_score < 0.22:
            return None

    title = _pick_first(item.get("title", [])) or ""
    if not title:
        return None

    authors = []
    for a in item.get("author", []) or []:
        fam = a.get("family", "")
        given = a.get("given", "")
        if fam or given:
            authors.append((fam + (", " + given) if fam and given else fam or given).strip())

    year = ""
    issued = item.get("issued", {}).get("date-parts", [])
    if issued and isinstance(issued, list) and issued[0]:
        year = str(issued[0][0])

    venue = _pick_first(item.get("container-title", [])) or _pick_first(item.get("publisher", "")) or ""
    abstract = re.sub(r"<[^>]+>", "", item.get("abstract", "") or "")

    pdf_url = ""
    for l in item.get("link", []) or []:
        if l.get("content-type") == "application/pdf" and l.get("URL"):
            pdf_url = l.get("URL")
            break

    return Paper(
        title=title, authors=", ".join(authors), year=year, venue=venue,
        abstract=abstract, doi=item.get("DOI", "") or "", url=item.get("URL", ""),
        source="Crossref", sections=[], equations=[],
        extra={"queried_at": _now_iso(), "type": item.get("type", ""), "pdf_url": pdf_url},
    )


# ---------- Text extraction ----------

def strip_html_to_text(html: str) -> str:
    if not html:
        return ""
    html = re.sub(r"(?is)<script.*?>.*?</script>", " ", html)
    html = re.sub(r"(?is)<style.*?>.*?</style>", " ", html)
    html = re.sub(r"(?is)<noscript.*?>.*?</noscript>", " ", html)
    html = re.sub(r"(?i)<br\s*/?>", "\n", html)
    html = re.sub(r"(?i)</(p|div|section|article|li|ul|ol|table|tr|td|th|h[1-6])>", "\n", html)
    plain = re.sub(r"<[^>]+>", " ", html)
    plain = unescape(plain)
    lines = [ln.strip() for ln in plain.splitlines()]
    lines = [ln for ln in lines if ln and len(ln) >= 2 and not re.match(r"^[$&*#_=]{2,}|^[\d\s]*$", ln)]
    return "\n".join(lines)


def filter_redundant_lines(lines: List[str]) -> List[str]:
    out = []
    seen = set()
    for ln in lines:
        compact = re.sub(r"\s+", " ", ln).strip()
        if not compact:
            continue
        if len(compact) > 180 and compact.lower().startswith("javascript"):
            continue
        if compact.lower().startswith("cookie") and "policy" in compact.lower():
            continue
        key = compact.lower()
        if key in seen:
            continue
        seen.add(key)
        out.append(compact)
    return out


def fetch_url_text(url: str) -> str:
    if not url:
        return ""
    try:
        r = requests.get(url, headers={"User-Agent": "paper-notion-summarizer/2.0"}, timeout=40)
    except Exception:
        return ""
    if r.status_code != 200 or not r.content:
        return ""
    return r.text or ""


def fetch_pdf_text(pdf_url: str) -> str:
    if not pdf_url or PdfReader is None:
        return ""
    try:
        r = requests.get(pdf_url, headers={"User-Agent": "paper-notion-summarizer/2.0"}, timeout=60, allow_redirects=True)
    except Exception:
        return ""
    if r.status_code != 200 or not r.content:
        return ""
    try:
        reader = PdfReader(BytesIO(r.content))
    except Exception:
        return ""
    pages = []
    for page in reader.pages:
        try:
            text = page.extract_text() or ""
        except Exception:
            text = ""
        if text:
            pages.append(text)
    return "\n".join(pages)


# ---------- Section splitting ----------

def _section_candidate(line: str) -> Optional[str]:
    normalized = re.sub(r"\s+", " ", line.strip()).lower()
    normalized = re.sub(r"^(?:[0-9]+(?:\.[0-9]+)*|[ivxlcdm]+|section|sec\.?)[\s\-\.:)]*", "", normalized, flags=re.I)
    for pattern, canonical in SECTION_PATTERNS:
        if re.search(pattern, normalized, flags=re.I):
            return canonical
    if re.search(r"^acknowledg|acknowledgements|references|notes|appendix|supplementary", normalized, flags=re.I):
        if re.search(r"appendix|supplement", normalized, flags=re.I):
            return "Appendix"
        return "Discussion"
    return None


def split_sections(text: str) -> List[Section]:
    if not text:
        return []
    lines = list(text.splitlines())
    sections: List[Section] = []
    current_title = "Full Text"
    current_lines: List[str] = []

    for line in lines:
        cleaned = re.sub(r"\s+", " ", line.strip()).strip()
        cleaned = re.sub(r"^[=\-*]+|[=\-*]+$", "", cleaned).strip()
        if not cleaned:
            continue
        candidate = _section_candidate(cleaned)
        if candidate:
            if current_lines:
                sections.append(Section(title=current_title, content="\n".join(current_lines).strip()))
                current_lines = []
            current_title = candidate
            continue
        if re.match(r"^fig\.|^figure\s+\d+|^table\s+\d+|^eq\.|^equation\b", cleaned, flags=re.I):
            continue
        if len(cleaned) > 180 and any(tag in cleaned.lower() for tag in ["cookie", "javascript", "analytics", "advertisement"]):
            continue
        current_lines.append(cleaned)

    if current_lines:
        sections.append(Section(title=current_title, content="\n".join(current_lines).strip()))

    # Merge tiny fragments
    merged: List[Section] = []
    for sec in sections:
        if not sec.content or len(sec.content) < 80:
            if merged:
                merged[-1].content += "\n" + sec.title + "\n" + sec.content
            else:
                merged.append(sec)
            continue
        merged.append(sec)

    # De-dup
    deduped: List[Section] = []
    seen = set()
    for sec in merged:
        key = _normalize_title(sec.title)
        if key in seen and deduped:
            deduped[-1].content += "\n" + sec.content
            continue
        seen.add(key)
        deduped.append(sec)
    return deduped


def detect_equations(text: str) -> List[str]:
    if not text:
        return []
    formulas: List[str] = []
    inline = re.findall(r"\$[^\$]{4,220}\$", text)
    block = re.findall(r"\\\[(.*?)\\\]", text, flags=re.S)
    aligned = re.findall(r"\\begin\{align\*?\}(.*?)\\end\{align\*?\}", text, flags=re.S)
    eq_env = re.findall(r"\\begin\{equation\*?\}(.*?)\\end\{equation\*?\}", text, flags=re.S)
    formulas.extend(inline)
    formulas.extend([f"\\[{x.strip()}\\]" for x in block])
    formulas.extend([x.strip() for x in aligned])
    formulas.extend([x.strip() for x in eq_env])

    candidates = []
    for line in text.splitlines():
        if re.search(r"(\\frac|\\sum|\\int|\\log|\\argmax|\\argmin|\\rightarrow|\\left|\\right|\\infty|\\nabla|\\mathbf|\\alpha|\\beta|\\gamma)", line):
            compact = line.strip()
            if 20 < len(compact) < 500:
                candidates.append(compact)
    formulas.extend(candidates)

    uniq = []
    seen = set()
    for f in formulas:
        norm = re.sub(r"\s+", " ", f).strip()
        if not norm or norm in seen:
            continue
        seen.add(norm)
        uniq.append(norm)
    return uniq[:60]


def load_fulltext(paper: Paper, skip_fulltext: bool) -> Paper:
    if skip_fulltext:
        paper.sections = [Section("Abstract", paper.abstract)]
        paper.equations = detect_equations(paper.abstract)
        return paper

    candidate_text = ""
    source = (paper.extra.get("pdf_url") or "").strip()
    if source:
        candidate_text = fetch_pdf_text(source)

    if not candidate_text and paper.url and "arxiv.org" in paper.url:
        html = fetch_url_text(paper.url)
        plain_html = strip_html_to_text(html)
        candidate_text = "\n".join(filter_redundant_lines(plain_html.splitlines()))

    if not candidate_text and paper.url and "doi.org" in paper.url:
        html = fetch_url_text(paper.url)
        if html:
            plain_html = strip_html_to_text(html)
            candidate_text = "\n".join(filter_redundant_lines(plain_html.splitlines()))

    if candidate_text:
        sections = split_sections(candidate_text)
        if paper.abstract and (not sections or _normalize_title(sections[0].title) != "abstract"):
            sections.insert(0, Section("Abstract", paper.abstract))
        paper.sections = sections
        all_text = "\n".join([s.content for s in sections if s.content])
        paper.equations = detect_equations(all_text)
    else:
        paper.sections = [Section("Abstract", paper.abstract)]
        paper.equations = detect_equations(paper.abstract)
    return paper


# ---------- CLI ----------

def parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="Extract paper metadata and full text to JSON")
    p.add_argument("query", nargs="?", help="Paper title, DOI, or arXiv URL")
    p.add_argument("--title", dest="title_opt", help="Paper title (alternative to positional arg)")
    p.add_argument("--doi", help="Explicit DOI")
    p.add_argument("--arxiv-id", help="Explicit arXiv ID")
    p.add_argument("--skip-fulltext", action="store_true", help="Extract abstract only (skip PDF)")
    p.add_argument("--output", "-o", help="Output file path (default: stdout)")
    return p.parse_args()


def main() -> int:
    args = parse_args()
    query = args.title_opt or args.query
    if not query:
        print("Error: provide a paper title, DOI, or arXiv URL")
        return 1

    # Auto-detect arXiv ID from URL
    if not args.arxiv_id and query:
        m = re.search(r"(?:https?://)?(?:www\.)?arxiv\.org/(?:abs|pdf)/([\w\.\-]+)", query)
        if m:
            args.arxiv_id = m.group(1).replace(".pdf", "")

    # Auto-detect DOI
    if not args.doi and query:
        m = re.search(r"10\.\d{4,9}/[^\s]+", query)
        if m:
            args.doi = m.group(0)

    paper = None
    if args.arxiv_id:
        paper = fetch_from_arxiv(query, arxiv_id=args.arxiv_id)
    if not paper and args.doi:
        paper = fetch_from_crossref(query, doi=args.doi)
    if not paper:
        paper = fetch_from_crossref(query)
    if not paper:
        paper = fetch_from_arxiv(query)
    if not paper:
        print("Could not fetch paper metadata.")
        paper = Paper(
            title=query, authors="", year="", venue="", abstract="",
            doi=args.doi or "", url="", source="manual",
            sections=[Section("Abstract", "")], equations=[],
            extra={"note": "failed-to-fetch", "queried_at": _now_iso()},
        )

    paper = load_fulltext(paper, skip_fulltext=args.skip_fulltext)

    output = {
        "title": paper.title, "authors": paper.authors, "year": paper.year,
        "venue": paper.venue, "abstract": paper.abstract, "doi": paper.doi,
        "url": paper.url, "source": paper.source,
        "sections": [{"title": s.title, "content": s.content} for s in paper.sections],
        "equations": paper.equations, "extra": paper.extra,
    }
    json_str = json.dumps(output, ensure_ascii=False, indent=2)

    if args.output:
        Path(args.output).parent.mkdir(parents=True, exist_ok=True)
        Path(args.output).write_text(json_str, encoding="utf-8")
        print(f"Extracted to: {args.output}")
    else:
        print(json_str)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/push_to_notion.py
content: #!/usr/bin/env python3
"""Push an LLM-written structured paper summary to Notion.

Input: a JSON file with the following structure:
{
  "title": "Paper Title",
  "metadata": {
    "authors": "...",
    "year": "...",
    "venue": "...",
    "doi": "...",
    "url": "...",
    "source": "arXiv/Crossref"
  },
  "sections": [
    {"heading": "One-line Summary", "content": "..."},
    {"heading": "Problem & Motivation", "content": "..."},
    {"heading": "Key Contributions", "content": "1. ..."},
    {"heading": "Method", "content": "..."},
    {"heading": "Experiments", "content": "..."},
    {"heading": "Ablation & Analysis", "content": "..."},
    {"heading": "Limitations & Future Work", "content": "..."},
    {"heading": "Overall Assessment", "content": "..."}
  ]
}
"""

from __future__ import annotations

import argparse
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional

import requests

NOTION_API_VERSION = "2025-09-03"
BLOCK_MAX_TEXT = 1800
CHUNK_BATCH = 80
EMOJI_RE = re.compile(
    "["
    "\U0001F1E6-\U0001F1FF"
    "\U0001F300-\U0001F5FF"
    "\U0001F600-\U0001F64F"
    "\U0001F680-\U0001F6FF"
    "\U0001F700-\U0001F77F"
    "\U0001F780-\U0001F7FF"
    "\U0001F800-\U0001F8FF"
    "\U0001F900-\U0001F9FF"
    "\U0001FA00-\U0001FAFF"
    "\U00002600-\U000026FF"
    "\U00002700-\U000027BF"
    "\U00002B00-\U00002BFF"
    "\uFE0F"
    "\u200D"
    "]+",
    flags=re.UNICODE,
)

# Metadata heading labels recognized across languages (for dedup)
_META_HEADINGS = {
    "metadata", "0. metadata",
    "메타데이터", "0. 메타데이터",
    "メタデータ", "0. メタデータ",
    "métadonnées", "0. métadonnées",
    "metadaten", "0. metadaten",
}


def _warn(message: str) -> None:
    print(f"[WARN] {message}", file=sys.stderr)


def _sanitize_token(value: str) -> str:
    if not value:
        return ""
    return "".join(ch for ch in value.strip() if 31 < ord(ch) < 127).strip()


def _strip_emoji(text: str) -> str:
    if not text:
        return ""
    cleaned = EMOJI_RE.sub("", text)
    return re.sub(r"\s+", " ", cleaned).strip()


def _read_api_key(override: Optional[str] = None) -> str:
    if override:
        return _sanitize_token(override)
    key = os.environ.get("NOTION_API_KEY")
    if key:
        return _sanitize_token(key)
    key_file = Path.home() / ".config" / "notion" / "api_key"
    if key_file.exists():
        return _sanitize_token(key_file.read_text(encoding="utf-8"))
    return ""


def notion_request(method: str, path: str, token: str, payload: Optional[dict] = None) -> dict:
    headers = {
        "Authorization": f"Bearer {token}",
        "Notion-Version": NOTION_API_VERSION,
        "Content-Type": "application/json",
    }
    url = f"https://api.notion.com/v1{path}"
    r = requests.request(method, url, headers=headers, json=payload, timeout=40)
    if r.status_code >= 400:
        raise RuntimeError(f"Notion API error {r.status_code}: {r.text[:400]}")
    if r.status_code == 204:
        return {}
    return r.json()


def get_block_children(block_id: str, token: str, page_size: int = 100) -> List[dict]:
    cursor = None
    items = []
    while True:
        path = f"/blocks/{block_id}/children?page_size={page_size}"
        if cursor:
            path += f"&start_cursor={cursor}"
        data = notion_request("GET", path, token)
        items.extend(data.get("results", []))
        if not data.get("has_more"):
            break
        cursor = data.get("next_cursor")
    return items


def clear_block_children(page_id: str, token: str) -> None:
    blocks = get_block_children(page_id, token)
    for block in blocks:
        notion_request("DELETE", f"/blocks/{block['id']}", token)


def append_blocks(page_id: str, token: str, blocks: List[Dict], batch_size: int = CHUNK_BATCH) -> None:
    for i in range(0, len(blocks), batch_size):
        notion_request("PATCH", f"/blocks/{page_id}/children", token, {"children": blocks[i : i + batch_size]})


def _normalize_title(t: str) -> str:
    return re.sub(r"\s+", " ", t).strip().lower()


def search_page_by_title(title: str, token: str) -> Optional[str]:
    payload = {
        "query": title,
        "filter": {"value": "page", "property": "object"},
        "page_size": 10,
    }
    try:
        data = notion_request("POST", "/search", token, payload)
    except Exception:
        data = notion_request("POST", "/search", token, {"query": title, "page_size": 10})

    target = _normalize_title(title)
    for item in data.get("results", []):
        if item.get("object") != "page":
            continue
        try:
            rich = item.get("properties", {}).get("title", {}).get("title", [])
            item_title = "".join([x.get("plain_text", "") for x in rich])
        except Exception:
            item_title = ""
        if item_title and _normalize_title(item_title) == target:
            return item.get("id")
    return None


def chunk_text(text: str, max_len: int = BLOCK_MAX_TEXT) -> List[str]:
    """Split text into Notion-safe chunks, preserving line breaks."""
    if not text:
        return [""]
    text = text.strip()
    if len(text) <= max_len:
        return [text]

    chunks = []
    current = []
    cur_len = 0
    for line in text.split("\n"):
        line_len = len(line) + 1
        if cur_len + line_len > max_len and current:
            chunks.append("\n".join(current))
            current = [line]
            cur_len = line_len
        else:
            current.append(line)
            cur_len += line_len
    if current:
        chunks.append("\n".join(current))
    return chunks


def rich_text_block(text: str, t: str = "paragraph") -> Dict:
    text = text if text else "(No content)"
    return {
        "object": "block",
        "type": t,
        t: {"rich_text": [{"type": "text", "text": {"content": text}}]},
    }


def heading_block(level: int, text: str) -> Dict:
    level = max(1, min(3, int(level)))
    return {
        "object": "block",
        "type": f"heading_{level}",
        f"heading_{level}": {"rich_text": [{"type": "text", "text": {"content": text}}]},
    }


def divider_block() -> Dict:
    return {"object": "block", "type": "divider", "divider": {}}


def bulleted_list_block(text: str) -> Dict:
    return {
        "object": "block",
        "type": "bulleted_list_item",
        "bulleted_list_item": {"rich_text": [{"type": "text", "text": {"content": text}}]},
    }


def numbered_list_block(text: str) -> Dict:
    return {
        "object": "block",
        "type": "numbered_list_item",
        "numbered_list_item": {"rich_text": [{"type": "text", "text": {"content": text}}]},
    }


def equation_block(expression: str) -> Dict:
    return {"object": "block", "type": "equation", "equation": {"expression": expression}}


def _parse_content_to_blocks(content: str, heading_level: int = 3, section_heading: str = "") -> List[Dict]:
    """Parse markdown-like content into Notion blocks.

    Supports: numbered/bulleted lists, sub-headings (###), paragraphs, LaTeX equations ($$).
    Programming code fences are converted to plain text with a warning.
    """
    blocks: List[Dict] = []
    lines = content.strip().split("\n")
    math_fence_langs = {"latex", "tex", "math", "equation"}

    i = 0
    paragraph_buffer: List[str] = []

    def flush_paragraph() -> None:
        if paragraph_buffer:
            text = "\n".join(paragraph_buffer).strip()
            if text:
                for chunk in chunk_text(text):
                    blocks.append(rich_text_block(chunk))
            paragraph_buffer.clear()

    def append_equation_or_text(expr: str) -> None:
        expression = expr.strip()
        if not expression:
            return
        if "\n" not in expression and len(expression) <= 1000:
            blocks.append(equation_block(expression))
            return
        rendered = f"$$\n{expression}\n$$" if "\n" in expression else f"$${expression}$$"
        for chunk in chunk_text(rendered):
            blocks.append(rich_text_block(chunk))

    while i < len(lines):
        line = lines[i].rstrip()

        if not line.strip():
            flush_paragraph()
            i += 1
            continue

        # Markdown fenced block
        if line.strip().startswith("```"):
            flush_paragraph()
            opening = line.strip()
            fence_lang = opening[3:].strip().lower()
            i += 1
            fence_lines: List[str] = []
            while i < len(lines) and not lines[i].strip().startswith("```"):
                fence_lines.append(lines[i].rstrip())
                i += 1
            if i < len(lines) and lines[i].strip().startswith("```"):
                i += 1

            fence_text = "\n".join(fence_lines).strip()
            if fence_lang in math_fence_langs:
                append_equation_or_text(fence_text)
                continue

            _warn(
                f'Code block detected in section="{section_heading or "unknown"}", '
                f'lang="{fence_lang or "plain"}". '
                "Converting to plain text (code blocks are not supported in summaries)."
            )
            if fence_text:
                for chunk in chunk_text(fence_text):
                    blocks.append(rich_text_block(chunk))
            continue

        # Sub-heading
        if line.startswith("### "):
            flush_paragraph()
            blocks.append(heading_block(heading_level, line[4:].strip()))
            i += 1
            continue

        # Numbered list
        m = re.match(r"^\s*(\d+)\.\s+(.+)", line)
        if m:
            flush_paragraph()
            blocks.append(numbered_list_block(m.group(2).strip()))
            i += 1
            continue

        # Bulleted list
        m = re.match(r"^\s*[-*]\s+(.+)", line)
        if m:
            flush_paragraph()
            blocks.append(bulleted_list_block(m.group(1).strip()))
            i += 1
            continue

        # Block equation
        if line.strip().startswith("$$"):
            flush_paragraph()
            stripped = line.strip()
            if stripped.count("$$") >= 2 and stripped != "$$":
                eq_text = stripped.replace("$$", "").strip()
                i += 1
            else:
                eq_lines = [stripped.replace("$$", "")]
                i += 1
                while i < len(lines) and "$$" not in lines[i]:
                    eq_lines.append(lines[i].strip())
                    i += 1
                if i < len(lines):
                    eq_lines.append(lines[i].strip().replace("$$", ""))
                    i += 1
                eq_text = "\n".join(l for l in eq_lines if l).strip()
            append_equation_or_text(eq_text)
            continue

        paragraph_buffer.append(line)
        i += 1

    flush_paragraph()
    return blocks


def build_notion_blocks(summary: Dict) -> List[Dict]:
    """Build Notion blocks from a structured summary JSON."""
    blocks = []
    meta = summary.get("metadata", {})

    # Title heading
    blocks.append(heading_block(1, summary.get("title", "Untitled")))
    blocks.append(divider_block())

    # Metadata section
    blocks.append(heading_block(2, "0. Metadata"))
    meta_lines = [
        f"Authors: {meta.get('authors', 'N/A')}",
        f"Year: {meta.get('year', 'N/A')}",
        f"Venue: {meta.get('venue', 'N/A')}",
    ]
    if meta.get("doi"):
        meta_lines.append(f"DOI: {meta['doi']}")
    if meta.get("url"):
        meta_lines.append(f"URL: {meta['url']}")
    meta_lines.append(f"Collected: {datetime.now(timezone.utc).strftime('%Y-%m-%d')}")

    for line in meta_lines:
        blocks.append(bulleted_list_block(line))

    blocks.append(divider_block())

    # Content sections
    visible_idx = 0
    for section in summary.get("sections", []):
        raw_heading = section.get("heading", "") or ""
        heading = _strip_emoji(raw_heading) or "Section"
        if heading != raw_heading:
            _warn(f'Emoji stripped from heading: "{raw_heading}" -> "{heading}"')

        # Skip metadata section if already rendered above
        if heading.lower().strip() in {h.lower() for h in _META_HEADINGS}:
            continue

        visible_idx += 1
        content = section.get("content", "")

        # Strip leading number prefix to avoid double-numbering
        stripped_heading = re.sub(r"^\d+\.\s*", "", heading).strip() or heading

        blocks.append(heading_block(2, f"{visible_idx}. {stripped_heading}"))
        content_blocks = _parse_content_to_blocks(content, section_heading=stripped_heading)
        blocks.extend(content_blocks)
        blocks.append(divider_block())

    return blocks


def create_or_update_page(
    summary: Dict,
    parent_page_id: str,
    token: str,
    force_update: bool = False,
    dry_run: bool = False,
) -> str:
    title = summary.get("title", "Untitled")
    all_blocks = build_notion_blocks(summary)

    if dry_run:
        return json.dumps({"title": title, "block_count": len(all_blocks), "mode": "dry-run"}, ensure_ascii=False, indent=2)

    existing = search_page_by_title(title, token)

    if existing and force_update:
        clear_block_children(existing, token)
        append_blocks(existing, token, all_blocks)
        return existing

    if existing and not force_update:
        append_blocks(existing, token, all_blocks)
        return existing

    payload = {
        "parent": {"page_id": parent_page_id},
        "properties": {"title": {"title": [{"text": {"content": title}}]}},
    }
    page = notion_request("POST", "/pages", token, payload)
    page_id = page.get("id", "")
    append_blocks(page_id, token, all_blocks)
    return page_id


def parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="Push a structured paper summary to Notion")
    p.add_argument("summary_json", help="Path to summary JSON file")
    p.add_argument("--parent-page-id", help="Notion parent page ID")
    p.add_argument("--notion-key", help="Notion API token")
    p.add_argument("--force-update", action="store_true", help="Overwrite existing page content")
    p.add_argument("--dry-run", action="store_true", help="Preview without uploading")
    return p.parse_args()


def main() -> int:
    args = parse_args()
    [REDACTED_SECRET])
    if not [REDACTED_SECRET]"NOTION_API_KEY is required")
        return 1

    parent = args.parent_page_id or os.environ.get("NOTION_PARENT_PAGE_ID", "")
    if not parent:
        print("parent page id required: --parent-page-id or NOTION_PARENT_PAGE_ID")
        return 1

    summary_path = Path(args.summary_json)
    if not summary_path.exists():
        print(f"Summary file not found: {summary_path}")
        return 1

    summary = json.loads(summary_path.read_text(encoding="utf-8"))
    result = create_or_update_page(summary, parent, token, force_update=args.force_update, dry_run=args.dry_run)

    if args.dry_run:
        print(result)
    else:
        print(f"Notion page created/updated: {result}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


Security scan verdicts — ClawScan: suspicious | static analysis:  | VirusTotal: suspicious | SkillSpector: clean (LOW)