Skill: elio040208/arxiv-paper-reader (version 1.0.3)

--- name: arxiv-paper-reader description: Search arXiv by keyword, filter by submitted date range, fetch arXiv papers from an arXiv ID or URL, convert papers into Markdown and PDF files in the workspace, and maintain daily topic archives with summary files. Use when the user wants to search arXiv, browse recent keyword-matched papers, ingest a specific paper, summarize arXiv content, or run a recurring topic sync. metadata: {"openclaw":{"os":["win32","linux","darwin"],"requires":{"anyBins":["python","python3"]}}} --- # arXiv Paper Reader Use the bundled Python scripts before reasoning about arXiv content. They handle: - searching arXiv by keyword - filtering keyword results by submitted date range - downloading arXiv metadata and paper content - converting papers to Markdown and PDF in the workspace - syncing configured topics into daily archive folders ## Inputs - Accept raw arXiv IDs like `1706.03762` or URLs such as `https://arxiv.org/abs/1706.03762`. - Only accept raw IDs or HTTPS arXiv URLs on `arxiv.org`, `www.arxiv.org`, or `export.arxiv.org`. - Accept keyword searches such as `transformer`, `diffusion`, or `computer vision`. - Accept optional submitted-date windows using `YYYY-MM-DD`. - Do not use category filters or alias-based domain shortcuts; search is intentionally keyword-only. ## Search workflow 1. Pick a Python command: - Prefer `python` - Fall back to `python3` 2. If the user wants search results or the latest papers for a topic, run: ```bash python {baseDir}/scripts/search_arxiv.py --query "<keywords>" --limit <n> ``` 3. Read `search_results.md` and `search_results.json`. 4. Use `{baseDir}/references/search-usage.md` to present the results. 5. If the user asks for the latest papers matching a keyword, pass `--sort submittedDate`. 6. If the user wants the default best-match ranking, omit `--sort` and let the script use relevance order. 7. If the user gives a date window, add `--start-date YYYY-MM-DD --end-date YYYY-MM-DD`. ## Topic sync workflow 1. Tell the user to maintain `{rootDir}/topics.json`, or seed it from `{baseDir}/references/topics.example.json`. 2. For recurring daily updates, run: ```bash python {baseDir}/scripts/sync_arxiv_topics.py --daily --root-dir <root-dir> ``` 3. For manual backfill, run: ```bash python {baseDir}/scripts/sync_arxiv_topics.py --start-date YYYY-MM-DD --end-date YYYY-MM-DD --root-dir <root-dir> ``` 4. Read `<root-dir>/runs/<capture-date>/run_manifest.md` first. 5. Each captured paper lives at `topics/<topic-slug>/<capture-date>/<paper-id>__<title-slug>/`. 6. Expect each paper directory to contain `paper.pdf`, `paper.md`, `metadata.json`, and `summary.md`. 7. The batch summary is template-based and grounded in the abstract plus converted Markdown; treat it as a review aid, not a substitute for reading the paper. ## Fetch workflow 1. Choose an output directory: - If the user gives one, use it. - Otherwise write to `./artifacts/arxiv/<paper-id>/` in the current workspace. 2. Run the converter: ```bash python {baseDir}/scripts/arxiv_to_md.py <paper-id-or-url> --output-dir <target-dir> ``` 3. Read the generated `paper.pdf`, `paper.md`, and `metadata.json`. 4. Summarize the paper in Markdown. 5. Save the summary to `<target-dir>/summary.md` if the user asked for files. Otherwise return the summary directly in chat. ## Summary format Use the headings in `{baseDir}/references/summary-format.md`. Keep the summary grounded in the generated Markdown. If the conversion falls back to abstract-only mode, say so explicitly in the summary. ## Safety - Pass IDs, URLs, and keywords as single CLI arguments. Do not splice untrusted text into shell pipelines. - Only pass raw arXiv IDs or HTTPS arXiv URLs; reject arbitrary third-party URLs. - TLS verification is strict. If requests fail because your machine lacks a valid CA bundle, install `certifi` or fix the system trust store. - arXiv source archives are processed in-memory, only `.tex` members are read, and suspicious paths plus oversized payloads are rejected before parsing. - Date windows use arXiv `submittedDate` and inclusive `YYYY-MM-DD` boundaries. - Do not invent claims that are not supported by `paper.md` or `search_results.md`. - Do not reintroduce hardcoded category or alias mappings; keep search behavior keyword-only.

path: scripts/arxiv_api.py
content: #!/usr/bin/env python3
"""Shared helpers for talking to the arXiv API."""

from __future__ import annotations

import re
import ssl
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from datetime import date

ARXIV_QUERY_URL = "https://export.arxiv.org/api/query?{query_string}"
ARXIV_API_ID_URL = "https://export.arxiv.org/api/query?id_list={paper_id}"
ARXIV_ABS_URL = "https://arxiv.org/abs/{paper_id}"
ARXIV_HTML_URL = "https://arxiv.org/html/{paper_id}"
ARXIV_PDF_URL = "https://arxiv.org/pdf/{paper_id}.pdf"
ARXIV_EPRINT_URL = "https://arxiv.org/e-print/{paper_id}"
USER_AGENT = "openclaw-arxiv-paper-reader/0.2"
DEFAULT_TIMEOUT = 30.0
DEFAULT_LIMIT = 5
MAX_LIMIT = 20
VALID_SORTS = {"relevance", "submittedDate", "lastUpdatedDate"}
VALID_ORDERS = {"ascending", "descending"}
ATOM_NS = {
    "atom": "http://www.w3.org/2005/Atom",
    "opensearch": "http://a9.com/-/spec/opensearch/1.1/",
}
ARXIV_ALLOWED_HOSTS = {"arxiv.org", "www.arxiv.org", "export.arxiv.org"}


@dataclass
class PaperMetadata:
    paper_id: str
    title: str
    abstract: str
    authors: list[str]
    categories: list[str]
    published: str
    updated: str


@dataclass
class SearchResult:
    paper_id: str
    title: str
    abstract: str
    authors: list[str]
    categories: list[str]
    published: str
    updated: str
    abstract_url: str
    pdf_url: str


@dataclass
class SearchResponse:
    entries: list[SearchResult]
    total_results: int
    returned_results: int
    search_query: str
    sort_by: str
    sort_order: str
    start_date: str | None
    end_date: str | None


def normalize_space(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


def _validate_identifier(candidate: str, original_value: str) -> str:
    if candidate.endswith(".pdf"):
        candidate = candidate[:-4]

    if not candidate:
        raise ValueError("empty arXiv identifier")

    if not re.fullmatch(r"[A-Za-z0-9._/-]+(?:v\d+)?", candidate):
        raise ValueError(f"unsupported arXiv identifier: {original_value}")

    return candidate


def _extract_identifier_from_arxiv_url(url: str, *, allow_http: bool) -> str:
    parsed = urllib.parse.urlparse(url)
    scheme = (parsed.scheme or "").lower()
    if scheme != "https" and not (allow_http and scheme == "http"):
        raise ValueError("only HTTPS arXiv URLs are supported; use a raw arXiv ID or a valid arXiv URL")

    host = (parsed.hostname or "").lower()
    if host not in ARXIV_ALLOWED_HOSTS:
        raise ValueError("only arXiv URLs are supported; use a raw arXiv ID or a valid arXiv URL")

    path = parsed.path.strip("/")
    if path.startswith("abs/"):
        return _validate_identifier(path[len("abs/") :], url)
    if path.startswith("html/"):
        return _validate_identifier(path[len("html/") :], url)
    if path.startswith("e-print/"):
        return _validate_identifier(path[len("e-print/") :], url)
    if path.startswith("pdf/") and path.endswith(".pdf"):
        return _validate_identifier(path[len("pdf/") : -4], url)

    raise ValueError("unsupported arXiv URL path; use /abs/, /pdf/, /html/, or /e-print/")


def normalize_identifier(value: str, *, allow_http_url: bool = False) -> str:
    candidate = value.strip()
    candidate = re.sub(r"^arxiv:\s*", "", candidate, flags=re.IGNORECASE)

    if re.match(r"^https?://", candidate, flags=re.IGNORECASE):
        return _extract_identifier_from_arxiv_url(candidate, allow_http=allow_http_url)

    return _validate_identifier(candidate, value)


def safe_dir_name(value: str) -> str:
    return re.sub(r"[^A-Za-z0-9._-]+", "_", value)


def build_ssl_context() -> ssl.SSLContext:
    try:
        import certifi  # type: ignore

        return ssl.create_default_context(cafile=certifi.where())
    except Exception:
        return ssl.create_default_context()


def request_bytes(url: str, timeout: float = DEFAULT_TIMEOUT) -> bytes:
    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    context = build_ssl_context()
    try:
        with urllib.request.urlopen(request, timeout=timeout, context=context) as response:
            return response.read()
    except ssl.SSLCertVerificationError as error:
        raise RuntimeError(
            "TLS certificate verification failed while fetching arXiv content. "
            "Install certifi or fix the system CA bundle."
        ) from error
    except urllib.error.URLError as error:
        reason = getattr(error, "reason", None)
        if isinstance(reason, ssl.SSLCertVerificationError) or "CERTIFICATE_VERIFY_FAILED" in str(reason or error):
            raise RuntimeError(
                "TLS certificate verification failed while fetching arXiv content. "
                "Install certifi or fix the system CA bundle."
            ) from error
        raise


def decode_bytes(payload: bytes) -> str:
    for encoding in ("utf-8", "utf-8-sig", "latin-1"):
        try:
            return payload.decode(encoding)
        except UnicodeDecodeError:
            continue
    return payload.decode("utf-8", errors="replace")


def request_text(url: str, timeout: float = DEFAULT_TIMEOUT) -> str:
    return decode_bytes(request_bytes(url, timeout))


def build_query_text(query: str) -> str:
    text = normalize_space(query).replace('"', "")
    if not text:
        raise ValueError("query cannot be empty")
    if re.fullmatch(r"[A-Za-z0-9._+-]+", text):
        return f"all:{text}"
    return f'all:"{text}"'


def build_search_query(query: str | None) -> str:
    if not query or not normalize_space(query):
        raise ValueError("query is required")
    return build_query_text(query)


def validate_date_value(value: str, field_name: str) -> str:
    try:
        parsed = date.fromisoformat(value)
    except ValueError as error:
        raise ValueError(f"{field_name} must use YYYY-MM-DD format") from error
    return parsed.isoformat()


def validate_date_range(start_date: str | None, end_date: str | None) -> tuple[str | None, str | None]:
    if start_date is None and end_date is None:
        return None, None
    if not start_date or not end_date:
        raise ValueError("start_date and end_date must be provided together")

    normalized_start = validate_date_value(start_date, "start_date")
    normalized_end = validate_date_value(end_date, "end_date")
    if normalized_start > normalized_end:
        raise ValueError("start_date must be less than or equal to end_date")
    return normalized_start, normalized_end


def build_submitted_date_query(start_date: str | None, end_date: str | None) -> str | None:
    normalized_start, normalized_end = validate_date_range(start_date, end_date)
    if normalized_start is None or normalized_end is None:
        return None
    start_token = normalized_start.replace("-", "")
    end_token = normalized_end.replace("-", "")
    return f"submittedDate:[{start_token}0000 TO {end_token}2359]"


def build_search_query_with_dates(query: str | None, start_date: str | None, end_date: str | None) -> tuple[str, str | None, str | None]:
    query_part = build_search_query(query)
    date_part = build_submitted_date_query(start_date, end_date)
    if not date_part:
        return query_part, None, None
    normalized_start, normalized_end = validate_date_range(start_date, end_date)
    return f"{query_part} AND {date_part}", normalized_start, normalized_end


def validate_limit(limit: int) -> int:
    if not 1 <= limit <= MAX_LIMIT:
        raise ValueError(f"limit must be between 1 and {MAX_LIMIT}")
    return limit


def default_sort_for() -> str:
    return "relevance"


def validate_sort(sort_by: str) -> str:
    if sort_by not in VALID_SORTS:
        raise ValueError(f"unsupported sort: {sort_by}")
    return sort_by


def validate_order(sort_order: str) -> str:
    if sort_order not in VALID_ORDERS:
        raise ValueError(f"unsupported order: {sort_order}")
    return sort_order


def parse_search_entry(entry: ET.Element) -> SearchResult:
    paper_id = normalize_identifier(entry.findtext("atom:id", default="", namespaces=ATOM_NS), allow_http_url=True)
    authors = [
        normalize_space(node.text or "")
        for node in entry.findall("atom:author/atom:name", ATOM_NS)
        if normalize_space(node.text or "")
    ]
    categories = [node.attrib["term"] for node in entry.findall("atom:category", ATOM_NS) if "term" in node.attrib]
    return SearchResult(
        paper_id=paper_id,
        title=normalize_space(entry.findtext("atom:title", default="", namespaces=ATOM_NS)),
        abstract=normalize_space(entry.findtext("atom:summary", default="", namespaces=ATOM_NS)),
        authors=authors,
        categories=categories,
        published=normalize_space(entry.findtext("atom:published", default="", namespaces=ATOM_NS)),
        updated=normalize_space(entry.findtext("atom:updated", default="", namespaces=ATOM_NS)),
        abstract_url=ARXIV_ABS_URL.format(paper_id=paper_id),
        pdf_url=ARXIV_PDF_URL.format(paper_id=paper_id),
    )


def parse_search_response(
    xml_text: str,
    search_query: str,
    sort_by: str,
    sort_order: str,
    start_date: str | None,
    end_date: str | None,
) -> SearchResponse:
    root = ET.fromstring(xml_text)
    entries = [parse_search_entry(entry) for entry in root.findall("atom:entry", ATOM_NS)]
    total_results = int(root.findtext("opensearch:totalResults", default="0", namespaces=ATOM_NS))
    return SearchResponse(
        entries=entries,
        total_results=total_results,
        returned_results=len(entries),
        search_query=search_query,
        sort_by=sort_by,
        sort_order=sort_order,
        start_date=start_date,
        end_date=end_date,
    )


def fetch_metadata(paper_id: str, timeout: float = DEFAULT_TIMEOUT) -> PaperMetadata:
    xml_text = request_text(ARXIV_API_ID_URL.format(paper_id=urllib.parse.quote(paper_id)), timeout)
    root = ET.fromstring(xml_text)
    entry = root.find("atom:entry", ATOM_NS)
    if entry is None:
        raise RuntimeError(f"paper not found in arXiv API: {paper_id}")

    authors = [
        normalize_space(node.text or "")
        for node in entry.findall("atom:author/atom:name", ATOM_NS)
        if normalize_space(node.text or "")
    ]
    categories = [node.attrib["term"] for node in entry.findall("atom:category", ATOM_NS) if "term" in node.attrib]

    return PaperMetadata(
        paper_id=paper_id,
        title=normalize_space(entry.findtext("atom:title", default="", namespaces=ATOM_NS)),
        abstract=normalize_space(entry.findtext("atom:summary", default="", namespaces=ATOM_NS)),
        authors=authors,
        categories=categories,
        published=normalize_space(entry.findtext("atom:published", default="", namespaces=ATOM_NS)),
        updated=normalize_space(entry.findtext("atom:updated", default="", namespaces=ATOM_NS)),
    )


def search_papers(
    query: str | None,
    *,
    start_date: str | None = None,
    end_date: str | None = None,
    sort_by: str | None = None,
    sort_order: str = "descending",
    limit: int = DEFAULT_LIMIT,
    timeout: float = DEFAULT_TIMEOUT,
) -> SearchResponse:
    limit = validate_limit(limit)
    search_query, normalized_start, normalized_end = build_search_query_with_dates(query, start_date, end_date)
    resolved_sort = validate_sort(sort_by or default_sort_for())
    resolved_order = validate_order(sort_order)
    params = urllib.parse.urlencode(
        {
            "search_query": search_query,
            "start": 0,
            "max_results": limit,
            "sortBy": resolved_sort,
            "sortOrder": resolved_order,
        }
    )
    xml_text = request_text(ARXIV_QUERY_URL.format(query_string=params), timeout)
    return parse_search_response(
        xml_text,
        search_query,
        resolved_sort,
        resolved_order,
        normalized_start,
        normalized_end,
    )


path: scripts/arxiv_to_md.py
content: #!/usr/bin/env python3
"""Fetch an arXiv paper and convert it to Markdown."""

from __future__ import annotations

import argparse
import gzip
import io
import json
import re
import sys
import tarfile
import textwrap
import urllib.error
import urllib.parse
import zipfile
from dataclasses import asdict
from html.parser import HTMLParser
from pathlib import Path, PurePosixPath

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from arxiv_api import (  # noqa: E402
    ARXIV_ABS_URL,
    ARXIV_EPRINT_URL,
    ARXIV_HTML_URL,
    ARXIV_PDF_URL,
    PaperMetadata,
    fetch_metadata,
    normalize_identifier,
    request_bytes,
    request_text,
    safe_dir_name,
)

MAX_ARCHIVE_MEMBERS = 5000
MAX_ACCEPTED_TEX_FILES = 200
MAX_TEX_FILE_BYTES = 5 * 1024 * 1024
MAX_TOTAL_TEX_BYTES = 25 * 1024 * 1024


def decode_bytes(payload: bytes) -> str:
    for encoding in ("utf-8", "utf-8-sig", "latin-1"):
        try:
            return payload.decode(encoding)
        except UnicodeDecodeError:
            continue
    return payload.decode("utf-8", errors="replace")


def normalize_space(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


class HTMLToMarkdownParser(HTMLParser):
    SKIP_TAGS = {"script", "style", "svg", "noscript", "nav", "footer", "header", "form", "button"}

    def __init__(self, capture_tag: str | None = None) -> None:
        super().__init__(convert_charrefs=True)
        self.capture_tag = capture_tag
        self.capture_depth = 0
        self.in_capture = capture_tag is None
        self.skip_depth = 0
        self.parts: list[str] = []
        self.list_stack: list[dict[str, int | str]] = []
        self.preformatted = False
        self.link_stack: list[str | None] = []

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        tag = tag.lower()
        attributes = dict(attrs)

        if self.capture_tag and tag == self.capture_tag:
            self.capture_depth += 1
            self.in_capture = True

        if not self.in_capture:
            return

        if tag in self.SKIP_TAGS:
            self.skip_depth += 1
            return

        if self.skip_depth:
            return

        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}:
            level = int(tag[1])
            self.parts.append(f"\n\n{'#' * level} ")
        elif tag in {"p", "section", "article", "main"}:
            self.parts.append("\n\n")
        elif tag == "br":
            self.parts.append("\n")
        elif tag == "blockquote":
            self.parts.append("\n\n> ")
        elif tag == "ul":
            self.list_stack.append({"type": "ul", "counter": 0})
            self.parts.append("\n")
        elif tag == "ol":
            self.list_stack.append({"type": "ol", "counter": 0})
            self.parts.append("\n")
        elif tag == "li":
            indent = "  " * max(len(self.list_stack) - 1, 0)
            if self.list_stack and self.list_stack[-1]["type"] == "ol":
                self.list_stack[-1]["counter"] = int(self.list_stack[-1]["counter"]) + 1
                bullet = f"{self.list_stack[-1]['counter']}. "
            else:
                bullet = "- "
            self.parts.append(f"\n{indent}{bullet}")
        elif tag in {"strong", "b"}:
            self.parts.append("**")
        elif tag in {"em", "i"}:
            self.parts.append("*")
        elif tag == "code" and not self.preformatted:
            self.parts.append("`")
        elif tag == "pre":
            self.preformatted = True
            self.parts.append("\n\n```text\n")
        elif tag == "a":
            self.link_stack.append(attributes.get("href"))

    def handle_endtag(self, tag: str) -> None:
        tag = tag.lower()

        if self.skip_depth and tag in self.SKIP_TAGS:
            self.skip_depth -= 1
            return

        if not self.in_capture or self.skip_depth:
            if self.capture_tag and tag == self.capture_tag and self.capture_depth:
                self.capture_depth -= 1
                if self.capture_depth == 0:
                    self.in_capture = False
            return

        if tag in {"h1", "h2", "h3", "h4", "h5", "h6", "p", "section"}:
            self.parts.append("\n\n")
        elif tag in {"strong", "b"}:
            self.parts.append("**")
        elif tag in {"em", "i"}:
            self.parts.append("*")
        elif tag == "code" and not self.preformatted:
            self.parts.append("`")
        elif tag == "pre":
            self.preformatted = False
            self.parts.append("\n```\n")
        elif tag == "a":
            href = self.link_stack.pop() if self.link_stack else None
            if href:
                self.parts.append(f" ({href})")
        elif tag in {"ul", "ol"} and self.list_stack:
            self.list_stack.pop()
            self.parts.append("\n")

        if self.capture_tag and tag == self.capture_tag and self.capture_depth:
            self.capture_depth -= 1
            if self.capture_depth == 0:
                self.in_capture = False

    def handle_data(self, data: str) -> None:
        if not self.in_capture or self.skip_depth:
            return

        if self.preformatted:
            self.parts.append(data)
            return

        collapsed = re.sub(r"\s+", " ", data)
        if not collapsed.strip():
            if self.parts and not self.parts[-1].endswith((" ", "\n")):
                self.parts.append(" ")
            return

        self.parts.append(collapsed)

    def markdown(self) -> str:
        text = "".join(self.parts)
        text = text.replace("\r\n", "\n").replace("\r", "\n")
        text = re.sub(r"[ \t]+\n", "\n", text)
        text = re.sub(r"\n{3,}", "\n\n", text)
        text = re.sub(r"\n +", "\n", text)
        return text.strip()


def html_to_markdown(html_text: str) -> str:
    lowered = html_text.lower()
    capture_tag = None
    if "<article" in lowered:
        capture_tag = "article"
    elif "<main" in lowered:
        capture_tag = "main"

    parser = HTMLToMarkdownParser(capture_tag=capture_tag)
    parser.feed(html_text)
    parser.close()
    return parser.markdown()


def content_looks_complete(markdown_text: str) -> bool:
    word_count = len(re.findall(r"\b\w+\b", markdown_text))
    heading_count = len(re.findall(r"^\s{0,3}#{1,6}\s", markdown_text, flags=re.MULTILINE))
    return word_count >= 250 or (word_count >= 150 and heading_count >= 3)


def fetch_html_markdown(paper_id: str, timeout: float) -> str | None:
    try:
        html_text = request_text(ARXIV_HTML_URL.format(paper_id=urllib.parse.quote(paper_id)), timeout)
    except urllib.error.HTTPError as error:
        if error.code in {403, 404}:
            return None
        raise

    markdown_text = html_to_markdown(html_text)
    if not content_looks_complete(markdown_text):
        return None
    return markdown_text


def validate_archive_member_name(name: str) -> str:
    normalized = name.replace("\\", "/").strip()
    if not normalized:
        raise RuntimeError("archive member name cannot be empty")
    if normalized.startswith("/"):
        raise RuntimeError(f"unsafe archive member path: {name}")
    if re.match(r"^[A-Za-z]:", normalized):
        raise RuntimeError(f"unsafe archive member path: {name}")

    parts = PurePosixPath(normalized).parts
    if any(part == ".." for part in parts):
        raise RuntimeError(f"unsafe archive member path: {name}")

    return normalized


def validate_member_count(count: int) -> None:
    if count > MAX_ARCHIVE_MEMBERS:
        raise RuntimeError(f"archive contains too many members (limit: {MAX_ARCHIVE_MEMBERS})")


def validate_tex_file_size(size: int, name: str) -> None:
    if size > MAX_TEX_FILE_BYTES:
        raise RuntimeError(f"archive member {name} exceeds the per-file size limit ({MAX_TEX_FILE_BYTES} bytes)")


def update_total_tex_bytes(current_total: int, added_size: int) -> int:
    new_total = current_total + added_size
    if new_total > MAX_TOTAL_TEX_BYTES:
        raise RuntimeError(f"archive exceeds the total TeX size limit ({MAX_TOTAL_TEX_BYTES} bytes)")
    return new_total


def validate_tex_file_count(current_count: int) -> None:
    if current_count >= MAX_ACCEPTED_TEX_FILES:
        raise RuntimeError(f"archive contains too many TeX files (limit: {MAX_ACCEPTED_TEX_FILES})")


def read_gzip_payload(payload: bytes, max_bytes: int) -> bytes:
    with gzip.GzipFile(fileobj=io.BytesIO(payload)) as handle:
        expanded = handle.read(max_bytes + 1)
    if len(expanded) > max_bytes:
        raise RuntimeError(f"gzip payload exceeds the size limit ({max_bytes} bytes)")
    return expanded


def decode_single_tex_payload(payload: bytes, name: str) -> dict[str, str]:
    validate_tex_file_size(len(payload), name)
    update_total_tex_bytes(0, len(payload))
    validate_archive_member_name(name)
    return {name: decode_bytes(payload)}


def unpack_source_files(payload: bytes) -> dict[str, str]:
    source_files = extract_archive_files(payload)
    if source_files:
        return source_files

    try:
        expanded = read_gzip_payload(payload, MAX_TOTAL_TEX_BYTES)
    except OSError:
        expanded = None

    if expanded is not None:
        source_files = extract_archive_files(expanded)
        if source_files:
            return source_files
        return decode_single_tex_payload(expanded, "source.tex")

    return decode_single_tex_payload(payload, "source.tex")


def extract_archive_files(payload: bytes) -> dict[str, str]:
    buffer = io.BytesIO(payload)
    output: dict[str, str] = {}
    inspected_members = 0
    total_tex_bytes = 0

    if zipfile.is_zipfile(buffer):
        buffer.seek(0)
        with zipfile.ZipFile(buffer) as archive:
            for info in archive.infolist():
                inspected_members += 1
                validate_member_count(inspected_members)
                if info.is_dir() or not info.filename.lower().endswith(".tex"):
                    continue

                validate_tex_file_count(len(output))
                normalized_name = validate_archive_member_name(info.filename)
                validate_tex_file_size(info.file_size, normalized_name)
                total_tex_bytes = update_total_tex_bytes(total_tex_bytes, info.file_size)
                output[normalized_name] = decode_bytes(archive.read(info))
        return output

    for mode in ("r:*", "r:gz", "r:bz2", "r:xz"):
        buffer.seek(0)
        try:
            with tarfile.open(fileobj=buffer, mode=mode) as archive:
                for member in archive:
                    inspected_members += 1
                    validate_member_count(inspected_members)
                    if not member.isfile() or not member.name.lower().endswith(".tex"):
                        continue

                    validate_tex_file_count(len(output))
                    normalized_name = validate_archive_member_name(member.name)
                    validate_tex_file_size(member.size, normalized_name)
                    file_handle = archive.extractfile(member)
                    if file_handle is None:
                        continue
                    payload_bytes = file_handle.read(MAX_TEX_FILE_BYTES + 1)
                    validate_tex_file_size(len(payload_bytes), normalized_name)
                    total_tex_bytes = update_total_tex_bytes(total_tex_bytes, len(payload_bytes))
                    output[normalized_name] = decode_bytes(payload_bytes)
                if output:
                    return output
        except tarfile.TarError:
            continue

    return output


def choose_main_tex(source_files: dict[str, str]) -> str:
    if not source_files:
        raise RuntimeError("no TeX files found in arXiv source package")

    best_name = ""
    best_score = -1
    for name, text in source_files.items():
        score = len(text)
        if "\\begin{document}" in text:
            score += 100000
        score += text.count("\\section") * 1000
        score += text.count("\\subsection") * 500
        if "\\title{" in text:
            score += 2000
        if score > best_score:
            best_name = name
            best_score = score
    return best_name


def resolve_inputs(text: str, source_files: dict[str, str], depth: int = 0) -> str:
    if depth > 2:
        return text

    lookup: dict[str, str] = {}
    for name, content in source_files.items():
        normalized = Path(name).as_posix()
        candidates = {normalized, Path(normalized).name}
        if normalized.endswith(".tex"):
            candidates.add(normalized[:-4])
            candidates.add(Path(normalized).stem)
        for candidate in candidates:
            lookup.setdefault(candidate, content)

    pattern = re.compile(r"\\(?:input|include)\{([^}]+)\}")

    def replace(match: re.Match[str]) -> str:
        key = match.group(1).strip().replace("\\", "/")
        options = [key, f"{key}.tex", Path(key).name, f"{Path(key).name}.tex"]
        for option in options:
            if option in lookup:
                return "\n\n" + resolve_inputs(lookup[option], source_files, depth + 1) + "\n\n"
        return ""

    return pattern.sub(replace, text)


def strip_comments(text: str) -> str:
    cleaned_lines = []
    for line in text.splitlines():
        cleaned_lines.append(re.sub(r"(?<!\\)%.*", "", line))
    return "\n".join(cleaned_lines)


def convert_math_environment(match: re.Match[str]) -> str:
    body = match.group(1).strip()
    if not body:
        return ""
    return f"\n\n```math\n{body}\n```\n\n"


def latex_to_markdown(latex_text: str) -> str:
    text = strip_comments(latex_text)

    if "\\begin{document}" in text:
        text = text.split("\\begin{document}", 1)[1]
    if "\\end{document}" in text:
        text = text.split("\\end{document}", 1)[0]

    text = re.sub(r"\\begin\{abstract\}.*?\\end\{abstract\}", "", text, flags=re.DOTALL)
    text = re.sub(
        r"\\begin\{(?:equation\*?|align\*?|gather\*?|multline\*?)\}(.*?)\\end\{(?:equation\*?|align\*?|gather\*?|multline\*?)\}",
        convert_math_environment,
        text,
        flags=re.DOTALL,
    )
    text = re.sub(r"\\\[(.*?)\\\]", convert_math_environment, text, flags=re.DOTALL)
    text = re.sub(r"\$\$(.*?)\$\$", convert_math_environment, text, flags=re.DOTALL)

    replacements = [
        (r"\\part\*?\{([^{}]+)\}", r"\n\n# \1\n\n"),
        (r"\\chapter\*?\{([^{}]+)\}", r"\n\n# \1\n\n"),
        (r"\\section\*?\{([^{}]+)\}", r"\n\n## \1\n\n"),
        (r"\\subsection\*?\{([^{}]+)\}", r"\n\n### \1\n\n"),
        (r"\\subsubsection\*?\{([^{}]+)\}", r"\n\n#### \1\n\n"),
        (r"\\paragraph\*?\{([^{}]+)\}", r"\n\n**\1.** "),
        (r"\\href\{([^{}]+)\}\{([^{}]+)\}", r"[\2](\1)"),
        (r"\\url\{([^{}]+)\}", r"<\1>"),
        (r"\\(?:emph|textit|textsl)\{([^{}]+)\}", r"*\1*"),
        (r"\\(?:textbf|bfseries)\{([^{}]+)\}", r"**\1**"),
        (r"\\(?:texttt|verb)\{([^{}]+)\}", r"`\1`"),
        (r"\\item\b", r"\n- "),
        (r"\\\\", "\n"),
        (r"~", " "),
    ]

    for _ in range(3):
        previous = text
        for pattern, replacement in replacements:
            text = re.sub(pattern, replacement, text)
        if text == previous:
            break

    text = re.sub(r"\\begin\{(?:itemize|enumerate|description)\}", "", text)
    text = re.sub(r"\\end\{(?:itemize|enumerate|description)\}", "", text)
    text = re.sub(r"\\begin\{thebibliography\}", "\n\n## References\n\n", text)
    text = re.sub(r"\\end\{thebibliography\}", "", text)
    text = re.sub(r"\\(?:cite|citet|citep|eqref|autoref|ref|pageref)\{[^{}]*\}", "[ref]", text)
    text = re.sub(r"\\(?:label|footnote|thanks|index|bibliography|bibliographystyle)\{[^{}]*\}", "", text)
    text = re.sub(r"\\[A-Za-z@]+(?:\*?)?(?:\[[^\]]*\])?", "", text)
    text = text.replace("{", "").replace("}", "")

    text = text.replace("\r\n", "\n").replace("\r", "\n")
    text = re.sub(r"[ \t]+\n", "\n", text)
    text = re.sub(r"\n{3,}", "\n\n", text)
    text = re.sub(r"(?m)^- +", "- ", text)
    text = re.sub(r"\n +", "\n", text)
    return text.strip()


def fetch_latex_markdown(paper_id: str, timeout: float) -> str | None:
    try:
        payload = request_bytes(ARXIV_EPRINT_URL.format(paper_id=urllib.parse.quote(paper_id)), timeout)
    except urllib.error.HTTPError as error:
        if error.code in {403, 404}:
            return None
        raise

    source_files = unpack_source_files(payload)
    if not source_files:
        return None

    main_tex = choose_main_tex(source_files)
    combined = resolve_inputs(source_files[main_tex], source_files)
    markdown_text = latex_to_markdown(combined)
    if not markdown_text:
        return None
    return markdown_text


def assemble_markdown(metadata: PaperMetadata, body: str, source_kind: str) -> str:
    lines = [
        f"# {metadata.title}",
        "",
        f"- arXiv ID: `{metadata.paper_id}`",
        f"- Abstract page: {ARXIV_ABS_URL.format(paper_id=metadata.paper_id)}",
        f"- PDF: {ARXIV_PDF_URL.format(paper_id=metadata.paper_id)}",
        f"- Authors: {', '.join(metadata.authors) if metadata.authors else 'Unknown'}",
        f"- Categories: {', '.join(metadata.categories) if metadata.categories else 'Unknown'}",
        f"- Published: {metadata.published or 'Unknown'}",
        f"- Updated: {metadata.updated or 'Unknown'}",
        f"- Extracted from: {source_kind}",
        "",
        "## Abstract",
        "",
        metadata.abstract or "No abstract returned by the arXiv API.",
    ]

    if body.strip():
        lines.extend(["", "## Full Text", "", body.strip()])
    elif source_kind == "abstract-only":
        lines.extend(
            [
                "",
                "## Full Text",
                "",
                "_The converter could not obtain arXiv HTML or source files. Only the abstract is available._",
            ]
        )

    return "\n".join(lines).strip() + "\n"


def build_output_paths(output_dir: Path) -> dict[str, Path]:
    return {
        "paper_pdf": output_dir / "paper.pdf",
        "paper_md": output_dir / "paper.md",
        "metadata_json": output_dir / "metadata.json",
    }


def fetch_pdf_bytes(paper_id: str, timeout: float) -> bytes:
    pdf_url = ARXIV_PDF_URL.format(paper_id=urllib.parse.quote(paper_id))
    return request_bytes(pdf_url, timeout)


def fetch_paper_to_directory(
    paper_id: str,
    output_dir: Path,
    timeout: float = 30.0,
    extra_metadata: dict[str, str] | None = None,
) -> dict[str, str]:
    output_dir.mkdir(parents=True, exist_ok=True)
    output_paths = build_output_paths(output_dir)

    metadata = fetch_metadata(paper_id, timeout)
    output_paths["paper_pdf"].write_bytes(fetch_pdf_bytes(paper_id, timeout))

    body = fetch_html_markdown(paper_id, timeout)
    source_kind = "arXiv HTML"

    if body is None:
        body = fetch_latex_markdown(paper_id, timeout)
        source_kind = "arXiv source"

    if body is None:
        body = ""
        source_kind = "abstract-only"

    paper_markdown = assemble_markdown(metadata, body, source_kind)
    output_paths["paper_md"].write_text(paper_markdown, encoding="utf-8")

    metadata_payload = {
        **asdict(metadata),
        "abstract_url": ARXIV_ABS_URL.format(paper_id=paper_id),
        "pdf_url": ARXIV_PDF_URL.format(paper_id=paper_id),
        "source_kind": source_kind,
        "paper_pdf": str(output_paths["paper_pdf"]),
        "paper_md": str(output_paths["paper_md"]),
    }
    if extra_metadata:
        metadata_payload.update(extra_metadata)
    output_paths["metadata_json"].write_text(
        json.dumps(metadata_payload, indent=2, ensure_ascii=False) + "\n",
        encoding="utf-8",
    )

    return {
        "paper_id": paper_id,
        "output_dir": str(output_dir),
        "paper_pdf": str(output_paths["paper_pdf"]),
        "paper_md": str(output_paths["paper_md"]),
        "metadata_json": str(output_paths["metadata_json"]),
        "source_kind": source_kind,
    }


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description="Fetch an arXiv paper, convert it to Markdown, and save metadata alongside it.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=textwrap.dedent(
            """\
            Examples:
              python arxiv_to_md.py 1706.03762
              python arxiv_to_md.py https://arxiv.org/abs/2401.00001 --output-dir ./artifacts/arxiv/2401.00001
            """
        ),
    )
    parser.add_argument("paper", help="arXiv identifier or URL")
    parser.add_argument("--output-dir", type=Path, help="directory for paper.md and metadata.json")
    parser.add_argument("--timeout", type=float, default=30.0, help="network timeout in seconds")
    args = parser.parse_args(argv)

    try:
        paper_id = normalize_identifier(args.paper)
        output_dir = args.output_dir or Path.cwd() / "artifacts" / "arxiv" / safe_dir_name(paper_id)
        result = fetch_paper_to_directory(paper_id, output_dir, timeout=args.timeout)
        print(json.dumps(result, ensure_ascii=False))
        return 0
    except Exception as error:  # pragma: no cover - surfaced in CLI output
        print(f"error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/search_arxiv.py
content: #!/usr/bin/env python3
"""Search arXiv papers by keyword."""

from __future__ import annotations

import argparse
import json
import sys
import textwrap
from dataclasses import asdict
from pathlib import Path

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from arxiv_api import DEFAULT_LIMIT, SearchResponse, safe_dir_name, search_papers  # noqa: E402


def default_output_dir(query: str, sort_by: str | None, start_date: str | None, end_date: str | None) -> Path:
    label = safe_dir_name(query) or "search"
    if sort_by:
        label = f"{label}-{sort_by}"
    if start_date and end_date:
        label = f"{label}-{start_date}-to-{end_date}"
    return Path.cwd() / "artifacts" / "arxiv-search" / safe_dir_name(label)


def render_markdown(response: SearchResponse) -> str:
    lines = [
        "# arXiv Search Results",
        "",
        f"- Search query: `{response.search_query}`",
        f"- Sort: `{response.sort_by}`",
        f"- Order: `{response.sort_order}`",
        f"- Start date: `{response.start_date}`" if response.start_date else "- Start date: none",
        f"- End date: `{response.end_date}`" if response.end_date else "- End date: none",
        f"- Total matches: {response.total_results}",
        f"- Returned results: {response.returned_results}",
        "",
    ]

    if not response.entries:
        lines.extend(
            [
                "No papers matched the search.",
                "",
                "Try a broader keyword or switch the sort order.",
            ]
        )
        return "\n".join(lines).strip() + "\n"

    for index, entry in enumerate(response.entries, start=1):
        lines.extend(
            [
                f"{index}. {entry.title} (`{entry.paper_id}`)",
                f"   - Authors: {', '.join(entry.authors) if entry.authors else 'Unknown'}",
                f"   - Categories: {', '.join(entry.categories) if entry.categories else 'Unknown'}",
                f"   - Published: {entry.published or 'Unknown'}",
                f"   - Updated: {entry.updated or 'Unknown'}",
                f"   - Abstract URL: {entry.abstract_url}",
                f"   - PDF: {entry.pdf_url}",
                f"   - Abstract: {entry.abstract or 'No abstract available.'}",
                "",
            ]
        )

    return "\n".join(lines).strip() + "\n"


def write_search_outputs(output_dir: Path, response: SearchResponse) -> dict[str, Path]:
    output_dir.mkdir(parents=True, exist_ok=True)
    json_path = output_dir / "search_results.json"
    md_path = output_dir / "search_results.md"
    payload = {
        "search_query": response.search_query,
        "sort_by": response.sort_by,
        "sort_order": response.sort_order,
        "start_date": response.start_date,
        "end_date": response.end_date,
        "total_results": response.total_results,
        "returned_results": response.returned_results,
        "results": [asdict(entry) for entry in response.entries],
    }
    json_path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
    md_path.write_text(render_markdown(response), encoding="utf-8")
    return {"json": json_path, "md": md_path}


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description="Search arXiv by keyword, then write JSON and Markdown result files.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=textwrap.dedent(
            """\
            Examples:
              python search_arxiv.py --query transformer
              python search_arxiv.py --query transformer --sort submittedDate --limit 5
              python search_arxiv.py --query "computer vision diffusion" --sort submittedDate
              python search_arxiv.py --query "agent security" --start-date 2026-03-01 --end-date 2026-03-07 --sort submittedDate
            """
        ),
    )
    parser.add_argument("--query", help="plain text keywords to search", required=True)
    parser.add_argument("--start-date", help="inclusive submittedDate lower bound in YYYY-MM-DD", default=None)
    parser.add_argument("--end-date", help="inclusive submittedDate upper bound in YYYY-MM-DD", default=None)
    parser.add_argument("--sort", choices=["relevance", "submittedDate", "lastUpdatedDate"], default=None)
    parser.add_argument("--order", choices=["ascending", "descending"], default="descending")
    parser.add_argument("--limit", type=int, default=DEFAULT_LIMIT)
    parser.add_argument("--output-dir", type=Path, help="directory for search_results.json and search_results.md")
    parser.add_argument("--timeout", type=float, default=30.0, help="network timeout in seconds")
    args = parser.parse_args(argv)

    try:
        response = search_papers(
            args.query,
            start_date=args.start_date,
            end_date=args.end_date,
            sort_by=args.sort,
            sort_order=args.order,
            limit=args.limit,
            timeout=args.timeout,
        )
        output_dir = args.output_dir or default_output_dir(args.query, response.sort_by, response.start_date, response.end_date)
        outputs = write_search_outputs(output_dir, response)
        print(
            json.dumps(
                {
                    "output_dir": str(output_dir),
                    "search_results_json": str(outputs["json"]),
                    "search_results_md": str(outputs["md"]),
                    "search_query": response.search_query,
                    "sort_by": response.sort_by,
                    "sort_order": response.sort_order,
                    "start_date": response.start_date,
                    "end_date": response.end_date,
                    "total_results": response.total_results,
                    "returned_results": response.returned_results,
                },
                ensure_ascii=False,
            )
        )
        return 0
    except Exception as error:  # pragma: no cover - surfaced in CLI output
        print(f"error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/sync_arxiv_topics.py
content: #!/usr/bin/env python3
"""Sync keyword-based arXiv topics into a daily local archive."""

from __future__ import annotations

import argparse
import json
import re
import sys
import textwrap
from datetime import date, datetime, timedelta
from pathlib import Path

SCRIPT_DIR = Path(__file__).resolve().parent
SKILL_DIR = SCRIPT_DIR.parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from arxiv_api import DEFAULT_LIMIT, VALID_ORDERS, VALID_SORTS, safe_dir_name, search_papers  # noqa: E402
from arxiv_to_md import fetch_paper_to_directory  # noqa: E402
from search_arxiv import write_search_outputs  # noqa: E402

DEFAULT_ROOT_DIR = Path.cwd() / "artifacts" / "arxiv-monitor"
DEFAULT_STATE_NAME = "sync_state.json"
DEFAULT_TOPICS_NAME = "topics.json"
SUMMARY_HEADINGS = [
    "## One-line takeaway",
    "## Problem and motivation",
    "## Method",
    "## Main findings",
    "## Limits and caveats",
    "## Follow-up questions",
]


def load_json(path: Path) -> dict:
    return json.loads(path.read_text(encoding="utf-8"))


def write_json(path: Path, payload: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")


def today_local() -> str:
    return datetime.now().astimezone().date().isoformat()


def normalize_slug(value: str, fallback: str) -> str:
    slug = safe_dir_name(value).replace("_", "-").strip("._-").lower()
    return slug or fallback


def validate_limit(value: object, field_name: str) -> int:
    if not isinstance(value, int):
        raise ValueError(f"{field_name} must be an integer")
    if value < 1:
        raise ValueError(f"{field_name} must be greater than or equal to 1")
    return value


def validate_sort_value(value: object, field_name: str) -> str:
    if not isinstance(value, str) or value not in VALID_SORTS:
        raise ValueError(f"{field_name} must be one of: {', '.join(sorted(VALID_SORTS))}")
    return value


def validate_order_value(value: object, field_name: str) -> str:
    if not isinstance(value, str) or value not in VALID_ORDERS:
        raise ValueError(f"{field_name} must be one of: {', '.join(sorted(VALID_ORDERS))}")
    return value


def parse_iso_date(value: str, field_name: str) -> date:
    try:
        return date.fromisoformat(value)
    except ValueError as error:
        raise ValueError(f"{field_name} must use YYYY-MM-DD format") from error


def read_topics_config(path: Path) -> dict:
    if not path.exists():
        sample_path = SKILL_DIR / "references" / "topics.example.json"
        raise FileNotFoundError(
            f"topics config not found: {path}. Copy {sample_path} to that location and edit it for your interests."
        )
    config = load_json(path)
    if not isinstance(config, dict):
        raise ValueError("topics config must be a JSON object")
    return config


def flatten_topics(config: dict) -> tuple[dict[str, object], list[dict[str, object]]]:
    defaults = config.get("defaults")
    groups = config.get("groups")

    if not isinstance(defaults, dict):
        raise ValueError("topics config defaults must be an object")
    if not isinstance(groups, list) or not groups:
        raise ValueError("topics config groups must be a non-empty list")

    resolved_defaults = {
        "per_topic_limit": validate_limit(defaults.get("per_topic_limit", DEFAULT_LIMIT), "defaults.per_topic_limit"),
        "sort_by": validate_sort_value(defaults.get("sort_by", "submittedDate"), "defaults.sort_by"),
        "sort_order": validate_order_value(defaults.get("sort_order", "descending"), "defaults.sort_order"),
    }

    flattened: list[dict[str, object]] = []
    used_slugs: set[str] = set()
    topic_index = 0

    for group in groups:
        if not isinstance(group, dict):
            raise ValueError("each group must be an object")
        group_name = group.get("name")
        if not isinstance(group_name, str) or not group_name.strip():
            raise ValueError("each group must define a non-empty name")
        topics = group.get("topics")
        if not isinstance(topics, list):
            raise ValueError(f"group {group_name} must define a topics list")

        for topic in topics:
            topic_index += 1
            if not isinstance(topic, dict):
                raise ValueError(f"group {group_name} contains an invalid topic entry")
            if topic.get("enabled", True) is False:
                continue

            topic_name = topic.get("name")
            query = topic.get("query")
            if not isinstance(topic_name, str) or not topic_name.strip():
                raise ValueError(f"group {group_name} has a topic without a non-empty name")
            if not isinstance(query, str) or not query.strip():
                raise ValueError(f"topic {topic_name} must define a non-empty query")

            raw_slug = topic.get("slug")
            if raw_slug is not None and (not isinstance(raw_slug, str) or not raw_slug.strip()):
                raise ValueError(f"topic {topic_name} has an invalid slug")
            base_slug = normalize_slug(raw_slug or query, f"topic-{topic_index}")
            topic_slug = base_slug
            suffix = 2
            while topic_slug in used_slugs:
                topic_slug = f"{base_slug}-{suffix}"
                suffix += 1
            used_slugs.add(topic_slug)

            per_topic_limit = validate_limit(
                topic.get("per_topic_limit", resolved_defaults["per_topic_limit"]),
                f"topic {topic_name} per_topic_limit",
            )

            flattened.append(
                {
                    "group_name": group_name.strip(),
                    "topic_name": topic_name.strip(),
                    "query": query.strip(),
                    "topic_slug": topic_slug,
                    "per_topic_limit": per_topic_limit,
                    "sort_by": resolved_defaults["sort_by"],
                    "sort_order": resolved_defaults["sort_order"],
                }
            )

    if not flattened:
        raise ValueError("topics config does not contain any enabled topics")
    return resolved_defaults, flattened


def load_state(path: Path) -> dict:
    if not path.exists():
        return {"topics": {}}
    payload = load_json(path)
    if not isinstance(payload, dict):
        raise ValueError("sync state must be a JSON object")
    topics = payload.get("topics")
    if not isinstance(topics, dict):
        raise ValueError("sync state topics must be an object")
    return payload


def determine_sync_window(
    *,
    daily: bool,
    topic_slug: str,
    state: dict,
    today_value: str,
    start_date: str | None,
    end_date: str | None,
) -> tuple[str | None, str | None, str]:
    if not daily:
        if not start_date or not end_date:
            raise ValueError("start_date and end_date are required when --daily is not used")
        return start_date, end_date, "backfill"

    topics_state = state.setdefault("topics", {})
    topic_state = topics_state.get(topic_slug, {})
    if not isinstance(topic_state, dict):
        topic_state = {}

    today_date = parse_iso_date(today_value, "today")
    previous_end = topic_state.get("end_date")
    if previous_end:
        previous_date = parse_iso_date(str(previous_end), f"state end_date for {topic_slug}")
        next_date = previous_date + timedelta(days=1)
        if next_date > today_date:
            return None, None, "up_to_date"
        return next_date.isoformat(), today_value, "daily"

    return today_value, today_value, "daily"


def build_paper_dir(root_dir: Path, topic_slug: str, capture_date: str, paper_id: str, title: str) -> Path:
    title_slug = normalize_slug(title, paper_id.replace("/", "-"))
    return root_dir / "topics" / topic_slug / capture_date / f"{safe_dir_name(paper_id)}__{title_slug}"


def paper_dir_complete(paper_dir: Path) -> bool:
    required = ["paper.pdf", "paper.md", "metadata.json", "summary.md"]
    return all((paper_dir / filename).exists() for filename in required)


def read_paper_sections(markdown_text: str) -> tuple[str, str]:
    abstract_match = re.search(r"## Abstract\s+(.*?)(?:\n## Full Text|\Z)", markdown_text, flags=re.DOTALL)
    full_text_match = re.search(r"## Full Text\s+(.*)\Z", markdown_text, flags=re.DOTALL)
    abstract = abstract_match.group(1).strip() if abstract_match else ""
    full_text = full_text_match.group(1).strip() if full_text_match else ""
    return abstract, full_text


def split_sentences(text: str) -> list[str]:
    compact = re.sub(r"\s+", " ", text).strip()
    if not compact:
        return []
    parts = re.split(r"(?<=[.!?])\s+", compact)
    return [part.strip() for part in parts if part.strip()]


def first_meaningful_paragraph(full_text: str) -> str:
    blocks = [block.strip() for block in re.split(r"\n\s*\n", full_text) if block.strip()]
    for block in blocks:
        if block.startswith("#"):
            continue
        if len(block) >= 80:
            return re.sub(r"\s+", " ", block)
    for block in blocks:
        if not block.startswith("#"):
            return re.sub(r"\s+", " ", block)
    return ""


def heading_candidates(full_text: str) -> list[str]:
    headings = re.findall(r"^#{2,6}\s+(.*)$", full_text, flags=re.MULTILINE)
    return [heading.strip() for heading in headings if heading.strip()]


def build_summary_markdown(metadata: dict[str, object], paper_markdown: str) -> str:
    title = str(metadata.get("title") or "Untitled paper").strip()
    abstract, full_text = read_paper_sections(paper_markdown)
    sentences = split_sentences(abstract)
    takeaway = sentences[0] if sentences else f"This paper studies {title}."
    problem = " ".join(sentences[:2]) if sentences else (abstract or f"The paper is about {title}.")

    method_paragraph = first_meaningful_paragraph(full_text)
    if not method_paragraph:
        method_paragraph = "The converter could not recover enough full-text detail, so review paper.md for the full methodology."

    finding_items = sentences[2:5]
    if not finding_items:
        finding_items = heading_candidates(full_text)[:3]
    if not finding_items:
        finding_items = ["Review the abstract and paper.md for the primary empirical or conceptual findings."]

    limit_items = [
        "This summary is auto-generated from arXiv metadata, the abstract, and converted Markdown, so nuanced claims may be compressed.",
    ]
    if metadata.get("source_kind") == "abstract-only":
        limit_items.append("Only the abstract was available during conversion, so method and evidence details may be incomplete.")
    else:
        limit_items.append("Validate implementation details, dataset setup, and ablations against paper.md before relying on this summary.")

    follow_up_items = [
        "Which evaluation setup or benchmark best demonstrates the claimed gains?",
        "What assumptions or threat model boundaries matter most for real deployment?",
        "What evidence would you want to inspect next in paper.md before trusting the conclusions?",
    ]

    lines = [
        f"# {title}",
        "",
        SUMMARY_HEADINGS[0],
        "",
        takeaway,
        "",
        SUMMARY_HEADINGS[1],
        "",
        problem,
        "",
        SUMMARY_HEADINGS[2],
        "",
        method_paragraph,
        "",
        SUMMARY_HEADINGS[3],
        "",
    ]
    lines.extend([f"- {item}" for item in finding_items])
    lines.extend(
        [
            "",
            SUMMARY_HEADINGS[4],
            "",
        ]
    )
    lines.extend([f"- {item}" for item in limit_items])
    lines.extend(
        [
            "",
            SUMMARY_HEADINGS[5],
            "",
        ]
    )
    lines.extend([f"- {item}" for item in follow_up_items])
    return "\n".join(lines).strip() + "\n"


def build_summary_path(paper_dir: Path) -> Path:
    return paper_dir / "summary.md"


def build_existing_result(paper_id: str, paper_dir: Path, source_kind: str = "existing") -> dict[str, str]:
    return {
        "paper_id": paper_id,
        "output_dir": str(paper_dir),
        "paper_pdf": str(paper_dir / "paper.pdf"),
        "paper_md": str(paper_dir / "paper.md"),
        "metadata_json": str(paper_dir / "metadata.json"),
        "source_kind": source_kind,
    }


def render_manifest_markdown(manifest: dict) -> str:
    lines = [
        "# arXiv Topic Sync Run",
        "",
        f"- Mode: `{manifest['mode']}`",
        f"- Capture date: `{manifest['capture_date']}`",
        f"- Run timestamp: `{manifest['run_at']}`",
        f"- Root dir: `{manifest['root_dir']}`",
        f"- Start date: `{manifest['start_date']}`" if manifest.get("start_date") else "- Start date: none",
        f"- End date: `{manifest['end_date']}`" if manifest.get("end_date") else "- End date: none",
        f"- Topics processed: {len(manifest['topics'])}",
        f"- Papers captured: {len(manifest['papers'])}",
        "",
        "## Topics",
        "",
    ]

    if not manifest["topics"]:
        lines.append("No topics were processed.")
    else:
        for index, topic in enumerate(manifest["topics"], start=1):
            lines.extend(
                [
                    f"{index}. {topic['topic_name']} (`{topic['topic_slug']}`)",
                    f"   - Group: {topic['group_name']}",
                    f"   - Query: {topic['query']}",
                    f"   - Window: {topic.get('start_date') or 'none'} -> {topic.get('end_date') or 'none'}",
                    f"   - Status: {topic['status']}",
                    f"   - Returned results: {topic.get('returned_results', 0)}",
                    f"   - New papers: {topic.get('new_papers', 0)}",
                ]
            )
            if topic.get("search_results_md"):
                lines.append(f"   - Search results: {topic['search_results_md']}")
            if topic.get("error"):
                lines.append(f"   - Error: {topic['error']}")
            lines.append("")

    lines.extend(["## Papers", ""])
    if not manifest["papers"]:
        lines.append("No papers were captured in this run.")
    else:
        for index, paper in enumerate(manifest["papers"], start=1):
            lines.extend(
                [
                    f"{index}. {paper['title']} (`{paper['paper_id']}`)",
                    f"   - Topic: {paper['topic_name']} (`{paper['topic_slug']}`)",
                    f"   - Group: {paper['group_name']}",
                    f"   - Status: {paper['status']}",
                    f"   - Output dir: {paper['output_dir']}",
                    f"   - PDF: {paper['paper_pdf']}",
                    f"   - Markdown: {paper['paper_md']}",
                    f"   - Metadata: {paper['metadata_json']}",
                    f"   - Summary: {paper['summary_md']}",
                    "",
                ]
            )
    return "\n".join(lines).strip() + "\n"


def sync_topics(
    *,
    root_dir: Path,
    config_path: Path,
    daily: bool,
    start_date: str | None,
    end_date: str | None,
    timeout: float,
    capture_date: str | None = None,
) -> tuple[dict, int]:
    capture_value = capture_date or today_local()
    config = read_topics_config(config_path)
    _, topics = flatten_topics(config)
    state_path = root_dir / DEFAULT_STATE_NAME
    state = load_state(state_path)

    run_dir = root_dir / "runs" / capture_value
    run_dir.mkdir(parents=True, exist_ok=True)

    manifest = {
        "mode": "daily" if daily else "backfill",
        "capture_date": capture_value,
        "run_at": datetime.now().astimezone().isoformat(),
        "root_dir": str(root_dir),
        "config_path": str(config_path),
        "state_path": str(state_path),
        "start_date": start_date,
        "end_date": end_date,
        "topics": [],
        "papers": [],
    }

    failures = 0

    for topic in topics:
        topic_slug = str(topic["topic_slug"])
        topic_name = str(topic["topic_name"])
        group_name = str(topic["group_name"])
        query = str(topic["query"])
        limit = int(topic["per_topic_limit"])
        sort_by = str(topic["sort_by"])
        sort_order = str(topic["sort_order"])

        topic_record = {
            "group_name": group_name,
            "topic_name": topic_name,
            "topic_slug": topic_slug,
            "query": query,
            "limit": limit,
            "sort_by": sort_by,
            "sort_order": sort_order,
            "status": "pending",
            "start_date": None,
            "end_date": None,
            "search_results_json": None,
            "search_results_md": None,
            "total_results": 0,
            "returned_results": 0,
            "new_papers": 0,
            "skipped_papers": 0,
            "error": None,
        }
        manifest["topics"].append(topic_record)

        try:
            window_start, window_end, mode_label = determine_sync_window(
                daily=daily,
                topic_slug=topic_slug,
                state=state,
                today_value=capture_value,
                start_date=start_date,
                end_date=end_date,
            )
            topic_record["start_date"] = window_start
            topic_record["end_date"] = window_end

            if mode_label == "up_to_date":
                topic_record["status"] = "up_to_date"
                continue

            topic_dir = root_dir / "topics" / topic_slug / capture_value
            topic_dir.mkdir(parents=True, exist_ok=True)

            response = search_papers(
                query,
                start_date=window_start,
                end_date=window_end,
                sort_by=sort_by,
                sort_order=sort_order,
                limit=limit,
                timeout=timeout,
            )
            search_outputs = write_search_outputs(topic_dir, response)
            topic_record["search_results_json"] = str(search_outputs["json"])
            topic_record["search_results_md"] = str(search_outputs["md"])
            topic_record["total_results"] = response.total_results
            topic_record["returned_results"] = response.returned_results

            for entry in response.entries:
                paper_dir = build_paper_dir(root_dir, topic_slug, capture_value, entry.paper_id, entry.title)
                summary_path = build_summary_path(paper_dir)
                if paper_dir_complete(paper_dir):
                    topic_record["skipped_papers"] += 1
                    result = build_existing_result(entry.paper_id, paper_dir)
                    paper_status = "existing"
                else:
                    extra_metadata = {
                        "topic_name": topic_name,
                        "topic_slug": topic_slug,
                        "group_name": group_name,
                        "capture_date": capture_value,
                        "summary_md": str(summary_path),
                    }
                    result = fetch_paper_to_directory(
                        entry.paper_id,
                        paper_dir,
                        timeout=timeout,
                        extra_metadata=extra_metadata,
                    )
                    paper_markdown = Path(result["paper_md"]).read_text(encoding="utf-8")
                    metadata_payload = load_json(Path(result["metadata_json"]))
                    summary_path.write_text(build_summary_markdown(metadata_payload, paper_markdown), encoding="utf-8")
                    topic_record["new_papers"] += 1
                    paper_status = "captured"

                manifest["papers"].append(
                    {
                        "group_name": group_name,
                        "topic_name": topic_name,
                        "topic_slug": topic_slug,
                        "capture_date": capture_value,
                        "paper_id": entry.paper_id,
                        "title": entry.title,
                        "status": paper_status,
                        "output_dir": result["output_dir"],
                        "paper_pdf": result["paper_pdf"],
                        "paper_md": result["paper_md"],
                        "metadata_json": result["metadata_json"],
                        "summary_md": str(summary_path),
                    }
                )

            topic_record["status"] = "success"
            if daily:
                state.setdefault("topics", {}).setdefault(topic_slug, {})
                state["topics"][topic_slug] = {
                    "end_date": window_end,
                    "updated_at": manifest["run_at"],
                }
        except Exception as error:
            failures += 1
            topic_record["status"] = "failed"
            topic_record["error"] = str(error)

    manifest_path_json = run_dir / "run_manifest.json"
    manifest_path_md = run_dir / "run_manifest.md"
    write_json(manifest_path_json, manifest)
    manifest_path_md.write_text(render_manifest_markdown(manifest), encoding="utf-8")

    if daily:
        write_json(state_path, state)

    exit_code = 1 if failures else 0
    return {
        "mode": manifest["mode"],
        "root_dir": str(root_dir),
        "config_path": str(config_path),
        "state_path": str(state_path),
        "capture_date": capture_value,
        "start_date": start_date,
        "end_date": end_date,
        "run_manifest_json": str(manifest_path_json),
        "run_manifest_md": str(manifest_path_md),
        "topics_processed": len(manifest["topics"]),
        "papers_captured": len(manifest["papers"]),
        "failures": failures,
    }, exit_code


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description="Sync configured arXiv keyword topics into a daily local archive.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=textwrap.dedent(
            """\
            Examples:
              python sync_arxiv_topics.py --daily
              python sync_arxiv_topics.py --start-date 2026-03-01 --end-date 2026-03-07
              python sync_arxiv_topics.py --daily --root-dir ./artifacts/arxiv-monitor
            """
        ),
    )
    parser.add_argument("--daily", action="store_true", help="run the stateful daily sync workflow")
    parser.add_argument("--start-date", default=None, help="inclusive submittedDate lower bound in YYYY-MM-DD")
    parser.add_argument("--end-date", default=None, help="inclusive submittedDate upper bound in YYYY-MM-DD")
    parser.add_argument("--root-dir", type=Path, default=DEFAULT_ROOT_DIR, help="root archive directory")
    parser.add_argument("--timeout", type=float, default=30.0, help="network timeout in seconds")
    args = parser.parse_args(argv)

    if args.daily and (args.start_date or args.end_date):
        print("error: --daily cannot be combined with --start-date/--end-date", file=sys.stderr)
        return 1
    if not args.daily and (not args.start_date or not args.end_date):
        print("error: either use --daily or provide both --start-date and --end-date", file=sys.stderr)
        return 1

    config_path = args.root_dir / DEFAULT_TOPICS_NAME
    try:
        result, exit_code = sync_topics(
            root_dir=args.root_dir,
            config_path=config_path,
            daily=args.daily,
            start_date=args.start_date,
            end_date=args.end_date,
            timeout=args.timeout,
        )
        print(json.dumps(result, ensure_ascii=False))
        return exit_code
    except Exception as error:  # pragma: no cover - surfaced in CLI output
        print(f"error: {error}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: references/search-usage.md
content: # Search Result Handling

Use this workflow when the user asks to search arXiv or find the latest papers for a topic.

1. Run `search_arxiv.py` first.
2. Read `search_results.md`.
3. Show the first few candidates with explicit arXiv IDs.
4. Ask the user to name one or more arXiv IDs if they want full Markdown conversion or summaries.
5. For requests like "latest transformer papers", pass `--query transformer --sort submittedDate`.
6. For requests like "latest diffusion papers in computer vision", pass `--query "computer vision diffusion" --sort submittedDate`.
7. If the user gives a date window, add `--start-date YYYY-MM-DD --end-date YYYY-MM-DD`.

Keep result presentation grounded in the generated files. Do not imply that a paper has already been fetched into `paper.md` until `arxiv_to_md.py` runs.


path: references/summary-format.md
content: # Summary Template

Use this structure:

## One-line takeaway

State the main contribution in one sentence.

## Problem and motivation

Explain what problem the paper tries to solve and why it matters.

## Method

Describe the key approach, architecture, dataset, or experimental setup.

## Main findings

List the most important empirical or theoretical results.

## Limits and caveats

Call out assumptions, weak spots, missing comparisons, or incomplete evidence.

## Follow-up questions

List three to five concrete questions that deserve deeper review.


path: references/topics.example.json
content: {
  "defaults": {
    "per_topic_limit": 5,
    "sort_by": "submittedDate",
    "sort_order": "descending"
  },
  "groups": [
    {
      "name": "Agent安全",
      "topics": [
        {
          "name": "agent security",
          "query": "agent security",
          "slug": "agent-security",
          "enabled": true
        },
        {
          "name": "prompt injection",
          "query": "prompt injection",
          "slug": "prompt-injection",
          "enabled": true
        },
        {
          "name": "indirect prompt injection",
          "query": "indirect prompt injection",
          "slug": "indirect-prompt-injection",
          "enabled": true
        },
        {
          "name": "tool misuse",
          "query": "tool misuse",
          "slug": "tool-misuse",
          "enabled": true
        },
        {
          "name": "action grounding",
          "query": "action grounding",
          "slug": "action-grounding",
          "enabled": true
        },
        {
          "name": "browser agent attack",
          "query": "browser agent attack",
          "slug": "browser-agent-attack",
          "enabled": true
        },
        {
          "name": "autonomous agent safety",
          "query": "autonomous agent safety",
          "slug": "autonomous-agent-safety",
          "enabled": true
        },
        {
          "name": "planner attack",
          "query": "planner attack",
          "slug": "planner-attack",
          "enabled": true
        },
        {
          "name": "memory poisoning",
          "query": "memory poisoning",
          "slug": "memory-poisoning",
          "enabled": true
        }
      ]
    },
    {
      "name": "深伪/真实性",
      "topics": [
        {
          "name": "deepfake detection",
          "query": "deepfake detection",
          "slug": "deepfake-detection",
          "enabled": true
        },
        {
          "name": "provenance",
          "query": "provenance",
          "slug": "provenance",
          "enabled": true
        },
        {
          "name": "synthetic media",
          "query": "synthetic media",
          "slug": "synthetic-media",
          "enabled": true
        },
        {
          "name": "multimodal misinformation",
          "query": "multimodal misinformation",
          "slug": "multimodal-misinformation",
          "enabled": true
        },
        {
          "name": "content authenticity",
          "query": "content authenticity",
          "slug": "content-authenticity",
          "enabled": true
        },
        {
          "name": "media forensics",
          "query": "media forensics",
          "slug": "media-forensics",
          "enabled": true
        }
      ]
    },
    {
      "name": "鲁棒性",
      "topics": [
        {
          "name": "adversarial robustness",
          "query": "adversarial robustness",
          "slug": "adversarial-robustness",
          "enabled": true
        },
        {
          "name": "multimodal robustness",
          "query": "multimodal robustness",
          "slug": "multimodal-robustness",
          "enabled": true
        },
        {
          "name": "jailbreak",
          "query": "jailbreak",
          "slug": "jailbreak",
          "enabled": true
        },
        {
          "name": "context poisoning",
          "query": "context poisoning",
          "slug": "context-poisoning",
          "enabled": true
        },
        {
          "name": "retrieval poisoning",
          "query": "retrieval poisoning",
          "slug": "retrieval-poisoning",
          "enabled": true
        }
      ]
    }
  ]
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)