Skill: jiahaowugit/release20260324 (version 1.0.0)

--- name: opg description: "Academic literature discovery and citation network analysis. Multi-source search across arXiv, DBLP, Semantic Scholar, and Google Scholar. Build citation networks (references from PDF parsing, citations from Google Scholar), get recommendations, monitor new papers, analyze topics, parse PDFs, import from Zotero, generate research summaries, export as BibTeX/CSV/Markdown/JSON, and generate interactive HTML graph visualizations. Use when user asks about finding papers, literature review, citation analysis, research trends, or visualizing citation networks." allowed-tools: Read, Write, Edit, Bash --- # OpenPaperGraph — Literature Discovery & Citation Analysis You are a research assistant with access to a CLI tool for academic literature discovery and analysis. ## Setup The CLI is located at: `SKILL_DIR/openpapergraph_cli.py` Before first use, ensure dependencies are installed: ```bash pip install httpx pymupdf scholarly ``` All commands output JSON to stdout. Run from the `SKILL_DIR` directory. ## Architecture: Multi-Source This tool reduces dependency on any single data source: | Task | Primary Sources | Fallback | |---|---|---| | Search | arXiv + DBLP + S2 | Deduplicated, sorted by citations | | References | Download PDF → parse reference list | S2 API | | Citations | Google Scholar | S2 API | | Citation counts | Google Scholar | S2 | | Recommendations | S2 Recommendations API | — | | Reference resolution | arXiv → S2 → CrossRef → OpenAlex | Multi-cascade | ## Available Commands ### 1. Search Papers Multi-source search across arXiv, DBLP, and Semantic Scholar. Supports conference filtering. ```bash python SKILL_DIR/openpapergraph_cli.py search "QUERY" --source SOURCE --venue VENUE --limit N ``` - `--source`: `all` (default, multi-source), `arxiv`, `dblp`, or `s2` - `--venue`: Filter by conference — `ICLR`, `NeurIPS`, `ICML`, `ACL`, `EMNLP`, `NAACL`, `WebConf`, `KDD` - `--limit`: Max results (default 20) **When to use**: User asks to find papers, search for literature, or look up specific topics/conferences. ### 2. Build Citation Network Construct a citation graph from seed papers. References come from PDF parsing (downloaded from arXiv/Unpaywall), citations from Google Scholar. Falls back to S2 when needed. ```bash python SKILL_DIR/openpapergraph_cli.py graph PAPER_ID1 PAPER_ID2 --depth 1 --output graph.json ``` - Paper IDs can be: S2 hex ID (`204e3073...`), arXiv ID (`ARXIV:1706.03762`), DOI (`DOI:10.1145/...`), paper title (`"attention is all you need"`), PDF path (`paper.pdf`), BibTeX file (`refs.bib`), or Zotero CSL-JSON export (`zotero.json`) - `--depth`: Expansion depth (1 or 2, default 1) - `--output`: Save graph to file for later analysis/export **When to use**: User wants to explore the citation landscape around specific papers. ### 3. Paper Recommendations Get related paper recommendations based on one or more papers (via S2 Recommendations API). ```bash python SKILL_DIR/openpapergraph_cli.py recommend PAPER_ID1 PAPER_ID2 --limit 10 ``` - Also accepts paper titles and PDF paths as input **When to use**: User wants to discover related or similar papers they may have missed. ### 4. Monitor New Papers Check for recently published papers on a research topic (multi-source: arXiv + DBLP + S2, citation counts enriched via Google Scholar). ```bash python SKILL_DIR/openpapergraph_cli.py monitor "TOPIC" --year-from 2025 --limit 20 ``` **When to use**: User wants to stay updated on latest publications in a field. ### 5. Topic Analysis Analyze a citation graph for topics, keyword distribution, year trends, and top authors. ```bash python SKILL_DIR/openpapergraph_cli.py analyze graph.json ``` **When to use**: User wants to understand the thematic structure of a set of papers. ### 6. Research Summary Generate a research summary from a citation graph. Uses LLM if any provider is configured, otherwise falls back to extractive analysis. ```bash python SKILL_DIR/openpapergraph_cli.py summary graph.json --style STYLE python SKILL_DIR/openpapergraph_cli.py summary graph.json --provider deepseek --model deepseek-chat ``` - `--style`: `overview` (default), `trends`, or `gaps` - `--provider`: LLM provider name (e.g. `openai`, `deepseek`, `qwen`, `zhipu`, `moonshot`) - `--model`: Override the provider's default model **When to use**: User wants a quick overview of a research area or to identify trends/gaps. ### 7. PDF Reference Extraction Extract references from a PDF paper, resolving via multi-source cascade (arXiv → S2 → CrossRef → OpenAlex). ```bash python SKILL_DIR/openpapergraph_cli.py pdf /path/to/paper.pdf python SKILL_DIR/openpapergraph_cli.py pdf /path/to/paper.pdf --use-grobid ``` - `--use-grobid`: Use GROBID for structured extraction (requires Docker service on port 8070) - Returns: resolved papers, unresolved references, and resolve rate **When to use**: User provides a PDF and wants to find/analyze its references. ### 7b. Build Graph from PDF Reference Lists Build a citation graph directly from one or more PDF papers' reference lists. ```bash python SKILL_DIR/openpapergraph_cli.py graph-from-pdf paper.pdf [paper2.pdf ...] --output graph.json python SKILL_DIR/openpapergraph_cli.py graph-from-pdf paper.pdf --depth 1 --include-unresolved -o graph.json ``` - `--depth 0` (default): Only PDF references. `--depth 1`: Also expand resolved papers. - `--include-unresolved`: Keep unresolved references as nodes in the graph (marked `resolved=false`) - `--use-grobid`: Use GROBID for structured extraction - References resolved via: arXiv → Semantic Scholar → CrossRef → OpenAlex (multi-source cascade) **When to use**: User has PDF papers and wants a citation graph faithful to the actual reference lists. ### 8. Zotero Import Import papers from a Zotero library or collection. ```bash python SKILL_DIR/openpapergraph_cli.py zotero --user-id ID --api-key KEY [--collection KEY] [--list-collections] ``` **When to use**: User wants to import their existing Zotero library for analysis. ### 9. Export Export a citation graph as BibTeX, CSV, Markdown, or JSON. All formats sort papers by year descending. ```bash python SKILL_DIR/openpapergraph_cli.py export graph.json --format bibtex --output refs.bib python SKILL_DIR/openpapergraph_cli.py export graph.json --format csv --output papers.csv python SKILL_DIR/openpapergraph_cli.py export graph.json --format markdown --output papers.md python SKILL_DIR/openpapergraph_cli.py export graph.json --format json --output papers.json ``` - `--format`: `bibtex` (default), `csv`, `markdown`, or `json` - CSV/Markdown/JSON include full fields: id, title, authors, year, citations, source, url, doi, arxiv_id, abstract **When to use**: User wants to save results for use in a reference manager, spreadsheet, or documentation. ### 9b. Export Interactive HTML Graph Export a citation graph as a self-contained interactive HTML visualization. ```bash python SKILL_DIR/openpapergraph_cli.py export-html graph.json --output graph.html python SKILL_DIR/openpapergraph_cli.py export-html graph.json --output graph.html --title "My Research" --summary --inline ``` - `--title`: Custom page title (default: "Paper Graph") - `--summary`: Pre-generate AI summary at export time (requires LLM API key in env). Result is embedded; API key is NOT. - `--inline`: Inline vis-network JS for fully offline use (~500KB larger, no CDN needed) - `--provider` / `--model`: Override LLM provider/model for `--summary` - **Layout**: Semantic left-to-right hierarchy — References (LEFT) → Seeds (CENTER) → Citations (RIGHT) - **Node types**: Seeds (purple stars), References (blue circles), Citations (green diamonds), with legend - **Features**: bidirectional hover linking, type filter, search/filter, in-page export, seed source management (add/remove seeds) - **Summary modes**: (A) Pre-generate with `--summary`, (B) Runtime API key (20+ providers), (C) Manual copy/paste (CORS-proof) - Security: API keys are **never** embedded in the HTML output **When to use**: User wants a visual, interactive exploration of the citation network, or wants to share a browsable graph. ### 9b. Interactive Graph Server (`serve`) Start a local HTTP server for interactive graph management. Unlike `export-html` (static, read-only), `serve` lets users add papers, convert nodes to seeds, remove seeds, and all changes persist to the graph JSON file. ```bash python SKILL_DIR/openpapergraph_cli.py serve graph.json --port 8787 ``` - `--port`: Server port (default: 8787) - `--title`: Custom page title - **Add papers**: "+ Add Paper" button in toolbar. Input via title/ID, BibTeX, or PDF upload. Toggle "Treat as Seed Paper" to control expansion. - **Seed**: Full expansion — fetches references + citations from S2/Google Scholar, adds nodes + edges - **Non-seed**: Lightweight — only checks relationships with existing seeds, no expansion - **Convert to seed**: Click any non-seed paper in the list → "⬆ Convert to Seed" button appears. Also available in the node tooltip when clicking graph nodes. - **Remove seed**: Seeds/Sources tab → "Remove" button. Deletes seed + exclusive connections. - **Persistent**: All changes immediately written to graph JSON file. Survives page refresh. - **Dedup**: Papers matched by DOI > arXiv ID > title+year similarity (no duplicates) **When to use**: User wants to interactively build and manage a citation network through the browser, with all changes persisted. Use `export-html` instead when you want a static file for sharing. ### 10. Remove Seed Paper Remove a seed paper and all papers exclusively connected to it from a graph. ```bash python SKILL_DIR/openpapergraph_cli.py remove-seed graph.json "paper_id_or_title" ``` - Accepts paper ID or title substring (fuzzy match) - Removes the seed + papers connected only to that seed (not shared with other seeds) - Cleans up all incident edges - Overwrites the graph file (use `-o` to save to a different file) ### 11. Remove Non-Seed Paper Remove a single non-seed paper from a graph. ```bash python SKILL_DIR/openpapergraph_cli.py remove-paper graph.json "paper_id_or_title" ``` - Accepts paper ID or title substring (fuzzy match) - Only works for non-seed papers (use `remove-seed` for seeds) - Cleans up all incident edges - Overwrites the graph file (use `-o` to save to a different file) ### 12. List Conferences Show supported conference venues for filtering. ```bash python SKILL_DIR/openpapergraph_cli.py conferences ``` ### 13. List LLM Providers Show all 20 supported LLM providers and whether their API key is configured. ```bash python SKILL_DIR/openpapergraph_cli.py llm-providers ``` ## Workflow Guidelines 1. **Start with search** — Help the user find relevant seed papers first (default: multi-source) 2. **Build a graph** — Use seed paper IDs to construct a citation network, save to a `.json` file 3. **Explore interactively** — Use `serve` to open the graph in browser, add papers, convert to seeds (`serve`) 4. **Analyze** — Run topic analysis or generate a summary on the saved graph 5. **Discover more** — Use recommendations to find papers the user may have missed 6. **Export** — Save results as BibTeX/CSV/Markdown/JSON for the user's reference manager 7. **Share** — Generate a static HTML graph for sharing/viewing (`export-html`) ## Output Format All commands output JSON to stdout. When presenting results to the user: - Show paper titles, authors, year, and citation counts in a readable format - For large result sets, summarize the top results and mention the total count - Paper IDs can be: S2 hex IDs, arXiv IDs (`ARXIV:xxxx`), DOIs (`DOI:xxxx`), paper titles, or PDF file paths - The `source` field in results indicates where each paper came from (arxiv, semantic_scholar, google_scholar, crossref, openalex, dblp) ## Environment Variables ### `S2_API_KEY` (Recommended) Semantic Scholar API key. Free at [semanticscholar.org/product/api](https://www.semanticscholar.org/product/api). - **Purpose**: Authenticates requests to the S2 API (paper search, citation data, recommendations) - **Why needed**: Without it, S2 enforces strict rate limiting — frequent calls return 429 errors - **Role**: S2 is the **fallback** in the multi-source architecture — when PDF download or Google Scholar fails, the system falls back to S2. Also the **exclusive source** for the `recommend` command ### LLM Provider API Key (Optional — any one of 20 providers) The `summary` command supports **20 LLM providers**. Set any one API key to enable LLM-powered summaries: **US**: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `DEEPSEEK_API_KEY`, `GROQ_API_KEY`, `TOGETHER_API_KEY`, `FIREWORKS_API_KEY`, `MISTRAL_API_KEY`, `XAI_API_KEY`, `PERPLEXITY_API_KEY`, `OPENROUTER_API_KEY` **Chinese**: `ZHIPUAI_API_KEY` (智谱), `MOONSHOT_API_KEY` (月之暗面), `BAICHUAN_API_KEY` (百川), `YI_API_KEY` (零一万物), `DASHSCOPE_API_KEY` (通义千问), `ARK_API_KEY` (豆包), `MINIMAX_API_KEY`, `STEPFUN_API_KEY` (阶跃星辰), `SENSENOVA_API_KEY` (商汤) **Custom**: Set `LLM_API_KEY` + `LLM_BASE_URL` + `LLM_MODEL` for any OpenAI-compatible endpoint. **Additional environment variables:** - `LLM_PROVIDER`: Explicitly select LLM provider (alternative to `--provider` CLI flag) - `LLM_MODEL`: Override default model for the selected provider (alternative to `--model` CLI flag) - `TMPDIR`: Custom directory for PDF download cache (defaults to system temp) Without any LLM key, `summary` uses extractive analysis and `export-html` hides the AI summary panel. All other commands are unaffected. Run `llm-providers` to check status. ## Cross-Tool Compatibility This CLI is designed to be called by any AI coding tool (Claude Code, OpenClaw, Codex, etc.): - All output is structured JSON on stdout - Errors go to stderr - Exit code 0 = success, 1 = argument error, 2 = runtime error - No interactive input required — all parameters via command-line flags

path: COMMERCIAL.md
content: # Commercial Use

OpenPaperGraph is available for noncommercial use under the [PolyForm Noncommercial License 1.0.0](LICENSE).

If you want to use OpenPaperGraph in a commercial setting, including but not limited to:

- paid services,
- commercial SaaS or hosted offerings,
- internal business operations,
- consulting or client work,
- bundling with commercial products,

please contact:

**[REDACTED_SECRET]**

Commercial licenses may be available separately.


path: README.md
content: <p align="center">
  <img alt="OpenPaperGraph" src="figures/logo-part-light-full.png" width="420">
</p>

<h3 align="center">The research copilot for the OpenClaw era</h3>

<p align="center">
  <img src="https://img.shields.io/badge/python-3.8+-blue.svg" alt="Python 3.8+">
  <img src="https://img.shields.io/badge/dependencies-3-green.svg" alt="3 Dependencies">
  <img src="https://img.shields.io/badge/LLM_providers-20+-orange.svg" alt="20+ LLM Providers">
  <img src="https://img.shields.io/badge/data_sources-8-purple.svg" alt="8 Data Sources">
  <img src="https://img.shields.io/badge/license-PolyForm%20Noncommercial%201.0.0-lightgrey.svg" alt="License">
</p>

<p align="center">
  <a href="README_CN.md">中文</a> | English
</p>

<p align="center">
  <b>Search 8 sources. Build citation graphs. Generate AI summaries.</b><br>
  Works as a Claude Code / OpenClaw <code>/opg</code> skill or standalone CLI.<br>
  No API keys required. 3 pip packages. One-line install.
</p>

---

## Why OpenPaperGraph?

In the age of AI agents like **Claude Code** and **OpenClaw**, researchers need tools that work *with* their workflow — not against it. OpenPaperGraph is designed to be the **research backbone** for AI-assisted development:

- **Agent-native** — JSON stdout, zero interactive input, composable with any AI tool
- **Multi-source resilient** — 8 data sources with automatic fallbacks; never blocked by a single API
- **Local-first** — your data stays on disk as simple JSON files, no cloud lock-in
- **Minimal footprint** — 3 dependencies, no database, no Docker (optional GROBID excepted)

---

## Quick Start

### Claude Code / OpenClaw Users

```bash
# One-line setup
bash install.sh --global

# Then in Claude Code:
#   /opg Search papers about LLM agent security
#   /opg Build a citation graph from ARXIV:1706.03762
```

### Standalone CLI

```bash
pip install httpx pymupdf scholarly

# Search → Build → Explore → Export
python openpapergraph_cli.py search "prompt injection" --limit 5 --pretty
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json
python openpapergraph_cli.py serve graph.json        # -> http://localhost:8787
python openpapergraph_cli.py export graph.json --format bibtex -o refs.bib
```

> No API keys needed. All core features work with free public APIs.

---

## Features at a Glance

| | Command | Description |
|---|---------|------------|
| **Discover** | `search` | Multi-source search across arXiv + DBLP + Semantic Scholar |
| | `recommend` | Related paper recommendations via S2 API |
| | `monitor` | Track recently published papers on a topic |
| **Build** | `graph` | Build citation networks from titles, IDs, PDFs, or BibTeX |
| | `graph-from-pdf` | Build graph directly from PDF reference lists |
| | `pdf` | Extract and resolve references from a PDF |
| | `zotero` | Import papers from Zotero library |
| **Manage** | `serve` | Interactive browser-based graph editor (persistent) |
| | `remove-seed` | Remove a seed paper and its exclusive connections |
| | `remove-paper` | Remove a non-seed paper from a graph |
| **Analyze** | `analyze` | Topic analysis: keywords, year trends, top authors |
| | `summary` | AI-powered or extractive research summaries |
| **Export** | `export` | BibTeX / CSV / Markdown / JSON |
| | `export-html` | Self-contained interactive HTML visualization |
| **Util** | `conferences` | List supported conference venue filters |
| | `llm-providers` | Show 20+ LLM providers and config status |

**Common flags**: `--output` / `-o` (save to file), `--pretty` (formatted JSON), `--quiet` (suppress progress)

---

## Typical Workflow

```
  Search          Build            Explore           Analyze          Export
  ──────>        ──────>          ──────>           ──────>         ──────>
  search    graph / graph-from-pdf   serve        analyze + summary    export / export-html
  recommend                      (browser UI)                       (bibtex, csv, md, html)
  monitor                        add / convert
                                 remove
```

```bash
# 1. Find seed papers
python openpapergraph_cli.py search "attention is all you need" --limit 5 --pretty

# 2. Build citation network
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json

# 3. Explore interactively (add papers, expand seeds — all changes saved)
python openpapergraph_cli.py serve graph.json

# 4. Analyze
python openpapergraph_cli.py analyze graph.json --pretty
python openpapergraph_cli.py summary graph.json --style overview --pretty

# 5. Export
python openpapergraph_cli.py export graph.json --format bibtex -o refs.bib
python openpapergraph_cli.py export-html graph.json -o graph.html --summary
```

---

## Architecture: Multi-Source by Design

OpenPaperGraph reduces dependency on any single data source by combining 8 academic databases with intelligent fallbacks:

| Task | Primary Sources | Fallback |
|------|----------------|----------|
| **Search** | arXiv + DBLP + Semantic Scholar | Deduplicated, sorted by citations |
| **References** | Download PDF -> parse reference list | S2 API |
| **Citations** | Google Scholar | S2 API |
| **Recommendations** | S2 Recommendations API | — |
| **PDF download** | arXiv -> Unpaywall (Open Access) | Direct URL |
| **Reference resolution** | arXiv -> CrossRef -> OpenAlex | Multi-cascade |

The tool continues to work even when individual APIs are down or rate-limited.

---

## Installation

### Prerequisites

- **Python 3.8+**
- **pip** package manager
- **Internet connection** (for querying academic APIs)

### Core Dependencies (3 packages)

```bash
pip install httpx pymupdf scholarly
```

| Package | Purpose |
|---------|---------|
| `httpx` | HTTP client for all API calls |
| `pymupdf` | PDF text extraction |
| `scholarly` | Google Scholar access (falls back to S2 if not installed) |

### Optional

```bash
pip install openai  # For LLM-powered summaries (20+ providers supported)
```

### API Keys (all optional)

```bash
export S2_API_KEY="..."          # Recommended: avoid S2 rate limiting (free key)
export OPENAI_API_KEY="sk-..."   # Or any of 20+ LLM providers for AI summaries
```

> **All commands work without any API keys.** arXiv, DBLP, CrossRef, OpenAlex, and Unpaywall are completely free.

---

## Commands Reference

<details>
<summary><b>search</b> — Multi-source paper search</summary>

```bash
python openpapergraph_cli.py search "transformer attention" --limit 10
python openpapergraph_cli.py search "large language model" --source arxiv
python openpapergraph_cli.py search "reinforcement learning" --source dblp --venue NeurIPS
```

| Flag | Default | Description |
|------|---------|-------------|
| `--source` | `all` | `all`, `arxiv`, `dblp`, or `s2` |
| `--venue` | — | `ICLR`, `NeurIPS`, `ICML`, `ACL`, `EMNLP`, `NAACL`, `WebConf`, `KDD` |
| `--limit` | 20 | Max results |
| `--offset` | 0 | Pagination offset |

</details>

<details>
<summary><b>graph</b> — Build citation network</summary>

Accepts multiple input formats:

| Input Type | Example |
|------------|---------|
| arXiv ID | `ARXIV:1706.03762` |
| DOI | `DOI:10.1145/3295222` |
| S2 hex ID | `204e3073870fae3d05bcbc2f6a8e263d9b72e776` |
| Paper title | `"attention is all you need"` |
| PDF file | `paper.pdf` |
| BibTeX file | `refs.bib` |
| Zotero JSON | `zotero_export.json` |

```bash
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json
python openpapergraph_cli.py graph "attention is all you need" "BERT" paper.pdf -o graph.json --depth 2
```

| Flag | Default | Description |
|------|---------|-------------|
| `--depth` | 1 | Expansion depth (2 = refs of refs) |
| `--include-unresolved` | false | Keep unresolved PDF references |
| `--use-grobid` | false | Use GROBID for PDF parsing |

#### Default Retrieval Limits

Each seed paper retrieves references and citations with the following built-in limits:

| Data | Source | Default Limit |
|------|--------|---------------|
| **References** | PDF parsing (local) | Unlimited (all refs in PDF) |
| **References** | S2 API (fallback) | 100 per paper |
| **Citations** | Google Scholar | 50 per paper |
| **Citations** | S2 API (fallback) | 50 per paper |
| **Depth > 1 expansion** | S2 API | 20 refs × top 10 cited papers |

These limits are hardcoded to balance completeness with API rate limits. They are not currently configurable via CLI flags. For papers with hundreds of citations, only the first 50 are retrieved; the remaining can be discovered by converting those papers to seeds via `serve`.

</details>

<details>
<summary><b>recommend</b> — Paper recommendations</summary>

```bash
python openpapergraph_cli.py recommend ARXIV:1706.03762 --limit 10 --pretty
```

Uses S2 Recommendations API. Accepts paper ID or title.

</details>

<details>
<summary><b>monitor</b> — Track new papers</summary>

```bash
python openpapergraph_cli.py monitor "LLM agent security" --year 2025 --limit 20
```

</details>

<details>
<summary><b>analyze</b> — Topic analysis</summary>

```bash
python openpapergraph_cli.py analyze graph.json --pretty
```

Returns: keywords, year distribution, top authors, topic clusters.

</details>

<details>
<summary><b>summary</b> — Research summary</summary>

```bash
python openpapergraph_cli.py summary graph.json --style overview --pretty
python openpapergraph_cli.py summary graph.json --provider deepseek --model deepseek-chat
```

Styles: `overview`, `trends`, `gaps`. Without LLM key, uses extractive analysis.

</details>

<details>
<summary><b>pdf</b> — PDF reference extraction</summary>

```bash
python openpapergraph_cli.py pdf paper.pdf --pretty
python openpapergraph_cli.py pdf paper.pdf --use-grobid  # Better accuracy
```

</details>

<details>
<summary><b>serve</b> — Interactive graph server</summary>

**Prerequisite**: You need a graph JSON file first. Build one with `graph` or `graph-from-pdf`:

```bash
# 1. Build a graph first
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json

# 2. Then start the server
python openpapergraph_cli.py serve graph.json --port 8787
# Open http://localhost:8787 in your browser
```

| Flag | Default | Description |
|------|---------|-------------|
| `--port` | 8787 | HTTP server port |
| `--title` | `"Paper Graph"` | Page title shown in browser |

#### Browser UI Features

**Graph Visualization** (powered by vis.js)
- Interactive force-directed network layout with drag, zoom, and pan
- Color-coded nodes: seeds (purple stars), references (blue, left), citations (green, right)
- Year-based gradient coloring with a visual legend
- Hover tooltips showing title, authors, year, citation count, and abstract
- Click a node to highlight its direct connections

**Toolbar & Search**
- Real-time search across titles, authors, and years
- Filter by node type (All / Seed / Reference / Citation) and source database
- Paper count statistics displayed as chips (seeds, refs, cites)

**Add Paper** (server mode only)
- Click `+ Add Paper` button to open a modal with three input tabs:
  - **Title** — enter a paper title; the system resolves it via arXiv + Semantic Scholar
  - **BibTeX** — paste a BibTeX entry for direct import
  - **PDF** — drag-and-drop or click to upload a PDF; extracts metadata and references automatically
- Toggle **"Treat as Seed Paper"** to trigger full expansion (references + citations)
- Same retrieval limits as `graph` command (50 citations, 100 S2 refs per paper)
- SSE-powered real-time progress bar during graph expansion

**Convert to Seed**
- Hover over any non-seed node and click **"Convert to Seed Paper"** in the tooltip
- Triggers full expansion: downloads PDF, parses references, fetches citations
- Confirmation dialog with live progress tracking

**Remove Seed**
- Source panel lists all seed papers with reference/citation counts
- Click **Remove** to delete a seed and its exclusive connections
- Confirmation dialog prevents accidental deletion

**Metadata Enrichment**
- **Enrich** button fills in missing metadata (abstracts, citation counts, DOIs) for all papers

**Persistence**
- All mutations (add, convert, remove, enrich) are saved to the JSON file immediately
- Refresh the page at any time — your changes are preserved

**Sidebar**
- Scrollable paper list sorted by citation count, grouped by type
- Click any paper to locate and highlight it in the graph
- External links to Semantic Scholar, arXiv, and DOI pages

</details>

<details>
<summary><b>export</b> / <b>export-html</b> — Export graph data</summary>

```bash
python openpapergraph_cli.py export graph.json --format bibtex -o refs.bib
python openpapergraph_cli.py export graph.json --format csv -o papers.csv
python openpapergraph_cli.py export graph.json --format markdown -o papers.md
python openpapergraph_cli.py export-html graph.json -o graph.html --summary --inline
```

</details>

<details>
<summary><b>remove-seed</b> / <b>remove-paper</b> — Remove papers from graph</summary>

```bash
# Remove seed + its exclusive connections
python openpapergraph_cli.py remove-seed graph.json "ClawWorm"

# Remove a non-seed paper
python openpapergraph_cli.py remove-paper graph.json "some paper title"
```

Both support fuzzy title matching. Use `-o` to save to a different file.

</details>

<details>
<summary><b>/opg</b> — Claude Code / OpenClaw skill</summary>

When installed as a skill (`bash install.sh --global`), all commands can be invoked via natural language in Claude Code or OpenClaw:

| What you want | What to type |
|---------------|-------------|
| Search papers | `/opg Search papers about transformer attention` |
| Build graph | `/opg Build a citation network from ARXIV:1706.03762` |
| Recommendations | `/opg Recommend papers similar to ARXIV:1706.03762` |
| Interactive viewer | `/opg Start a graph server for graph.json` |
| Analyze | `/opg Analyze the citation network in graph.json` |
| Export | `/opg Export the citation network as BibTeX` |
| Monitor | `/opg Monitor new papers on LLM agent security since 2025` |
| PDF extraction | `/opg Extract references from paper.pdf` |

| | `/opg` Skill | Python CLI |
|---|-------------|------------|
| **Invocation** | Natural language | Command-line arguments |
| **Output** | AI auto-formats results | Raw JSON (`--pretty` for formatted) |
| **Multi-step** | AI chains steps automatically | Run each step manually |
| **Setup** | `bash install.sh --global` | `pip install httpx pymupdf scholarly` |

</details>

---

## Environment Variables

### S2_API_KEY (Recommended)

Free at [semanticscholar.org/product/api](https://www.semanticscholar.org/product/api). Avoids rate limiting.

```bash
export S2_API_KEY="your_key_here"
```

### LLM API Keys (Optional — pick any one)

<details>
<summary><b>20+ supported providers</b></summary>

#### US Providers

| Provider | Env Variable | Default Model |
|----------|-------------|---------------|
| OpenAI | `OPENAI_API_KEY` | `gpt-4o-mini` |
| Anthropic | `ANTHROPIC_API_KEY` | `claude-sonnet-4-20250514` |
| Google Gemini | `GEMINI_API_KEY` | `gemini-2.0-flash` |
| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek-chat` |
| Groq | `GROQ_API_KEY` | `llama-3.1-8b-instant` |
| Together AI | `TOGETHER_API_KEY` | `Llama-3.1-8B-Instruct-Turbo` |
| Fireworks AI | `FIREWORKS_API_KEY` | `llama-v3p1-8b-instruct` |
| Mistral | `MISTRAL_API_KEY` | `mistral-small-latest` |
| xAI (Grok) | `XAI_API_KEY` | `grok-3-mini-fast` |
| Perplexity | `PERPLEXITY_API_KEY` | `sonar` |
| OpenRouter | `OPENROUTER_API_KEY` | `llama-3.1-8b-instruct:free` |

#### Chinese Providers

| Provider | Env Variable | Default Model |
|----------|-------------|---------------|
| Zhipu | `ZHIPUAI_API_KEY` | `glm-4-flash` |
| Moonshot | `MOONSHOT_API_KEY` | `moonshot-v1-8k` |
| Baichuan | `BAICHUAN_API_KEY` | `Baichuan2-Turbo` |
| Yi | `YI_API_KEY` | `yi-lightning` |
| Qwen | `DASHSCOPE_API_KEY` | `qwen-turbo` |
| Doubao | `ARK_API_KEY` | `doubao-lite-32k` |
| MiniMax | `MINIMAX_API_KEY` | `MiniMax-Text-01` |
| Stepfun | `STEPFUN_API_KEY` | `step-1-flash` |
| SenseTime | `SENSENOVA_API_KEY` | `SenseChat-Turbo` |

#### Custom / Self-hosted

```bash
export LLM_API_KEY="your_key"
export LLM_BASE_URL="http://localhost:11434/v1"  # e.g. Ollama
export LLM_MODEL="llama3"
```

</details>

---

## Graph JSON Format

```jsonc
{
  "nodes": [
    {
      "id": "ARXIV:1706.03762",
      "title": "Attention is All you Need",
      "authors": ["Ashish Vaswani", "Noam Shazeer"],
      "year": 2017,
      "citation_count": 169932,
      "is_seed": true,
      "source": "semantic_scholar"
    }
  ],
  "edges": [
    { "source": "paper_A_id", "target": "paper_B_id", "type": "cites" }
  ],
  "seed_papers": ["ARXIV:1706.03762"],
  "total_papers": 90
}
```

**Edge convention**: `A -> B` means **A cites B**.

---

## GROBID Setup (Optional)

[GROBID](https://github.com/kermitt2/grobid) provides structured PDF extraction with **70-90% resolve rate** (vs 40-60% with regex).

```bash
# Quick start
docker run -d --name grobid -p 8070:8070 lfoppiano/grobid:0.8.1
curl -s http://localhost:8070/api/isalive  # verify

# Use with any PDF command
python openpapergraph_cli.py pdf paper.pdf --use-grobid
```

<details>
<summary>Docker Compose (persistent)</summary>

```yaml
services:
  grobid:
    image: lfoppiano/grobid:0.8.1
    ports:
      - "8070:8070"
    restart: unless-stopped
```

</details>

---

## Python API

```python
import sys; sys.path.insert(0, "/path/to/skill-version")

from services.semantic_scholar import search
total, papers = search("transformer attention", limit=10)

from services.graph_builder import build_graph
graph = build_graph(paper_ids=["ARXIV:1706.03762"], depth=1)

from services.analysis import analyze, summarize
result = analyze(graph.nodes)
```

<details>
<summary><b>Available modules</b></summary>

| Module | Key Functions |
|--------|--------------|
| `services.semantic_scholar` | `search()`, `get_paper()`, `get_references()`, `get_citations()`, `recommend()` |
| `services.arxiv` | `search()` |
| `services.dblp` | `search()` |
| `services.google_scholar` | `search()`, `get_citations()`, `get_citation_count()` |
| `services.graph_builder` | `build_graph()`, `build_graph_from_pdfs()` |
| `services.graph_manager` | `expand_as_seed()`, `add_non_seed()`, `convert_to_seed()` |
| `services.graph_server` | `start_server()` |
| `services.pdf_parser` | `parse_pdf()` |
| `services.reference_resolver` | `resolve_references()` |
| `services.analysis` | `analyze()`, `summarize()` |
| `services.llm_client` | `llm_chat()`, `is_llm_available()`, `list_providers()` |
| `services.export` | `to_bibtex()`, `to_csv()`, `to_markdown()`, `to_json()` |
| `services.html_export` | `export_html()` |
| `services.paper_downloader` | `download_pdf()` |
| `services.zotero` | `get_collections()`, `import_papers()` |

</details>

---

## Troubleshooting

<details>
<summary><b>Common issues</b></summary>

**Q: HTTP 429 from Semantic Scholar**
Set `S2_API_KEY` (free at [semanticscholar.org/product/api](https://www.semanticscholar.org/product/api)).

**Q: Google Scholar returns no results**
GS may rate-limit automated access. Falls back to S2 automatically. Wait a few minutes or use `--source s2`.

**Q: `summary` returns extractive instead of LLM-generated**
No LLM key detected. Set any provider key. Run `llm-providers --pretty` to check.

**Q: Low PDF reference resolve rate**
Try `--use-grobid` (70-90% vs 40-60% with regex). See [GROBID Setup](#grobid-setup-optional).

**Q: `graph` command is slow**
Expected. Building involves PDF download, reference parsing, Google Scholar queries (2s rate limit), and multi-API resolution. A single seed with ~50 refs takes 2-5 minutes.

**Q: `serve` shows empty graph**
Build a graph first with `graph` or `graph-from-pdf`.

**Q: Where are PDFs cached?**
`$TMPDIR/opg_pdf_cache/` (usually `/tmp/opg_pdf_cache/`). Delete to clear.

</details>

---

## File Structure

```
skill-version/
├── README.md                     # This file
├── README_CN.md                  # Chinese version
├── SKILL.md                      # Claude Code / OpenClaw skill definition
├── install.sh                    # One-line installer
├── openpapergraph_cli.py         # CLI entry point (16 commands)
├── schemas.py                    # Data models (Paper, GraphData)
├── requirements.txt              # httpx + pymupdf + scholarly
└── services/
    ├── semantic_scholar.py       # S2 API + multi-source search
    ├── arxiv.py                  # arXiv search
    ├── dblp.py                   # DBLP search + venue filtering
    ├── google_scholar.py         # Google Scholar citations
    ├── paper_downloader.py       # PDF download (arXiv/Unpaywall/URL)
    ├── graph_builder.py          # Citation network construction
    ├── graph_manager.py          # Graph mutations (add/convert/remove)
    ├── graph_server.py           # Interactive HTTP server
    ├── pdf_parser.py             # PDF reference extraction
    ├── reference_resolver.py     # Multi-source reference resolution
    ├── bibtex_parser.py          # BibTeX & Zotero parser
    ├── zotero.py                 # Zotero library import
    ├── export.py                 # BibTeX/CSV/Markdown/JSON export
    ├── html_export.py            # Interactive HTML visualization
    ├── analysis.py               # Topic analysis + summarization
    └── llm_client.py             # 20+ provider LLM client
```

---

## Data Sources

| Source | Coverage | Used For |
|--------|----------|----------|
| **arXiv** | Preprints (CS, physics, math, bio) | Search, PDF download, resolution |
| **DBLP** | CS conferences & journals | Search, conference filtering |
| **Google Scholar** | All disciplines | Citations, citation counts |
| **Semantic Scholar** | 200M+ papers | Search fallback, recommendations |
| **CrossRef** | DOI-registered publications | Reference resolution |
| **OpenAlex** | 250M+ works | Reference resolution |
| **Unpaywall** | Open-access PDFs | PDF download |
| **Zotero** | User's library | Import collections |

---

## Contact

For questions, suggestions, or collaboration: **[REDACTED_SECRET]**

---

## License

OpenPaperGraph is licensed under the [PolyForm Noncommercial License 1.0.0](LICENSE).


path: README_CN.md
content: <p align="center">
  <img alt="OpenPaperGraph" src="figures/logo-part-light-full.png" width="420">
</p>

<h3 align="center">OpenClaw 时代的学术研究助手</h3>

<p align="center">
  <img src="https://img.shields.io/badge/python-3.8+-blue.svg" alt="Python 3.8+">
  <img src="https://img.shields.io/badge/dependencies-3-green.svg" alt="3 Dependencies">
  <img src="https://img.shields.io/badge/LLM_providers-20+-orange.svg" alt="20+ LLM Providers">
  <img src="https://img.shields.io/badge/data_sources-8-purple.svg" alt="8 Data Sources">
  <img src="https://img.shields.io/badge/license-PolyForm%20Noncommercial%201.0.0-lightgrey.svg" alt="License">
</p>

<p align="center">
  中文 | <a href="README.md">English</a>
</p>

<p align="center">
  <b>跨 8 大数据源搜索论文，构建引用网络，生成 AI 摘要。</b><br>
  可作为 Claude Code / OpenClaw 的 <code>/opg</code> 技能使用，也可独立运行。<br>
  无需 API Key，仅需 3 个 pip 包，一键安装。
</p>

---

## 为什么选择 OpenPaperGraph？

在 **Claude Code** 和 **OpenClaw** 等 AI Agent 时代，研究者需要与工作流无缝配合的工具。OpenPaperGraph 正是为此而生：

- **Agent 原生** — JSON 标准输出，零交互输入，可与任何 AI 工具组合
- **多源容灾** — 8 大数据源 + 自动降级，单个 API 宕机不影响使用
- **本地优先** — 数据以 JSON 文件存储在本地，无云端依赖
- **极简依赖** — 仅 3 个依赖包，无数据库，无 Docker（GROBID 可选）

---

## 快速开始

### Claude Code / OpenClaw 用户

```bash
# 一键安装
bash install.sh --global

# 然后在 Claude Code 中直接输入：
#   /opg 搜索关于 LLM agent 安全的论文
#   /opg 基于 ARXIV:1706.03762 构建引用网络
```

### 独立 CLI 使用

```bash
pip install httpx pymupdf scholarly

# 搜索 → 构建 → 探索 → 导出
python openpapergraph_cli.py search "prompt injection" --limit 5 --pretty
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json
python openpapergraph_cli.py serve graph.json        # -> http://localhost:8787
python openpapergraph_cli.py export graph.json --format bibtex -o refs.bib
```

> 无需 API Key。所有核心功能均可通过免费公开 API 使用。

---

## 功能一览

| | 命令 | 说明 |
|---|------|------|
| **发现** | `search` | 跨 arXiv + DBLP + Semantic Scholar 多源搜索 |
| | `recommend` | 通过 S2 API 获取相关论文推荐 |
| | `monitor` | 追踪某个主题的最新论文 |
| **构建** | `graph` | 从标题、ID、PDF 或 BibTeX 构建引用网络 |
| | `graph-from-pdf` | 从 PDF 参考文献列表直接构建引用图 |
| | `pdf` | 从 PDF 中提取并解析参考文献 |
| | `zotero` | 从 Zotero 文献库导入 |
| **管理** | `serve` | 基于浏览器的交互式图编辑器（持久化存储） |
| | `remove-seed` | 删除种子论文及其独占连接 |
| | `remove-paper` | 从图中删除非种子论文 |
| **分析** | `analyze` | 主题分析：关键词、年份趋势、高产作者 |
| | `summary` | AI 驱动或抽取式研究摘要 |
| **导出** | `export` | BibTeX / CSV / Markdown / JSON |
| | `export-html` | 自包含的交互式 HTML 可视化 |
| **工具** | `conferences` | 列出支持的会议过滤项 |
| | `llm-providers` | 显示 20+ LLM 提供商及配置状态 |

**通用参数**：`--output` / `-o`（保存到文件）、`--pretty`（格式化 JSON）、`--quiet`（静默模式）

---

## 典型工作流

```
  搜索          构建           探索            分析           导出
  ──────>      ──────>       ──────>        ──────>       ──────>
  search   graph/graph-from-pdf  serve     analyze+summary   export/export-html
  recommend                  (浏览器 UI)                    (bibtex, csv, md, html)
  monitor                    添加/转化/删除
```

```bash
# 1. 查找种子论文
python openpapergraph_cli.py search "attention is all you need" --limit 5 --pretty

# 2. 构建引用网络
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json

# 3. 交互式探索（添加论文、扩展种子 — 所有修改自动保存）
python openpapergraph_cli.py serve graph.json

# 4. 分析
python openpapergraph_cli.py analyze graph.json --pretty
python openpapergraph_cli.py summary graph.json --style overview --pretty

# 5. 导出
python openpapergraph_cli.py export graph.json --format bibtex -o refs.bib
python openpapergraph_cli.py export-html graph.json -o graph.html --summary
```

---

## 交互式图谱服务器（`serve`）

**前提**：需要先有一个 graph JSON 文件。使用 `graph` 或 `graph-from-pdf` 构建：

```bash
# 1. 先构建引用图
python openpapergraph_cli.py graph ARXIV:1706.03762 -o graph.json

# 2. 再启动服务器
python openpapergraph_cli.py serve graph.json --port 8787
# 在浏览器中打开 http://localhost:8787
```

| 参数 | 默认值 | 说明 |
|------|--------|------|
| `--port` | 8787 | HTTP 服务器端口 |
| `--title` | `"Paper Graph"` | 浏览器页面标题 |

### 浏览器前端功能

**图谱可视化**（基于 vis.js）
- 交互式力导向网络布局，支持拖拽、缩放、平移
- 节点颜色编码：种子论文（紫色星形）、参考文献（蓝色，左侧）、引用文献（绿色，右侧）
- 基于年份的渐变着色 + 可视化图例
- 悬浮提示显示标题、作者、年份、引用数和摘要
- 点击节点高亮其直接连接

**工具栏 & 搜索**
- 实时搜索标题、作者和年份
- 按节点类型（全部 / 种子 / 参考文献 / 引用文献）和数据源过滤
- 论文数量统计芯片（种子、参考文献、引用文献）

**添加论文**（服务器模式独有）
- 点击 `+ Add Paper` 按钮打开模态框，支持三种输入方式：
  - **标题** — 输入论文标题，系统通过 arXiv + Semantic Scholar 自动解析
  - **BibTeX** — 粘贴 BibTeX 条目直接导入
  - **PDF** — 拖放或点击上传 PDF，自动提取元数据和参考文献
- 勾选 **"Treat as Seed Paper"** 触发完整扩展（参考文献 + 引用文献）
- 与 `graph` 命令相同的检索限制（每篇论文最多 50 条引用、100 条 S2 参考文献）
- SSE 驱动的实时进度条

**转化为种子论文**
- 悬浮在非种子节点上，点击提示框中的 **"Convert to Seed Paper"**
- 触发完整扩展：下载 PDF、解析参考文献、抓取引用文献
- 带实时进度追踪的确认对话框

**删除种子论文**
- 来源面板列出所有种子论文及其参考/引用数量
- 点击 **Remove** 删除种子及其独占连接
- 确认对话框防止误删

**元数据补全**
- **Enrich** 按钮为所有论文补全缺失的摘要、引用数、DOI 等元数据

**持久化存储**
- 所有操作（添加、转化、删除、补全）立即写入 JSON 文件
- 随时刷新页面，所有更改均已保存

**侧边栏**
- 按引用数排序的可滚动论文列表，按类型分组
- 点击任意论文定位并高亮图中节点
- 外部链接跳转至 Semantic Scholar、arXiv 和 DOI 页面

### 默认检索限制

构建图谱（`graph` 命令）和服务器模式（`serve` 添加论文）使用相同的检索限制：

| 数据 | 来源 | 默认限制 |
|------|------|---------|
| **参考文献** | PDF 解析（本地） | 无限制（提取 PDF 中所有参考文献） |
| **参考文献** | S2 API（降级） | 每篇论文 100 条 |
| **引用文献** | Google Scholar | 每篇论文 50 条 |
| **引用文献** | S2 API（降级） | 每篇论文 50 条 |
| **depth > 1 扩展** | S2 API | 20 条参考文献 × 前 10 篇高引论文 |

这些限制为硬编码值，在完整性和 API 速率限制之间取得平衡，当前不可通过命令行参数修改。对于有数百条引用的论文，仅检索前 50 条；其余可通过在 `serve` 中将这些论文转化为种子来发现。

---

## 架构：多源设计

OpenPaperGraph 通过组合 8 大学术数据库 + 智能降级策略，降低对单一数据源的依赖：

| 任务 | 主要数据源 | 降级方案 |
|------|-----------|---------|
| **搜索** | arXiv + DBLP + Semantic Scholar | 自动去重，按引用量排序 |
| **参考文献** | 下载 PDF -> 解析参考文献列表 | S2 API |
| **引用** | Google Scholar | S2 API |
| **推荐** | S2 推荐 API | — |
| **PDF 下载** | arXiv -> Unpaywall（开放获取） | 直接 URL |
| **参考文献解析** | arXiv -> CrossRef -> OpenAlex | 多级级联 |

即使某个 API 宕机或被限流，工具仍可正常工作。

---

## 安装

### 环境要求

- **Python 3.8+**
- **pip** 包管理器
- **网络连接**（用于查询学术 API）

### 核心依赖（3 个包）

```bash
pip install httpx pymupdf scholarly
```

| 包 | 用途 |
|----|------|
| `httpx` | 所有 API 调用的 HTTP 客户端 |
| `pymupdf` | PDF 文本提取 |
| `scholarly` | Google Scholar 访问（未安装时自动降级到 S2） |

### 可选

```bash
pip install openai  # 用于 LLM 驱动的摘要（支持 20+ 提供商）
```

### API Key（均为可选）

```bash
export S2_API_KEY="..."          # 推荐：避免 S2 限流（免费申请）
export OPENAI_API_KEY="sk-..."   # 或任意 20+ LLM 提供商的 Key，用于 AI 摘要
```

> **所有命令无需 API Key 即可使用。** arXiv、DBLP、CrossRef、OpenAlex、Unpaywall 完全免费。

---

## `/opg` 技能用法（Claude Code / OpenClaw）

安装为技能后（`bash install.sh --global`），所有命令均可在 Claude Code 或 OpenClaw 中通过自然语言调用：

| 你想做什么 | 在 Claude Code 中输入 |
|-----------|---------------------|
| 搜索论文 | `/opg 搜索关于 transformer attention 的论文` |
| 构建引用图 | `/opg 基于 ARXIV:1706.03762 构建引用网络` |
| 获取推荐 | `/opg 推荐与 ARXIV:1706.03762 相似的论文` |
| 交互查看 | `/opg 启动 graph.json 的图服务器` |
| 分析网络 | `/opg 分析 graph.json 的引文网络` |
| 导出 | `/opg 将引用网络导出为 BibTeX` |
| 监控新论文 | `/opg 监控 2025 年以来 LLM agent 安全的新论文` |
| PDF 提取 | `/opg 从 paper.pdf 提取参考文献` |

| | `/opg` 技能 | Python CLI |
|---|-------------|------------|
| **调用方式** | 自然语言 | 命令行参数 |
| **输出** | AI 自动格式化 | 原始 JSON（`--pretty` 格式化） |
| **多步骤** | AI 自动串联步骤 | 手动逐步执行 |
| **安装** | `bash install.sh --global` | `pip install httpx pymupdf scholarly` |

---

## 环境变量

### S2_API_KEY（推荐）

免费申请：[semanticscholar.org/product/api](https://www.semanticscholar.org/product/api)，避免限流。

```bash
export S2_API_KEY="your_key_here"
```

### LLM API Key（可选 — 任选其一）

<details>
<summary><b>20+ 支持的提供商</b></summary>

#### 国际提供商

| 提供商 | 环境变量 | 默认模型 |
|--------|---------|---------|
| OpenAI | `OPENAI_API_KEY` | `gpt-4o-mini` |
| Anthropic | `ANTHROPIC_API_KEY` | `claude-sonnet-4-20250514` |
| Google Gemini | `GEMINI_API_KEY` | `gemini-2.0-flash` |
| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek-chat` |
| Groq | `GROQ_API_KEY` | `llama-3.1-8b-instant` |
| Together AI | `TOGETHER_API_KEY` | `Llama-3.1-8B-Instruct-Turbo` |
| Mistral | `MISTRAL_API_KEY` | `mistral-small-latest` |
| xAI (Grok) | `XAI_API_KEY` | `grok-3-mini-fast` |
| Perplexity | `PERPLEXITY_API_KEY` | `sonar` |
| OpenRouter | `OPENROUTER_API_KEY` | `llama-3.1-8b-instruct:free` |

#### 国内提供商

| 提供商 | 环境变量 | 默认模型 |
|--------|---------|---------|
| 智谱 AI | `ZHIPUAI_API_KEY` | `glm-4-flash` |
| 月之暗面 | `MOONSHOT_API_KEY` | `moonshot-v1-8k` |
| 百川 | `BAICHUAN_API_KEY` | `Baichuan2-Turbo` |
| 零一万物 | `YI_API_KEY` | `yi-lightning` |
| 通义千问 | `DASHSCOPE_API_KEY` | `qwen-turbo` |
| 豆包 | `ARK_API_KEY` | `doubao-lite-32k` |
| MiniMax | `MINIMAX_API_KEY` | `MiniMax-Text-01` |
| 阶跃星辰 | `STEPFUN_API_KEY` | `step-1-flash` |
| 商汤日日新 | `SENSENOVA_API_KEY` | `SenseChat-Turbo` |

#### 自托管 / 自定义

```bash
export LLM_API_KEY="your_key"
export LLM_BASE_URL="http://localhost:11434/v1"  # 例如 Ollama
export LLM_MODEL="llama3"
```

</details>

---

## GROBID 设置（可选）

[GROBID](https://github.com/kermitt2/grobid) 提供结构化 PDF 提取，参考文献解析率 **70-90%**（正则方式为 40-60%）。

```bash
docker run -d --name grobid -p 8070:8070 lfoppiano/grobid:0.8.1
python openpapergraph_cli.py pdf paper.pdf --use-grobid
```

---

## 数据源

| 数据源 | 覆盖范围 | 用途 |
|--------|---------|------|
| **arXiv** | 预印本（CS、物理、数学、生物） | 搜索、PDF 下载、解析 |
| **DBLP** | CS 会议与期刊 | 搜索、会议过滤 |
| **Google Scholar** | 全学科 | 引用数据、引用计数 |
| **Semantic Scholar** | 2 亿+ 论文 | 搜索降级、推荐 |
| **CrossRef** | DOI 注册出版物 | 参考文献解析 |
| **OpenAlex** | 2.5 亿+ 作品 | 参考文献解析 |
| **Unpaywall** | 开放获取 PDF | PDF 下载 |
| **Zotero** | 用户个人文献库 | 导入文献集 |

---

## 联系方式

问题、建议或合作请联系：**[REDACTED_SECRET]**

---

## 许可证

OpenPaperGraph 采用 [PolyForm Noncommercial License 1.0.0](LICENSE) 许可。


path: TRADEMARKS.md
content: # Trademark Notice

"OpenPaperGraph", the OpenPaperGraph logo, and any related names, logos, or visual identifiers used by this project are trademarks, service marks, or brand assets of the project maintainer, unless otherwise stated.

No trademark rights are granted under the software license for this repository, including the PolyForm Noncommercial License 1.0.0.

You may use the project name and logo only for nominative and referential purposes, such as:

- describing the project,
- linking to the official repository,
- stating factual compatibility or origin.

You may **not** use the OpenPaperGraph name or logo:

- in the name of a fork, modified version, hosted service, or redistributed product in a way that suggests affiliation, endorsement, or official status;
- in branding, marketing, domain names, app names, company names, or social media accounts without prior written permission;
- for any commercial offering without prior written permission.

If you want to use the OpenPaperGraph name or logo beyond the limited purposes above, please contact:

**[REDACTED_SECRET]**


path: graph_manager.py
content: """Graph management — add papers, convert to seed, persist to JSON.

Provides the core logic for interactive graph editing from the HTML viewer.
All mutations are written back to the graph JSON file (source of truth).
"""
import sys
import re
import json
from typing import Optional, List, Tuple

from schemas import Paper, GraphData


def _title_similarity(a: str, b: str) -> float:
    """Jaccard similarity on word sets, with containment boost.

    If the shorter title's words are mostly contained in the longer title
    (e.g. truncated title), use containment ratio instead of Jaccard.
    This handles cases like 'Paper Title: Full Subtitle' vs 'Paper Title: Full'.
    """
    wa = set(re.sub(r'[^a-z0-9\s]', '', a.lower()).split())
    wb = set(re.sub(r'[^a-z0-9\s]', '', b.lower()).split())
    if not wa or not wb:
        return 0.0
    intersection = len(wa & wb)
    jaccard = intersection / len(wa | wb)
    # Containment: what fraction of the *shorter* set is in the longer?
    shorter = min(len(wa), len(wb))
    containment = intersection / shorter if shorter > 0 else 0.0
    # Use containment if shorter title has >= 5 words and >= 80% contained
    if shorter >= 5 and containment >= 0.8:
        return max(jaccard, containment)
    return jaccard


def _normalize_key(paper: Paper) -> str:
    """Stable dedup key: prefer DOI > arXiv ID > normalized title+year."""
    if paper.doi:
        return f"doi:{paper.doi.lower()}"
    if paper.arxiv_id:
        aid = re.sub(r'v\\d+$', '', paper.arxiv_id)
        return f"arxiv:{aid.lower().replace('arxiv:', '')}"
    norm_title = re.sub(r'[^a-z0-9]', '', (paper.title or '').lower())
    return f"title:{norm_title}:{paper.year or 'x'}"


def find_existing(graph: GraphData, paper: Paper) -> Optional[Paper]:
    """Find a paper in the graph by DOI, arXiv ID, or fuzzy title match."""
    key = _normalize_key(paper)
    for node in graph.nodes:
        if _normalize_key(node) == key:
            return node
        if paper.title and node.title and _title_similarity(paper.title, node.title) >= 0.85:
            if paper.year and node.year and abs(paper.year - node.year) <= 1:
                return node
            if not paper.year or not node.year:
                return node
    return None


def _dedup_edges(edges: list) -> list:
    seen = set()
    result = []
    for e in edges:
        key = (e["source"], e["target"])
        if key not in seen:
            seen.add(key)
            result.append(e)
    return result


def dedup_and_merge_nodes(graph: GraphData) -> int:
    """Find duplicate nodes in graph, merge them, and rewrite edges.

    Merge rules:
    - seed paper wins over non-seed (keep seed's ID, merge non-seed's info in)
    - same status → merge into the one with more complete metadata
    - all edges pointing to/from the removed node are rewritten to the kept node
    - seed_papers list is updated accordingly

    Returns number of nodes removed.
    """
    merged_count = 0
    # Build groups of duplicate nodes using _normalize_key + fuzzy title
    # Process in order so we can track which nodes have been consumed
    consumed = set()  # IDs of nodes that were merged into another
    id_remap = {}     # old_id → new_id for edge rewriting

    node_list = list(graph.nodes)
    for i, a in enumerate(node_list):
        if a.id in consumed:
            continue
        for j in range(i + 1, len(node_list)):
            b = node_list[j]
            if b.id in consumed:
                continue
            if not _is_same_paper(a, b):
                continue

            # Found duplicate: decide which to keep
            # Rule: seed wins; if both same status, keep whichever has more info
            if a.is_seed and not b.is_seed:
                keep, remove = a, b
            elif b.is_seed and not a.is_seed:
                keep, remove = b, a
            else:
                # Same status — keep the one with more metadata
                def _info_score(p):
                    s = 0
                    if p.doi: s += 1
                    if p.arxiv_id: s += 1
                    if p.abstract: s += 1
                    if p.year: s += 1
                    if p.authors: s += len(p.authors)
                    if p.url: s += 1
                    if p.pdf_url: s += 1
                    s += p.citation_count
                    return s
                if _info_score(a) >= _info_score(b):
                    keep, remove = a, b
                else:
                    keep, remove = b, a

            # Merge metadata from remove into keep
            _merge_metadata(keep, remove)
            # Also merge fields that _merge_metadata doesn't cover
            if not keep.title and remove.title:
                keep.title = remove.title
            if not keep.authors and remove.authors:
                keep.authors = remove.authors
            if not keep.year and remove.year:
                keep.year = remove.year
            if not keep.source or keep.source == "unknown":
                keep.source = remove.source
            if remove.refs_expanded and not keep.refs_expanded:
                keep.refs_expanded = True
            if remove.cites_expanded and not keep.cites_expanded:
                keep.cites_expanded = True
            if remove.is_seed:
                keep.is_seed = True

            consumed.add(remove.id)
            id_remap[remove.id] = keep.id
            merged_count += 1
            print(f"[dedup] Merged '{remove.title[:50] if remove.title else remove.id}' → '{keep.title[:50] if keep.title else keep.id}'", file=sys.stderr)

    if not id_remap:
        return 0

    # Remove consumed nodes
    graph.nodes = [n for n in graph.nodes if n.id not in consumed]

    # Rewrite edges
    for e in graph.edges:
        if e["source"] in id_remap:
            e["source"] = id_remap[e["source"]]
        if e["target"] in id_remap:
            e["target"] = id_remap[e["target"]]

    # Dedup edges after rewrite (merging may create duplicates)
    graph.edges = _dedup_edges(graph.edges)

    # Remove self-loops
    graph.edges = [e for e in graph.edges if e["source"] != e["target"]]

    # Update seed_papers list
    new_seeds = []
    seen_seeds = set()
    for sid in graph.seed_papers:
        actual = id_remap.get(sid, sid)
        if actual not in seen_seeds:
            seen_seeds.add(actual)
            new_seeds.append(actual)
    graph.seed_papers = new_seeds

    print(f"[dedup] Removed {merged_count} duplicate nodes, graph now {len(graph.nodes)} papers, {len(graph.edges)} edges", file=sys.stderr)
    return merged_count


def _is_same_paper(a: Paper, b: Paper) -> bool:
    """Check if two Paper objects refer to the same real-world paper."""
    if a.id == b.id:
        return True
    # DOI match
    if a.doi and b.doi and a.doi.lower().strip() == b.doi.lower().strip():
        return True
    # arXiv ID match (strip version suffix)
    if a.arxiv_id and b.arxiv_id:
        aid_a = re.sub(r'v\d+$', '', a.arxiv_id.lower().replace('arxiv:', ''))
        aid_b = re.sub(r'v\d+$', '', b.arxiv_id.lower().replace('arxiv:', ''))
        if aid_a == aid_b:
            return True
    # Normalized key match
    if _normalize_key(a) == _normalize_key(b):
        return True
    # Fuzzy title match
    if a.title and b.title and _title_similarity(a.title, b.title) >= 0.85:
        if a.year and b.year:
            return abs(a.year - b.year) <= 1
        return True  # if either year unknown, trust title similarity
    return False


def save_graph(graph: GraphData, path: str):
    """Dedup + merge nodes, then persist graph to JSON file."""
    dedup_and_merge_nodes(graph)
    graph.edges = _dedup_edges(graph.edges)
    graph.total_papers = len(graph.nodes)
    graph.save(path)
    print(f"[graph-manager] Saved graph to {path}: {graph.total_papers} papers, {len(graph.edges)} edges", file=sys.stderr)


def enrich_incomplete_papers(graph: GraphData, graph_path: str) -> int:
    """Enrich papers that are missing year/authors/abstract by re-fetching metadata.

    Tries: S2 API (by paper ID) → arXiv search (by title) → S2 search (by title).
    Returns number of papers enriched.
    """
    from services import semantic_scholar as s2
    from services import arxiv

    incomplete = [p for p in graph.nodes
                  if p.resolved and (not p.year or not p.authors or not p.abstract)]

    if not incomplete:
        return 0

    print(f"[graph-manager] Enriching {len(incomplete)} papers with missing metadata...",
          file=sys.stderr)
    enriched = 0

    for paper in incomplete:
        found = None

        # Strategy 1: S2 API direct lookup by ID (most reliable)
        if paper.id and not paper.id.startswith(("pdf:", "bibtex:", "crossref:", "openalex:", "unresolved:")):
            try:
                result = s2.get_paper(paper.id)
                found = result.get("paper") if isinstance(result, dict) else None
            except Exception:
                pass

        # Strategy 2: arXiv search by title
        if not found and paper.title:
            try:
                _, results = arxiv.search(paper.title, limit=3)
                for r in results:
                    if _title_similarity(paper.title, r.title) >= 0.7:
                        found = r
                        break
            except Exception:
                pass

        # Strategy 3: S2 search by title
        if not found and paper.title:
            try:
                _, results = s2._s2_search(paper.title, limit=3)
                for r in results:
                    if _title_similarity(paper.title, r.title) >= 0.7:
                        found = r
                        break
            except Exception:
                pass

        if found:
            if not paper.year and found.year:
                paper.year = found.year
            if (not paper.authors or paper.authors == []) and found.authors:
                paper.authors = found.authors
            if not paper.abstract and found.abstract:
                paper.abstract = found.abstract
            if not paper.doi and found.doi:
                paper.doi = found.doi
            if not paper.arxiv_id and found.arxiv_id:
                paper.arxiv_id = found.arxiv_id
            if not paper.url and found.url:
                paper.url = found.url
            if found.citation_count and found.citation_count > paper.citation_count:
                paper.citation_count = found.citation_count
            enriched += 1
            print(f"[graph-manager]   Enriched: {paper.title[:50]} → year={paper.year}, "
                  f"authors={len(paper.authors or [])}", file=sys.stderr)

    if enriched:
        save_graph(graph, graph_path)
        print(f"[graph-manager] Enriched {enriched}/{len(incomplete)} papers", file=sys.stderr)

    return enriched


class RateLimitError(Exception):
    """Raised when all sources are rate-limited."""
    def __init__(self, sources: list, message: str = None):
        self.sources = sources
        self.message = message or f"Rate limited by: {', '.join(sources)}"
        super().__init__(self.message)


def resolve_paper_by_title(title: str) -> Optional[Paper]:
    """Resolve a paper by title, ID (S2 hex, ARXIV:, DOI:), or search."""
    from services import arxiv
    from services import semantic_scholar as s2

    title = title.strip()
    if not title:
        return None

    rate_limited_sources = []

    # Check if it's already an ID (S2 hex, ARXIV:xxx, DOI:xxx)
    s2_re = re.compile(r'^[0-9a-fA-F]{40}$')
    if s2_re.match(title) or re.match(r'^(ARXIV|DOI):', title, re.IGNORECASE):
        try:
            result = s2.get_paper(title)
            paper = result["paper"]
            print(f"[graph-manager] Resolved by ID: {paper.title} ({paper.id})", file=sys.stderr)
            return paper
        except Exception as e:
            err_str = str(e).lower()
            if "429" in err_str or "rate" in err_str:
                rate_limited_sources.append("Semantic Scholar")
            print(f"[graph-manager] ID lookup failed for {title}: {e}", file=sys.stderr)
            if rate_limited_sources:
                raise RateLimitError(rate_limited_sources)
            return None

    # Try arXiv search first — use short query (arXiv API returns junk for long queries)
    try:
        if ':' in title:
            short_query = title.split(':')[0].strip()
        else:
            words = title.split()
            short_query = ' '.join(words[:8])
        short_query = re.sub(r'[:\-,;/()]+', ' ', short_query).strip()
        short_query = re.sub(r'\s+', ' ', short_query)
        print(f"[graph-manager] arXiv search: '{short_query}'", file=sys.stderr)
        total, papers = arxiv.search(short_query, limit=10)
        for p in papers:
            if _title_similarity(title, p.title) >= 0.5:
                print(f"[graph-manager] Resolved via arXiv: {p.title} ({p.id})", file=sys.stderr)
                return p
    except Exception as e:
        err_str = str(e).lower()
        if "429" in err_str or "rate" in err_str:
            rate_limited_sources.append("arXiv")
        print(f"[graph-manager] arXiv search failed: {e}", file=sys.stderr)

    # Try S2 search
    try:
        total, papers = s2._s2_search(title, limit=5)
        for p in papers:
            if _title_similarity(title, p.title) >= 0.5:
                print(f"[graph-manager] Resolved via S2: {p.title} ({p.id})", file=sys.stderr)
                return p
    except Exception as e:
        err_str = str(e).lower()
        if "429" in err_str or "rate" in err_str:
            rate_limited_sources.append("Semantic Scholar")
        print(f"[graph-manager] S2 search failed: {e}", file=sys.stderr)

    # If all failures were rate limits, raise specific error
    if rate_limited_sources:
        raise RateLimitError(rate_limited_sources)

    return None


def resolve_paper_by_bibtex(bibtex: str) -> Optional[Paper]:
    """Parse BibTeX to extract title/DOI/arXiv, then resolve."""
    title = None
    doi = None
    arxiv_id = None
    year = None
    authors = []

    # Extract fields from BibTeX
    title_match = re.search(r'title\s*=\s*\{([^}]+)\}', bibtex, re.IGNORECASE)
    if title_match:
        title = title_match.group(1).strip()

    doi_match = re.search(r'doi\s*=\s*\{([^}]+)\}', bibtex, re.IGNORECASE)
    if doi_match:
        doi = doi_match.group(1).strip()

    arxiv_match = re.search(r'(?:eprint|arxiv)\s*=\s*\{([^}]+)\}', bibtex, re.IGNORECASE)
    if arxiv_match:
        arxiv_id = arxiv_match.group(1).strip()

    year_match = re.search(r'year\s*=\s*\{?(\d{4})\}?', bibtex, re.IGNORECASE)
    if year_match:
        year = int(year_match.group(1))

    author_match = re.search(r'author\s*=\s*\{([^}]+)\}', bibtex, re.IGNORECASE)
    if author_match:
        raw = author_match.group(1).strip()
        authors = [a.strip() for a in raw.split(' and ')]

    # Try DOI first via S2
    if doi:
        from services import semantic_scholar as s2
        try:
            result = s2.get_paper(f"DOI:{doi}")
            paper = result["paper"]
            print(f"[graph-manager] Resolved via DOI: {paper.title}", file=sys.stderr)
            return paper
        except Exception:
            pass

    # Try arXiv ID via S2
    if arxiv_id:
        from services import semantic_scholar as s2
        try:
            result = s2.get_paper(f"ARXIV:{arxiv_id}")
            paper = result["paper"]
            print(f"[graph-manager] Resolved via arXiv ID: {paper.title}", file=sys.stderr)
            return paper
        except Exception:
            pass

    # Fall back to title search
    if title:
        try:
            paper = resolve_paper_by_title(title)
        except RateLimitError:
            paper = None  # BibTeX has enough info to create a paper below
        if paper:
            # Enrich with BibTeX metadata
            if year and not paper.year:
                paper.year = year
            if authors and not paper.authors:
                paper.authors = authors
            if doi and not paper.doi:
                paper.doi = doi
            return paper

    # If nothing resolved, create a basic Paper from BibTeX fields
    if title:
        return Paper(
            id=f"bibtex:{re.sub(r'[^a-z0-9]', '', title.lower())[:40]}",
            title=title,
            authors=authors,
            year=year,
            doi=doi,
            arxiv_id=arxiv_id,
            resolved=False,
            source="bibtex",
        )

    return None


def resolve_paper_from_pdf(pdf_path: str) -> Tuple[Optional[Paper], list]:
    """Parse PDF, identify the paper, and extract references.

    Returns (paper, resolved_references).
    """
    from services.pdf_parser import parse_pdf
    from services.graph_builder import _identify_pdf_paper

    result = parse_pdf(pdf_path)
    paper = _identify_pdf_paper(pdf_path, result.get("text_length", 0))

    resolved_refs = [Paper(**p) for p in result.get("resolved_papers", [])]

    if not paper:
        import os
        fname = os.path.basename(pdf_path).replace(".pdf", "")
        paper = Paper(
            id=f"pdf:{fname}",
            title=f"[PDF] {fname}",
            resolved=False,
            source="pdf_upload",
        )

    return paper, resolved_refs


def _merge_metadata(existing: Paper, new: Paper):
    """Merge metadata from *new* into *existing* without overwriting present fields."""
    if not existing.doi and new.doi:
        existing.doi = new.doi
    if not existing.arxiv_id and new.arxiv_id:
        existing.arxiv_id = new.arxiv_id
    if not existing.abstract and new.abstract:
        existing.abstract = new.abstract
    if new.citation_count > existing.citation_count:
        existing.citation_count = new.citation_count
    if not existing.pdf_url and new.pdf_url:
        existing.pdf_url = new.pdf_url
    if not existing.url and new.url:
        existing.url = new.url


def _get_seed_references(paper: Paper, **kwargs) -> list:
    """Fetch references for a paper.

    Priority:
    1. PDF download → parse references (regex/GROBID)
    2. LLM extraction from PDF text (if LLM API configured)
    3. Return empty list (no S2/arXiv API calls for refs)
    """
    pdf_path = None

    # Step 1: Download PDF
    try:
        from services.paper_downloader import download_pdf
        pdf_path = download_pdf(paper)
    except Exception as e:
        print(f"[graph-manager] PDF download failed: {e}", file=sys.stderr)

    # Step 2: Parse references from PDF
    if pdf_path:
        try:
            from services.pdf_parser import parse_pdf
            result = parse_pdf(pdf_path)
            resolved = result.get("resolved_papers", [])
            if resolved:
                papers = [Paper(**p) for p in resolved]
                print(f"[graph-manager] PDF refs: {len(papers)} for {paper.title[:50]}", file=sys.stderr)
                return papers
        except Exception as e:
            print(f"[graph-manager] PDF ref extraction failed: {e}", file=sys.stderr)

    # Step 3: LLM fallback — extract references from PDF text via LLM
    if pdf_path:
        refs = _llm_extract_references(pdf_path, paper)
        if refs:
            return refs

    return []


def _llm_extract_references(pdf_path: str, paper: Paper) -> list:
    """Use LLM to extract references from PDF text when regex parsing fails."""
    try:
        from services.llm_client import is_llm_available, llm_chat
        if not is_llm_available():
            return []
    except ImportError:
        return []

    # Extract text from PDF
    try:
        import fitz  # PyMuPDF
        doc = fitz.open(pdf_path)
        # Get last ~30% of text (where references typically are)
        total_pages = len(doc)
        start_page = max(0, int(total_pages * 0.7))
        text = ""
        for i in range(start_page, total_pages):
            text += doc[i].get_text()
        doc.close()
        if len(text) < 100:
            return []
    except Exception as e:
        print(f"[graph-manager] LLM ref: PDF text extraction failed: {e}", file=sys.stderr)
        return []

    # Truncate to fit in LLM context
    text = text[:12000]

    prompt = f"""Extract all references/bibliography entries from this academic paper text.
For each reference, provide: title, authors (comma-separated), year, and venue/journal.

Return ONLY a JSON array, no markdown, no explanation. Example:
[{{"title": "Attention is All You Need", "authors": "Vaswani, Shazeer, Parmar", "year": 2017, "venue": "NeurIPS"}}]

If you cannot find references, return: []

Text:
{text}"""

    try:
        response = llm_chat(prompt, max_tokens=4000, temperature=0.1)
        if not response:
            return []

        import json as _json
        # Extract JSON from response (handle markdown code blocks)
        response = response.strip()
        if response.startswith("```"):
            response = response.split("\n", 1)[1] if "\n" in response else response
            response = response.rsplit("```", 1)[0] if "```" in response else response
            response = response.strip()

        refs_data = _json.loads(response)
        if not isinstance(refs_data, list):
            return []

        papers = []
        for r in refs_data:
            title = r.get("title", "").strip()
            if not title:
                continue
            authors_raw = r.get("authors", "")
            authors = [a.strip() for a in authors_raw.split(",")] if isinstance(authors_raw, str) else authors_raw
            year = r.get("year")
            if isinstance(year, str) and year.isdigit():
                year = int(year)

            # Create Paper with a title-based ID (will be deduped by find_existing)
            pid = f"llm:{re.sub(r'[^a-z0-9]', '', title.lower())[:50]}"
            papers.append(Paper(
                id=pid,
                title=title,
                authors=authors,
                year=year,
                resolved=False,
                source="llm_extraction",
            ))

        if papers:
            print(f"[graph-manager] LLM extracted {len(papers)} references for {paper.title[:50]}", file=sys.stderr)
        return papers

    except Exception as e:
        print(f"[graph-manager] LLM ref extraction failed: {e}", file=sys.stderr)
        return []


def _get_seed_citations(paper: Paper) -> list:
    """Fetch citations for a paper.

    Priority: Google Scholar → S2 API.
    If both fail twice total, give up and return what we have.
    """
    citations = []
    failures = 0
    MAX_FAILURES = 2

    # 1. Google Scholar (title-based search)
    if paper.title and failures < MAX_FAILURES:
        try:
            from services.google_scholar import get_citations
            gs_cites = get_citations(paper.title, limit=50)
            if gs_cites:
                citations.extend(gs_cites)
                print(f"[graph-manager] Google Scholar: {len(gs_cites)} citations for {paper.title[:50]}", file=sys.stderr)
            else:
                failures += 1
                print(f"[graph-manager] Google Scholar returned 0 citations ({failures}/{MAX_FAILURES} failures)", file=sys.stderr)
        except Exception as e:
            failures += 1
            print(f"[graph-manager] Google Scholar failed ({failures}/{MAX_FAILURES}): {e}", file=sys.stderr)

    # 2. S2 API fallback (only if GS failed and we haven't hit failure limit)
    if not citations and failures < MAX_FAILURES and paper.id and not paper.id.startswith("pdf:"):
        try:
            from services import semantic_scholar as s2
            s2_cites = s2.get_citations(paper.id, limit=50)
            if s2_cites:
                citations.extend(s2_cites)
                print(f"[graph-manager] S2 citations: {len(s2_cites)} for {paper.title[:50]}", file=sys.stderr)
            else:
                failures += 1
                print(f"[graph-manager] S2 citations returned 0 ({failures}/{MAX_FAILURES} failures)", file=sys.stderr)
        except Exception as e:
            failures += 1
            print(f"[graph-manager] S2 citations failed ({failures}/{MAX_FAILURES}): {e}", file=sys.stderr)

    if failures >= MAX_FAILURES:
        print(f"[graph-manager] Citation search hit {MAX_FAILURES} failures, skipping further attempts", file=sys.stderr)

    return citations


def expand_as_seed(graph: GraphData, paper: Paper, graph_path: str,
                   pre_parsed_refs: list = None) -> dict:
    """Expand a paper as seed: fetch references + citations, add to graph, persist.

    Args:
        pre_parsed_refs: If provided (e.g. from PDF upload), skip re-fetching references.

    Returns a summary dict with counts of added nodes/edges.
    """
    # --- Mark as seed & reconcile with existing node ---
    paper.is_seed = True
    if paper.id not in graph.seed_papers:
        graph.seed_papers.append(paper.id)

    existing = find_existing(graph, paper)
    if existing:
        existing.is_seed = True
        if existing.id not in graph.seed_papers:
            graph.seed_papers.append(existing.id)
        paper = existing
    elif paper.id not in {n.id for n in graph.nodes}:
        graph.nodes.append(paper)

    # Progress callback (used by server SSE)
    try:
        from services.graph_server import set_progress
    except ImportError:
        def set_progress(*a, **kw): pass

    # Skip if already expanded
    if paper.refs_expanded and paper.cites_expanded:
        print(f"[graph-manager] {paper.id} already expanded, skipping", file=sys.stderr)
        return {
            "status": "already_expanded",
            "paper_id": paper.id,
            "paper_title": paper.title,
            "new_nodes": 0, "new_edges": 0,
            "total_nodes": len(graph.nodes),
            "total_edges": len(graph.edges),
        }

    nodes_before = len(graph.nodes)
    edges_before = len(graph.edges)
    ids_before = {n.id for n in graph.nodes}

    # --- Fetch references (paper cites these) ---
    if pre_parsed_refs is not None:
        refs = pre_parsed_refs
        set_progress("refs", f"Using {len(refs)} pre-parsed PDF references...", 20)
        print(f"[graph-manager] Using {len(refs)} pre-parsed refs for {paper.title[:50]}", file=sys.stderr)
    else:
        set_progress("refs", "Downloading PDF and extracting references...", 10)
        refs = _get_seed_references(paper)
    set_progress("refs", f"Adding {len(refs)} references to graph...", 40)
    for ref in refs:
        existing_ref = find_existing(graph, ref)
        if existing_ref:
            actual_id = existing_ref.id
            _merge_metadata(existing_ref, ref)
        else:
            actual_id = ref.id
            if ref.id not in {n.id for n in graph.nodes}:
                graph.nodes.append(ref)
        graph.add_edge(paper.id, actual_id, "cites")
    paper.refs_expanded = True
    set_progress("cites", "Searching Google Scholar for citations...", 60)

    # --- Fetch citations (these cite paper) ---
    cites = _get_seed_citations(paper)
    for cite in cites:
        if not cite.id:
            continue
        existing_cite = find_existing(graph, cite)
        if existing_cite:
            actual_id = existing_cite.id
            _merge_metadata(existing_cite, cite)
        else:
            actual_id = cite.id
            if cite.id not in {n.id for n in graph.nodes}:
                graph.nodes.append(cite)
        graph.add_edge(actual_id, paper.id, "cites")
    paper.cites_expanded = True

    set_progress("cites", f"Added {len(cites)} citations. Checking cross-seed edges...", 80)

    # --- Check new papers against existing seeds (fast, no API calls) ---
    # Use the reference data we already have to find cross-seed edges
    newly_added_ids = {n.id for n in graph.nodes} - ids_before
    if newly_added_ids:
        # refs already fetched contain seed papers' IDs — check if any new paper
        # matches a reference of an existing seed (by comparing node IDs in edges)
        seed_ids = set(graph.seed_papers)
        cross_edges = 0
        for new_id in newly_added_ids:
            # Check if this new paper is referenced by any seed (already handled above)
            # Check if this new paper references any seed (from its own refs if available)
            for seed_id in seed_ids:
                if seed_id == new_id:
                    continue
                # If seed's references include this new paper, edge already exists
                # If new paper's title matches a known reference of another seed, add edge
                new_node = graph.find_node(new_id)
                seed_node = graph.find_node(seed_id)
                if new_node and seed_node:
                    # Check by examining existing refs data from PDF parse
                    for ref in refs:
                        if find_existing(graph, ref) and find_existing(graph, ref).id == seed_id:
                            if graph.add_edge(paper.id, seed_id, "cites"):
                                cross_edges += 1
                    for cite in cites:
                        if cite.id and find_existing(graph, cite) and find_existing(graph, cite).id == new_id:
                            if graph.add_edge(new_id, paper.id, "cites"):
                                cross_edges += 1
        if cross_edges:
            print(f"[graph-manager] Cross-seed edges from existing data: +{cross_edges}", file=sys.stderr)

    # --- Dedup edges & persist ---
    set_progress("save", "Saving graph...", 90)
    graph.edges = _dedup_edges(graph.edges)
    save_graph(graph, graph_path)

    # --- Auto-enrich papers missing metadata (authors, year, abstract) ---
    set_progress("enrich", "Enriching incomplete paper metadata...", 93)
    try:
        enriched = enrich_incomplete_papers(graph, graph_path)
        if enriched:
            set_progress("enrich", f"Enriched {enriched} papers with missing metadata", 98)
    except Exception as e:
        print(f"[graph-manager] Auto-enrich failed (non-fatal): {e}", file=sys.stderr)

    new_nodes = len(graph.nodes) - nodes_before
    new_edges = len(graph.edges) - edges_before
    set_progress("done", f"Done! +{new_nodes} papers, +{new_edges} edges", 100, done=True)

    return {
        "status": "ok",
        "paper_id": paper.id,
        "paper_title": paper.title,
        "new_nodes": new_nodes,
        "new_edges": new_edges,
        "total_nodes": len(graph.nodes),
        "total_edges": len(graph.edges),
    }


def _is_same_paper(a: Paper, b: Paper) -> bool:
    """Check if two Paper objects represent the same paper."""
    if a.doi and b.doi and a.doi.lower().strip() == b.doi.lower().strip():
        return True
    if a.arxiv_id and b.arxiv_id and a.arxiv_id.strip() == b.arxiv_id.strip():
        return True
    if a.title and b.title and _title_similarity(a.title, b.title) >= 0.85:
        if a.year and b.year:
            return abs(a.year - b.year) <= 1
        return True
    return False


def _quick_get_paper_refs_titles(paper: Paper) -> list:
    """Quickly get raw reference titles from a paper's PDF (no resolve, no API).

    Returns list of RawReference objects with title/doi/arxiv_id fields.
    Falls back to empty list on any failure.
    """
    try:
        from services.paper_downloader import download_pdf
        pdf_path = download_pdf(paper)
        if not pdf_path:
            return []
    except Exception as e:
        print(f"[add-non-seed] PDF download failed for quick ref check: {e}", file=sys.stderr)
        return []

    try:
        import fitz
        doc = fitz.open(pdf_path)
        text = "".join(page.get_text() for page in doc)
        doc.close()

        from services.pdf_parser import _extract_via_regex
        raw_refs = _extract_via_regex(text)
        print(f"[add-non-seed] Quick PDF ref extraction: {len(raw_refs)} raw refs", file=sys.stderr)
        return raw_refs
    except Exception as e:
        print(f"[add-non-seed] Quick PDF ref extraction failed: {e}", file=sys.stderr)
        return []


def add_non_seed(graph: GraphData, paper: Paper, graph_path: str,
                  pre_parsed_refs: list = None) -> dict:
    """Add a paper as non-seed with bidirectional relationship check against all seeds.

    Direction A (new_paper → seed): Does new paper cite any seed?
      A1: Try quick PDF download + raw ref title matching against seeds
      A2: Fallback — check existing graph edges where X→seed, X matches new paper

    Direction B (seed → new_paper): Does any seed cite new paper?
      Check existing graph edges where seed→X, X matches new paper
    """
    existing = find_existing(graph, paper)
    if existing:
        return {
            "status": "exists",
            "paper_id": existing.id,
            "paper_title": existing.title,
            "message": "Paper already exists in graph",
        }

    paper.is_seed = False
    graph.nodes.append(paper)

    new_edges = 0
    seed_ids = set(graph.seed_papers)
    if not seed_ids:
        save_graph(graph, graph_path)
        return {
            "status": "ok", "paper_id": paper.id, "paper_title": paper.title,
            "new_nodes": 1, "new_edges": 0,
            "total_nodes": len(graph.nodes), "total_edges": len(graph.edges),
        }

    # Build seed Paper lookup for _is_same_paper checks
    seed_nodes = {s: graph.find_node(s) for s in seed_ids}

    # ──────────────────────────────────────────────
    # Direction A: Does new paper cite any seed?
    # ──────────────────────────────────────────────
    a_found_seeds = set()  # seed IDs confirmed cited by new paper

    # A1: Use pre_parsed_refs (PDF upload) or quick PDF download
    raw_refs = pre_parsed_refs
    if not raw_refs:
        # Try quick PDF download + raw extraction (no resolve needed)
        raw_refs_raw = _quick_get_paper_refs_titles(paper)
        if raw_refs_raw:
            # Convert RawReference to mini Paper for _is_same_paper matching
            raw_refs = []
            for r in raw_refs_raw:
                raw_refs.append(Paper(
                    id=r.doi or r.arxiv_id or r.title or "",
                    title=r.title or "",
                    authors=r.authors if isinstance(r.authors, list) else [],
                    year=r.year,
                    citation_count=0,
                    doi=r.doi,
                    arxiv_id=r.arxiv_id,
                    resolved=False,
                ))

    if raw_refs:
        for ref in raw_refs:
            ref_paper = ref if isinstance(ref, Paper) else None
            if ref_paper is None:
                continue
            for sid, snode in seed_nodes.items():
                if sid in a_found_seeds:
                    continue
                if snode and _is_same_paper(ref_paper, snode):
                    if graph.add_edge(paper.id, sid, "cites"):
                        new_edges += 1
                        print(f"[add-non-seed] New paper cites seed {sid} (A1: ref match)", file=sys.stderr)
                    a_found_seeds.add(sid)

    # A2: Fallback for seeds not found in A1 — check graph edges where X→seed
    remaining_seeds = seed_ids - a_found_seeds
    if remaining_seeds:
        for e in graph.edges:
            if e["target"] in remaining_seeds and e["source"] != paper.id:
                source_node = graph.find_node(e["source"])
                if source_node and _is_same_paper(source_node, paper):
                    sid = e["target"]
                    if graph.add_edge(paper.id, sid, "cites"):
                        new_edges += 1
                        print(f"[add-non-seed] New paper cites seed {sid} (A2: edge match)", file=sys.stderr)
                    remaining_seeds.discard(sid)
                    if not remaining_seeds:
                        break

    # ──────────────────────────────────────────────
    # Direction B: Does any seed cite new paper?
    # ──────────────────────────────────────────────
    # B1: Check existing graph edges (seed→X, X matches new paper)
    b_found_seeds = set()
    for e in graph.edges:
        if e["source"] in seed_ids and e["target"] != paper.id:
            target_node = graph.find_node(e["target"])
            if target_node and _is_same_paper(target_node, paper):
                sid = e["source"]
                if graph.add_edge(sid, paper.id, "cites"):
                    new_edges += 1
                    print(f"[add-non-seed] Seed {sid} cites new paper (B1: edge match)", file=sys.stderr)
                b_found_seeds.add(sid)

    # B2: For seeds not found in B1, download seed's PDF and check raw refs
    # This catches cases where seed's references were incompletely extracted during graph build
    b_remaining = seed_ids - b_found_seeds
    if b_remaining:
        for sid in b_remaining:
            snode = seed_nodes.get(sid)
            if not snode:
                continue
            try:
                seed_raw_refs = _quick_get_paper_refs_titles(snode)
                if seed_raw_refs:
                    for r in seed_raw_refs:
                        ref_paper = Paper(
                            id=r.doi or r.arxiv_id or r.title or "",
                            title=r.title or "",
                            authors=r.authors if isinstance(r.authors, list) else [],
                            year=r.year,
                            citation_count=0,
                            doi=r.doi,
                            arxiv_id=r.arxiv_id,
                            resolved=False,
                        )
                        if _is_same_paper(ref_paper, paper):
                            if graph.add_edge(sid, paper.id, "cites"):
                                new_edges += 1
                                print(f"[add-non-seed] Seed {sid} cites new paper (B2: seed PDF ref match)", file=sys.stderr)
                            break
            except Exception as e:
                print(f"[add-non-seed] B2 seed PDF check failed for {sid}: {e}", file=sys.stderr)

    graph.edges = _dedup_edges(graph.edges)
    save_graph(graph, graph_path)

    return {
        "status": "ok",
        "paper_id": paper.id,
        "paper_title": paper.title,
        "new_nodes": 1,
        "new_edges": new_edges,
        "total_nodes": len(graph.nodes),
        "total_edges": len(graph.edges),
    }


def convert_to_seed(graph: GraphData, paper_id: str, graph_path: str) -> dict:
    """Convert an existing non-seed node to seed and expand it."""
    paper = None
    for node in graph.nodes:
        if node.id == paper_id:
            paper = node
            break

    if not paper:
        return {"status": "error", "message": f"Paper {paper_id} not found in graph"}

    if paper.is_seed:
        return {"status": "exists", "message": f"Paper is already a seed", "paper_id": paper.id}

    return expand_as_seed(graph, paper, graph_path)


def enrich_metadata(graph: GraphData, graph_path: str) -> dict:
    """Fill missing year/authors/abstract for papers using S2 batch API.

    Uses S2 POST /paper/batch endpoint (up to 500 IDs per request) for
    papers with S2 hex IDs, and individual lookups for arXiv/DOI IDs.
    """
    import time
    import httpx

    S2_BASE = "https://api.semanticscholar.org/graph/v1"
    FIELDS = "paperId,title,authors,year,citationCount,abstract,url,externalIds"

    def _s2_headers():
        import os
        h = {}
        key = os.environ.get("S2_API_KEY", "")
        if key:
            h["x-api-key"] = key
        return h

    # Find papers needing enrichment
    to_enrich = []
    for node in graph.nodes:
        if not node.year or not node.authors or not node.abstract:
            to_enrich.append(node)

    if not to_enrich:
        print("[enrich] All papers have metadata, nothing to do", file=sys.stderr)
        return {"enriched": 0, "total": len(graph.nodes)}

    print(f"[enrich] {len(to_enrich)} papers need metadata enrichment", file=sys.stderr)

    # Separate by ID type
    s2_hex_nodes = []  # Can use batch API
    other_nodes = []   # Need individual lookup
    for node in to_enrich:
        pid = node.id
        if pid.startswith("ARXIV:") or pid.startswith("DOI:") or pid.startswith("crossref:") or pid.startswith("openalex:"):
            other_nodes.append(node)
        elif pid.startswith("pdf:") or pid.startswith("unresolved:"):
            continue  # Skip synthetic IDs
        else:
            s2_hex_nodes.append(node)

    enriched_count = 0

    # Batch API for S2 hex IDs (up to 500 per request)
    BATCH_SIZE = 100
    for i in range(0, len(s2_hex_nodes), BATCH_SIZE):
        batch = s2_hex_nodes[i:i + BATCH_SIZE]
        ids = [n.id for n in batch]
        print(f"[enrich] Batch {i // BATCH_SIZE + 1}: {len(ids)} papers via S2 batch API",
              file=sys.stderr)
        try:
            with httpx.Client() as client:
                resp = client.post(
                    f"{S2_BASE}/paper/batch",
                    json={"ids": ids},
                    params={"fields": FIELDS},
                    headers=_s2_headers(),
                    timeout=30,
                )
                if resp.status_code == 429:
                    print("[enrich] Rate limited, waiting 5s...", file=sys.stderr)
                    time.sleep(5)
                    resp = client.post(
                        f"{S2_BASE}/paper/batch",
                        json={"ids": ids},
                        params={"fields": FIELDS},
                        headers=_s2_headers(),
                        timeout=30,
                    )
                resp.raise_for_status()
                results = resp.json()

            for node, data in zip(batch, results):
                if not data:
                    continue
                if not node.year and data.get("year"):
                    node.year = data["year"]
                if not node.authors and data.get("authors"):
                    node.authors = [a.get("name", "") for a in data["authors"]]
                if not node.abstract and data.get("abstract"):
                    node.abstract = data["abstract"]
                if not node.doi:
                    ext = data.get("externalIds") or {}
                    if ext.get("DOI"):
                        node.doi = ext["DOI"]
                    if ext.get("ArXiv"):
                        node.arxiv_id = ext["ArXiv"]
                if not node.url and data.get("url"):
                    node.url = data["url"]
                if data.get("citationCount") and (data["citationCount"] > (node.citation_count or 0)):
                    node.citation_count = data["citationCount"]
                enriched_count += 1

        except Exception as e:
            print(f"[enrich] Batch failed: {e}", file=sys.stderr)

        # Rate limit courtesy
        if i + BATCH_SIZE < len(s2_hex_nodes):
            time.sleep(1)

    # Individual lookups for ARXIV:/DOI: IDs
    for node in other_nodes:
        pid = node.id
        lookup_id = pid
        if pid.startswith("crossref:"):
            lookup_id = f"DOI:{pid[len('crossref:'):]}"
        elif pid.startswith("openalex:"):
            continue  # S2 can't look up OpenAlex IDs

        print(f"[enrich] Individual lookup: {lookup_id[:50]}", file=sys.stderr)
        try:
            with httpx.Client() as client:
                resp = client.get(
                    f"{S2_BASE}/paper/{lookup_id}",
                    params={"fields": FIELDS},
                    headers=_s2_headers(),
                    timeout=15,
                )
                if resp.status_code == 429:
                    time.sleep(3)
                    resp = client.get(
                        f"{S2_BASE}/paper/{lookup_id}",
                        params={"fields": FIELDS},
                        headers=_s2_headers(),
                        timeout=15,
                    )
                if resp.status_code == 404:
                    continue
                resp.raise_for_status()
                data = resp.json()

            if not node.year and data.get("year"):
                node.year = data["year"]
            if not node.authors and data.get("authors"):
                node.authors = [a.get("name", "") for a in data["authors"]]
            if not node.abstract and data.get("abstract"):
                node.abstract = data["abstract"]
            if not node.url and data.get("url"):
                node.url = data["url"]
            if data.get("citationCount") and (data["citationCount"] > (node.citation_count or 0)):
                node.citation_count = data["citationCount"]
            enriched_count += 1
            time.sleep(0.5)

        except Exception as e:
            print(f"[enrich] Failed for {lookup_id}: {e}", file=sys.stderr)

    save_graph(graph, graph_path)
    print(f"[enrich] Done: enriched {enriched_count}/{len(to_enrich)} papers", file=sys.stderr)
    return {"enriched": enriched_count, "total": len(graph.nodes)}


path: install.sh
content: #!/bin/bash
# OpenPaperGraph — One-line installer for Claude Code / OpenClaw
# Usage: bash install.sh [--global | --project | --check]
#
# --global   Install /opg skill globally (available in all Claude Code sessions)
# --project  Install /opg skill in the current project only
# --check    Verify installation status without making changes
# (no flag)  Interactive — asks which mode to use

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CLI_PATH="$SCRIPT_DIR/openpapergraph_cli.py"
SKILL_PATH="$SCRIPT_DIR/SKILL.md"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

ok()   { echo -e "  ${GREEN}✓${NC} $1"; }
warn() { echo -e "  ${YELLOW}!${NC} $1"; }
fail() { echo -e "  ${RED}✗${NC} $1"; }
info() { echo -e "  ${CYAN}→${NC} $1"; }

# ─── Check Prerequisites ──────────────────────────────────────────────

check_prerequisites() {
    echo ""
    echo "Checking prerequisites..."

    # Python 3.8+
    if command -v python3 &>/dev/null; then
        PY_VER=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
        PY_MAJOR=$(echo "$PY_VER" | cut -d. -f1)
        PY_MINOR=$(echo "$PY_VER" | cut -d. -f2)
        if [ "$PY_MAJOR" -ge 3 ] && [ "$PY_MINOR" -ge 8 ]; then
            ok "Python $PY_VER"
        else
            fail "Python $PY_VER (need 3.8+)"
            return 1
        fi
    else
        fail "Python 3 not found"
        return 1
    fi

    # Core dependencies
    local all_ok=true
    for pkg in httpx pymupdf scholarly; do
        if python3 -c "import $( [ "$pkg" = "pymupdf" ] && echo "fitz" || echo "$pkg" )" 2>/dev/null; then
            ok "$pkg installed"
        else
            fail "$pkg not installed"
            all_ok=false
        fi
    done

    # Optional: openai
    if python3 -c "import openai" 2>/dev/null; then
        ok "openai installed (LLM summaries enabled)"
    else
        warn "openai not installed (LLM summaries disabled, extractive mode only)"
    fi

    # CLI file exists
    if [ -f "$CLI_PATH" ]; then
        ok "CLI found: $CLI_PATH"
    else
        fail "CLI not found: $CLI_PATH"
        return 1
    fi

    # SKILL.md exists
    if [ -f "$SKILL_PATH" ]; then
        ok "SKILL.md found: $SKILL_PATH"
    else
        fail "SKILL.md not found: $SKILL_PATH"
        return 1
    fi

    # S2 API key
    if [ -n "${S2_API_KEY:-}" ]; then
        ok "S2_API_KEY set"
    else
        warn "S2_API_KEY not set (Semantic Scholar will be rate-limited)"
    fi

    # LLM API key (check common ones)
    local llm_found=false
    for key in OPENAI_API_KEY ANTHROPIC_API_KEY DEEPSEEK_API_KEY GEMINI_API_KEY DASHSCOPE_API_KEY ZHIPUAI_API_KEY MOONSHOT_API_KEY; do
        if [ -n "${!key:-}" ]; then
            ok "LLM key found: $key"
            llm_found=true
            break
        fi
    done
    if ! $llm_found; then
        warn "No LLM API key set (summary will use extractive mode)"
    fi

    if $all_ok; then
        return 0
    else
        return 1
    fi
}

# ─── Install Dependencies ─────────────────────────────────────────────

install_deps() {
    echo ""
    echo "Installing dependencies..."
    pip install httpx pymupdf scholarly && ok "Core dependencies installed" || { fail "pip install failed"; return 1; }
    echo ""
    info "Optional: run 'pip install openai' to enable LLM-powered summaries"
}

# ─── Install Skill (Global) ───────────────────────────────────────────

install_global() {
    echo ""
    echo "Installing /opg skill globally..."

    local target_dir="$HOME/.claude/commands/opg"
    mkdir -p "$target_dir"

    # Remove old installation (copy or broken symlink)
    if [ -e "$target_dir/SKILL.md" ] || [ -L "$target_dir/SKILL.md" ]; then
        rm -f "$target_dir/SKILL.md"
        info "Removed old installation"
    fi
    # Also remove legacy flat file if it exists
    if [ -e "$HOME/.claude/commands/opg.md" ] || [ -L "$HOME/.claude/commands/opg.md" ]; then
        rm -f "$HOME/.claude/commands/opg.md"
        info "Removed legacy opg.md"
    fi

    # Create symlink
    ln -sf "$SKILL_PATH" "$target_dir/SKILL.md"

    # Verify
    if [ -L "$target_dir/SKILL.md" ] && [ -f "$target_dir/SKILL.md" ]; then
        local resolved
        resolved=$(python3 -c "import os; print(os.path.dirname(os.path.realpath('$target_dir/SKILL.md')))")
        ok "Symlink created: $target_dir/SKILL.md"
        ok "SKILL_DIR resolves to: $resolved"
        if [ "$resolved" = "$SCRIPT_DIR" ]; then
            ok "Path resolution correct!"
        else
            warn "SKILL_DIR resolves to $resolved (expected $SCRIPT_DIR)"
        fi
    else
        fail "Symlink creation failed"
        return 1
    fi

    echo ""
    ok "Global install complete. Start a NEW Claude Code session and type /opg"
}

# ─── Install Skill (Project) ──────────────────────────────────────────

install_project() {
    echo ""
    echo "Installing /opg skill in current project..."

    local target_dir="$SCRIPT_DIR/.claude/commands/opg"
    mkdir -p "$target_dir"

    # Create symlink (relative for portability within repo)
    ln -sf "../../../SKILL.md" "$target_dir/SKILL.md"

    if [ -L "$target_dir/SKILL.md" ] && [ -f "$target_dir/SKILL.md" ]; then
        ok "Symlink created: $target_dir/SKILL.md -> ../../../SKILL.md"
        ok "Project-level /opg skill installed"
    else
        fail "Symlink creation failed"
        return 1
    fi

    echo ""
    ok "Project install complete. Start a NEW Claude Code session from this directory"
}

# ─── Check Installation Status ─────────────────────────────────────────

check_status() {
    echo ""
    echo "=== OpenPaperGraph Installation Status ==="

    check_prerequisites

    echo ""
    echo "Skill registration:"

    # Global
    if [ -L "$HOME/.claude/commands/opg/SKILL.md" ]; then
        local target
        target=$(readlink "$HOME/.claude/commands/opg/SKILL.md" 2>/dev/null || true)
        ok "Global: ~/.claude/commands/opg/SKILL.md -> $target"
    elif [ -f "$HOME/.claude/commands/opg/SKILL.md" ]; then
        warn "Global: ~/.claude/commands/opg/SKILL.md exists but is a COPY (should be symlink)"
    elif [ -f "$HOME/.claude/commands/opg.md" ]; then
        warn "Global: ~/.claude/commands/opg.md exists (legacy flat file, should be opg/SKILL.md symlink)"
    else
        info "Global: not installed"
    fi

    # Project
    if [ -L "$SCRIPT_DIR/.claude/commands/opg/SKILL.md" ]; then
        ok "Project: .claude/commands/opg/SKILL.md (symlinked)"
    elif [ -f "$SCRIPT_DIR/.claude/commands/opg/SKILL.md" ]; then
        warn "Project: .claude/commands/opg/SKILL.md exists but is a COPY"
    else
        info "Project: not installed"
    fi

    echo ""
    echo "Quick test:"
    if python3 "$CLI_PATH" conferences --quiet 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'  CLI works: {len(d)} conferences found')" 2>/dev/null; then
        ok "CLI is functional"
    else
        fail "CLI test failed"
    fi

    echo ""
}

# ─── Main ──────────────────────────────────────────────────────────────

echo ""
echo "╔══════════════════════════════════════════╗"
echo "║   OpenPaperGraph Installer               ║"
echo "║   Academic Literature Discovery CLI       ║"
echo "╚══════════════════════════════════════════╝"

case "${1:-}" in
    --global)
        install_deps
        install_global
        ;;
    --project)
        install_deps
        install_project
        ;;
    --check)
        check_status
        ;;
    *)
        echo ""
        echo "Usage:"
        echo "  bash install.sh --global    Install globally (recommended)"
        echo "  bash install.sh --project   Install for this project only"
        echo "  bash install.sh --check     Check installation status"
        echo ""
        echo "What would you like to do?"
        echo "  1) Global install (available in all Claude Code sessions)"
        echo "  2) Project install (only in this directory)"
        echo "  3) Check status"
        echo ""
        read -rp "Choose [1/2/3]: " choice
        case "$choice" in
            1) install_deps; install_global ;;
            2) install_deps; install_project ;;
            3) check_status ;;
            *) echo "Invalid choice"; exit 1 ;;
        esac
        ;;
esac

echo ""


path: openpapergraph_cli.py
content: #!/usr/bin/env python3
"""OpenPaperGraph CLI — Multi-source academic literature discovery and citation network analysis.

Sources: arXiv, DBLP, Semantic Scholar, Google Scholar, CrossRef, OpenAlex, Unpaywall.
References from PDF parsing, citations from Google Scholar, S2 as fallback.

Outputs structured JSON to stdout. Progress/errors go to stderr.
Exit codes: 0=success, 1=argument error, 2=API/runtime error.

Usage:
    python openpapergraph_cli.py search "transformer" --limit 10
    python openpapergraph_cli.py search "transformer" --source arxiv --venue NeurIPS --limit 10
    python openpapergraph_cli.py graph ARXIV:1706.03762 --depth 1 --output graph.json
    python openpapergraph_cli.py graph "attention is all you need" -o graph.json
    python openpapergraph_cli.py graph paper.pdf -o graph.json
    python openpapergraph_cli.py recommend ARXIV:1706.03762 --limit 10
    python openpapergraph_cli.py monitor "vision transformer" --year-from 2025
    python openpapergraph_cli.py analyze graph.json
    python openpapergraph_cli.py summary graph.json --style overview
    python openpapergraph_cli.py pdf paper.pdf
    python openpapergraph_cli.py graph-from-pdf paper.pdf -o graph.json
    python openpapergraph_cli.py zotero --user-id ID --api-key KEY
    python openpapergraph_cli.py export graph.json --format bibtex --output refs.bib
    python openpapergraph_cli.py export graph.json --format markdown -o papers.md
    python openpapergraph_cli.py export-html graph.json -o graph.html --title "My Research"
    python openpapergraph_cli.py conferences
"""
import argparse
import json
import sys
import os

# Add current directory to path for imports
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

from schemas import Paper, GraphData, SearchResult

# Regex for Semantic Scholar 40-char hex paper ID
import re
_S2_ID_RE = re.compile(r'^[0-9a-fA-F]{40}$')
_ARXIV_PREFIX_RE = re.compile(r'^ARXIV:', re.IGNORECASE)
_DOI_PREFIX_RE = re.compile(r'^DOI:', re.IGNORECASE)


def output(data, pretty=False, output_file=None):
    """Write JSON to stdout, or save to file if output_file is specified."""
    indent = 2 if pretty else None
    text = json.dumps(data, indent=indent, ensure_ascii=False)
    if output_file:
        with open(output_file, "w", encoding="utf-8") as f:
            f.write(text + "\n")
        print(f"  Output saved to: {output_file}", file=sys.stderr)
    else:
        print(text)


def _out(args, data):
    """Shorthand: output data with pretty and output_file from args."""
    output(data, pretty=args.pretty, output_file=getattr(args, 'output', None))


def resolve_paper_id(identifier: str) -> str:
    """Smart resolve: accept S2 ID, ARXIV:, DOI:, PDF path, or paper title.

    Returns a Semantic Scholar paper ID string.
    - 40-char hex → use as-is
    - ARXIV:xxxx or DOI:xxxx → use as-is (S2 accepts these prefixes)
    - *.pdf file path → extract title from PDF, search S2 for paper_id
    - anything else → treat as paper title, search arXiv then S2 for paper_id
    """
    identifier = identifier.strip()

    # Already a S2 hex ID
    if _S2_ID_RE.match(identifier):
        return identifier

    # Already prefixed (ARXIV: or DOI:)
    if _ARXIV_PREFIX_RE.match(identifier) or _DOI_PREFIX_RE.match(identifier):
        return identifier

    # PDF file → should be handled directly by cmd_graph/cmd_recommend, not here
    if identifier.lower().endswith('.pdf') and os.path.isfile(identifier):
        raise ValueError(f"PDF '{identifier}' should be handled directly (use graph-from-pdf or pass to cmd_graph/cmd_recommend)")

    # Treat as paper title → search
    print(f"  Resolving title: \"{identifier}\"", file=sys.stderr)
    return _search_for_paper_id(identifier)


def _title_similarity(a: str, b: str) -> float:
    """Simple Jaccard similarity on lowered word sets."""
    wa = set(re.sub(r'[^a-z0-9\s]', '', a.lower()).split())
    wb = set(re.sub(r'[^a-z0-9\s]', '', b.lower()).split())
    if not wa or not wb:
        return 0.0
    return len(wa & wb) / len(wa | wb)


def _strip_arxiv_version(arxiv_id: str) -> str:
    """Strip version suffix from arXiv ID: ARXIV:2105.02723v1 → ARXIV:2105.02723"""
    return re.sub(r'v\d+$', '', arxiv_id)


def _search_for_paper_id(title: str) -> str:
    """Search arXiv first, then S2. Return ARXIV:ID directly (S2 API accepts it)."""
    from services import arxiv, semantic_scholar as s2

    # Try arXiv first (default source) — return ARXIV:xxxx directly
    try:
        total, papers = arxiv.search(title, limit=5)
        for p in papers:
            sim = _title_similarity(title, p.title)
            if sim >= 0.5:
                clean_id = _strip_arxiv_version(p.id)
                print(f"  Found on arXiv: {p.title} → {clean_id} (sim={sim:.2f})", file=sys.stderr)
                return clean_id  # "ARXIV:xxxx" — S2 API natively accepts this format
        if papers:
            print(f"  arXiv results not similar enough (best: {papers[0].title})", file=sys.stderr)
    except Exception as e:
        print(f"  arXiv search failed: {e}", file=sys.stderr)

    # Fallback to S2 direct search (uses _s2_search for S2-only)
    try:
        total, papers = s2._s2_search(title, limit=3)
        for p in papers:
            sim = _title_similarity(title, p.title)
            if sim >= 0.5:
                print(f"  Found on S2: {p.title} → {p.id} (sim={sim:.2f})", file=sys.stderr)
                return p.id
        if papers:
            # If no good match, take the first S2 result (S2 relevance ranking)
            print(f"  S2 best match: {papers[0].title} → {papers[0].id}", file=sys.stderr)
            return papers[0].id
    except Exception as e:
        print(f"  S2 search failed: {e}", file=sys.stderr)

    raise ValueError(f"Could not find paper matching: \"{title}\"")


def cmd_search(args):
    from services import semantic_scholar as s2, arxiv, dblp

    if args.source == "all":
        # s2.search() is now multi-source (arXiv + DBLP + S2) with dedup
        total, papers = s2.search(args.query, limit=args.limit, offset=args.offset)
        _out(args, SearchResult(total=total, papers=papers).to_dict())

    elif args.source == "arxiv":
        total, papers = arxiv.search(args.query, limit=args.limit, offset=args.offset)
        _out(args, SearchResult(total=total, papers=papers).to_dict())

    elif args.source == "dblp":
        total, papers = dblp.search(args.query, limit=args.limit, offset=args.offset, venue=args.venue)
        _out(args, SearchResult(total=total, papers=papers).to_dict())

    else:  # s2 — use S2-only search for explicit S2 source
        query = f"{args.query} venue:{args.venue}" if args.venue else args.query
        total, papers = s2._s2_search(query, limit=args.limit, offset=args.offset)
        _out(args, SearchResult(total=total, papers=papers).to_dict())


def cmd_graph(args):
    from services.graph_builder import build_graph, build_graph_from_pdfs

    # Separate file inputs by type
    pdf_paths = []
    bib_paths = []  # .bib and .json (CSL-JSON)
    non_files = []

    for p in args.paper_ids:
        lower = p.lower()
        if lower.endswith('.pdf') and os.path.isfile(p):
            pdf_paths.append(p)
        elif (lower.endswith('.bib') or lower.endswith('.json')) and os.path.isfile(p):
            bib_paths.append(p)
        else:
            non_files.append(p)

    # Parse BibTeX / CSL-JSON files → extract seed paper IDs
    bib_seed_ids = []
    if bib_paths:
        from services.bibtex_parser import parse_file
        for bib_path in bib_paths:
            print(f"  Parsing bibliography file: {bib_path}", file=sys.stderr)
            bib_papers = parse_file(bib_path)
            for bp in bib_papers:
                # Try to resolve to S2 ID for graph building
                pid = None
                if bp.arxiv_id:
                    pid = f"ARXIV:{bp.arxiv_id}"
                elif bp.doi:
                    pid = f"DOI:{bp.doi}"
                else:
                    # Search by title
                    try:
                        pid = _search_for_paper_id(bp.title)
                    except Exception as e:
                        print(f"  Could not resolve: {bp.title[:50]}... ({e})", file=sys.stderr)
                if pid:
                    bib_seed_ids.append(pid)
            print(f"  Resolved {len(bib_seed_ids)} papers from {bib_path}", file=sys.stderr)

    all_non_pdf_ids = non_files + bib_seed_ids

    if pdf_paths:
        # PDF inputs → directly parse references (no roundabout title search)
        print(f"  Direct PDF parsing: {len(pdf_paths)} file(s)", file=sys.stderr)
        graph = build_graph_from_pdfs(pdf_paths, depth=args.depth)

        # If there are also non-PDF IDs, build their graph and merge
        if all_non_pdf_ids:
            resolved_ids = [resolve_paper_id(pid) if not (pid.startswith('ARXIV:') or pid.startswith('DOI:') or _S2_ID_RE.match(pid)) else pid for pid in all_non_pdf_ids]
            id_graph = build_graph(resolved_ids, depth=args.depth)
            # Merge nodes and edges
            existing_ids = {n.id for n in graph.nodes}
            for node in id_graph.nodes:
                if node.id not in existing_ids:
                    graph.nodes.append(node)
                    existing_ids.add(node.id)
            existing_edges = {(e["source"], e["target"]) for e in graph.edges}
            for edge in id_graph.edges:
                if (edge["source"], edge["target"]) not in existing_edges:
                    graph.edges.append(edge)
            graph.total_papers = len(graph.nodes)
    else:
        # All non-PDF inputs → resolve IDs (S2 hex, ARXIV:, DOI:, or title)
        resolved_ids = []
        for pid in all_non_pdf_ids:
            if pid.startswith('ARXIV:') or pid.startswith('DOI:') or _S2_ID_RE.match(pid):
                resolved_ids.append(pid)
            else:
                resolved_ids.append(resolve_paper_id(pid))
        if not resolved_ids:
            raise ValueError("No paper IDs could be resolved. Check your input files.")
        graph = build_graph(resolved_ids, depth=args.depth)

    # Auto-enrich papers missing metadata (authors, year, abstract)
    incomplete = [p for p in graph.nodes if not p.year or not p.authors]
    if incomplete and args.output:
        print(f"  Enriching {len(incomplete)} papers with missing metadata...", file=sys.stderr)
        try:
            from services.graph_manager import enrich_incomplete_papers
            enriched = enrich_incomplete_papers(graph, args.output)
            if enriched:
                print(f"  Enriched {enriched}/{len(incomplete)} papers", file=sys.stderr)
        except Exception as e:
            print(f"  Auto-enrich failed (non-fatal): {e}", file=sys.stderr)

    _out(args, graph.to_dict())


def cmd_recommend(args):
    from services.semantic_scholar import recommend

    # Separate PDF files from paper IDs/titles
    pdf_paths = [p for p in args.paper_ids if p.lower().endswith('.pdf') and os.path.isfile(p)]
    non_pdfs = [p for p in args.paper_ids if p not in pdf_paths]

    resolved_ids = []

    # PDF inputs → parse references, use resolved paper IDs as seeds
    if pdf_paths:
        from services.pdf_parser import parse_pdf
        for pdf_path in pdf_paths:
            print(f"  Parsing PDF for recommendation seeds: {pdf_path}", file=sys.stderr)
            result = parse_pdf(pdf_path)
            for paper in result.get("resolved_papers", []):
                pid = paper.get("paper_id") or paper.get("id")
                if pid:
                    resolved_ids.append(pid)
        print(f"  Extracted {len(resolved_ids)} seed paper(s) from PDF(s)", file=sys.stderr)

    # Non-PDF inputs → resolve as usual (S2 hex, ARXIV:, DOI:, or title)
    for pid in non_pdfs:
        resolved_ids.append(resolve_paper_id(pid))

    if not resolved_ids:
        raise ValueError("No paper IDs resolved. Check your input.")

    papers = recommend(resolved_ids, limit=args.limit)
    _out(args, {"papers": [p.to_dict() for p in papers]})


def cmd_monitor(args):
    from services.semantic_scholar import search_recent
    papers = search_recent(args.query, year_from=args.year_from, limit=args.limit)
    _out(args, {"papers": [p.to_dict() for p in papers]})


def cmd_analyze(args):
    from services.analysis import analyze
    graph = GraphData.load(args.graph_file)
    result = analyze(graph.nodes)
    _out(args, result)


def cmd_summary(args):
    import os as _os
    # Apply --provider and --model overrides via env vars
    if getattr(args, 'provider', None):
        _os.environ["LLM_PROVIDER"] = args.provider
    if getattr(args, 'model', None):
        _os.environ["LLM_MODEL"] = args.model
    from services.analysis import summarize
    graph = GraphData.load(args.graph_file)
    result = summarize(graph.nodes, style=args.style)
    _out(args, result)


def cmd_llm_providers(args):
    from services.llm_client import list_providers
    result = list_providers()
    _out(args, {"providers": result, "total": len(result)})


def cmd_pdf(args):
    from services.pdf_parser import parse_pdf
    result = parse_pdf(args.file, use_grobid=getattr(args, 'use_grobid', False))
    _out(args, result)


def cmd_graph_from_pdf(args):
    from services.graph_builder import build_graph_from_pdfs
    graph = build_graph_from_pdfs(
        pdf_paths=args.files,
        depth=args.depth,
        include_unresolved=args.include_unresolved,
        use_grobid=args.use_grobid,
    )

    # Auto-enrich papers missing metadata
    incomplete = [p for p in graph.nodes if not p.year or not p.authors]
    if incomplete and args.output:
        print(f"  Enriching {len(incomplete)} papers with missing metadata...", file=sys.stderr)
        try:
            from services.graph_manager import enrich_incomplete_papers
            enriched = enrich_incomplete_papers(graph, args.output)
            if enriched:
                print(f"  Enriched {enriched}/{len(incomplete)} papers", file=sys.stderr)
        except Exception as e:
            print(f"  Auto-enrich failed (non-fatal): {e}", file=sys.stderr)

    _out(args, graph.to_dict())


def cmd_zotero(args):
    from services.zotero import import_papers, get_collections
    if args.list_collections:
        cols = get_collections(args.user_id, args.api_key)
        _out(args, cols)
    else:
        papers = import_papers(args.user_id, args.api_key, collection_key=args.collection, limit=args.limit)
        _out(args, {"papers": [p.to_dict() for p in papers]})


def cmd_export(args):
    from services.export import to_bibtex, to_csv, to_markdown, to_json
    graph = GraphData.load(args.graph_file)
    fmt = args.format
    if fmt == "csv":
        content = to_csv(graph.nodes)
    elif fmt == "markdown":
        content = to_markdown(graph.nodes)
    elif fmt == "json":
        content = to_json(graph.nodes)
    else:
        content = to_bibtex(graph.nodes)

    out_file = getattr(args, 'output', None)
    if out_file:
        with open(out_file, "w", encoding="utf-8") as f:
            f.write(content)
        print(f"  Output saved to: {out_file}", file=sys.stderr)
    else:
        print(content)


def cmd_export_html(args):
    """Export graph as an interactive HTML file."""
    from services.html_export import export_html

    graph = GraphData.load(args.graph_file)

    # Pre-generate summary if --summary flag is set
    pre_summary = None
    if getattr(args, 'summary', False):
        from services.analysis import summarize
        from services.llm_client import is_llm_available, get_provider_info
        if getattr(args, 'provider', None):
            os.environ["LLM_PROVIDER"] = args.provider
        if getattr(args, 'model', None):
            os.environ["LLM_MODEL"] = args.model
        print("  Pre-generating summary...", file=sys.stderr)
        pre_summary = summarize(graph.nodes, style="overview")
        method = pre_summary.get("method", "unknown")
        print(f"  Summary generated ({method}) — will be embedded in HTML", file=sys.stderr)
    else:
        print("  No --summary flag — users can enter API key at runtime in the HTML", file=sys.stderr)

    out_file = getattr(args, 'output', None) or "graph.html"
    title = getattr(args, 'title', None) or "Paper Graph"
    inline_js = getattr(args, 'inline', False)
    export_html(graph, output_path=out_file, title=title, pre_summary=pre_summary, inline_js=inline_js)


def cmd_serve(args):
    """Start interactive graph management server."""
    from services.graph_server import start_server
    start_server(args.graph_file, port=args.port, title=getattr(args, 'title', 'Paper Graph'))


def cmd_remove_seed(args):
    """Remove a seed paper and its exclusive connections from a graph."""
    graph = GraphData.load(args.graph_file)
    paper_id = args.paper_id
    # Support fuzzy title matching
    if not graph.find_node(paper_id):
        for n in graph.nodes:
            if n.title and paper_id.lower() in n.title.lower():
                paper_id = n.id
                print(f"  Matched: {n.title}", file=sys.stderr)
                break
    if paper_id not in graph.seed_papers:
        _out(args, {"status": "error", "message": f"'{args.paper_id}' is not a seed paper"})
        return
    stats = graph.remove_seed(paper_id)
    out_path = args.output or args.graph_file
    graph.save(out_path)
    _out(args, {"status": "ok", "graph_file": out_path, **stats})


def cmd_remove_paper(args):
    """Remove a non-seed paper from a graph."""
    graph = GraphData.load(args.graph_file)
    paper_id = args.paper_id
    # Support fuzzy title matching
    if not graph.find_node(paper_id):
        for n in graph.nodes:
            if n.title and paper_id.lower() in n.title.lower():
                paper_id = n.id
                print(f"  Matched: {n.title}", file=sys.stderr)
                break
    stats = graph.remove_node(paper_id)
    if "error" in stats:
        _out(args, {"status": "error", "message": stats["error"]})
        return
    if stats["removed_nodes"] == 0:
        _out(args, {"status": "error", "message": f"Paper '{args.paper_id}' not found in graph"})
        return
    out_path = args.output or args.graph_file
    graph.save(out_path)
    _out(args, {"status": "ok", "graph_file": out_path, **stats})


def cmd_conferences(args):
    from services.dblp import VENUE_KEYS
    _out(args, list(VENUE_KEYS.keys()))


def main():
    # Common args inherited by all subcommands
    common = argparse.ArgumentParser(add_help=False)
    common.add_argument("--pretty", action="store_true", help="Pretty-print JSON output")
    common.add_argument("--quiet", action="store_true", help="Suppress stderr progress")
    common.add_argument("--output", "-o", default=None, help="Save output to file (default: print to stdout)")

    parser = argparse.ArgumentParser(
        prog="openpapergraph_cli",
        description="Academic literature discovery and citation network analysis CLI.",
    )
    sub = parser.add_subparsers(dest="command", required=True)

    # search
    p = sub.add_parser("search", help="Search papers", parents=[common])
    p.add_argument("query", help="Search query")
    p.add_argument("--source", default="all", choices=["s2", "arxiv", "dblp", "all"], help="Search source (default: all — multi-source arXiv+DBLP+S2)")
    p.add_argument("--venue", default=None, help="Conference filter: ICLR, NeurIPS, ICML, ACL, EMNLP, NAACL, WebConf, KDD")
    p.add_argument("--limit", type=int, default=20)
    p.add_argument("--offset", type=int, default=0)

    # graph
    p = sub.add_parser("graph", help="Build citation network", parents=[common])
    p.add_argument("paper_ids", nargs="+", help="Paper IDs, titles, or PDF paths (auto-resolved)")
    p.add_argument("--depth", type=int, default=1)

    # recommend
    p = sub.add_parser("recommend", help="Get paper recommendations", parents=[common])
    p.add_argument("paper_ids", nargs="+", help="Paper IDs, titles, or PDF paths (auto-resolved)")
    p.add_argument("--limit", type=int, default=20)

    # monitor
    p = sub.add_parser("monitor", help="Check for recent papers on a topic", parents=[common])
    p.add_argument("query", help="Topic to monitor")
    p.add_argument("--year-from", type=int, default=2024)
    p.add_argument("--limit", type=int, default=20)

    # analyze
    p = sub.add_parser("analyze", help="Topic analysis on a graph", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")

    # summary
    p = sub.add_parser("summary", help="Generate research summary", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("--style", default="overview", choices=["overview", "trends", "gaps"])
    p.add_argument("--provider", default=None, help="LLM provider (e.g. openai, deepseek, qwen, zhipu, moonshot)")
    p.add_argument("--model", default=None, help="LLM model name (overrides provider default)")

    # pdf
    p = sub.add_parser("pdf", help="Extract references from PDF", parents=[common])
    p.add_argument("file", help="Path to PDF file")
    p.add_argument("--use-grobid", action="store_true", help="Use GROBID for structured extraction (needs Docker)")

    # graph-from-pdf
    p = sub.add_parser("graph-from-pdf", help="Build citation graph from PDF reference lists", parents=[common])
    p.add_argument("files", nargs="+", help="PDF file paths")
    p.add_argument("--depth", type=int, default=0, help="Expansion depth (0=PDF refs only, 1=expand via S2)")
    p.add_argument("--include-unresolved", action="store_true", help="Include unresolved references as nodes")
    p.add_argument("--use-grobid", action="store_true", help="Use GROBID for structured extraction (needs Docker)")

    # zotero
    p = sub.add_parser("zotero", help="Import from Zotero", parents=[common])
    p.add_argument("--user-id", required=True)
    p.add_argument("--api-key", required=True)
    p.add_argument("--collection", default=None, help="Collection key")
    p.add_argument("--list-collections", action="store_true")
    p.add_argument("--limit", type=int, default=50)

    # export
    p = sub.add_parser("export", help="Export graph as BibTeX/CSV/Markdown/JSON", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("--format", default="bibtex", choices=["bibtex", "csv", "markdown", "json"])

    # export-html
    p = sub.add_parser("export-html", help="Export graph as interactive HTML visualization", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("--title", default="Paper Graph", help="Page title")
    p.add_argument("--summary", action="store_true", help="Pre-generate AI summary at export time (requires LLM API key in env)")
    p.add_argument("--inline", action="store_true", help="Inline vis-network JS for fully offline use (~500KB larger)")
    p.add_argument("--provider", default=None, help="LLM provider for --summary (e.g. openai, deepseek)")
    p.add_argument("--model", default=None, help="LLM model for --summary")

    # conferences
    sub.add_parser("conferences", help="List supported conferences", parents=[common])

    # llm-providers
    sub.add_parser("llm-providers", help="List supported LLM providers and their status", parents=[common])

    # remove-seed
    p = sub.add_parser("remove-seed", help="Remove seed paper + exclusive connections", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("paper_id", help="Paper ID or title substring to match")

    # remove-paper
    p = sub.add_parser("remove-paper", help="Remove a non-seed paper from graph", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("paper_id", help="Paper ID or title substring to match")

    # serve — interactive graph management server
    p = sub.add_parser("serve", help="Start interactive graph server (HTML viewer + management)", parents=[common])
    p.add_argument("graph_file", help="Path to graph JSON file")
    p.add_argument("--port", type=int, default=8787, help="Server port (default: 8787)")
    p.add_argument("--title", default="Paper Graph", help="Page title")

    args = parser.parse_args()

    if args.quiet:
        import logging
        logging.disable(logging.CRITICAL)

    commands = {
        "search": cmd_search, "graph": cmd_graph, "graph-from-pdf": cmd_graph_from_pdf,
        "recommend": cmd_recommend, "monitor": cmd_monitor, "analyze": cmd_analyze,
        "summary": cmd_summary, "pdf": cmd_pdf, "zotero": cmd_zotero,
        "export": cmd_export, "export-html": cmd_export_html,
        "conferences": cmd_conferences, "llm-providers": cmd_llm_providers,
        "remove-seed": cmd_remove_seed, "remove-paper": cmd_remove_paper,
        "serve": cmd_serve,
    }

    try:
        commands[args.command](args)
    except KeyboardInterrupt:
        sys.exit(130)
    except Exception as e:
        print(json.dumps({"error": str(e)}), file=sys.stderr)
        sys.exit(2)


if __name__ == "__main__":
    main()


path: requirements.txt
content: httpx
pymupdf
scholarly


path: schemas.py
content: """Lightweight data models using dataclasses — no Pydantic dependency."""
from dataclasses import dataclass, field, asdict
from typing import List, Optional
import json


@dataclass
class RawReference:
    """A reference extracted from a PDF before resolution."""
    raw_text: str
    title: Optional[str] = None
    authors: List[str] = field(default_factory=list)
    year: Optional[int] = None
    doi: Optional[str] = None
    arxiv_id: Optional[str] = None

    def to_dict(self):
        return asdict(self)


@dataclass
class Paper:
    id: str
    title: str
    authors: List[str] = field(default_factory=list)
    year: Optional[int] = None
    citation_count: int = 0
    abstract: Optional[str] = None
    url: Optional[str] = None
    is_seed: bool = False
    resolved: bool = True
    source: str = "semantic_scholar"
    doi: Optional[str] = None
    arxiv_id: Optional[str] = None
    pdf_url: Optional[str] = None
    refs_expanded: bool = False
    cites_expanded: bool = False

    def to_dict(self):
        return asdict(self)


@dataclass
class GraphData:
    nodes: List[Paper] = field(default_factory=list)
    edges: List[dict] = field(default_factory=list)
    seed_papers: List[str] = field(default_factory=list)
    depth: int = 1
    total_papers: int = 0

    def to_dict(self):
        return {
            "nodes": [n.to_dict() for n in self.nodes],
            "edges": self.edges,
            "seed_papers": self.seed_papers,
            "depth": self.depth,
            "total_papers": self.total_papers,
        }

    def save(self, path: str):
        with open(path, "w") as f:
            json.dump(self.to_dict(), f, indent=2, ensure_ascii=False)

    @classmethod
    def load(cls, path: str) -> "GraphData":
        with open(path) as f:
            data = json.load(f)
        # Filter out unknown keys so older/newer JSON files load gracefully
        paper_fields = {f.name for f in Paper.__dataclass_fields__.values()}
        nodes = []
        for n in data.get("nodes", []):
            filtered = {k: v for k, v in n.items() if k in paper_fields}
            nodes.append(Paper(**filtered))
        return cls(
            nodes=nodes,
            edges=data.get("edges", []),
            seed_papers=data.get("seed_papers", []),
            depth=data.get("depth", 1),
            total_papers=data.get("total_papers", len(nodes)),
        )

    # --- Helper methods for graph mutations ---

    def node_ids(self) -> set:
        return {n.id for n in self.nodes}

    def edge_keys(self) -> set:
        return {(e["source"], e["target"]) for e in self.edges}

    def find_node(self, node_id: str) -> Optional[Paper]:
        for n in self.nodes:
            if n.id == node_id:
                return n
        return None

    def add_node(self, paper: Paper) -> bool:
        """Add node if not duplicate. If exists, upgrade metadata/seed status.
        Returns True if a new node was added."""
        existing = self.find_node(paper.id)
        if existing is not None:
            # Upgrade seed status (one-way: non-seed -> seed)
            if paper.is_seed and not existing.is_seed:
                existing.is_seed = True
                if paper.id not in self.seed_papers:
                    self.seed_papers.append(paper.id)
            # Fill in missing metadata
            if not existing.abstract and paper.abstract:
                existing.abstract = paper.abstract
            if not existing.doi and paper.doi:
                existing.doi = paper.doi
            if not existing.arxiv_id and paper.arxiv_id:
                existing.arxiv_id = paper.arxiv_id
            if not existing.url and paper.url:
                existing.url = paper.url
            if not existing.pdf_url and paper.pdf_url:
                existing.pdf_url = paper.pdf_url
            if paper.citation_count > existing.citation_count:
                existing.citation_count = paper.citation_count
            if paper.refs_expanded:
                existing.refs_expanded = True
            if paper.cites_expanded:
                existing.cites_expanded = True
            return False
        self.nodes.append(paper)
        if paper.is_seed and paper.id not in self.seed_papers:
            self.seed_papers.append(paper.id)
        self.total_papers = len(self.nodes)
        return True

    def add_edge(self, source: str, target: str, edge_type: str = "cites") -> bool:
        """Add edge if not duplicate. Returns True if added."""
        key = (source, target)
        if key in self.edge_keys():
            return False
        self.edges.append({"source": source, "target": target, "type": edge_type})
        return True

    def remove_seed(self, seed_id: str) -> dict:
        """Remove a seed paper and its exclusive connections.
        Returns stats about what was removed."""
        if seed_id not in self.seed_papers:
            return {"removed_nodes": 0, "removed_edges": 0}

        # Find connected nodes
        connected = set()
        for e in self.edges:
            if e["source"] == seed_id:
                connected.add(e["target"])
            if e["target"] == seed_id:
                connected.add(e["source"])

        # Find exclusive nodes (not connected to any other seed)
        other_seeds = [s for s in self.seed_papers if s != seed_id]
        exclusive = set()
        for nid in connected:
            if nid in self.seed_papers:
                continue
            is_exclusive = True
            for e in self.edges:
                if (e["source"] == nid and e["target"] in other_seeds) or \
                   (e["target"] == nid and e["source"] in other_seeds):
                    is_exclusive = False
                    break
            if is_exclusive:
                exclusive.add(nid)

        to_remove = {seed_id} | exclusive
        old_edge_count = len(self.edges)
        self.nodes = [n for n in self.nodes if n.id not in to_remove]
        self.edges = [e for e in self.edges
                      if e["source"] not in to_remove and e["target"] not in to_remove]
        self.seed_papers = [s for s in self.seed_papers if s != seed_id]
        self.total_papers = len(self.nodes)

        return {
            "removed_nodes": len(to_remove),
            "removed_edges": old_edge_count - len(self.edges),
        }

    def remove_node(self, node_id: str) -> dict:
        """Remove a single non-seed node and its incident edges."""
        if node_id in self.seed_papers:
            return {"error": "Cannot remove seed paper with remove_node. Use remove_seed()."}
        node = self.find_node(node_id)
        if not node:
            return {"removed_nodes": 0, "removed_edges": 0}
        before_edges = len(self.edges)
        self.nodes = [n for n in self.nodes if n.id != node_id]
        self.edges = [e for e in self.edges
                      if e["source"] != node_id and e["target"] != node_id]
        self.total_papers = len(self.nodes)
        return {
            "removed_nodes": 1,
            "removed_edges": before_edges - len(self.edges),
        }


@dataclass
class SearchResult:
    total: int
    papers: List[Paper]

    def to_dict(self):
        return {"total": self.total, "papers": [p.to_dict() for p in self.papers]}


path: services/analysis.py
content: """Topic analysis and AI summary — pure functions, no framework dependency."""
import re
import os
import sys
from collections import Counter
from typing import List

from schemas import Paper

STOP_WORDS = {
    "the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "of", "with",
    "by", "from", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had",
    "do", "does", "did", "will", "would", "could", "should", "may", "might", "can",
    "this", "that", "these", "those", "it", "its", "we", "our", "they", "their", "them",
    "not", "no", "as", "if", "than", "more", "most", "also", "such", "which", "each",
    "both", "between", "through", "over", "under", "about", "into", "during", "before",
    "after", "above", "below", "all", "any", "some", "other", "new", "using", "based",
    "show", "shows", "shown", "proposed", "propose", "paper", "method", "methods",
    "approach", "results", "result", "model", "models", "data", "use", "used",
}


def extract_keywords(texts: List[str], top_n: int = 50) -> list:
    counts = Counter()
    for text in texts:
        if not text:
            continue
        words = re.findall(r'\b[a-z]{3,}\b', text.lower())
        counts.update(w for w in words if w not in STOP_WORDS)
    return [{"word": w, "count": c} for w, c in counts.most_common(top_n)]


def detect_topics(papers: List[Paper], num_topics: int = 5) -> list:
    abstracts = [p.abstract for p in papers if p.abstract]
    if not abstracts:
        return []
    all_kw = extract_keywords(abstracts, top_n=100)
    if not all_kw:
        return []
    seeds = [kw["word"] for kw in all_kw[:num_topics * 3:3]][:num_topics]
    topics = []
    for i, seed in enumerate(seeds):
        matching = [p for p in papers if p.abstract and seed in p.abstract.lower()]
        if not matching:
            continue
        group_kw = extract_keywords([p.abstract for p in matching if p.abstract], top_n=8)
        topics.append({"id": i, "label": seed.capitalize(), "words": group_kw, "paper_count": len(matching)})
    return topics


def analyze(papers: List[Paper]) -> dict:
    topics = detect_topics(papers)
    year_counts = Counter(p.year for p in papers if p.year)
    year_dist = [{"year": y, "count": c} for y, c in sorted(year_counts.items())]
    author_counts = Counter()
    for p in papers:
        for a in p.authors:
            if a:
                author_counts[a] += 1
    top_authors = [{"name": n, "paper_count": c} for n, c in author_counts.most_common(15)]
    keyword_cloud = extract_keywords([p.abstract for p in papers if p.abstract], top_n=40)
    return {"topics": topics, "year_distribution": year_dist, "top_authors": top_authors, "keyword_cloud": keyword_cloud}


def _extractive_summary(papers: List[Paper], style: str = "overview") -> dict:
    """Extractive (keyword/statistical) summary — no LLM required."""
    top = sorted(papers, key=lambda p: p.citation_count, reverse=True)
    years = [p.year for p in papers if p.year]
    year_counts = Counter(years)
    total = len(papers)
    avg_cite = sum(p.citation_count for p in papers) // max(total, 1)
    year_range = f"{min(years)}-{max(years)}" if years else "unknown"

    stop = {"the", "a", "an", "and", "or", "for", "of", "in", "on", "to", "with", "by", "from", "is", "are", "using", "based", "via"}
    word_counts = Counter()
    for p in papers:
        word_counts.update(w for w in p.title.lower().split() if len(w) > 3 and w not in stop)
    top_kw = [w for w, _ in word_counts.most_common(10)]

    parts = [f"This collection contains {total} papers spanning {year_range}, with an average of {avg_cite} citations per paper."]
    if top[:3]:
        parts.append("Most influential: " + "; ".join(f'"{p.title}" ({p.citation_count} cites)' for p in top[:3]) + ".")
    if top_kw:
        parts.append(f"Key themes: {', '.join(top_kw[:7])}.")

    findings = []
    for p in top[:5]:
        if p.abstract:
            sentences = p.abstract.split(". ")
            if sentences and len(sentences[-1]) > 20:
                findings.append(f"[{p.year}] {sentences[-1].strip()}")

    trends = []
    if top_kw[:3]:
        trends.append(f"Dominant topics: {', '.join(top_kw[:3])}")
    sorted_years = sorted(year_counts.items())
    if len(sorted_years) >= 3:
        early = sum(c for _, c in sorted_years[:len(sorted_years)//2])
        late = sum(c for _, c in sorted_years[len(sorted_years)//2:])
        if late > early * 1.5:
            trends.append("Publication rate accelerating")

    directions = []
    if len(top_kw) >= 2:
        directions.append(f"Explore intersections between {top_kw[0]} and {top_kw[-1]}")
    directions.append("Check recent preprints extending top-cited works")

    return {"summary": " ".join(parts), "key_findings": findings[:5], "research_trends": trends,
            "suggested_directions": directions, "method": "extractive"}


def _build_llm_prompt(papers: List[Paper], style: str) -> str:
    """Build a prompt for LLM-powered summarization."""
    top = sorted(papers, key=lambda p: p.citation_count, reverse=True)[:20]
    total = len(papers)
    years = [p.year for p in papers if p.year]
    year_range = f"{min(years)}-{max(years)}" if years else "unknown"

    paper_list = []
    for i, p in enumerate(top, 1):
        entry = f"{i}. \"{p.title}\" ({p.year}, {p.citation_count} citations)"
        if p.authors:
            entry += f" — {', '.join(p.authors[:3])}"
        if p.abstract:
            entry += f"\n   Abstract: {p.abstract[:300]}..."
        paper_list.append(entry)

    style_instructions = {
        "overview": "Provide a comprehensive overview of the research area: main themes, key contributions, and how the papers relate to each other.",
        "trends": "Focus on research trends: what topics are gaining momentum, how the field has evolved over time, and where it is heading.",
        "gaps": "Identify research gaps: what questions remain unanswered, what areas are underexplored, and what promising directions lack sufficient work.",
    }

    return f"""Analyze this collection of {total} academic papers spanning {year_range}.

{style_instructions.get(style, style_instructions['overview'])}

Top papers (by citations):
{chr(10).join(paper_list)}

Respond in JSON format with exactly these keys:
- "summary": A 3-5 sentence overview paragraph
- "key_findings": Array of 3-5 key findings (one sentence each)
- "research_trends": Array of 3-5 research trends
- "suggested_directions": Array of 3-5 suggested future directions
"""


def _llm_summary(papers: List[Paper], style: str = "overview") -> dict:
    """LLM-powered summary — uses any configured provider via llm_client."""
    import json as _json
    from services.llm_client import llm_chat, get_provider_info

    prompt = _build_llm_prompt(papers, style)
    raw = llm_chat(prompt, temperature=0.3, max_tokens=2000)
    if not raw:
        return None

    # Parse JSON from LLM response (handle markdown code blocks)
    text = raw.strip()
    if text.startswith("```"):
        text = re.sub(r'^```(?:json)?\s*', '', text)
        text = re.sub(r'\s*```$', '', text)

    try:
        result = _json.loads(text)
        provider_info = get_provider_info() or {}
        result["method"] = "llm"
        result["provider"] = provider_info.get("display_name", "unknown")
        result["model"] = provider_info.get("model", "unknown")
        return result
    except _json.JSONDecodeError:
        # LLM returned non-JSON — wrap raw text as summary
        provider_info = get_provider_info() or {}
        return {
            "summary": text,
            "key_findings": [],
            "research_trends": [],
            "suggested_directions": [],
            "method": "llm",
            "provider": provider_info.get("display_name", "unknown"),
            "model": provider_info.get("model", "unknown"),
        }


def summarize(papers: List[Paper], style: str = "overview") -> dict:
    """Generate a research summary — LLM-powered if a provider is configured, extractive otherwise."""
    if not papers:
        return {"summary": "No papers.", "key_findings": [], "research_trends": [], "suggested_directions": [], "method": "extractive"}

    # Try LLM first
    from services.llm_client import is_llm_available
    if is_llm_available():
        result = _llm_summary(papers, style)
        if result:
            return result
        print("[summary] LLM call failed, falling back to extractive analysis", file=sys.stderr)

    return _extractive_summary(papers, style)


path: services/arxiv.py
content: """arXiv API search (sync)."""
import re
from typing import List, Tuple
from xml.etree import ElementTree

import httpx

from schemas import Paper

ARXIV_API = "https://export.arxiv.org/api/query"
NS = {"atom": "http://www.w3.org/2005/Atom"}


def _parse_entry(entry) -> Paper:
    title = (entry.findtext("atom:title", "", NS) or "").strip().replace("\n", " ")
    abstract = (entry.findtext("atom:summary", "", NS) or "").strip().replace("\n", " ")
    authors = [a.findtext("atom:name", "", NS) for a in entry.findall("atom:author", NS)]
    published = entry.findtext("atom:published", "", NS) or ""
    year = int(published[:4]) if len(published) >= 4 else None
    id_url = entry.findtext("atom:id", "", NS) or ""
    arxiv_id = id_url.split("/abs/")[-1] if "/abs/" in id_url else id_url
    # Strip version suffix (e.g., v1, v2) — S2 API doesn't accept versioned IDs
    arxiv_id = re.sub(r'v\d+$', '', arxiv_id)

    return Paper(
        id=f"ARXIV:{arxiv_id}",
        title=title,
        authors=authors,
        year=year,
        citation_count=0,
        abstract=abstract[:1000] if abstract else None,
        url=id_url,
    )


def search(query: str, limit: int = 20, offset: int = 0) -> Tuple[int, List[Paper]]:
    with httpx.Client(timeout=30) as client:
        resp = client.get(ARXIV_API, params={
            "search_query": f"all:{query}",
            "start": offset,
            "max_results": limit,
            "sortBy": "relevance",
            "sortOrder": "descending",
        })
        resp.raise_for_status()

    root = ElementTree.fromstring(resp.text)
    total_str = root.findtext("{http://a9.com/-/spec/opensearch/1.1/}totalResults", "0")
    total = int(total_str) if total_str.isdigit() else 0
    papers = []
    for entry in root.findall("atom:entry", NS):
        try:
            papers.append(_parse_entry(entry))
        except Exception:
            pass
    return total, papers


def get_by_id(arxiv_id: str) -> Paper:
    """Fetch a single paper by arXiv ID (e.g. '2603.15727')."""
    # Strip ARXIV: prefix and version suffix
    raw_id = re.sub(r'^ARXIV:', '', arxiv_id)
    raw_id = re.sub(r'v\d+$', '', raw_id)

    with httpx.Client(timeout=30) as client:
        resp = client.get(ARXIV_API, params={"id_list": raw_id, "max_results": 1})
        resp.raise_for_status()

    root = ElementTree.fromstring(resp.text)
    entries = root.findall("atom:entry", NS)
    for entry in entries:
        # arXiv returns an entry even for invalid IDs, but with no title
        title = (entry.findtext("atom:title", "", NS) or "").strip()
        if title and title != "Error":
            return _parse_entry(entry)
    raise ValueError(f"arXiv paper not found: {arxiv_id}")


path: services/bibtex_parser.py
content: """Parse BibTeX (.bib) files and Zotero exports into Paper objects.

Supports:
- Standard .bib files
- Zotero BibTeX export
- Zotero CSL-JSON export (.json)

Field mapping:
  title    -> Paper.title
  author   -> Paper.authors (split on " and ")
  year     -> Paper.year
  doi      -> Paper.doi
  url      -> Paper.url
  abstract -> Paper.abstract
  eprint/arxivid -> Paper.arxiv_id
"""

import json
import re
import sys
from typing import List, Optional, Tuple

from schemas import Paper


# ---------------------------------------------------------------------------
# BibTeX parsing (no external dependency)
# ---------------------------------------------------------------------------

def _parse_bibtex_entries(text: str) -> List[dict]:
    """Parse BibTeX text into a list of entry dicts."""
    entries = []
    # Match @type{key, ... }
    # We need to handle nested braces
    pattern = re.compile(r'@(\w+)\s*\{([^,]*),', re.IGNORECASE)
    pos = 0
    while pos < len(text):
        m = pattern.search(text, pos)
        if not m:
            break
        entry_type = m.group(1).lower()
        entry_key = m.group(2).strip()
        # Find the matching closing brace
        start = m.end()
        depth = 1
        i = start
        while i < len(text) and depth > 0:
            if text[i] == '{':
                depth += 1
            elif text[i] == '}':
                depth -= 1
            i += 1
        body = text[start:i - 1] if depth == 0 else text[start:]
        # Parse fields
        fields = _parse_bibtex_fields(body)
        fields['_type'] = entry_type
        fields['_key'] = entry_key
        entries.append(fields)
        pos = i

    return entries


def _parse_bibtex_fields(body: str) -> dict:
    """Extract key=value pairs from a BibTeX entry body."""
    fields = {}
    # Match field = {value} or field = "value" or field = number
    i = 0
    while i < len(body):
        # Skip whitespace and commas
        while i < len(body) and body[i] in ' \t\n\r,':
            i += 1
        if i >= len(body):
            break
        # Find field name
        match = re.match(r'(\w+)\s*=\s*', body[i:])
        if not match:
            i += 1
            continue
        field_name = match.group(1).lower()
        i += match.end()
        if i >= len(body):
            break
        # Parse value
        value, consumed = _parse_bibtex_value(body[i:])
        fields[field_name] = value
        i += consumed
    return fields


def _parse_bibtex_value(text: str) -> Tuple[str, int]:
    """Parse a BibTeX value starting at the current position.
    Returns (value_string, chars_consumed)."""
    text = text.lstrip()
    if not text:
        return "", 0
    offset = len(text) - len(text.lstrip())

    if text[0] == '{':
        # Brace-delimited value
        depth = 1
        i = 1
        while i < len(text) and depth > 0:
            if text[i] == '{':
                depth += 1
            elif text[i] == '}':
                depth -= 1
            i += 1
        value = text[1:i - 1] if depth == 0 else text[1:]
        return _clean_bibtex_value(value), i
    elif text[0] == '"':
        # Quote-delimited value
        i = 1
        while i < len(text) and text[i] != '"':
            if text[i] == '\\':
                i += 1  # skip escaped char
            i += 1
        value = text[1:i]
        return _clean_bibtex_value(value), i + 1
    else:
        # Bare number or string
        m = re.match(r'(\S+)', text)
        if m:
            return m.group(1).strip(','), m.end()
        return "", 0


def _clean_bibtex_value(value: str) -> str:
    """Clean up a BibTeX value: remove extra braces, LaTeX commands."""
    # Remove nested braces (common in titles)
    value = re.sub(r'\{([^{}]*)\}', r'\1', value)
    # Remove common LaTeX commands
    value = re.sub(r'\\textit\{([^}]*)\}', r'\1', value)
    value = re.sub(r'\\textbf\{([^}]*)\}', r'\1', value)
    value = re.sub(r'\\emph\{([^}]*)\}', r'\1', value)
    value = re.sub(r'\\[a-zA-Z]+\s*', '', value)  # remove remaining commands
    # Clean up whitespace
    value = re.sub(r'\s+', ' ', value).strip()
    return value


def parse_bibtex(text: str) -> List[Paper]:
    """Parse BibTeX text and return a list of Paper objects."""
    entries = _parse_bibtex_entries(text)
    papers = []
    for entry in entries:
        if entry.get('_type') in ('string', 'comment', 'preamble'):
            continue

        title = entry.get('title', '').strip()
        if not title:
            continue

        # Parse authors
        author_str = entry.get('author', '')
        authors = [a.strip() for a in author_str.split(' and ') if a.strip()] if author_str else []

        # Parse year
        year = None
        year_str = entry.get('year', '')
        if year_str:
            m = re.search(r'(\d{4})', year_str)
            if m:
                year = int(m.group(1))

        # Extract identifiers
        doi = entry.get('doi', '').strip() or None
        url = entry.get('url', '').strip() or None
        abstract = entry.get('abstract', '').strip() or None

        # arXiv ID
        arxiv_id = None
        eprint = entry.get('eprint', '') or entry.get('arxivid', '')
        if eprint:
            arxiv_id = eprint.strip()
        elif url and 'arxiv.org' in url:
            m = re.search(r'(\d{4}\.\d{4,5})', url)
            if m:
                arxiv_id = m.group(1)

        # Generate ID
        if arxiv_id:
            paper_id = f"ARXIV:{arxiv_id}"
        elif doi:
            paper_id = f"DOI:{doi}"
        else:
            paper_id = f"bib:{entry.get('_key', title[:30])}"

        papers.append(Paper(
            id=paper_id,
            title=title,
            authors=authors,
            year=year,
            doi=doi,
            url=url,
            abstract=abstract,
            arxiv_id=arxiv_id,
            source="bibtex",
            is_seed=True,
        ))

    print(f"[bibtex] Parsed {len(papers)} papers from BibTeX", file=sys.stderr)
    return papers


def parse_bibtex_file(path: str) -> List[Paper]:
    """Parse a .bib file and return Paper objects."""
    with open(path, 'r', encoding='utf-8', errors='replace') as f:
        text = f.read()
    return parse_bibtex(text)


# ---------------------------------------------------------------------------
# Zotero CSL-JSON parsing
# ---------------------------------------------------------------------------

def parse_csl_json(text: str) -> List[Paper]:
    """Parse Zotero CSL-JSON export and return Paper objects."""
    try:
        data = json.loads(text)
    except json.JSONDecodeError as e:
        print(f"[csl-json] Failed to parse JSON: {e}", file=sys.stderr)
        return []

    if isinstance(data, dict):
        # Some exports wrap items in a top-level object
        data = data.get('items', [data])

    papers = []
    for item in data:
        if not isinstance(item, dict):
            continue

        title = item.get('title', '').strip()
        if not title:
            continue

        # Authors
        authors = []
        for a in item.get('author', []):
            if isinstance(a, dict):
                parts = [a.get('given', ''), a.get('family', '')]
                name = ' '.join(p for p in parts if p).strip()
                if name:
                    authors.append(name)

        # Year
        year = None
        issued = item.get('issued', {})
        if isinstance(issued, dict):
            date_parts = issued.get('date-parts', [[]])
            if date_parts and date_parts[0]:
                try:
                    year = int(date_parts[0][0])
                except (ValueError, IndexError):
                    pass

        # Identifiers
        doi = item.get('DOI', '').strip() or None
        url = item.get('URL', '').strip() or None
        abstract = item.get('abstract', '').strip() or None

        arxiv_id = None
        if url and 'arxiv.org' in (url or ''):
            m = re.search(r'(\d{4}\.\d{4,5})', url)
            if m:
                arxiv_id = m.group(1)

        if arxiv_id:
            paper_id = f"ARXIV:{arxiv_id}"
        elif doi:
            paper_id = f"DOI:{doi}"
        else:
            paper_id = f"csl:{item.get('id', title[:30])}"

        papers.append(Paper(
            id=paper_id,
            title=title,
            authors=authors,
            year=year,
            doi=doi,
            url=url,
            abstract=abstract,
            arxiv_id=arxiv_id,
            source="zotero_csl",
            is_seed=True,
        ))

    print(f"[csl-json] Parsed {len(papers)} papers from CSL-JSON", file=sys.stderr)
    return papers


def parse_csl_json_file(path: str) -> List[Paper]:
    """Parse a Zotero CSL-JSON file and return Paper objects."""
    with open(path, 'r', encoding='utf-8') as f:
        text = f.read()
    return parse_csl_json(text)


# ---------------------------------------------------------------------------
# Auto-detect and parse
# ---------------------------------------------------------------------------

def parse_file(path: str) -> List[Paper]:
    """Auto-detect file type and parse. Supports .bib and .json (CSL-JSON)."""
    lower = path.lower()
    if lower.endswith('.bib'):
        return parse_bibtex_file(path)
    elif lower.endswith('.json'):
        return parse_csl_json_file(path)
    else:
        # Try BibTeX first, then CSL-JSON
        with open(path, 'r', encoding='utf-8', errors='replace') as f:
            text = f.read()
        if text.lstrip().startswith('@') or '@article' in text.lower() or '@inproceedings' in text.lower():
            return parse_bibtex(text)
        else:
            return parse_csl_json(text)


path: services/dblp.py
content: """DBLP API search (sync) — best for CS conference proceedings."""
from typing import List, Tuple, Optional

import httpx

from schemas import Paper

DBLP_API = "https://dblp.org/search/publ/api"

VENUE_KEYS = {
    "ICLR": "iclr", "NeurIPS": "neurips", "ICML": "icml",
    "ACL": "acl", "EMNLP": "emnlp", "NAACL": "naacl",
    "WebConf": "www", "KDD": "kdd",
}


def _parse_hit(hit: dict) -> Optional[Paper]:
    info = hit.get("info", {})
    title = info.get("title", "").rstrip(".")
    if not title:
        return None
    authors_data = info.get("authors", {}).get("author", [])
    if isinstance(authors_data, dict):
        authors_data = [authors_data]
    authors = [a.get("text", a) if isinstance(a, dict) else str(a) for a in authors_data]
    year_str = info.get("year", "")
    year = int(year_str) if str(year_str).isdigit() else None
    doi = info.get("doi", "")
    url = info.get("ee", "") or info.get("url", "")
    paper_id = f"DOI:{doi}" if doi else f"DBLP:{info.get('key', '')}"
    return Paper(id=paper_id, title=title, authors=authors, year=year,
                 url=url or (f"https://doi.org/{doi}" if doi else None))


def search(query: str, limit: int = 20, offset: int = 0, venue: Optional[str] = None) -> Tuple[int, List[Paper]]:
    search_query = query
    if venue:
        keyword = VENUE_KEYS.get(venue, venue.lower())
        search_query = f"{query} {keyword}"
    with httpx.Client(timeout=30) as client:
        resp = client.get(DBLP_API, params={"q": search_query, "format": "json", "h": limit, "f": offset})
        resp.raise_for_status()
    data = resp.json()
    result = data.get("result", {})
    total = int(result.get("hits", {}).get("@total", 0))
    hits = result.get("hits", {}).get("hit", [])
    papers = [p for h in hits if (p := _parse_hit(h))]
    return total, papers


path: services/export.py
content: """Export papers to BibTeX, CSV, Markdown, or JSON."""
import csv
import io
import json
from typing import List

from schemas import Paper


def _sort_by_year(papers: List[Paper], ascending: bool = False) -> List[Paper]:
    """Sort papers by year. None-year papers go last."""
    return sorted(papers, key=lambda p: (0 if p.year else 1, (p.year or 0) * (1 if ascending else -1), (p.title or "").lower()))


def to_bibtex(papers: List[Paper]) -> str:
    entries = []
    for p in _sort_by_year(papers):
        author_str = " and ".join(p.authors) if p.authors else "Unknown"
        key = p.id.replace("/", "_").replace(":", "_")[:30]
        entries.append(
            f"@article{{{key},\n"
            f"  title = {{{p.title}}},\n"
            f"  author = {{{author_str}}},\n"
            f"  year = {{{p.year or ''}}},\n"
            f"  url = {{{p.url or ''}}},\n"
            f"}}\n"
        )
    return "\n".join(entries)


def to_csv(papers: List[Paper]) -> str:
    buf = io.StringIO()
    writer = csv.writer(buf)
    writer.writerow(["id", "title", "authors", "year", "citation_count", "source", "url", "doi", "arxiv_id", "abstract"])
    for p in _sort_by_year(papers):
        writer.writerow([
            p.id, p.title, "; ".join(p.authors), p.year or "", p.citation_count,
            p.source or "", p.url or "", p.doi or "", p.arxiv_id or "",
            (p.abstract or "")[:500],
        ])
    return buf.getvalue()


def to_markdown(papers: List[Paper]) -> str:
    """Export papers as a Markdown list sorted by year descending."""
    lines = [f"# Paper List ({len(papers)} papers)\n"]
    for i, p in enumerate(_sort_by_year(papers), 1):
        lines.append(f"## {i}. {p.title or 'Untitled'}\n")
        lines.append(f"- **Year:** {p.year or 'N/A'}")
        lines.append(f"- **Authors:** {', '.join(p.authors) if p.authors else 'Unknown'}")
        lines.append(f"- **Citations:** {p.citation_count}")
        if p.source:
            lines.append(f"- **Source:** {p.source}")
        if p.url:
            lines.append(f"- **URL:** {p.url}")
        if p.doi:
            lines.append(f"- **DOI:** {p.doi}")
        if p.arxiv_id:
            lines.append(f"- **arXiv:** {p.arxiv_id}")
        if p.abstract:
            lines.append(f"- **Abstract:** {p.abstract[:500]}{'…' if len(p.abstract) > 500 else ''}")
        lines.append(f"- **ID:** {p.id}")
        lines.append("")
    return "\n".join(lines)


def to_json(papers: List[Paper]) -> str:
    """Export papers as formatted JSON sorted by year descending."""
    sorted_papers = _sort_by_year(papers)
    return json.dumps([p.to_dict() for p in sorted_papers], indent=2, ensure_ascii=False)


path: services/google_scholar.py
content: """Google Scholar client — citations, citation counts, and search.

Uses the `scholarly` library for structured access to Google Scholar.
Falls back gracefully if scholarly is not installed or Google blocks requests.

Install: pip install scholarly
"""
import sys
import time
import logging
from typing import List, Optional

from schemas import Paper

logger = logging.getLogger(__name__)

# Rate limiting
_LAST_REQUEST_TIME = 0
_MIN_INTERVAL = 2.0  # seconds between requests


def _rate_limit():
    """Enforce minimum interval between Google Scholar requests."""
    global _LAST_REQUEST_TIME
    now = time.time()
    elapsed = now - _LAST_REQUEST_TIME
    if elapsed < _MIN_INTERVAL:
        time.sleep(_MIN_INTERVAL - elapsed)
    _LAST_REQUEST_TIME = time.time()


def _scholarly():
    """Lazy import scholarly with availability check.

    Returns the scholarly.scholarly object which has search_pubs, citedby, etc.
    """
    try:
        from scholarly import scholarly as _mod
        return _mod
    except ImportError:
        return None


def _parse_scholar_paper(pub: dict) -> Paper:
    """Convert a scholarly publication dict to a Paper object."""
    bib = pub.get("bib", {})
    title = bib.get("title", "")
    authors = bib.get("author", [])
    if isinstance(authors, str):
        authors = [a.strip() for a in authors.split(" and ")]
    year = None
    year_str = bib.get("pub_year", "")
    if str(year_str).isdigit():
        year = int(year_str)

    citation_count = pub.get("num_citations", 0)
    url = pub.get("pub_url") or pub.get("eprint_url") or ""

    # Generate a stable ID
    scholar_id = pub.get("author_pub_id") or pub.get("cites_id", [""])[0] or ""
    paper_id = f"GS:{scholar_id}" if scholar_id else f"GS:{title[:50]}"

    return Paper(
        id=paper_id,
        title=title,
        authors=authors if isinstance(authors, list) else [authors],
        year=year,
        citation_count=citation_count,
        url=url,
        source="google_scholar",
    )


def search(query: str, limit: int = 10) -> List[Paper]:
    """Search Google Scholar for papers matching query."""
    scholarly = _scholarly()
    if not scholarly:
        logger.debug("scholarly not installed, skipping Google Scholar search")
        return []

    papers = []
    try:
        _rate_limit()
        print(f"[GS] Searching: {query[:50]}...", file=sys.stderr)
        search_query = scholarly.search_pubs(query)
        for i, pub in enumerate(search_query):
            if i >= limit:
                break
            papers.append(_parse_scholar_paper(pub))
            if i < limit - 1:
                _rate_limit()
    except Exception as e:
        print(f"[GS] Search failed: {e}", file=sys.stderr)

    return papers


def get_citations(title: str, limit: int = 50) -> List[Paper]:
    """Get papers that cite a given paper (by title).

    Searches Google Scholar for the paper, then retrieves its citing papers.
    """
    scholarly = _scholarly()
    if not scholarly:
        logger.debug("scholarly not installed, skipping GS citations")
        return []

    try:
        _rate_limit()
        print(f"[GS] Finding citations for: {title[:50]}...", file=sys.stderr)

        # Find the paper first
        search_query = scholarly.search_pubs(title)
        pub = next(search_query, None)
        if not pub:
            print(f"[GS] Paper not found: {title[:50]}", file=sys.stderr)
            return []

        # Get citing papers
        _rate_limit()
        cites = []
        try:
            citedby = scholarly.citedby(pub)
            for i, citing_pub in enumerate(citedby):
                if i >= limit:
                    break
                cites.append(_parse_scholar_paper(citing_pub))
                if i < limit - 1:
                    _rate_limit()
        except Exception as e:
            print(f"[GS] citedby iteration failed: {e}", file=sys.stderr)

        print(f"[GS] Found {len(cites)} citing papers", file=sys.stderr)
        return cites

    except Exception as e:
        print(f"[GS] get_citations failed: {e}", file=sys.stderr)
        return []


def get_citation_count(title: str) -> Optional[int]:
    """Get the citation count for a paper by title."""
    scholarly = _scholarly()
    if not scholarly:
        return None

    try:
        _rate_limit()
        search_query = scholarly.search_pubs(title)
        pub = next(search_query, None)
        if pub:
            return pub.get("num_citations", 0)
    except Exception as e:
        logger.debug(f"GS citation count failed for '{title[:30]}': {e}")

    return None


def enrich_citation_counts(papers: List[Paper]) -> List[Paper]:
    """Batch-enrich citation counts for papers via Google Scholar.

    Only enriches papers that have citation_count == 0 (no existing count).
    Skips if scholarly is not available. Returns papers list (modified in place).
    """
    scholarly = _scholarly()
    if not scholarly:
        return papers

    enriched = 0
    for paper in papers:
        if paper.citation_count and paper.citation_count > 0:
            continue  # Already has citation count
        try:
            count = get_citation_count(paper.title)
            if count is not None and count > 0:
                paper.citation_count = count
                enriched += 1
        except Exception:
            continue
        # Stop after enriching a few to avoid rate limits
        if enriched >= 10:
            break

    if enriched > 0:
        print(f"[GS] Enriched citation counts for {enriched} papers", file=sys.stderr)

    return papers


path: services/zotero.py
content: """Zotero API client (sync)."""
from typing import List, Optional

import httpx

from schemas import Paper

ZOTERO_API = "https://api.zotero.org"


def get_collections(user_id: str, api_key: str) -> list:
    headers = {"Zotero-API-Key": api_key, "Zotero-API-Version": "3"}
    with httpx.Client(timeout=30) as client:
        resp = client.get(f"{ZOTERO_API}/users/{user_id}/collections", headers=headers, params={"format": "json"})
        resp.raise_for_status()
    return [{"key": c["data"]["key"], "name": c["data"]["name"], "num_items": c["meta"].get("numItems", 0)}
            for c in resp.json()]


def import_papers(user_id: str, api_key: str, collection_key: Optional[str] = None, limit: int = 50) -> List[Paper]:
    base = f"{ZOTERO_API}/users/{user_id}/items"
    if collection_key:
        base = f"{ZOTERO_API}/users/{user_id}/collections/{collection_key}/items"
    headers = {"Zotero-API-Key": api_key, "Zotero-API-Version": "3"}
    with httpx.Client(timeout=30) as client:
        resp = client.get(base, headers=headers, params={"limit": limit, "format": "json"})
        resp.raise_for_status()

    papers = []
    for item in resp.json():
        data = item.get("data", {})
        if data.get("itemType") in ("attachment", "note"):
            continue
        creators = data.get("creators", [])
        authors = [f"{c.get('firstName', '')} {c.get('lastName', '')}".strip() for c in creators]
        authors = [a for a in authors if a]
        date_str = data.get("date", "")
        year = None
        for part in date_str.split("-"):
            if part.isdigit() and len(part) == 4:
                year = int(part)
                break
        doi = data.get("DOI", "")
        papers.append(Paper(
            id=f"DOI:{doi}" if doi else f"zotero:{item.get('key', '')}",
            title=data.get("title", "Untitled"), authors=authors, year=year,
            abstract=data.get("abstractNote"),
            url=data.get("url") or (f"https://doi.org/{doi}" if doi else None),
        ))
    return papers


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)