Skill: peterkouss/arxiv-master-search (version 0.1.0)

# arXiv 检索与分析 Skill ## 概述 本 skill 提供强大的 arXiv 学术论文检索、下载和分析功能，帮助智能体高效进行学术文献调研。支持关键词检索、作者检索、日期范围过滤、分类筛选等多种检索方式，并提供批量下载、元数据提取、论文总结等功能。 所有配置均通过命令行参数传入，无需配置文件。可选参数均有合理默认值。 ## 目录结构 ``` arxiv-search-skill/ ├── skill.md # 本文件 - 使用说明与规范 ├── README.md # 快速入门指南 ├── requirements.txt # Python 依赖 ├── arxiv_search.py # 统一入口（可选） ├── scripts/ │ ├── __init__.py │ ├── utils.py # 工具函数 │ ├── search.py # 检索模块 │ ├── download.py # 下载模块 │ ├── metadata.py # 元数据处理模块 │ ├── batch_search.py # 批量检索模块 │ └── summarize.py # 论文总结模块 ├── output/ # 输出目录 │ ├── pdfs/ # 下载的 PDF 文件 │ ├── metadata/ # 元数据 JSON 文件 │ └── summaries/ # 生成的摘要文件 └── examples/ # 使用示例 ├── example_queries.jsonl └── ids.txt ``` ## 安装与配置 ### 1. 安装依赖 ```bash pip install -r requirements.txt ``` ### 2. 切换到 scripts 目录 所有脚本在 `scripts/` 目录下运行： ```bash cd scripts ``` ## 核心功能 --- ## 1. 检索模块 (search.py) ### 功能 在 arXiv 中按关键词、标题、摘要、作者、分类等检索论文，支持日期范围过滤。 ### 使用方法 ```bash python search.py --query "quantum computing" --max-results 50 ``` ### 命令行参数 #### 检索条件 (至少提供一个) | 参数 | 简写 | 说明 | 必需 | |------|------|------|------| | `--query` | `-q` | 通用查询字符串 | 否 | | `--title` | `-t` | 标题关键词 | 否 | | `--abstract` | `-a` | 摘要关键词 | 否 | | `--author` | `-au` | 作者名 | 否 | | `--category` | `-c` | arXiv 分类 (如 cs.AI, quant-ph) | 否 | #### 日期过滤 (可选) | 参数 | 简写 | 说明 | 默认值 | |------|------|------|--------| | `--start-date` | `-s` | 起始日期 (YYYY-MM-DD) | - | | `--end-date` | `-e` | 结束日期 (YYYY-MM-DD) | - | #### 检索控制 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--max-results` | 最大结果数 | 100 | | `--sort-by` | 排序方式: relevance, lastUpdatedDate, submittedDate | relevance | | `--sort-order` | 排序顺序: descending, ascending | descending | | `--timeout` | 请求超时秒数 | 30 | | `--retry-attempts` | 重试次数 | 3 | #### 速率限制 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--requests-per-second` | 每秒最大请求数 | 2.0 | | `--min-delay` | 请求间最小延迟秒数 | 0.5 | #### 输入输出 | 参数 | 简写 | 说明 | |------|------|------| | `--id-list` | - | 从文件读取 arXiv ID 列表进行检索 | | `--output` | `-o` | 输出元数据 JSON 文件路径 | #### 日志 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--log-level` | 日志级别: DEBUG, INFO, WARNING, ERROR | INFO | | `--log-file` | 日志文件路径 | - | ### 高级查询语法 ```bash # 标题中包含关键词 python search.py --query "ti:quantum AND ti:computing" # 摘要中包含关键词 python search.py --query "abs:machine AND abs:learning" # 组合条件 python search.py --query "cat:cs.AI AND (ti:neural OR abs:network)" ``` ### 使用示例 ```bash # 基础关键词检索 python search.py --query "quantum computing" --max-results 50 # 标题检索 python search.py --title "transformer" --max-results 30 # 作者检索 python search.py --author "Smith, J" --max-results 100 # 分类检索 python search.py --category "cs.AI" --start-date "2023-01-01" # 组合检索并保存结果 python search.py --query "deep learning" --category "cs.LG" --max-results 50 --output results.json ``` ### 输出示例 ```json { "search_metadata": { "query": "quantum computing", "original_query": { "query": "quantum computing", "title": null, "abstract": null, "author": null, "category": null, "start_date": null, "end_date": null }, "timestamp": "2023-01-15T10:30:00Z", "returned_results": 50 }, "papers": [ { "arxiv_id": "2301.01234", "arxiv_url": "https://arxiv.org/abs/2301.01234", "pdf_url": "https://arxiv.org/pdf/2301.01234.pdf", "title": "Quantum Computing for Machine Learning", "authors": [ {"name": "John Smith"}, {"name": "Alice Johnson"} ], "abstract": "We present a novel approach...", "categories": ["quant-ph", "cs.LG"], "primary_category": "quant-ph", "published_date": "2023-01-05", "updated_date": "2023-01-10", "doi": "10.1234/arxiv.2301.01234", "journal_ref": null, "comments": "15 pages, 5 figures" } ] } ``` --- ## 2. 下载模块 (download.py) ### 功能 下载单个或批量论文 PDF 文件。 ### 使用方法 ```bash python download.py --id 2301.01234 ``` ### 命令行参数 #### 输入源 (选择一个) | 参数 | 说明 | 必需 | |------|------|------| | `--id` | arXiv ID（可多次指定） | 否 | | `--id-list` | 包含 arXiv ID 列表的文本文件 | 否 | | `--metadata` | 包含论文元数据的 JSON 文件 | 否 | #### 输出选项 (可选) | 参数 | 简写 | 说明 | 默认值 | |------|------|------|--------| | `--output-dir` | `-o` | 输出目录 | output/pdfs | | `--skip-existing` | - | 跳过已存在的文件 | 默认启用 | | `--no-skip-existing` | - | 覆盖已存在的文件 | - | #### 下载控制 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--parallel` | 并行下载数 | 3 | | `--chunk-size` | 下载块大小字节 | 8192 | | `--verify-ssl` | 验证 SSL 证书 | 默认启用 | | `--no-verify-ssl` | 不验证 SSL 证书 | - | #### 速率限制 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--requests-per-second` | 每秒最大请求数 | 2.0 | | `--min-delay` | 请求间最小延迟秒数 | 0.5 | #### 日志 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--log-level` | 日志级别: DEBUG, INFO, WARNING, ERROR | INFO | | `--log-file` | 日志文件路径 | - | ### 使用示例 ```bash # 下载单个论文 python download.py --id 2301.01234 # 下载多个论文 python download.py --id 2301.01234 --id 2302.05678 # 从 ID 列表文件下载 python download.py --id-list ids.txt # 从元数据文件下载 python download.py --metadata results.json # 指定输出目录 python download.py --id 2301.01234 --output-dir my_papers/ # 强制覆盖已存在的文件 python download.py --id 2301.01234 --no-skip-existing ``` --- ## 3. 元数据处理模块 (metadata.py) ### 功能 提取论文元数据，支持多种格式导出（JSON、BibTeX、CSV、RIS），以及合并多个元数据文件。 ### 使用方法 ```bash python metadata.py --input results.json --format bibtex --output refs.bib ``` ### 命令行参数 #### 输入方式 (选择一个) | 参数 | 说明 | 必需 | |------|------|------| | `--id` | 单个 arXiv ID | 否 | | `--input` | `-i` | 输入元数据 JSON 文件 | 否 | | `--merge` | 合并多个元数据文件 | 否 | #### 输出选项 | 参数 | 简写 | 说明 | 默认值 | |------|------|------|--------| | `--format` | `-f` | 输出格式: json, bibtex, csv, ris | json | | `--output` | `-o` | 输出文件路径 | - | #### 日志 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--log-level` | 日志级别: DEBUG, INFO, WARNING, ERROR | INFO | | `--log-file` | 日志文件路径 | - | ### 使用示例 ```bash # 获取单个论文元数据 python metadata.py --id 2301.01234 # 从元数据文件导出 BibTeX python metadata.py --input results.json --format bibtex --output refs.bib # 导出 CSV python metadata.py --input results.json --format csv --output papers.csv # 导出 RIS python metadata.py --input results.json --format ris --output papers.ris # 合并多个元数据文件 python metadata.py --merge result1.json result2.json --output merged.json ``` ### 输出格式 #### BibTeX 格式 ```bibtex @article{smith:2023:quantum, title = {Quantum Computing for Machine Learning}, author = {Smith, John and Johnson, Alice}, year = {2023}, month = {01}, journal = {arXiv preprint 2301.01234}, eprint = {2301.01234}, archivePrefix = {arXiv}, primaryClass = {quant-ph}, url = {https://arxiv.org/abs/2301.01234}, abstract = {We present a novel approach...} } ``` --- ## 4. 批量检索模块 (batch_search.py) ### 功能 从 JSONL 文件批量执行多个检索任务。 ### 使用方法 ```bash python batch_search.py --input queries.jsonl --output batch_results/ ``` ### 命令行参数 #### 输入 (必需) | 参数 | 简写 | 说明 | |------|------|------| | `--input` | `-i` | 输入 JSONL 文件路径 | #### 输出 (可选) | 参数 | 简写 | 说明 | 默认值 | |------|------|------|--------| | `--output` | `-o` | 输出目录 | output/metadata | | `--no-individual` | - | 不保存单独的查询结果 | - | #### 检索控制 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--timeout` | 请求超时秒数 | 30 | | `--retry-attempts` | 重试次数 | 3 | | `--requests-per-second` | 每秒最大请求数 | 2.0 | | `--min-delay` | 请求间最小延迟秒数 | 0.5 | #### 日志 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--log-level` | 日志级别: DEBUG, INFO, WARNING, ERROR | INFO | | `--log-file` | 日志文件路径 | - | ### 输入文件格式 (JSONL) ```jsonl {"query": "deep learning", "max_results": 50} {"query": "transformer architecture", "category": "cs.CL", "max_results": 30, "name": "transformer_2023"} {"author": "Bengio, Y", "max_results": 100, "name": "bengio_papers"} {"category": "cs.AI", "start_date": "2023-01-01", "max_results": 50} ``` JSONL 每行一个查询对象，支持的字段： - `query` / `q`: 通用查询 - `title` / `t`: 标题关键词 - `abstract` / `a`: 摘要关键词 - `author` / `au`: 作者名 - `category` / `c` / `categories`: 分类 - `start_date` / `start-date` / `s`: 起始日期 - `end_date` / `end-date` / `e`: 结束日期 - `max_results` / `max-results` / `m`: 最大结果数 - `sort_by` / `sort-by`: 排序方式 - `sort_order` / `sort-order`: 排序顺序 - `name`: 查询名称（用于输出文件名） ### 使用示例 ```bash # 从 JSONL 文件运行批量检索 python batch_search.py --input queries.jsonl # 指定输出目录 python batch_search.py --input queries.jsonl --output batch_results/ # 不保存单独查询结果，只保存合并文件 python batch_search.py --input queries.jsonl --no-individual ``` --- ## 5. 论文总结模块 (summarize.py) ### 功能 生成论文摘要和关键点总结，以及文献综述概览。 ### 使用方法 ```bash python summarize.py --metadata results.json --overview --topic "deep learning" ``` ### 命令行参数 #### 输入方式 (选择一个) | 参数 | 说明 | 必需 | |------|------|------| | `--id` | 单个 arXiv ID | 否 | | `--metadata` | 包含论文元数据的 JSON 文件 | 否 | #### 输出选项 (可选) | 参数 | 简写 | 说明 | 默认值 | |------|------|------|--------| | `--output` | `-o` | 输出文件路径 | - | | `--output-dir` | - | 输出目录（用于批量模式） | output/summaries | #### 功能选项 (可选) | 参数 | 说明 | |------|------| | `--overview` | 生成文献综述概览 | | `--topic` | 研究主题（用于综述） | #### 日志 (可选) | 参数 | 说明 | 默认值 | |------|------|--------| | `--log-level` | 日志级别: DEBUG, INFO, WARNING, ERROR | INFO | | `--log-file` | 日志文件路径 | - | ### 总结内容 单篇论文总结包括： - 标题、作者、发表日期 - 分类、URL - 短摘要（前 500 字符） - 关键点： - 方法关键词 - 任务关键词 - 问题陈述 - 贡献点 文献综述包括： - 论文集合统计 - 分类分布 - 年份分布 - 方法和任务统计 - 关键洞察 ### 使用示例 ```bash # 总结单篇论文 python summarize.py --id 2301.01234 # 从元数据文件批量生成总结 python summarize.py --metadata results.json # 生成文献综述 python summarize.py --metadata results.json --overview --topic "deep learning" --output overview.json ``` --- ## 常见 arXiv 分类 | 分类 | 说明 | |------|------| | cs.AI | Artificial Intelligence | | cs.CL | Computation and Language | | cs.CV | Computer Vision and Pattern Recognition | | cs.LG | Machine Learning | | cs.NE | Neural and Evolutionary Computing | | cs.RO | Robotics | | quant-ph | Quantum Physics | | stat.ML | Machine Learning (Statistics) | | math.OC | Optimization and Control | | physics.data-an | Data Analysis, Statistics and Probability | --- ## API 使用示例 (Python 模块) ### 基础检索 ```python from search import ArxivSearch searcher = ArxivSearch() # 简单关键词检索 results = searcher.search( query="transformer architecture", max_results=20, sort_by="submittedDate" ) # 高级检索 results = searcher.search( query="cat:cs.CL AND abs:transformer", start_date="2023-01-01", end_date="2023-12-31", max_results=50 ) ``` ### 批量下载 ```python from download import PaperDownloader downloader = PaperDownloader(output_dir="output/pdfs") # 下载单个论文 downloader.download_by_id("2301.01234") # 批量下载 arxiv_ids = ["2301.01234", "2302.05678", "2303.09012"] papers = [{"arxiv_id": id} for id in arxiv_ids] downloader.download_batch(papers, skip_existing=True) ``` ### 元数据处理 ```python from metadata import MetadataExtractor extractor = MetadataExtractor() # 获取元数据 metadata = extractor.get_metadata("2301.01234") # 导出为 BibTeX papers = [metadata] extractor.export_bibtex(papers, "references.bib") ``` --- ## 完整工作流示例 ```bash # 1. 定义检索查询列表 cat > my_queries.jsonl << EOF {"query": "large language model", "max_results": 50, "name": "llm"} {"query": "chain-of-thought reasoning", "max_results": 30, "name": "cot"} {"category": "cs.CL", "max_results": 100, "start_date": "2023-01-01", "name": "cl_2023"} EOF # 2. 执行批量检索 python batch_search.py --input my_queries.jsonl --output my_results/ # 3. 下载相关论文 python download.py --metadata my_results/merged_metadata.json --output-dir my_pdfs/ # 4. 生成文献综述 python summarize.py --metadata my_results/merged_metadata.json --overview --topic "LLMs" --output llm_overview.json # 5. 导出 BibTeX python metadata.py --input my_results/merged_metadata.json --format bibtex --output references.bib ``` --- ## 输出规范 ### 元数据 JSON 格式规范 见 search.py 输出示例。 ### 错误处理规范 所有脚本遵循以下错误码规范： | 错误码 | 含义 | |--------|------| | 0 | 成功 | | 1 | 通用错误 | | 2 | 参数错误 | | 3 | 网络错误 | | 4 | 文件IO错误 | | 5 | API 限流 | ### 日志规范 ``` [INFO] 2023-01-15 10:30:00 - Starting search for query: "quantum computing" [INFO] 2023-01-15 10:30:02 - Found 1500 results, returning first 50 [WARNING] 2023-01-15 10:30:03 - Rate limit approaching, slowing down [ERROR] 2023-01-15 10:30:05 - Failed to download 2301.01234: Connection timeout ``` --- ## 高级技巧 ### 1. 检索优化 - 使用引号进行精确匹配: `"graph neural network"` - 使用通配符: `transformer*` - 使用布尔运算符: `AND`, `OR`, `ANDNOT` - 字段限定: `ti:`, `abs:`, `au:`, `cat:` ### 2. 分类组合 ```bash # 多个分类 python search.py --query "cat:cs.AI OR cat:cs.LG" # 排除分类 python search.py --query "cat:cs.* ANDNOT cat:cs.RO" ``` ### 3. 时间范围检索 ```bash # 最近一个月 (Linux/Mac) python search.py --query "deep learning" --start-date "$(date -d '1 month ago' +%Y-%m-%d)" # 特定年份 python search.py --query "transformer" --start-date "2023-01-01" --end-date "2023-12-31" ``` --- ## 常见问题 ### Q: 如何提高检索速度？ A: 使用 `--max-results` 限制返回数量，使用分类过滤减少结果集。 ### Q: 遇到 rate limit 怎么办？ A: 脚本会自动处理限流。可通过 `--requests-per-second` 和 `--min-delay` 调整速率限制参数。 ### Q: 下载的 PDF 文件名格式是什么？ A: 默认格式: `{arxiv_id}_{title_slug}.pdf`，例如: `2301.01234_quantum-computing-for-machine-learning.pdf` ### Q: 所有参数都有默认值吗？ A: 是的，所有可选参数都有合理的默认值。只需提供必需参数（如检索条件、输入源等）即可。 --- ## 更新日志 ### v1.1.0 - 移除配置文件，所有配置改为命令行参数 - 所有可选参数添加默认值 - 更新文档，添加完整参数说明 ### v1.0.0 - 初始版本发布 - 支持基础检索、下载、元数据提取 - 支持批量操作

path: README.md
content: # arXiv 检索与分析 Skill

一个功能完善的 arXiv 学术论文检索、下载和分析工具集，帮助智能体高效进行文献调研。

## 功能特性

- **多种检索方式**: 关键词、标题、摘要、作者、分类、日期范围
- **批量操作**: 支持批量检索、批量下载
- **格式导出**: JSON、BibTeX、CSV、RIS 格式
- **智能分析**: 论文总结、文献综述、趋势分析
- **速率限制**: 内置限流保护，避免被 arXiv 封禁
- **无需配置文件**: 所有配置通过命令行参数传入，带默认值

## 快速开始

### 安装依赖

```bash
pip install -r requirements.txt
```

### 切换到 scripts 目录

```bash
cd scripts
```

### 基础检索

```bash
# 关键词检索
python search.py --query "quantum computing" --max-results 50

# 保存结果到文件
python search.py --query "deep learning" --output results.json
```

### 下载论文

```bash
# 下载单个论文
python download.py --id 2301.01234

# 从元数据文件批量下载
python download.py --metadata results.json
```

### 导出参考文献

```bash
# 导出 BibTeX
python metadata.py --input results.json --format bibtex --output refs.bib

# 导出 CSV
python metadata.py --input results.json --format csv --output papers.csv
```

### 批量检索

```bash
# 从 JSONL 文件执行批量检索
python batch_search.py --input ../examples/example_queries.jsonl
```

## 目录结构

```
arxiv-search-skill/
├── skill.md                    # 完整使用说明与规范
├── README.md                   # 本文件
├── requirements.txt            # Python 依赖
├── scripts/
│   ├── search.py               # 检索模块
│   ├── download.py             # 下载模块
│   ├── metadata.py             # 元数据处理模块
│   ├── batch_search.py         # 批量检索模块
│   ├── summarize.py            # 论文总结模块
│   └── utils.py                # 工具函数
├── output/                     # 输出目录
│   ├── pdfs/                   # 下载的 PDF 文件
│   ├── metadata/               # 元数据 JSON 文件
│   └── summaries/              # 生成的摘要文件
└── examples/                   # 示例文件
    ├── example_queries.jsonl
    └── ids.txt
```

## 模块说明

| 模块 | 功能 |
|------|------|
| **search.py** | 论文检索（关键词、作者、分类、日期） |
| **download.py** | 论文 PDF 下载（单个/批量） |
| **metadata.py** | 元数据处理（BibTeX/CSV/RIS 导出） |
| **batch_search.py** | 批量检索（JSONL 输入） |
| **summarize.py** | 论文总结与文献综述 |

所有模块都支持 `--help` 查看完整参数列表。

## 快速参数参考

### search.py
```bash
python search.py --query "关键词" --max-results 50 --output results.json
```

### download.py
```bash
python download.py --id arXivID --output-dir my_papers/
```

### metadata.py
```bash
python metadata.py --input results.json --format bibtex --output refs.bib
```

### batch_search.py
```bash
python batch_search.py --input queries.jsonl --output batch_results/
```

### summarize.py
```bash
python summarize.py --metadata results.json --overview --topic "研究主题"
```

## 详细文档

请参考 [skill.md](./skill.md) 查看完整的使用说明、参数文档、API 文档和输出规范。

## 常见 arXiv 分类

| 分类 | 说明 |
|------|------|
| cs.AI | Artificial Intelligence |
| cs.CL | Computation and Language |
| cs.CV | Computer Vision |
| cs.LG | Machine Learning |
| cs.NE | Neural and Evolutionary Computing |
| quant-ph | Quantum Physics |
| stat.ML | Machine Learning (Statistics) |

## 完整工作流示例

```bash
cd scripts

# 1. 定义检索查询
cat > my_queries.jsonl << EOF
{"query": "large language model", "max_results": 50, "name": "llm"}
{"query": "chain-of-thought", "max_results": 30, "name": "cot"}
EOF

# 2. 执行批量检索
python batch_search.py --input my_queries.jsonl --output my_results/

# 3. 下载论文
python download.py --metadata my_results/merged_metadata.json

# 4. 生成文献综述
python summarize.py --metadata my_results/merged_metadata.json --overview --topic "LLMs" --output llm_overview.json

# 5. 导出 BibTeX
python metadata.py --input my_results/merged_metadata.json --format bibtex --output references.bib
```

## 许可证

本项目提供用于学术研究目的。


path: arxiv_search.py
content: #!/usr/bin/env python3
"""
arXiv 检索与分析 Skill - 主入口
便捷的统一命令行接口
"""

import sys
import argparse
from pathlib import Path


def main():
    """主入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 检索与分析 Skill",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
子命令:
  search      - 检索论文
  download    - 下载论文
  metadata    - 元数据处理
  batch       - 批量检索
  summarize   - 论文总结

使用示例:
  arxiv_search.py search --query "deep learning"
  arxiv_search.py download --id 2301.01234
  arxiv_search.py metadata --input results.json --format bibtex
        """
    )

    parser.add_argument("subcommand", help="子命令 (search, download, metadata, batch, summarize)")
    parser.add_argument("args", nargs=argparse.REMAINDER, help="子命令参数")

    # 如果没有参数，显示帮助
    if len(sys.argv) == 1:
        parser.print_help()
        return 0

    # 解析子命令
    args = parser.parse_args()

    # 导入并运行相应的模块
    script_dir = Path(__file__).parent / "scripts"
    sys.path.insert(0, str(Path(__file__).parent))

    subcommand_map = {
        "search": "search",
        "download": "download",
        "metadata": "metadata",
        "batch": "batch_search",
        "summarize": "summarize",
    }

    if args.subcommand not in subcommand_map:
        print(f"未知的子命令: {args.subcommand}")
        parser.print_help()
        return 1

    module_name = subcommand_map[args.subcommand]

    # 构建新的 argv
    module_path = script_dir / f"{module_name}.py"
    new_argv = [str(module_path)] + args.args

    # 执行模块
    import subprocess

    return subprocess.call([sys.executable] + new_argv)


if __name__ == "__main__":
    sys.exit(main())


path: examples/ids.txt
content: # 示例 arXiv ID 列表
# 每行一个 ID，可以包含版本号或完整 URL

2301.01234
2302.05678
2303.09012v1
https://arxiv.org/abs/2304.12345
2305.67890


path: requirements.txt
content: # arXiv Search Skill Dependencies

# 核心 arXiv API
arxiv>=1.4.8

# HTTP 请求
requests>=2.28.0
urllib3>=1.26.0

# 数据处理
pandas>=1.5.0
numpy>=1.23.0

# YAML 配置
PyYAML>=6.0

# 并行处理
tqdm>=4.64.0

# PDF 处理
PyPDF2>=3.0.0
pdfminer.six>=20221105

# 日期处理
python-dateutil>=2.8.0

# 日志
colorlog>=6.7.0

# 命令行界面
click>=8.1.0

# 可选 - 用于总结功能 (需要额外配置)
# openai>=1.0.0
# transformers>=4.30.0
# torch>=2.0.0


path: scripts/__init__.py
content: # arXiv Search Skill
# 用于检索、下载和分析 arXiv 论文的工具集

__version__ = "1.0.0"
__author__ = "ArXiv Search Skill"

# 延迟导入，避免循环依赖和未安装依赖时的问题
__all__ = ["search", "download", "metadata", "batch_search", "summarize", "utils"]

def __getattr__(name):
    if name in __all__:
        import importlib
        return importlib.import_module(f'.{name}', __name__)
    raise AttributeError(f"module {__name__} has no attribute {name}")


path: scripts/batch_search.py
content: #!/usr/bin/env python3
"""
batch_search.py - 批量检索模块
支持从 JSONL 文件批量执行多个检索任务

命令行参数:
  输入:
    --input, -i           输入 JSONL 文件路径 (必需)

  输出:
    --output, -o          输出目录 (默认: output/metadata)
    --no-individual       不保存单独的查询结果

  检索控制 (可选):
    --timeout             请求超时秒数 (默认: 30)
    --retry-attempts      重试次数 (默认: 3)
    --requests-per-second 每秒最大请求数 (默认: 2.0)
    --min-delay           请求间最小延迟秒数 (默认: 0.5)

  日志 (可选):
    --log-level           日志级别: DEBUG, INFO, WARNING, ERROR (默认: INFO)
    --log-file            日志文件路径
"""

import argparse
import sys
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Any, List

from tqdm import tqdm

# 支持作为模块导入和独立脚本运行
try:
    from .utils import (
        setup_logging,
        get_logger,
        save_json,
        load_jsonl,
        ensure_dir,
    )
    from .search import ArxivSearch
except ImportError:
    from utils import (
        setup_logging,
        get_logger,
        save_json,
        load_jsonl,
        ensure_dir,
    )
    from search import ArxivSearch


class BatchSearch:
    """批量检索器"""

    def __init__(
        self,
        output_dir: str = "output/metadata",
        timeout: int = 30,
        retry_attempts: int = 3,
        requests_per_second: float = 2.0,
        min_delay: float = 0.5,
        log_level: str = "INFO",
        log_file: str = None,
    ):
        """
        初始化批量检索器

        Args:
            output_dir: 输出目录 (默认: output/metadata)
            timeout: 请求超时秒数 (默认: 30)
            retry_attempts: 重试次数 (默认: 3)
            requests_per_second: 每秒最大请求数 (默认: 2.0)
            min_delay: 请求间最小延迟秒数 (默认: 0.5)
            log_level: 日志级别
            log_file: 日志文件路径
        """
        setup_logging(level=log_level, log_file=log_file)
        self.logger = get_logger()

        self.output_dir = Path(output_dir)
        ensure_dir(self.output_dir)

        self.searcher = ArxivSearch(
            timeout=timeout,
            retry_attempts=retry_attempts,
            requests_per_second=requests_per_second,
            min_delay=min_delay,
            log_level=log_level,
            log_file=log_file,
        )

    def parse_query_spec(self, spec: Dict[str, Any]) -> Dict[str, Any]:
        """
        解析查询规范

        Args:
            spec: 查询规范字典

        Returns:
            解析后的参数字典
        """
        params = {}

        field_map = {
            "query": "query",
            "q": "query",
            "title": "title",
            "t": "title",
            "abstract": "abstract",
            "a": "abstract",
            "author": "author",
            "au": "author",
            "category": "category",
            "c": "category",
            "categories": "category",
            "start_date": "start_date",
            "start-date": "start_date",
            "s": "start_date",
            "end_date": "end_date",
            "end-date": "end_date",
            "e": "end_date",
            "max_results": "max_results",
            "max-results": "max_results",
            "m": "max_results",
            "sort_by": "sort_by",
            "sort-by": "sort_by",
            "sort_order": "sort_order",
            "sort-order": "sort_order",
        }

        for src, dst in field_map.items():
            if src in spec:
                params[dst] = spec[src]

        return params

    def run_query(self, query_spec: Dict[str, Any], index: int = 0) -> Dict[str, Any]:
        """
        运行单个查询

        Args:
            query_spec: 查询规范
            index: 查询索引

        Returns:
            检索结果
        """
        params = self.parse_query_spec(query_spec)
        query_name = query_spec.get("name", f"query_{index:03d}")

        self.logger.info(f"执行查询 [{query_name}]: {params}")

        try:
            results = self.searcher.search(**params)
            results["search_metadata"]["query_name"] = query_name
            results["search_metadata"]["query_spec"] = query_spec

            return {
                "success": True,
                "query_name": query_name,
                "results": results,
                "paper_count": len(results["papers"]),
            }

        except Exception as e:
            self.logger.error(f"查询 [{query_name}] 失败: {e}")
            return {
                "success": False,
                "query_name": query_name,
                "error": str(e),
                "query_spec": query_spec,
            }

    def run_batch(
        self,
        query_specs: List[Dict[str, Any]],
        save_individual: bool = True,
    ) -> Dict[str, Any]:
        """
        运行批量检索

        Args:
            query_specs: 查询规范列表
            save_individual: 是否保存每个查询的单独结果

        Returns:
            批量检索结果汇总
        """
        all_results = []
        all_papers = []
        summary = {
            "total_queries": len(query_specs),
            "successful": 0,
            "failed": 0,
            "total_papers": 0,
            "queries": [],
        }

        self.logger.info(f"开始批量检索: {len(query_specs)} 个查询")

        for i, spec in enumerate(tqdm(query_specs, desc="检索进度")):
            result = self.run_query(spec, i)
            all_results.append(result)

            if result["success"]:
                summary["successful"] += 1
                papers = result["results"]["papers"]
                all_papers.extend(papers)
                summary["total_papers"] += len(papers)

                summary["queries"].append({
                    "name": result["query_name"],
                    "paper_count": result["paper_count"],
                    "success": True,
                })

                if save_individual:
                    output_file = self.output_dir / f"{result['query_name']}.json"
                    save_json(result["results"], output_file)

            else:
                summary["failed"] += 1
                summary["queries"].append({
                    "name": result["query_name"],
                    "error": result["error"],
                    "success": False,
                })

        merged = {
            "search_metadata": {
                "type": "batch_search",
                "timestamp": datetime.now(timezone.utc).isoformat() + "Z",
                "summary": summary,
            },
            "papers": all_papers,
            "individual_results": all_results,
        }

        merged_file = self.output_dir / "merged_metadata.json"
        save_json(merged, merged_file)

        self.logger.info(f"批量检索完成: 成功={summary['successful']}, 失败={summary['failed']}, 论文总数={summary['total_papers']}")
        return merged


def main():
    """命令行入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 批量检索工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
输入文件格式 (JSONL):
  {"query": "deep learning", "max_results": 50}
  {"author": "Bengio, Y", "max_results": 100, "name": "bengio_papers"}

示例:
  # 从 JSONL 文件运行批量检索
  python batch_search.py --input queries.jsonl

  # 指定输出目录
  python batch_search.py --input queries.jsonl --output batch_results/
        """
    )

    parser.add_argument("--input", "-i", required=True, help="输入 JSONL 文件路径")
    parser.add_argument("--output", "-o", default="output/metadata", help="输出目录 (默认: output/metadata)")
    parser.add_argument("--no-individual", action="store_true", help="不保存单独的查询结果")

    # 检索控制
    parser.add_argument("--timeout", type=int, default=30, help="请求超时秒数 (默认: 30)")
    parser.add_argument("--retry-attempts", type=int, default=3, help="重试次数 (默认: 3)")
    parser.add_argument("--requests-per-second", type=float, default=2.0, help="每秒最大请求数 (默认: 2.0)")
    parser.add_argument("--min-delay", type=float, default=0.5, help="请求间最小延迟秒数 (默认: 0.5)")

    # 日志
    parser.add_argument("--log-level", choices=["DEBUG", "INFO", "WARNING", "ERROR"], default="INFO", help="日志级别 (默认: INFO)")
    parser.add_argument("--log-file", help="日志文件路径")

    args = parser.parse_args()

    setup_logging(level=args.log_level, log_file=args.log_file)
    logger = get_logger()

    # 检查输入文件
    input_path = Path(args.input)
    if not input_path.exists():
        logger.error(f"输入文件不存在: {args.input}")
        return 1

    # 加载查询规范
    try:
        query_specs = load_jsonl(input_path)
        logger.info(f"加载了 {len(query_specs)} 个查询")
    except Exception as e:
        logger.error(f"无法加载输入文件: {e}")
        return 1

    # 运行批量检索
    batch_search = BatchSearch(
        output_dir=args.output,
        timeout=args.timeout,
        retry_attempts=args.retry_attempts,
        requests_per_second=args.requests_per_second,
        min_delay=args.min_delay,
        log_level=args.log_level,
        log_file=args.log_file,
    )

    try:
        results = batch_search.run_batch(
            query_specs,
            save_individual=not args.no_individual,
        )

        # 打印摘要
        summary = results["search_metadata"]["summary"]
        print("\n==== 批量检索摘要 ====")
        print(f"总查询数: {summary['total_queries']}")
        print(f"成功: {summary['successful']}")
        print(f"失败: {summary['failed']}")
        print(f"总论文数: {summary['total_papers']}")

        if summary["failed"] > 0:
            print("\n失败的查询:")
            for q in summary["queries"]:
                if not q["success"]:
                    print(f"  - {q['name']}: {q.get('error', 'unknown')}")

        return 0

    except Exception as e:
        logger.error(f"批量检索失败: {e}")
        import traceback
        traceback.print_exc()
        return 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/download.py
content: #!/usr/bin/env python3
"""
download.py - arXiv 论文下载模块
提供单个或批量论文 PDF 下载功能

命令行参数:
  输入源 (选择一个):
    --id                  arXiv ID（可多次指定）
    --id-list             包含 arXiv ID 列表的文本文件
    --metadata            包含论文元数据的 JSON 文件

  输出选项:
    --output-dir, -o      输出目录 (默认: output/pdfs)
    --skip-existing       跳过已存在的文件 (默认)
    --no-skip-existing    覆盖已存在的文件

  下载控制 (可选):
    --parallel            并行下载数 (默认: 3)
    --chunk-size          下载块大小字节 (默认: 8192)
    --verify-ssl          验证 SSL 证书 (默认: true)

  速率限制 (可选):
    --requests-per-second 每秒最大请求数 (默认: 2.0)
    --min-delay           请求间最小延迟秒数 (默认: 0.5)

  日志 (可选):
    --log-level           日志级别: DEBUG, INFO, WARNING, ERROR (默认: INFO)
    --log-file            日志文件路径
"""

import argparse
import sys
from pathlib import Path
from typing import Dict, Any, List, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests
from tqdm import tqdm

# 支持作为模块导入和独立脚本运行
try:
    from .utils import (
        setup_logging,
        get_logger,
        ensure_dir,
        sanitize_filename,
        slugify,
        parse_arxiv_id,
        RateLimiter,
        load_json,
    )
except ImportError:
    from utils import (
        setup_logging,
        get_logger,
        ensure_dir,
        sanitize_filename,
        slugify,
        parse_arxiv_id,
        RateLimiter,
        load_json,
    )


class PaperDownloader:
    """论文下载器"""

    def __init__(
        self,
        output_dir: str = "output/pdfs",
        max_parallel: int = 3,
        chunk_size: int = 8192,
        skip_existing: bool = True,
        verify_ssl: bool = True,
        requests_per_second: float = 2.0,
        min_delay: float = 0.5,
        log_level: str = "INFO",
        log_file: str = None,
    ):
        """
        初始化下载器

        Args:
            output_dir: 输出目录 (默认: output/pdfs)
            max_parallel: 并行下载数 (默认: 3)
            chunk_size: 下载块大小字节 (默认: 8192)
            skip_existing: 跳过已存在的文件 (默认: true)
            verify_ssl: 验证 SSL 证书 (默认: true)
            requests_per_second: 每秒最大请求数 (默认: 2.0)
            min_delay: 请求间最小延迟秒数 (默认: 0.5)
            log_level: 日志级别
            log_file: 日志文件路径
        """
        setup_logging(level=log_level, log_file=log_file)
        self.logger = get_logger()

        self.output_dir = Path(output_dir)
        ensure_dir(self.output_dir)

        self.chunk_size = chunk_size
        self.skip_existing = skip_existing
        self.verify_ssl = verify_ssl
        self.max_parallel = max_parallel

        self.rate_limiter = RateLimiter(requests_per_second, min_delay)
        self.session = requests.Session()

    def _get_pdf_url(self, arxiv_id: str) -> str:
        """获取 PDF URL"""
        normalized_id = parse_arxiv_id(arxiv_id)
        return f"https://arxiv.org/pdf/{normalized_id}.pdf"

    def _generate_filename(self, paper: Dict[str, Any]) -> str:
        """生成文件名"""
        arxiv_id = paper.get("arxiv_id", "unknown")
        title = paper.get("title", "paper")
        title_slug = slugify(title, max_length=60)
        return sanitize_filename(f"{arxiv_id}_{title_slug}.pdf")

    def download_by_id(
        self,
        arxiv_id: str,
        filename: str = None,
        paper_metadata: Dict[str, Any] = None,
    ) -> Optional[Path]:
        """
        通过 arXiv ID 下载论文

        Args:
            arxiv_id: arXiv ID
            filename: 输出文件名，如 None 则自动生成
            paper_metadata: 论文元数据（用于生成文件名）

        Returns:
            下载的文件路径，失败返回 None
        """
        normalized_id = parse_arxiv_id(arxiv_id)
        pdf_url = self._get_pdf_url(normalized_id)

        # 确定文件名
        if not filename:
            if paper_metadata:
                filename = self._generate_filename(paper_metadata)
            else:
                filename = f"{normalized_id}.pdf"

        filepath = self.output_dir / filename

        # 检查是否已存在
        if self.skip_existing and filepath.exists():
            self.logger.debug(f"文件已存在，跳过: {filepath}")
            return filepath

        self.logger.info(f"下载: {arxiv_id} -> {filename}")

        try:
            self.rate_limiter.wait()

            response = self.session.get(
                pdf_url,
                stream=True,
                verify=self.verify_ssl,
                timeout=60,
            )
            response.raise_for_status()

            total_size = int(response.headers.get("content-length", 0))

            with open(filepath, "wb") as f:
                if total_size > 0:
                    with tqdm(
                        desc=filename,
                        total=total_size,
                        unit="iB",
                        unit_scale=True,
                        unit_divisor=1024,
                        leave=False,
                    ) as pbar:
                        for chunk in response.iter_content(chunk_size=self.chunk_size):
                            size = f.write(chunk)
                            pbar.update(size)
                else:
                    for chunk in response.iter_content(chunk_size=self.chunk_size):
                        f.write(chunk)

            self.logger.info(f"下载完成: {filepath}")
            return filepath

        except Exception as e:
            self.logger.error(f"下载失败 {arxiv_id}: {e}")
            if filepath.exists():
                filepath.unlink()
            return None

    def download_batch(
        self,
        papers: List[Dict[str, Any]],
        skip_existing: bool = None,
    ) -> Dict[str, Any]:
        """
        批量下载论文

        Args:
            papers: 论文元数据列表
            skip_existing: 是否跳过已存在的文件

        Returns:
            下载结果统计
        """
        if skip_existing is not None:
            self.skip_existing = skip_existing

        results = {
            "total": len(papers),
            "success": [],
            "failed": [],
            "skipped": [],
        }

        # 先检查哪些已存在
        to_download = []
        for paper in papers:
            arxiv_id = paper.get("arxiv_id")
            if not arxiv_id:
                continue

            filename = self._generate_filename(paper)
            filepath = self.output_dir / filename

            if self.skip_existing and filepath.exists():
                results["skipped"].append({"arxiv_id": arxiv_id, "filepath": str(filepath)})
            else:
                to_download.append(paper)

        self.logger.info(f"总共 {len(papers)} 篇，跳过 {len(results['skipped'])} 篇，需下载 {len(to_download)} 篇")

        # 并行下载
        if to_download:
            with ThreadPoolExecutor(max_workers=self.max_parallel) as executor:
                futures = {}

                for paper in to_download:
                    arxiv_id = paper.get("arxiv_id")
                    future = executor.submit(self.download_by_id, arxiv_id, paper_metadata=paper)
                    futures[future] = arxiv_id

                for future in tqdm(
                    as_completed(futures),
                    total=len(futures),
                    desc="下载进度",
                ):
                    arxiv_id = futures[future]
                    try:
                        filepath = future.result()
                        if filepath:
                            results["success"].append({"arxiv_id": arxiv_id, "filepath": str(filepath)})
                        else:
                            results["failed"].append(arxiv_id)
                    except Exception as e:
                        self.logger.error(f"下载任务失败 {arxiv_id}: {e}")
                        results["failed"].append(arxiv_id)

        self.logger.info(f"下载统计: 成功={len(results['success'])}, 失败={len(results['failed'])}, 跳过={len(results['skipped'])}")
        return results


def main():
    """命令行入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 论文下载工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  # 下载单个论文
  python download.py --id 2301.01234

  # 下载多个论文
  python download.py --id 2301.01234 --id 2302.05678

  # 从 ID 列表文件下载
  python download.py --id-list ids.txt

  # 从元数据文件下载
  python download.py --metadata results.json

  # 指定输出目录
  python download.py --id 2301.01234 --output-dir my_papers/
        """
    )

    # 输入源（多选一）
    input_group = parser.add_mutually_exclusive_group(required=True)
    input_group.add_argument("--id", action="append", dest="arxiv_ids", help="arXiv ID（可多次指定）")
    input_group.add_argument("--id-list", help="包含 arXiv ID 列表的文本文件")
    input_group.add_argument("--metadata", help="包含论文元数据的 JSON 文件")

    # 输出选项
    parser.add_argument("--output-dir", "-o", default="output/pdfs", help="输出目录 (默认: output/pdfs)")
    parser.add_argument("--skip-existing", action="store_true", default=None, help="跳过已存在的文件 (默认)")
    parser.add_argument("--no-skip-existing", action="store_false", dest="skip_existing", help="覆盖已存在的文件")

    # 下载控制
    parser.add_argument("--parallel", type=int, default=3, help="并行下载数 (默认: 3)")
    parser.add_argument("--chunk-size", type=int, default=8192, help="下载块大小字节 (默认: 8192)")
    parser.add_argument("--verify-ssl", action="store_true", default=True, help="验证 SSL 证书 (默认: true)")
    parser.add_argument("--no-verify-ssl", action="store_false", dest="verify_ssl", help="不验证 SSL 证书")

    # 速率限制
    parser.add_argument("--requests-per-second", type=float, default=2.0, help="每秒最大请求数 (默认: 2.0)")
    parser.add_argument("--min-delay", type=float, default=0.5, help="请求间最小延迟秒数 (默认: 0.5)")

    # 日志
    parser.add_argument("--log-level", choices=["DEBUG", "INFO", "WARNING", "ERROR"], default="INFO", help="日志级别 (默认: INFO)")
    parser.add_argument("--log-file", help="日志文件路径")

    args = parser.parse_args()

    # 初始化下载器
    downloader = PaperDownloader(
        output_dir=args.output_dir,
        max_parallel=args.parallel,
        chunk_size=args.chunk_size,
        skip_existing=args.skip_existing if args.skip_existing is not None else True,
        verify_ssl=args.verify_ssl,
        requests_per_second=args.requests_per_second,
        min_delay=args.min_delay,
        log_level=args.log_level,
        log_file=args.log_file,
    )

    logger = get_logger()

    # 收集要下载的论文
    papers = []

    if args.arxiv_ids:
        for arxiv_id in args.arxiv_ids:
            papers.append({"arxiv_id": parse_arxiv_id(arxiv_id)})

    elif args.id_list:
        id_list_path = Path(args.id_list)
        if not id_list_path.exists():
            logger.error(f"ID 列表文件不存在: {args.id_list}")
            return 1

        with open(id_list_path, "r", encoding="utf-8") as f:
            for line in f:
                line = line.strip()
                if line and not line.startswith("#"):
                    papers.append({"arxiv_id": parse_arxiv_id(line)})

    elif args.metadata:
        metadata_path = Path(args.metadata)
        if not metadata_path.exists():
            logger.error(f"元数据文件不存在: {args.metadata}")
            return 1

        metadata = load_json(metadata_path)
        if "papers" in metadata:
            papers = metadata["papers"]
        elif isinstance(metadata, list):
            papers = metadata
        else:
            logger.error("无法识别的元数据格式")
            return 1

    if not papers:
        logger.warning("没有需要下载的论文")
        return 0

    logger.info(f"准备下载 {len(papers)} 篇论文")

    # 执行下载
    results = downloader.download_batch(papers, skip_existing=args.skip_existing)

    # 打印结果
    print("\n==== 下载结果 ====")
    print(f"总计: {results['total']}")
    print(f"成功: {len(results['success'])}")
    print(f"失败: {len(results['failed'])}")
    print(f"跳过: {len(results['skipped'])}")

    if results["failed"]:
        print(f"\n失败的 ID: {', '.join(results['failed'])}")

    return 0 if len(results["failed"]) == 0 else 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/metadata.py
content: #!/usr/bin/env python3
"""
metadata.py - 元数据处理模块
提供元数据提取、格式转换、BibTeX 导出等功能

命令行参数:
  输入方式 (选择一个):
    --id                  单个 arXiv ID
    --input, -i           输入元数据 JSON 文件
    --merge               合并多个元数据文件

  输出选项:
    --format, -f          输出格式: json, bibtex, csv, ris (默认: json)
    --output, -o          输出文件路径

  日志 (可选):
    --log-level           日志级别: DEBUG, INFO, WARNING, ERROR (默认: INFO)
    --log-file            日志文件路径
"""

import argparse
import sys
import csv
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Any, List, Optional, TextIO

# 支持作为模块导入和独立脚本运行
try:
    from .utils import (
        setup_logging,
        get_logger,
        load_json,
        save_json,
        parse_arxiv_id,
    )
    from .search import ArxivSearch
except ImportError:
    from utils import (
        setup_logging,
        get_logger,
        load_json,
        save_json,
        parse_arxiv_id,
    )
    from search import ArxivSearch


class MetadataExtractor:
    """元数据提取器"""

    def __init__(
        self,
        log_level: str = "INFO",
        log_file: str = None,
    ):
        """
        初始化提取器

        Args:
            log_level: 日志级别
            log_file: 日志文件路径
        """
        setup_logging(level=log_level, log_file=log_file)
        self.logger = get_logger()
        self.searcher = ArxivSearch(log_level=log_level, log_file=log_file)

    def get_metadata(self, arxiv_id: str) -> Optional[Dict[str, Any]]:
        """
        获取单个论文的元数据

        Args:
            arxiv_id: arXiv ID

        Returns:
            论文元数据
        """
        results = self.searcher.search_by_id([arxiv_id])
        if results["papers"]:
            return results["papers"][0]
        return None

    def to_bibtex(self, paper: Dict[str, Any]) -> str:
        """
        将论文元数据转换为 BibTeX 格式

        Args:
            paper: 论文元数据

        Returns:
            BibTeX 字符串
        """
        arxiv_id = paper.get("arxiv_id", "")
        title = paper.get("title", "").replace("{", "\\{").replace("}", "\\}")
        authors = paper.get("authors", [])

        author_strs = []
        for author in authors:
            name = author.get("name", "")
            if "," in name:
                author_strs.append(name)
            else:
                parts = name.split()
                if len(parts) > 1:
                    author_strs.append(f"{parts[-1]}, {' '.join(parts[:-1])}")
                else:
                    author_strs.append(name)

        author_field = " and ".join(author_strs)

        first_author = authors[0].get("name", "").split(",")[0].lower() if authors else "unknown"
        year = paper.get("published_date", "")[:4] if paper.get("published_date") else "0000"
        first_word = paper.get("title", "").split()[0].lower() if paper.get("title") else "paper"
        citation_key = f"{first_author}:{year}:{first_word}"

        bibtex = f"@article{{{citation_key},\n"
        bibtex += f"  title = {{{title}}},\n"
        bibtex += f"  author = {{{author_field}}},\n"

        if paper.get("published_date"):
            bibtex += f"  year = {{{paper['published_date'][:4]}}},\n"
            bibtex += f"  month = {{{paper['published_date'][5:7]}}},\n"

        if paper.get("primary_category"):
            bibtex += f"  journal = {{arXiv preprint {paper['arxiv_id']}}},\n"

        bibtex += f"  eprint = {{{paper['arxiv_id']}}},\n"
        bibtex += f"  archivePrefix = {{arXiv}},\n"

        if paper.get("primary_category"):
            bibtex += f"  primaryClass = {{{paper['primary_category']}}},\n"

        bibtex += f"  url = {{{paper.get('arxiv_url', '')}}},\n"
        abstract_short = paper.get('abstract', '')[:500]
        if len(paper.get('abstract', '')) > 500:
            abstract_short += "..."
        bibtex += f"  abstract = {{{abstract_short}}}\n"
        bibtex += "}\n"

        return bibtex

    def export_bibtex(self, papers: List[Dict[str, Any]], output_path: str) -> None:
        """批量导出为 BibTeX"""
        output_path = Path(output_path)
        output_path.parent.mkdir(parents=True, exist_ok=True)

        with open(output_path, "w", encoding="utf-8") as f:
            f.write(f"% BibTeX export generated by arXiv Search Skill\n")
            f.write(f"% Generated on: {datetime.now(timezone.utc).isoformat()}Z\n")
            f.write(f"% Total papers: {len(papers)}\n\n")

            for paper in papers:
                bibtex = self.to_bibtex(paper)
                f.write(bibtex)
                f.write("\n")

        self.logger.info(f"BibTeX 已导出到: {output_path}")

    def to_ris(self, paper: Dict[str, Any]) -> str:
        """转换为 RIS 格式"""
        ris = "TY  - JOUR\n"

        if paper.get("title"):
            ris += f"T1  - {paper['title']}\n"

        for author in paper.get("authors", []):
            ris += f"AU  - {author.get('name', '')}\n"

        if paper.get("published_date"):
            date_parts = paper["published_date"].split("-")
            py_line = f"PY  - {date_parts[0]}"
            if len(date_parts) > 1:
                py_line += f"/{date_parts[1]}"
            if len(date_parts) > 2:
                py_line += f"/{date_parts[2]}"
            ris += py_line + "\n"

        if paper.get("abstract"):
            ris += f"AB  - {paper['abstract']}\n"

        ris += f"AN  - {paper.get('arxiv_id', '')}\n"
        ris += f"UR  - {paper.get('arxiv_url', '')}\n"

        if paper.get("categories"):
            ris += f"KW  - {', '.join(paper['categories'])}\n"

        ris += "ER  - \n\n"
        return ris

    def export_ris(self, papers: List[Dict[str, Any]], output_path: str) -> None:
        """导出 RIS 格式"""
        output_path = Path(output_path)
        output_path.parent.mkdir(parents=True, exist_ok=True)

        with open(output_path, "w", encoding="utf-8") as f:
            for paper in papers:
                f.write(self.to_ris(paper))

        self.logger.info(f"RIS 已导出到: {output_path}")

    def export_csv(self, papers: List[Dict[str, Any]], output_path: str) -> None:
        """导出 CSV 格式"""
        output_path = Path(output_path)
        output_path.parent.mkdir(parents=True, exist_ok=True)

        fieldnames = [
            "arxiv_id",
            "title",
            "authors",
            "published_date",
            "updated_date",
            "primary_category",
            "categories",
            "arxiv_url",
            "pdf_url",
            "doi",
            "abstract",
        ]

        with open(output_path, "w", encoding="utf-8", newline="") as f:
            writer = csv.DictWriter(f, fieldnames=fieldnames)
            writer.writeheader()

            for paper in papers:
                row = {
                    "arxiv_id": paper.get("arxiv_id", ""),
                    "title": paper.get("title", ""),
                    "authors": "; ".join([a.get("name", "") for a in paper.get("authors", [])]),
                    "published_date": paper.get("published_date", ""),
                    "updated_date": paper.get("updated_date", ""),
                    "primary_category": paper.get("primary_category", ""),
                    "categories": ", ".join(paper.get("categories", [])),
                    "arxiv_url": paper.get("arxiv_url", ""),
                    "pdf_url": paper.get("pdf_url", ""),
                    "doi": paper.get("doi", ""),
                    "abstract": paper.get("abstract", "")[:1000],
                }
                writer.writerow(row)

        self.logger.info(f"CSV 已导出到: {output_path}")

    def merge_metadata(self, metadata_files: List[str], dedup_by: str = "arxiv_id") -> Dict[str, Any]:
        """合并多个元数据文件"""
        all_papers = []
        seen_ids = set()

        for filepath in metadata_files:
            try:
                data = load_json(filepath)
                papers = data.get("papers", []) if isinstance(data, dict) else data

                for paper in papers:
                    paper_id = paper.get(dedup_by)
                    if paper_id and paper_id not in seen_ids:
                        seen_ids.add(paper_id)
                        all_papers.append(paper)

            except Exception as e:
                self.logger.warning(f"无法加载文件 {filepath}: {e}")

        self.logger.info(f"合并完成，共 {len(all_papers)} 篇论文（去重后）")

        return {
            "search_metadata": {
                "merged_from": [str(Path(f).name) for f in metadata_files],
                "timestamp": datetime.now(timezone.utc).isoformat() + "Z",
                "returned_results": len(all_papers),
            },
            "papers": all_papers,
        }


def main():
    """命令行入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 元数据处理工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  # 获取单个论文元数据
  python metadata.py --id 2301.01234

  # 从元数据文件导出 BibTeX
  python metadata.py --input results.json --format bibtex --output refs.bib

  # 导出 CSV
  python metadata.py --input results.json --format csv --output papers.csv

  # 合并多个元数据文件
  python metadata.py --merge result1.json result2.json --output merged.json
        """
    )

    # 输入方式
    input_group = parser.add_mutually_exclusive_group(required=True)
    input_group.add_argument("--id", help="单个 arXiv ID")
    input_group.add_argument("--input", "-i", help="输入元数据 JSON 文件")
    input_group.add_argument("--merge", nargs="+", help="合并多个元数据文件")

    # 输出格式
    parser.add_argument(
        "--format", "-f",
        choices=["json", "bibtex", "csv", "ris"],
        default="json",
        help="输出格式 (默认: json)",
    )
    parser.add_argument("--output", "-o", help="输出文件路径")

    # 日志
    parser.add_argument("--log-level", choices=["DEBUG", "INFO", "WARNING", "ERROR"], default="INFO", help="日志级别 (默认: INFO)")
    parser.add_argument("--log-file", help="日志文件路径")

    args = parser.parse_args()

    extractor = MetadataExtractor(
        log_level=args.log_level,
        log_file=args.log_file,
    )

    logger = get_logger()

    try:
        papers = []

        if args.id:
            paper = extractor.get_metadata(args.id)
            if paper:
                papers = [paper]
            else:
                logger.error(f"无法获取论文: {args.id}")
                return 1

        elif args.input:
            data = load_json(args.input)
            if isinstance(data, dict) and "papers" in data:
                papers = data["papers"]
            elif isinstance(data, list):
                papers = data
            else:
                logger.error("无法识别的输入格式")
                return 1

        elif args.merge:
            merged = extractor.merge_metadata(args.merge)
            papers = merged["papers"]

            if args.format == "json" and args.output:
                save_json(merged, args.output)
                logger.info(f"合并结果已保存到: {args.output}")

        if not papers:
            logger.warning("没有处理的论文")
            return 0

        if args.format == "bibtex":
            if args.output:
                extractor.export_bibtex(papers, args.output)
            else:
                for paper in papers:
                    print(extractor.to_bibtex(paper))

        elif args.format == "ris":
            if args.output:
                extractor.export_ris(papers, args.output)
            else:
                for paper in papers:
                    print(extractor.to_ris(paper))

        elif args.format == "csv":
            if args.output:
                extractor.export_csv(papers, args.output)
            else:
                logger.error("CSV 格式需要指定 --output 文件")
                return 1

        else:  # json
            output_data = {
                "metadata": {
                    "generated_at": datetime.now(timezone.utc).isoformat() + "Z",
                    "count": len(papers),
                },
                "papers": papers,
            }
            if args.output:
                save_json(output_data, args.output)
                logger.info(f"已保存到: {args.output}")
            else:
                import json
                print(json.dumps(output_data, indent=2, ensure_ascii=False))

        print(f"\n处理完成: {len(papers)} 篇论文")
        return 0

    except Exception as e:
        logger.error(f"执行失败: {e}")
        import traceback
        traceback.print_exc()
        return 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/search.py
content: #!/usr/bin/env python3
"""
search.py - arXiv 检索模块
提供关键词检索、作者检索、分类检索、日期过滤等功能

命令行参数:
  检索条件 (至少提供一个):
    --query, -q          通用查询字符串
    --title, -t          标题关键词
    --abstract, -a       摘要关键词
    --author, -au        作者名
    --category, -c       arXiv 分类 (如 cs.AI, quant-ph)

  日期过滤:
    --start-date, -s     起始日期 (YYYY-MM-DD)
    --end-date, -e       结束日期 (YYYY-MM-DD)

  检索控制 (可选):
    --max-results, -m    最大结果数 (默认: 100)
    --sort-by             排序方式: relevance, lastUpdatedDate, submittedDate (默认: relevance)
    --sort-order          排序顺序: descending, ascending (默认: descending)
    --timeout             请求超时秒数 (默认: 30)
    --retry-attempts      重试次数 (默认: 3)

  速率限制 (可选):
    --requests-per-second 每秒最大请求数 (默认: 2.0)
    --min-delay           请求间最小延迟秒数 (默认: 0.5)

  输出:
    --output, -o          输出元数据 JSON 文件路径
    --id-list             从文件读取 arXiv ID 列表进行检索

  日志 (可选):
    --log-level           日志级别: DEBUG, INFO, WARNING, ERROR (默认: INFO)
    --log-file            日志文件路径
"""

import argparse
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Any, List, Optional

import arxiv

# 支持作为模块导入和独立脚本运行
try:
    from .utils import (
        setup_logging,
        get_logger,
        save_json,
        slugify,
        RateLimiter,
        parse_arxiv_id,
        load_json,
    )
except ImportError:
    from utils import (
        setup_logging,
        get_logger,
        save_json,
        slugify,
        RateLimiter,
        parse_arxiv_id,
        load_json,
    )


class ArxivSearch:
    """arXiv 检索类"""

    def __init__(
        self,
        timeout: int = 30,
        retry_attempts: int = 3,
        requests_per_second: float = 2.0,
        min_delay: float = 0.5,
        log_level: str = "INFO",
        log_file: str = None,
    ):
        """
        初始化检索器

        Args:
            timeout: 请求超时秒数
            retry_attempts: 重试次数
            requests_per_second: 每秒最大请求数
            min_delay: 请求间最小延迟秒数
            log_level: 日志级别
            log_file: 日志文件路径
        """
        setup_logging(level=log_level, log_file=log_file)
        self.logger = get_logger()

        self.rate_limiter = RateLimiter(requests_per_second, min_delay)

        # 初始化 arXiv 客户端
        self.client = arxiv.Client(
            page_size=100,
            delay_seconds=min_delay,
            num_retries=retry_attempts,
        )

    def build_query(
        self,
        query: str = None,
        title: str = None,
        abstract: str = None,
        author: str = None,
        category: str = None,
    ) -> str:
        """
        构建 arXiv 查询字符串

        Args:
            query: 通用查询字符串
            title: 标题关键词
            abstract: 摘要关键词
            author: 作者名
            category: arXiv 分类

        Returns:
            组合后的查询字符串
        """
        conditions = []

        if query:
            conditions.append(f"({query})")

        if title:
            title_terms = " AND ".join([f'ti:"{term}"' for term in title.split()])
            conditions.append(f"({title_terms})")

        if abstract:
            abstract_terms = " AND ".join([f'abs:"{term}"' for term in abstract.split()])
            conditions.append(f"({abstract_terms})")

        if author:
            author = author.strip()
            if "," in author:
                conditions.append(f'au:"{author}"')
            else:
                conditions.append(f'au:"{author}"')

        if category:
            categories = [c.strip() for c in category.split(",")]
            if len(categories) == 1:
                conditions.append(f"cat:{categories[0]}")
            else:
                cat_conditions = " OR ".join([f"cat:{c}" for c in categories])
                conditions.append(f"({cat_conditions})")

        if not conditions:
            raise ValueError("至少需要提供一个检索条件")

        return " AND ".join(conditions)

    def _convert_result(self, paper: arxiv.Result) -> Dict[str, Any]:
        """
        将 arxiv.Result 转换为标准化字典
        """
        arxiv_id = parse_arxiv_id(paper.entry_id)

        return {
            "arxiv_id": arxiv_id,
            "arxiv_url": paper.entry_id,
            "pdf_url": paper.pdf_url,
            "title": paper.title.strip(),
            "authors": [{"name": author.name} for author in paper.authors],
            "abstract": paper.summary.strip(),
            "categories": paper.categories,
            "primary_category": paper.primary_category,
            "published_date": paper.published.strftime("%Y-%m-%d") if paper.published else None,
            "updated_date": paper.updated.strftime("%Y-%m-%d") if paper.updated else None,
            "doi": paper.doi,
            "journal_ref": paper.journal_ref,
            "comments": paper.comment,
        }

    def _filter_by_date(
        self,
        papers: List[Dict[str, Any]],
        start_date: str = None,
        end_date: str = None,
    ) -> List[Dict[str, Any]]:
        """按日期范围过滤论文"""
        if not start_date and not end_date:
            return papers

        filtered = []
        for paper in papers:
            pub_date = paper.get("published_date") or paper.get("updated_date")
            if not pub_date:
                continue

            try:
                paper_date = datetime.strptime(pub_date, "%Y-%m-%d").date()

                if start_date:
                    start = datetime.strptime(start_date, "%Y-%m-%d").date()
                    if paper_date < start:
                        continue

                if end_date:
                    end = datetime.strptime(end_date, "%Y-%m-%d").date()
                    if paper_date > end:
                        continue

                filtered.append(paper)
            except (ValueError, TypeError):
                continue

        return filtered

    def search(
        self,
        query: str = None,
        title: str = None,
        abstract: str = None,
        author: str = None,
        category: str = None,
        start_date: str = None,
        end_date: str = None,
        max_results: int = 100,
        sort_by: str = "relevance",
        sort_order: str = "descending",
    ) -> Dict[str, Any]:
        """
        执行检索

        Args:
            query: 通用查询字符串
            title: 标题关键词
            abstract: 摘要关键词
            author: 作者名
            category: arXiv 分类
            start_date: 起始日期 (YYYY-MM-DD)
            end_date: 结束日期 (YYYY-MM-DD)
            max_results: 最大结果数 (默认: 100)
            sort_by: 排序方式: relevance, lastUpdatedDate, submittedDate (默认: relevance)
            sort_order: 排序顺序: descending, ascending (默认: descending)

        Returns:
            检索结果字典
        """
        # 映射排序参数
        sort_by_map = {
            "relevance": arxiv.SortCriterion.Relevance,
            "lastUpdatedDate": arxiv.SortCriterion.LastUpdatedDate,
            "submittedDate": arxiv.SortCriterion.SubmittedDate,
        }

        sort_order_map = {
            "descending": arxiv.SortOrder.Descending,
            "ascending": arxiv.SortOrder.Ascending,
        }

        # 构建查询
        query_str = self.build_query(
            query=query,
            title=title,
            abstract=abstract,
            author=author,
            category=category,
        )

        self.logger.info(f"执行检索: {query_str}")

        # 创建检索对象
        search = arxiv.Search(
            query=query_str,
            max_results=max_results * 2,
            sort_by=sort_by_map.get(sort_by, arxiv.SortCriterion.Relevance),
            sort_order=sort_order_map.get(sort_order, arxiv.SortOrder.Descending),
        )

        # 执行检索
        papers = []
        try:
            self.rate_limiter.wait()

            for i, result in enumerate(self.client.results(search)):
                if i >= max_results * 2:
                    break
                papers.append(self._convert_result(result))

        except Exception as e:
            self.logger.error(f"检索失败: {e}")
            raise

        # 日期过滤
        if start_date or end_date:
            papers = self._filter_by_date(papers, start_date, end_date)
            self.logger.info(f"日期过滤后剩余 {len(papers)} 篇论文")

        # 限制结果数
        papers = papers[:max_results]

        self.logger.info(f"检索完成，返回 {len(papers)} 篇论文")

        return {
            "search_metadata": {
                "query": query_str,
                "original_query": {
                    "query": query,
                    "title": title,
                    "abstract": abstract,
                    "author": author,
                    "category": category,
                    "start_date": start_date,
                    "end_date": end_date,
                },
                "timestamp": datetime.now(timezone.utc).isoformat() + "Z",
                "returned_results": len(papers),
            },
            "papers": papers,
        }

    def search_by_id(self, arxiv_ids: List[str]) -> Dict[str, Any]:
        """
        通过 arXiv ID 列表检索论文

        Args:
            arxiv_ids: arXiv ID 列表

        Returns:
            检索结果
        """
        papers = []
        failed_ids = []

        for arxiv_id in arxiv_ids:
            try:
                self.rate_limiter.wait()
                normalized_id = parse_arxiv_id(arxiv_id)
                search = arxiv.Search(id_list=[normalized_id])

                for result in self.client.results(search):
                    papers.append(self._convert_result(result))
                    break

            except Exception as e:
                self.logger.warning(f"获取 {arxiv_id} 失败: {e}")
                failed_ids.append(arxiv_id)

        return {
            "search_metadata": {
                "query": "id_list",
                "requested_ids": arxiv_ids,
                "failed_ids": failed_ids,
                "timestamp": datetime.now(timezone.utc).isoformat() + "Z",
                "returned_results": len(papers),
            },
            "papers": papers,
        }


def main():
    """命令行入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 论文检索工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  # 基础关键词检索
  python search.py --query "quantum computing" --max-results 50

  # 标题检索
  python search.py --title "transformer" --max-results 30

  # 作者检索
  python search.py --author "Smith, J" --max-results 100

  # 分类检索
  python search.py --category "cs.AI" --start-date "2023-01-01"

  # 组合检索并保存结果
  python search.py --query "deep learning" --category "cs.LG" --max-results 50 --output results.json
        """
    )

    # 检索条件
    parser.add_argument("--query", "-q", help="通用查询字符串")
    parser.add_argument("--title", "-t", help="标题关键词")
    parser.add_argument("--abstract", "-a", help="摘要关键词")
    parser.add_argument("--author", "-au", help="作者名")
    parser.add_argument("--category", "-c", help="arXiv 分类 (如 cs.AI, quant-ph)")

    # 日期过滤
    parser.add_argument("--start-date", "-s", help="起始日期 (YYYY-MM-DD)")
    parser.add_argument("--end-date", "-e", help="结束日期 (YYYY-MM-DD)")

    # 检索控制
    parser.add_argument("--max-results", "-m", type=int, default=100, help="最大结果数 (默认: 100)")
    parser.add_argument(
        "--sort-by",
        choices=["relevance", "lastUpdatedDate", "submittedDate"],
        default="relevance",
        help="排序方式 (默认: relevance)",
    )
    parser.add_argument(
        "--sort-order",
        choices=["descending", "ascending"],
        default="descending",
        help="排序顺序 (默认: descending)",
    )
    parser.add_argument("--timeout", type=int, default=30, help="请求超时秒数 (默认: 30)")
    parser.add_argument("--retry-attempts", type=int, default=3, help="重试次数 (默认: 3)")

    # 速率限制
    parser.add_argument("--requests-per-second", type=float, default=2.0, help="每秒最大请求数 (默认: 2.0)")
    parser.add_argument("--min-delay", type=float, default=0.5, help="请求间最小延迟秒数 (默认: 0.5)")

    # 输入输出
    parser.add_argument("--id-list", help="从文件读取 arXiv ID 列表进行检索")
    parser.add_argument("--output", "-o", help="输出元数据 JSON 文件路径")

    # 日志
    parser.add_argument("--log-level", choices=["DEBUG", "INFO", "WARNING", "ERROR"], default="INFO", help="日志级别 (默认: INFO)")
    parser.add_argument("--log-file", help="日志文件路径")

    args = parser.parse_args()

    # 初始化检索器
    searcher = ArxivSearch(
        timeout=args.timeout,
        retry_attempts=args.retry_attempts,
        requests_per_second=args.requests_per_second,
        min_delay=args.min_delay,
        log_level=args.log_level,
        log_file=args.log_file,
    )

    logger = get_logger()

    try:
        if args.id_list:
            # 从 ID 列表检索
            id_list_path = Path(args.id_list)
            if not id_list_path.exists():
                logger.error(f"ID 列表文件不存在: {args.id_list}")
                return 1

            arxiv_ids = []
            with open(id_list_path, "r", encoding="utf-8") as f:
                for line in f:
                    line = line.strip()
                    if line and not line.startswith("#"):
                        arxiv_ids.append(line)

            logger.info(f"从列表加载 {len(arxiv_ids)} 个 arXiv ID")
            results = searcher.search_by_id(arxiv_ids)

        else:
            # 执行条件检索
            if not any([args.query, args.title, args.abstract, args.author, args.category]):
                logger.error("至少需要提供一个检索条件 (--query, --title, --abstract, --author, --category)")
                parser.print_help()
                return 1

            results = searcher.search(
                query=args.query,
                title=args.title,
                abstract=args.abstract,
                author=args.author,
                category=args.category,
                start_date=args.start_date,
                end_date=args.end_date,
                max_results=args.max_results,
                sort_by=args.sort_by,
                sort_order=args.sort_order,
            )

        # 输出结果
        if args.output:
            save_json(results, args.output)
            logger.info(f"结果已保存到: {args.output}")
        else:
            import json
            print(json.dumps(results, indent=2, ensure_ascii=False))

        # 打印摘要信息
        print(f"\n==== 检索摘要 ====")
        print(f"返回论文数: {results['search_metadata']['returned_results']}")
        if results["papers"]:
            print(f"\n前 5 篇论文:")
            for i, paper in enumerate(results["papers"][:5]):
                title = paper["title"][:80] + "..." if len(paper["title"]) > 80 else paper["title"]
                print(f"  {i+1}. [{paper['arxiv_id']}] {title}")

        return 0

    except Exception as e:
        logger.error(f"执行失败: {e}")
        import traceback
        traceback.print_exc()
        return 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/summarize.py
content: #!/usr/bin/env python3
"""
summarize.py - 论文摘要/总结模块
提供论文元数据分析、摘要生成、文献综述辅助等功能

命令行参数:
  输入方式 (选择一个):
    --id                  单个 arXiv ID
    --metadata            包含论文元数据的 JSON 文件

  输出选项:
    --output, -o          输出文件路径
    --output-dir          输出目录（用于批量模式，默认: output/summaries）

  功能选项:
    --overview            生成文献综述概览
    --topic               研究主题（用于综述）

  日志 (可选):
    --log-level           日志级别: DEBUG, INFO, WARNING, ERROR (默认: INFO)
    --log-file            日志文件路径
"""

import argparse
import sys
import re
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Any, List, Optional
from collections import Counter, defaultdict

# 支持作为模块导入和独立脚本运行
try:
    from .utils import (
        setup_logging,
        get_logger,
        ensure_dir,
        load_json,
        save_json,
        parse_arxiv_id,
    )
    from .search import ArxivSearch
except ImportError:
    from utils import (
        setup_logging,
        get_logger,
        ensure_dir,
        load_json,
        save_json,
        parse_arxiv_id,
    )
    from search import ArxivSearch


class PaperSummarizer:
    """论文摘要生成器"""

    def __init__(
        self,
        output_dir: str = "output/summaries",
        log_level: str = "INFO",
        log_file: str = None,
    ):
        """
        初始化摘要生成器

        Args:
            output_dir: 输出目录 (默认: output/summaries)
            log_level: 日志级别
            log_file: 日志文件路径
        """
        setup_logging(level=log_level, log_file=log_file)
        self.logger = get_logger()

        self.output_dir = Path(output_dir)
        ensure_dir(self.output_dir)

        self.searcher = ArxivSearch(log_level=log_level, log_file=log_file)

        self.method_keywords = [
            "algorithm", "model", "method", "approach", "framework", "architecture",
            "network", "neural", "deep", "learning", "training", "optimization",
            "regression", "classification", "clustering", "embedding", "attention",
            "transformer", "cnn", "rnn", "lstm", "gan", "vae", "reinforcement",
            "bayesian", "probabilistic", "stochastic", "deterministic", "gradient",
            "backpropagation", "regularization", "normalization", "dropout",
        ]

        self.task_keywords = [
            "classification", "detection", "segmentation", "recognition", "generation",
            "translation", "summarization", "paraphrasing", "question", "answering",
            "reasoning", "inference", "prediction", "forecasting", "recommendation",
            "retrieval", "search", "matching", "alignment", "synthesis", "editing",
            "compression", "denoising", "restoration", "enhancement", "understanding",
        ]

    def extract_key_points(self, abstract: str) -> Dict[str, Any]:
        """
        从摘要中提取关键点

        Args:
            abstract: 摘要文本

        Returns:
            关键点字典
        """
        abstract_lower = abstract.lower()

        methods = []
        for keyword in self.method_keywords:
            pattern = r'\b' + re.escape(keyword) + r'\b'
            if re.search(pattern, abstract_lower):
                methods.append(keyword)

        tasks = []
        for keyword in self.task_keywords:
            pattern = r'\b' + re.escape(keyword) + r'\b'
            if re.search(pattern, abstract_lower):
                tasks.append(keyword)

        sentences = re.split(r'[.!?]+', abstract)
        problem_statement = sentences[0].strip() if sentences else ""

        contributions = []
        contribution_patterns = [
            r'we (?:propose|introduce|present|describe|develop|create) ([^.]+)',
            r'this (?:paper|work|study) ([^.]+)',
            r'we (?:show|demonstrate|prove|find) ([^.]+)',
            r'our (?:method|approach|model|framework) ([^.]+)',
        ]

        for pattern in contribution_patterns:
            matches = re.findall(pattern, abstract, re.IGNORECASE)
            contributions.extend([m.strip() for m in matches if m.strip()])

        return {
            "methods": methods,
            "tasks": tasks,
            "problem_statement": problem_statement,
            "contributions": contributions[:5],
        }

    def summarize_paper(self, paper: Dict[str, Any]) -> Dict[str, Any]:
        """
        生成单篇论文的摘要

        Args:
            paper: 论文元数据

        Returns:
            论文摘要
        """
        abstract = paper.get("abstract", "")
        key_points = self.extract_key_points(abstract) if abstract else {}

        summary = {
            "arxiv_id": paper.get("arxiv_id"),
            "title": paper.get("title"),
            "authors": [a.get("name", "") for a in paper.get("authors", [])],
            "published_date": paper.get("published_date"),
            "categories": paper.get("categories", []),
            "primary_category": paper.get("primary_category"),
            "url": paper.get("arxiv_url"),
            "pdf_url": paper.get("pdf_url"),
            "abstract_short": abstract[:500] + "..." if len(abstract) > 500 else abstract,
            "key_points": key_points,
            "generated_at": datetime.now(timezone.utc).isoformat() + "Z",
        }

        return summary

    def analyze_collection(self, papers: List[Dict[str, Any]]) -> Dict[str, Any]:
        """
        分析论文集合，生成统计和趋势

        Args:
            papers: 论文列表

        Returns:
            集合分析结果
        """
        category_counts = Counter()
        primary_category_counts = Counter()
        for paper in papers:
            for cat in paper.get("categories", []):
                category_counts[cat] += 1
            if paper.get("primary_category"):
                primary_category_counts[paper["primary_category"]] += 1

        year_counts = Counter()
        for paper in papers:
            date = paper.get("published_date") or paper.get("updated_date")
            if date:
                year = date[:4]
                year_counts[year] += 1

        all_methods = Counter()
        all_tasks = Counter()

        for paper in papers:
            abstract = paper.get("abstract", "")
            if abstract:
                key_points = self.extract_key_points(abstract)
                for method in key_points.get("methods", []):
                    all_methods[method] += 1
                for task in key_points.get("tasks", []):
                    all_tasks[task] += 1

        papers_by_year = defaultdict(list)
        for paper in papers:
            date = paper.get("published_date") or paper.get("updated_date")
            if date:
                year = date[:4]
                papers_by_year[year].append(paper)

        return {
            "collection_summary": {
                "total_papers": len(papers),
                "date_range": {
                    "earliest": min([p.get("published_date", "9999-99-99") for p in papers]) if papers else None,
                    "latest": max([p.get("published_date", "0000-00-00") for p in papers]) if papers else None,
                },
            },
            "categories": {
                "primary": dict(primary_category_counts.most_common()),
                "all": dict(category_counts.most_common()),
            },
            "year_distribution": dict(year_counts.most_common()),
            "methods": dict(all_methods.most_common(20)),
            "tasks": dict(all_tasks.most_common(20)),
            "papers_by_year": {
                year: len(papers)
                for year, papers in sorted(papers_by_year.items())
            },
        }

    def generate_literature_overview(
        self,
        papers: List[Dict[str, Any]],
        topic: str = None,
    ) -> Dict[str, Any]:
        """
        生成文献综述概览

        Args:
            papers: 论文列表
            topic: 研究主题

        Returns:
            文献综述
        """
        collection_analysis = self.analyze_collection(papers)
        paper_summaries = [self.summarize_paper(p) for p in papers]

        overview = {
            "topic": topic,
            "generated_at": datetime.now(timezone.utc).isoformat() + "Z",
            "collection_analysis": collection_analysis,
            "paper_summaries": paper_summaries,
            "key_insights": self._generate_key_insights(collection_analysis, paper_summaries),
        }

        return overview

    def _generate_key_insights(
        self,
        analysis: Dict[str, Any],
        summaries: List[Dict[str, Any]],
    ) -> List[str]:
        """生成关键洞察"""
        insights = []

        total = analysis["collection_summary"]["total_papers"]
        insights.append(f"共分析 {total} 篇论文")

        primary_cats = analysis["categories"]["primary"]
        if primary_cats:
            top_cat = list(primary_cats.keys())[0]
            insights.append(f"主要研究领域: {top_cat} ({primary_cats[top_cat]} 篇)")

        methods = analysis["methods"]
        if methods:
            top_methods = list(methods.keys())[:3]
            insights.append(f"常用方法: {', '.join(top_methods)}")

        tasks = analysis["tasks"]
        if tasks:
            top_tasks = list(tasks.keys())[:3]
            insights.append(f"研究任务: {', '.join(top_tasks)}")

        return insights


def main():
    """命令行入口"""
    parser = argparse.ArgumentParser(
        description="arXiv 论文总结工具",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
示例:
  # 总结单篇论文
  python summarize.py --id 2301.01234

  # 从元数据文件批量生成总结
  python summarize.py --metadata results.json

  # 生成文献综述
  python summarize.py --metadata results.json --overview --topic "deep learning"
        """
    )

    # 输入方式
    input_group = parser.add_mutually_exclusive_group(required=True)
    input_group.add_argument("--id", help="单个 arXiv ID")
    input_group.add_argument("--metadata", help="包含论文元数据的 JSON 文件")

    # 输出选项
    parser.add_argument("--output", "-o", help="输出文件路径")
    parser.add_argument("--output-dir", default="output/summaries", help="输出目录（用于批量模式，默认: output/summaries）")

    # 功能选项
    parser.add_argument("--overview", action="store_true", help="生成文献综述概览")
    parser.add_argument("--topic", help="研究主题（用于综述）")

    # 日志
    parser.add_argument("--log-level", choices=["DEBUG", "INFO", "WARNING", "ERROR"], default="INFO", help="日志级别 (默认: INFO)")
    parser.add_argument("--log-file", help="日志文件路径")

    args = parser.parse_args()

    summarizer = PaperSummarizer(
        output_dir=args.output_dir,
        log_level=args.log_level,
        log_file=args.log_file,
    )

    logger = get_logger()

    try:
        papers = []

        if args.id:
            paper = summarizer.searcher.search_by_id([args.id])["papers"]
            if paper:
                papers = paper
            else:
                logger.error(f"无法获取论文: {args.id}")
                return 1

        elif args.metadata:
            data = load_json(args.metadata)
            if isinstance(data, dict) and "papers" in data:
                papers = data["papers"]
            elif isinstance(data, list):
                papers = data
            else:
                logger.error("无法识别的输入格式")
                return 1

        if not papers:
            logger.warning("没有要处理的论文")
            return 0

        if args.overview:
            overview = summarizer.generate_literature_overview(papers, topic=args.topic)

            if args.output:
                save_json(overview, args.output)
                logger.info(f"综述已保存到: {args.output}")
            else:
                import json
                print(json.dumps(overview, indent=2, ensure_ascii=False))

            print("\n==== 文献综述摘要 ====")
            print(f"论文总数: {overview['collection_analysis']['collection_summary']['total_papers']}")
            print("\n关键洞察:")
            for insight in overview["key_insights"]:
                print(f"  - {insight}")

        else:
            if len(papers) == 1:
                summary = summarizer.summarize_paper(papers[0])
                if args.output:
                    save_json(summary, args.output)
                    logger.info(f"总结已保存到: {args.output}")
                else:
                    import json
                    print(json.dumps(summary, indent=2, ensure_ascii=False))
            else:
                output_dir = Path(args.output_dir) if args.output_dir else summarizer.output_dir
                ensure_dir(output_dir)

                summaries = []
                for paper in papers:
                    summary = summarizer.summarize_paper(paper)
                    summaries.append(summary)

                    arxiv_id = summary["arxiv_id"]
                    paper_file = output_dir / f"{arxiv_id}_summary.json"
                    save_json(summary, paper_file)

                if args.output:
                    save_json(summaries, args.output)
                    logger.info(f"汇总已保存到: {args.output}")

                print(f"\n已生成 {len(summaries)} 篇论文的总结")

        return 0

    except Exception as e:
        logger.error(f"执行失败: {e}")
        import traceback
        traceback.print_exc()
        return 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/utils.py
content: """
utils.py - 工具函数模块
提供通用辅助函数、日志设置等
"""

import os
import re
import time
import logging
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Any, List, Optional
from urllib.parse import urlparse

# 可选依赖
try:
    import colorlog
    HAS_COLORLOG = True
except ImportError:
    HAS_COLORLOG = False


def setup_logging(level: str = "INFO", log_file: str = None) -> logging.Logger:
    """
    设置日志配置

    Args:
        level: 日志级别 (DEBUG, INFO, WARNING, ERROR)
        log_file: 日志文件路径，如 None 则不记录到文件

    Returns:
        配置好的 logger 实例
    """
    log_format = "[%(levelname)s] %(asctime)s - %(message)s"
    date_format = "%Y-%m-%d %H:%M:%S"

    logger = logging.getLogger("arxiv_search")
    logger.setLevel(getattr(logging, level.upper()))
    logger.propagate = False

    if logger.handlers:
        logger.handlers.clear()

    # 控制台输出
    if HAS_COLORLOG:
        console_formatter = colorlog.ColoredFormatter(
            "%(log_color)s" + log_format,
            datefmt=date_format,
            log_colors={
                "DEBUG": "cyan",
                "INFO": "green",
                "WARNING": "yellow",
                "ERROR": "red",
                "CRITICAL": "red,bg_white",
            }
        )
    else:
        console_formatter = logging.Formatter(log_format, datefmt=date_format)

    console_handler = logging.StreamHandler()
    console_handler.setFormatter(console_formatter)
    logger.addHandler(console_handler)

    # 文件输出
    if log_file:
        log_path = Path(log_file)
        log_path.parent.mkdir(parents=True, exist_ok=True)

        file_formatter = logging.Formatter(log_format, datefmt=date_format)
        file_handler = logging.FileHandler(log_path, encoding="utf-8")
        file_handler.setFormatter(file_formatter)
        logger.addHandler(file_handler)

    return logger


def get_logger() -> logging.Logger:
    """
    获取全局 logger，如果未配置则使用默认配置

    Returns:
        logger 实例
    """
    logger = logging.getLogger("arxiv_search")
    if not logger.handlers:
        setup_logging()
    return logger


def deep_update(base_dict: Dict[str, Any], update_dict: Dict[str, Any]) -> None:
    """
    深度更新字典

    Args:
        base_dict: 基础字典
        update_dict: 更新字典
    """
    for key, value in update_dict.items():
        if key in base_dict and isinstance(base_dict[key], dict) and isinstance(value, dict):
            deep_update(base_dict[key], value)
        else:
            base_dict[key] = value


def slugify(text: str, max_length: int = 80) -> str:
    """
    将文本转换为 URL 友好的 slug

    Args:
        text: 输入文本
        max_length: 最大长度

    Returns:
        slug 化的文本
    """
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s-]", "", text)
    text = re.sub(r"[\s_-]+", "-", text)
    text = text.strip("-")
    return text[:max_length]


def sanitize_filename(filename: str) -> str:
    """
    清理文件名，移除非法字符

    Args:
        filename: 原始文件名

    Returns:
        清理后的文件名
    """
    invalid_chars = '<>:"/\\|?*'
    for char in invalid_chars:
        filename = filename.replace(char, "_")
    filename = filename.strip()
    filename = filename[:200]
    return filename


def ensure_dir(path: str) -> Path:
    """
    确保目录存在，不存在则创建

    Args:
        path: 目录路径

    Returns:
        Path 对象
    """
    dir_path = Path(path)
    dir_path.mkdir(parents=True, exist_ok=True)
    return dir_path


def parse_arxiv_id(arxiv_id: str) -> str:
    """
    解析 arXiv ID，支持多种格式

    Args:
        arxiv_id: arXiv ID 或 URL

    Returns:
        标准化的 arXiv ID

    Examples:
        >>> parse_arxiv_id("2301.01234")
        "2301.01234"
        >>> parse_arxiv_id("https://arxiv.org/abs/2301.01234v1")
        "2301.01234"
    """
    # 如果是 URL，提取 ID
    if arxiv_id.startswith(("http://", "https://")):
        parsed = urlparse(arxiv_id)
        path = parsed.path
        # 匹配 /abs/xxx 或 /pdf/xxx.pdf - 支持带点和斜杠的旧式 ID
        # 新模式: 2301.01234, 旧式: hep-th/9901001
        match = re.search(r"/(?:abs|pdf)/([^?#]+?)(?:v\d+)?(?:\.pdf)?$", path)
        if match:
            arxiv_id = match.group(1)

    # 移除版本号 (v1, v2 等) - 只在末尾匹配
    arxiv_id = re.sub(r"v\d+$", "", arxiv_id)

    return arxiv_id


def save_json(data: Any, filepath: str, indent: int = 2, ensure_ascii: bool = False) -> None:
    """
    保存数据为 JSON 文件

    Args:
        data: 要保存的数据
        filepath: 文件路径
        indent: 缩进空格数
        ensure_ascii: 是否确保 ASCII
    """
    filepath = Path(filepath)
    filepath.parent.mkdir(parents=True, exist_ok=True)

    with open(filepath, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=indent, ensure_ascii=ensure_ascii, default=str)


def load_json(filepath: str) -> Any:
    """
    从 JSON 文件加载数据

    Args:
        filepath: 文件路径

    Returns:
        加载的数据
    """
    with open(filepath, "r", encoding="utf-8") as f:
        return json.load(f)


def load_jsonl(filepath: str) -> List[Any]:
    """
    从 JSONL 文件加载数据

    Args:
        filepath: 文件路径

    Returns:
        数据列表
    """
    results = []
    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                results.append(json.loads(line))
    return results


class RateLimiter:
    """
    简单的速率限制器
    """

    def __init__(self, requests_per_second: float = 2.0, min_delay: float = 0.5):
        """
        初始化速率限制器

        Args:
            requests_per_second: 每秒请求数
            min_delay: 最小延迟（秒）
        """
        self.min_interval = 1.0 / requests_per_second if requests_per_second > 0 else 0
        self.min_interval = max(self.min_interval, min_delay)
        self.last_request_time = 0.0

    def wait(self) -> None:
        """等待直到可以发送下一个请求"""
        current_time = time.time()
        elapsed = current_time - self.last_request_time
        if elapsed < self.min_interval:
            time.sleep(self.min_interval - elapsed)
        self.last_request_time = time.time()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)