Skill: tartykisser/ai-paper-researcher (version 1.0.0)

--- name: ai-paper-researcher description: An arXiv paper search engine designed specifically for scientific researchers. Supports dual modes: "Broad Search" and "Top-Tier Strict Filtering", along with relevance and date sorting. Automatically checks for duplicates, downloads PDFs, and maintains a local CSV paper library. --- # AI Academic Paper Researcher ## 1. Skill Positioning & Core Objective This skill aims to assist researchers in the AI field by searching for arXiv literature and automating PDF downloads and local file management. **Core Principle:** All download records must rely on the local `workspace/paper_list/paper_list.csv` for deduplication to prevent repeated downloads. ## 2. Tools & Dependencies - **Execution Script:** `python arxiv_tool.py` - **Target Conference List:** The `target.csv` file located in the same directory as this skill (contains the names of top-tier conferences or journals the user follows, e.g., CVPR, NeurIPS, ICLR). ## 3. Sorting Strategy Selection Before executing any search, you must decide which sorting parameter (`--sort`) to use based on the user's intent: - **Searching for Classic Theories / Well-known Algorithms (Classic/Influential):** If the user searches for specific well-known algorithms (e.g., "Adam", "ResNet") or foundational papers in core fields, you **MUST use `--sort relevance`**. Otherwise, because arXiv defaults to returning a large number of newly submitted papers, classic older papers will be pushed out of the search results. - **Tracking Latest Frontiers (Latest Trends):** If the user explicitly requests "latest", "this year", or "recent weeks" papers, please use `--sort date`. ## 4. Two Retrieval Modes Infer the required mode based on the user's query: ### Mode A: Broad Search (All Relevant Mode) **Trigger Condition:** The user only provides a research direction without restricting the papers to be published in top-tier conferences. **Execution Logic:** 1. Run `python arxiv_tool.py search "[query]" --max 15 --sort [selected sorting strategy]`. 2. Ignore the `comment` field in the JSON response. 3. Exclude papers where `is_downloaded: true` in the results. 4. Select the papers most relevant to the user's needs and proceed directly to the download process. ### Mode B: Top-Tier Conference/Journal Strict Filtering (Top-Tier Verification Mode) **Trigger Condition:** The user explicitly requests "top-tier conferences", "top journals", or specifies certain conferences (e.g., "Help me find Adam-related papers from past ICLR conferences"). **Execution Logic:** 1. **Read Target List:** Use the file reading tool to view the contents of `target.csv` to get the list of target conferences/journals. 2. **Initial Search:** Run `python arxiv_tool.py search "[query]" --max 30 --sort [selected sorting strategy]`. *(Note: The script automatically fetches the latest version of the paper, so if it has been accepted by a top conference, the comment will contain the relevant information.)* 3. **LLM Semantic Verification (CRITICAL):** - Carefully review the `comment` field in the JSON of each candidate paper. - Determine whether any conference listed in `target.csv` is present in the `comment`. - **Note on Variations:** Be tolerant of abbreviations, year suffixes, or non-standard formatting of conference names when matching (e.g., `Accepted to ICLR 2015`, `NeurIPS'23`, `Appears in CVPR`). As long as it semantically refers to the target conference, consider it a successful match. - If the `comment` is empty, or does not contain a publication statement for the target conference, you **MUST exclude** the paper. 4. Exclude already downloaded papers (`is_downloaded: true`). 5. Proceed to the download process for the successfully verified papers. ## 5. Download & File Persistence 1. For the filtered papers, execute the download command one by one: `python arxiv_tool.py download [arxiv_id]`. 2. Collect the script's return results. ## 6. Reporting Standard After completing the search and download, report the final results to the user: - Explicitly state which retrieval mode was used (Mode A/B) and which sorting method (Date/Relevance). - List the successfully downloaded papers (Format: `[ArXiv ID] Title - (Matched conference, if any)`).

path: arxiv_tool.py
content: #!/usr/bin/env python3
"""
arXiv Research Assistant (Local File System Version)
"""

import arxiv
import argparse
import json
import os
import csv
import re
import requests
import time
from pathlib import Path

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
WORKSPACE_DIR = os.path.abspath(os.path.join(SCRIPT_DIR, '..', '..', 'paper_list'))
CSV_PATH = os.path.join(WORKSPACE_DIR, 'paper_list.csv')

class LocalArxivManager:
    def __init__(self):
        self.client = arxiv.Client()
        self._init_workspace()

    def _init_workspace(self):
        Path(WORKSPACE_DIR).mkdir(parents=True, exist_ok=True)
        if not os.path.exists(CSV_PATH):
            with open(CSV_PATH, mode='w', newline='', encoding='utf-8') as f:
                writer = csv.writer(f)
                writer.writerow(['arxiv ID', 'title', 'abstract'])

    def _get_downloaded_ids(self) -> set:
        downloaded = set()
        if os.path.exists(CSV_PATH):
            with open(CSV_PATH, mode='r', encoding='utf-8') as f:
                reader = csv.DictReader(f)
                for row in reader:
                    downloaded.add(row['arxiv ID'])
        return downloaded

    def _sanitize_filename(self, text: str) -> str:
        text = re.sub(r'[\\/*?:"<>|]', "", text)
        text = text.replace('\n', ' ').strip()
        return text

    def search(self, query: str, max_results: int = 15, sort_by: str = "relevance") -> list:
        
        sort_criterion = arxiv.SortCriterion.Relevance
        if sort_by == "date":
            sort_criterion = arxiv.SortCriterion.SubmittedDate

        ai_categories = "(cat:cs.AI OR cat:cs.CV OR cat:cs.LG OR cat:cs.CL OR cat:cs.NE OR cat:cs.RO OR cat:stat.ML)"
        
        if "cat:" not in query:
            full_query = f"({query}) AND {ai_categories}"
        else:
            full_query = query

        search = arxiv.Search(
            query=full_query,
            max_results=max_results,
            sort_by=sort_criterion
        )
        
        downloaded_ids = self._get_downloaded_ids()
        results = []
        
        for paper in self.client.results(search):
            paper_id = paper.entry_id.split("/")[-1].split("v")[0] 
            results.append({
                "arxiv_id": paper_id,
                "title": paper.title,
                "abstract": paper.summary.replace('\n', ' '),
                "comment": paper.comment if paper.comment else "",
                "published": paper.published.strftime('%Y-%m-%d') if paper.published else "",
                "is_downloaded": paper_id in downloaded_ids
            })
            
        return results

    def download(self, arxiv_id: str) -> dict:
        downloaded_ids = self._get_downloaded_ids()
        
        clean_id = arxiv_id.replace("arXiv:", "").replace("arxiv:", "").split("v")[0]
        
        if clean_id in downloaded_ids:
            return {"status": "skipped", "message": f"Paper {clean_id} is already in paper_list.csv, skip download."}

        search = arxiv.Search(id_list=[clean_id])
        papers = list(self.client.results(search))
        
        if not papers:
            return {"status": "error", "message": f"No papers were found with the ID {clean_id}."}
            
        paper = papers[0]
        sanitized_title = self._sanitize_filename(paper.title)
        filename = f"{clean_id}_{sanitized_title}.pdf"
        
        try:
            pdf_url = paper.pdf_url
            if not pdf_url.endswith('.pdf'):
                pdf_url += '.pdf'
                
            headers = {
                'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
            }
            
            max_retries = 3
            for attempt in range(max_retries):
                time.sleep(3)
                
                response = requests.get(pdf_url, stream=True, timeout=60, headers=headers)
                
                if response.status_code == 429:
                    if attempt < max_retries - 1:
                        wait_time = (attempt + 1) * 10
                        print(f"A 429 rate limit has been triggered. We will wait {wait_time} seconds before attempting the {attempt + 2}th retry...")
                        time.sleep(wait_time)
                        continue
                    else:
                        response.raise_for_status()
                else:
                    response.raise_for_status()
                    break
            
            filepath = os.path.join(WORKSPACE_DIR, filename)
            with open(filepath, 'wb') as f:
                for chunk in response.iter_content(chunk_size=8192):
                    if chunk:
                        f.write(chunk)
            
            with open(CSV_PATH, mode='a', newline='', encoding='utf-8') as f:
                writer = csv.writer(f)
                writer.writerow([clean_id, paper.title, paper.summary.replace('\n', ' ')])
                
            return {
                "status": "success", 
                "message": f"Successful streaming download and recording: {filename}",
                "path": filepath
            }
        except Exception as e:
            filepath = os.path.join(WORKSPACE_DIR, filename)
            if os.path.exists(filepath):
                os.remove(filepath)
            return {"status": "error", "message": f"Streaming download failed: {str(e)}"}

def main():
    parser = argparse.ArgumentParser(description="ArXiv CLI for OpenClaw")
    subparsers = parser.add_subparsers(dest="command")
    
    search_parser = subparsers.add_parser("search")
    search_parser.add_argument("query", help="Search keywords")
    search_parser.add_argument("--max", type=int, default=15, help="Maximum number of results")
    search_parser.add_argument("--sort", choices=["relevance", "date"], default="relevance", help="Sorting method")
    
    dl_parser = subparsers.add_parser("download")
    dl_parser.add_argument("arxiv_id", help="ArXiv ID")
    
    args = parser.parse_args()
    manager = LocalArxivManager()
    
    if args.command == "search":
        results = manager.search(args.query, args.max, args.sort)
        print(json.dumps(results, indent=2, ensure_ascii=False))
        
    elif args.command == "download":
        result = manager.download(args.arxiv_id)
        print(json.dumps(result, ensure_ascii=False))

if __name__ == "__main__":
    main()

path: README.md
content: <div align="center">

# 🧠 AI Academic Paper Researcher

**An intelligent, fully-automated arXiv research assistant designed for AI scientists and enthusiasts. Search, filter by top-tier conferences, download PDFs, and build your local library—all driven by natural language.**

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT-0](https://img.shields.io/badge/License-MIT--0-green.svg)](https://choosealicense.com/licenses/mit-0/)
[![OpenClaw Skill](https://img.shields.io/badge/OpenClaw-Skill-purple.svg)](#)

<br>

[**English**](./README.md) | [**简体中文**](./README_zh-CN.md)
</div>

---

## ✨ Why This Tool?

Navigating the ocean of daily arXiv submissions can be overwhelming. This tool acts as your personal academic scout. Whether you are a professional researcher tracking the latest advancements in specific subfields, or an enthusiast looking for foundational papers, this skill bridges the gap between raw arXiv data and curated, high-signal knowledge.

### 🚀 Key Features

- **🧠 Dual-Mode Intelligence**: 
  - **Broad Search**: Cast a wide net for the latest trends.
  - **Top-Tier Strict Filtering**: Uses LLM semantic verification to ensure papers were actually accepted to your target conferences (e.g., CVPR, NeurIPS, ICLR).
- **⏱️ Smart Sorting**: Automatically toggles between *Date* (for tracking new releases) and *Relevance* (for digging up classic/foundational algorithms).
- **🛡️ Domain Isolation**: Automatically appends AI-specific categories (`cs.CV`, `cs.LG`, `cs.AI`, etc.) to filter out noise from physics or biology.
- **📂 Local Library Management**: Silently downloads PDFs, standardizes file names, and logs everything into a local CSV to prevent duplicate downloads.
- **🛡️ Anti-Ban Protections**: Built-in exponential backoff and polite delays to handle arXiv's strict API rate limits (429 errors).

---

## 🧩 Architecture: How It Works

This skill is composed of several working parts that seamlessly integrate with your AI Agent (like OpenClaw):

| Component | Description | Role |
| :--- | :--- | :--- |
| **`SKILL.md`** | The "Brain" | Instructs the AI agent on how to interpret your queries, choose the right sorting strategy, and verify conference acceptances via the `comment` field. |
| **`arxiv_tool.py`** | The "Muscle" | A robust Python CLI tool that queries the arXiv API, handles HTTP streams, enforces rate limits, and downloads the PDFs. |
| **`target.csv`** | The "Filter" | A simple, user-defined list of top-tier venues you care about (e.g., `CVPR`, `ICCV`, `ICLR`). The LLM checks papers against this list. |
| **`paper_list.csv`** | The "Memory" | Automatically generated in your workspace. Tracks downloaded ArXiv IDs to ensure you never download the same paper twice. |

---

## 🛠️ Setup & Installation

**1. Install Dependencies**
Ensure you have the required Python packages installed in your environment:
```bash
pip install arxiv requests
```

**2. Directory Structure**
Ensure the files are placed correctly within your workspace:

```text
workspace/
├── paper_list/                  # Auto-generated: PDFs and paper_list.csv go here
└── skills/
    └── ai-paper-researcher/
        ├── SKILL.md             # The agent prompt
        ├── arxiv_tool.py        # The python script
        └── target.csv           # Your conference targets (e.g., CVPR, NeurIPS)
```

**3. Configure Targets**
Add your favorite conferences to `target.csv` (one per line):

```csv
target
CVPR
ICCV
NeurIPS
ICLR
```

---

## 🗣️ How to Use (Natural Language Prompting)

Once loaded into your AI agent, you can simply ask it to do the heavy lifting. Here are the best ways to prompt the system:

### 🔭 Scenario 1: Tracking the Latest Frontiers (Mode A)

*Perfect for weekly literature reviews in fast-moving domains.*

> "Invoke the `ai-paper-researcher` skill. Find the latest papers on **Agent security** from the past month. Sort by date, skip the top-tier conference filter, and download the most relevant ones to my workspace."

### 🏛️ Scenario 2: Deep Dive into Top-Tier Classics (Mode B)

*Perfect for finding foundational work that has been peer-reviewed and verified.*

> "Use the `ai-paper-researcher` skill. I want to research **Agent security**. Sort by relevance to find the most impactful papers. Strictly filter the results so you only download papers accepted to the venues listed in my `target.csv`. Give me a summary of what you downloaded."

### 🎯 Scenario 3: Precision Direct Download

*Perfect for grabbing specific papers you already know about without searching.*

> "Using the `ai-paper-researcher` skill, bypass the search and directly download the original Adam optimizer paper (ID: `1412.6980`) and the Focal Loss paper (ID: `1708.02002`)."

---

## 💻 CLI Usage (For Developers)

If you prefer to bypass the AI agent and use the terminal directly, the Python script is a fully functional standalone tool.

```bash
# Search for papers by relevance (returns JSON with comment fields)
python arxiv_tool.py search "diffusion models" --max 10 --sort relevance

# Search for the newest papers
python arxiv_tool.py search "diffusion models" --sort date

# Download a specific paper (handles naming and CSV logging automatically)
python arxiv_tool.py download 1412.6980
```

---

## 📜 Acknowledgments & License

This project was built upon the excellent foundational work from [Ractorrr/arxiv-mcp](https://github.com/Ractorrr/arxiv-mcp) (🔍 AI-powered arXiv research assistant). It has been heavily modified to remove database dependencies (MongoDB), introduce strict local file-system tracking, add LLM-driven semantic conference filtering, and implement robust streaming downloads to bypass arXiv API limitations.

**License:** [MIT License](https://choosealicense.com/licenses/mit/) - Free to use, modify, and distribute without attribution requirements.

path: README_zh-CN.md
content: 
<div align="center">

# 🧠 AI Academic Paper Researcher (AI 学术论文研究助手)

**专为 AI 科学家和爱好者设计的智能、全自动 arXiv 调研助手。通过自然语言驱动，实现搜索、顶会精确过滤、PDF 自动下载及本地文献库构建。**

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://choosealicense.com/licenses/mit/)
[![OpenClaw Skill](https://img.shields.io/badge/OpenClaw-Skill-purple.svg)](#)

<br>

[**English**](./README.md) | [**简体中文**](./README_zh-CN.md)

</div>

---

## ✨ 为什么需要这个工具？

面对每天海量的 arXiv 提交论文，往往让人感到不知所措。这个工具将成为您的私人学术侦察员。无论您是追踪特定子领域最新进展的专业研究人员，还是寻找基础经典论文的爱好者，本技能都能在原始 arXiv 数据和精选的高价值知识之间架起桥梁。

### 🚀 核心特性

- **🧠 双模式智能检索**: 
  - **广泛搜索 (Broad Search)**: 撒网捕捉最新前沿趋势。
  - **顶会严格过滤 (Top-Tier Strict Filtering)**: 使用大语言模型 (LLM) 进行语义校验，确保下载的论文确实被您的目标会议（如 CVPR, NeurIPS, ICLR）接收。
- **⏱️ 智能排序**: 自动在“按时间”（追踪最新发布）和“按相关度”（挖掘经典/基础算法）之间进行智能切换。
- **🛡️ 领域隔离**: 自动追加 AI 专属分类标签（`cs.CV`, `cs.LG`, `cs.AI` 等），精准过滤掉来自物理或生物学等其他领域的重名噪音。
- **📂 本地文献库管理**: 静默流式下载 PDF，标准化规范文件名，并将记录自动追加到本地 CSV 中以防止重复下载。
- **🛡️ 防封禁保护**: 内置指数退避算法（Exponential backoff）和请求延迟，从容应对 arXiv 严格的 API 速率限制（429 报错）。

---

## 🧩 架构：它是如何工作的？

本技能由几个协同工作的核心组件构成，可与您的 AI Agent（如 OpenClaw）无缝集成：

| 组件 | 描述 | 作用 |
| :--- | :--- | :--- |
| **`SKILL.md`** | “大脑” | 指导 AI Agent 如何理解您的查询，选择正确的排序策略，并通过 `comment` 字段校验会议接收情况。 |
| **`arxiv_tool.py`** | “肌肉” | 健壮的 Python CLI 工具，负责请求 arXiv API、处理 HTTP 数据流、强制执行速率限制以及下载 PDF。 |
| **`target.csv`** | “过滤器” | 简单的用户自定义列表，包含您关注的顶会名称（如 `CVPR`, `ICCV`, `ICLR`）。LLM 将依据此列表校验论文。 |
| **`paper_list.csv`** | “记忆” | 在工作区中自动生成。记录已下载的 ArXiv ID，确保您永远不会重复下载同一篇论文。 |

---

## 🛠️ 安装与配置

**1. 安装依赖环境**
请确保您的 Python 环境中已安装以下包：
```bash
pip install arxiv requests
```

**2. 目录结构**
确保文件在您的 workspace 工作区中放置正确：

```text
workspace/
├── paper_list/                  # 自动生成：PDF 文件和 paper_list.csv 会存放在这里
└── skills/
    └── ai-paper-researcher/
        ├── SKILL.md             # Agent 提示词指令
        ├── arxiv_tool.py        # Python 核心脚本
        └── target.csv           # 您的目标会议名单 (如 CVPR, NeurIPS)
```

**3. 配置目标会议**
将您关注的会议名称添加到 `target.csv` 中（每行一个）：

```csv
target
CVPR
ICCV
NeurIPS
ICLR
```

---

## 🗣️ 如何使用（自然语言提示）

将技能加载到您的 AI Agent 后，只需用自然语言发号施令即可。以下是最佳的 Prompt 模板：

### 🔭 场景 1：追踪最新前沿（模式 A）

*非常适合在快速发展的领域进行每周文献梳理。*

> “调用 `ai-paper-researcher` 技能。帮我检索近一个月关于 **Agent security** 的最新论文。请按时间（date）排序，跳过顶会过滤步骤，并直接将最相关的论文下载到我的本地工作区。”

### 🏛️ 场景 2：深度挖掘顶会经典（模式 B）

*非常适合寻找经过同行评审和验证的基础性高价值工作。*

> “使用 `ai-paper-researcher` 技能。我想深入调研 **Agent security** 方向。请按相关度（relevance）排序，找出最具影响力的论文。请严格过滤结果，只下载那些已经被我 `target.csv` 列表中会议接收的论文。完成后请给我一份下载总结。”

### 🎯 场景 3：精准直接下载

*当您已经知道具体的 ID，不需要搜索时，直接绕过检索流程。*

> “使用 `ai-paper-researcher` 技能，跳过搜索步骤，直接帮我下载原始的 Adam 优化器论文 (ID: `1412.6980`) 以及 Focal Loss 论文 (ID: `1708.02002`)。”

---

## 💻 命令行使用（面向开发者）

如果您更喜欢绕过 AI Agent 直接在终端中使用，该 Python 脚本也是一个功能完备的独立工具：

```bash
# 按相关度搜索论文（返回带有 comment 字段的 JSON，供二次开发使用）
python arxiv_tool.py search "diffusion models" --max 10 --sort relevance

# 搜索最新发布的论文
python arxiv_tool.py search "diffusion models" --sort date

# 下载指定论文（自动处理命名及 CSV 记录持久化）
python arxiv_tool.py download 1412.6980
```

---

## 📜 致谢与开源协议

本项目的核心灵感和基础框架源自 [Ractorrr/arxiv-mcp](https://github.com/Ractorrr/arxiv-mcp) (🔍 AI-powered arXiv research assistant)。
在此基础上，项目进行了重度修改：剥离了复杂的数据库依赖 (MongoDB)，引入了严格的本地文件系统追踪机制，新增了基于 LLM 驱动的语义级会议过滤管线，并实现了高度鲁棒的流式下载策略以绕过 arXiv 的 API 限制。

**开源协议:** [MIT License](https://choosealicense.com/licenses/mit/) - 允许免费使用、修改和分发，无强制署名要求。

path: target.csv
content: target
ICLR

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)