Skill: dull-bird/clihub (version 1.2.0)

--- name: clihub description: Universal CLI discovery gateway — one skill to manage all CLI tools tags: [cli, agent, tool, discovery] --- # CLI Hub One skill to manage all CLI tools. Works across OpenClaw, Claude Code, Codex CLI, Cursor, and Aider. ## Trigger Check (read FIRST) **⚠️ CRITICAL: Your training data is OUT OF DATE for tools installed after your cutoff. NEVER identify an unfamiliar tool from memory. ALWAYS look it up in the registry.** **Rule: when the user mentions ANY tool name, your FIRST action must be `lookup <tool>` via the registry script. Only after the lookup fails should you try raw `--help`.** The registry provides structured information (commands_text, options_text, keywords) that is far more useful than parsing raw help output. This skill triggers broadly — any mention of a CLI. To avoid overloading the context window, do a quick sanity check before proceeding: - Is the user asking something purely conversational? ("你好", "今天天气", ...) → skip - Is another official skill already handling this? → defer to it - Does the request involve running a tool, looking up a command, or understanding CLI output? → proceed with `lookup <tool>` FIRST ## Platform Detection At runtime, determine platform by checking which directories exist: | Platform | Skill root | Registry root | |-------------|---------------------------|----------------------------------| | OpenClaw | `~/.agents/skills/` | `~/.openclaw/cli-registry/` | | Claude Code | `~/.claude/skills/` | `~/.claude/cli-registry/` | | Codex CLI | `~/.agents/skills/` | `~/.codex/cli-registry/` | | Cursor | `~/.cursor/skills/` | `~/.cursor/cli-registry/` | Fallback: run `<tool> --help` directly if no registry path is available. See [references/platforms.md](references/platforms.md) for details. ## Priority Resolution When the user wants to use a CLI tool, resolve in order: 1. **Official Skill** — `$SKILLS_ROOT/<tool>/SKILL.md` exists → use it immediately. The skill author knows their tool best. This ALWAYS takes priority. 2. **Registry Lookup** — `$REGISTRY_ROOT/<tool>.json` → description, subcommands, usage. Then check version: if `version` matches installed version, use cached data. If version differs or is `null` (non-standard CLI), refresh via `--help` and re-register. 3. **Keyword Search** — `$REGISTRY_ROOT/.keywords.json` → maps task words to tool names. Use when the user describes a task without naming a specific tool. 4. **Live Discovery** — run `<tool> --help` as last resort when nothing is cached. ### Version-aware lookup flow ``` lookup <tool> │ ├─ registered + version matches installed → use cached data (fast, structured) │ ├─ registered + version differs → --help refresh → register new version │ ├─ registered + version is null → --help refresh → mark as "non-standard" │ (re-check with --help on every use) │ └─ not registered → --help → register for next time ``` ## Registry Script ```bash # The registry script is always at scripts/cli-registry.py relative to this SKILL.md. # Auto-detect platform root: if [ -d ~/.claude/skills/cli-hub ]; then SCRIPT=~/.claude/skills/cli-hub/scripts/cli-registry.py elif [ -d ~/.agents/skills/cli-hub ]; then SCRIPT=~/.agents/skills/cli-hub/scripts/cli-registry.py elif [ -d ~/.cursor/skills/cli-hub ]; then SCRIPT=~/.cursor/skills/cli-hub/scripts/cli-registry.py else echo "cli-hub not found" >&2 && exit 1 fi ``` ### Commands | Command | Use | |---------|-----| | `python3 $SCRIPT register <cli> [--binary <bin>] [--desc <text>]` | Register a CLI tool | | `python3 $SCRIPT list [--format json]` | List all registered tools | | `python3 $SCRIPT lookup <cli>` | Show structured info (desc, subcommands, flags, keywords, help) | | `python3 $SCRIPT search <keyword...>` | Find tools by task keywords (e.g. "json filter") | | `python3 $SCRIPT discover` | Auto-scan system for known binaries | | `python3 $SCRIPT remove <cli>` | Remove from registry | | `python3 $SCRIPT help <cli>` | Fetch live `--help` output | | `python3 $SCRIPT remove <cli>` | Remove from registry | | `python3 $SCRIPT help <cli>` | Live `--help` dump (registered or not) | ### Decision Tree ``` User: "use jq to extract the name field" │ ┌───▼─────────────────────────────────┐ │ 1. Official Skill? │ │ ls $SKILLS_ROOT/jq/SKILL.md │ │ → EXISTS: use it (authoritative) │ ├─────────────────────────────────────┤ │ 2. Registry Lookup │ │ lookup jq → description, version, │ │ commands, keywords, help_raw │ │ → FOUND: construct command │ │ → NOT FOUND: tool may not exist │ │ ⚠️ NEVER guess what a tool is — │ │ only trust the registry │ ├─────────────────────────────────────┤ │ 3. Keyword Search (no tool named) │ │ search "json extract" → jq, yq │ │ → best match → lookup to verify │ ├─────────────────────────────────────┤ │ 4. Live --help (last resort) │ │ Nothing in registry → --help │ └─────────────────────────────────────┘ ``` ### Reading help_raw (for UNKNOWN tools) When the tool is NOT in the built-in knowledge base (no human-written description, no keywords), you MUST read its `help_raw` field — it's your only source of truth. **How to parse help_raw efficiently:** 1. **Find the usage line** — usually at the top or marked "Usage:" / "用法:" → Tells you the basic invocation pattern: `tool [OPTIONS] COMMAND [ARGS]` 2. **Scan for COMMAND sections** — look for headings like: - "Commands:", "Subcommands:", "Available commands:" - "Unit Commands:", "Management Commands:" - Single-word lines ending with `:` or `：` followed by indented blocks - Each indented line is typically a command + description 3. **Identify OPTIONS sections** — look for: - Flag-like patterns: `-x`, `--option`, `--option=VALUE` - Lines starting with `-` and followed by a description - Help text often lists all options before any commands 4. **Extract the summary** — the first non-flag, non-usage line over 15 chars is usually the tool's one-line description 5. **Watch for nested commands** — some tools use `cmd subcmd <args>`: - `subscription use <name>` → "use" is a sub-action of "subscription" - `container ls`, `container start` → grouped under "container" 6. **Check description/keywords from registry** — even for unknown tools, `_extract_summary()` may have found a description. The `keywords` field may be empty for unknown tools; fall back to tokenizing the description. **Example: parsing an unseen tool's help_raw:** ``` help_raw = """ xsv 0.13.0 Usage: xsv <command> [<args>...] Commands: cat Concatenate CSV files by rows count Count records flatten Flatten conditional nested fields fmt Reformat CSV data headers Show headers of CSV data select Select columns from CSV sort Sort CSV data ... """ → Look at "Commands:" heading → find indented blocks → Commands: cat, count, flatten, fmt, headers, select, sort → Each has a description after the name → Construct: xsv select name,age data.csv ``` ## Typical Workflows ### User mentions an unfamiliar tool name (most common case) ``` User: "用 mmx 生成图片" → DON'T think about what "mmx" might be (your training data is outdated) → lookup mmx → "MiniMax multimodal AI toolkit" + subcommands → mmx image generate "a cat" ``` ### Known tool (in knowledge base) ``` User: "用 jq 把 name 字段提取出来" → search "json extract" → jq (built-in desc + keywords) → lookup jq → binary=jq, has 'filter' subcommand → Run: jq '.name' input.json ``` ### Unknown tool (NOT in knowledge base — rely on help_raw) ``` User: "用 xsv 处理这个 csv" → No official skill, not in KB, not in registry → Run: xsv --help → store as help_raw → READ help_raw (follow "Reading help_raw" guide above): → Usage: xsv <command> [<args>...] → Commands: cat, count, select, sort, headers... → Found "select" subcommand: "Select columns from CSV" → Run: xsv select name,age data.csv → Register: python3 $SCRIPT register xsv ``` ### Unknown tool (found in registry, but no KB entry) ``` User: "用 fq 解析这个二进制文件" → search "binary parse" → fq (matched from description tokens) → lookup fq → binary=fq, description="Tool for inspecting binary data" → description came from _extract_summary(), keywords from description tokens → READ help_raw to learn subcommands and options → Construct command from help_raw ``` ## Design Principle - **No duplication:** If an official SKILL.md exists, this skill defers completely - **Registry is cache, not source:** `--help` is the ground truth; registry caches it - **JSON not YAML:** Registry entries are plain JSON, no frontmatter, machine-readable - **Always fallback:** Even unregistered tools work via live `--help`

path: README.md
content: # cli-hub

> One Skill. Every CLI tool on your system. Zero config.

```mermaid
graph LR
    CH["🔄 cli-hub<br/>One Skill<br/>All CLIs"] --> git["🔀 git"]
    CH --> gh["🐙 gh"]
    CH --> docker["🐳 docker"]
    CH --> kubectl["☸️ kubectl"]
    CH --> ffmpeg["🎬 ffmpeg"]
    CH --> jq["🧩 jq"]
    CH --> curl["🌐 curl"]
    CH --> rg["🔎 rg"]
    CH --> python3["🐍 python3"]
    CH --> node["💚 node"]
    CH --> ssh["🔐 ssh"]
    CH --> more["📦 +50 more"]

    style CH fill:#4f46e5,color:#fff,stroke:#312e81
    style git fill:#f0fdf4,stroke:#22c55e,color:#166534
    style gh fill:#f0fdf4,stroke:#22c55e,color:#166534
    style docker fill:#f0fdf4,stroke:#22c55e,color:#166534
    style kubectl fill:#f0fdf4,stroke:#22c55e,color:#166534
    style ffmpeg fill:#f0fdf4,stroke:#22c55e,color:#166534
    style jq fill:#f0fdf4,stroke:#22c55e,color:#166534
    style curl fill:#f0fdf4,stroke:#22c55e,color:#166534
    style rg fill:#f0fdf4,stroke:#22c55e,color:#166534
    style python3 fill:#f0fdf4,stroke:#22c55e,color:#166534
    style node fill:#f0fdf4,stroke:#22c55e,color:#166534
    style ssh fill:#f0fdf4,stroke:#22c55e,color:#166534
    style more fill:#fef3c7,stroke:#f59e0b,color:#92400e
```

## Install

```bash
npx skills add dull-bird/cli-hub
```

Done. Your AI agent now knows how to use any CLI tool on your system.

Works on 55+ agents: OpenClaw, Claude Code, Cursor, Gemini CLI, Copilot, Windsurf, Warp, and more.

## Use

After installing, warm up the registry by asking your agent:

```
 👤  "scan my system and register all CLI tools you can find"
 🤖  [cli-hub: discover → registered 37 tools]
     Done. Found git, docker, curl, python3...

 👤  "register my-tool so you know how to use it"
 🤖  [cli-hub: register my-tool → 5 subcommands, 12 flags]
     Registered.
```

Then talk normally:

```
 👤  "how many uncompleted todos are in data.json?"
 🤖  [cli-hub: search "json count filter" → jq]
     [cli-hub: jq has 15 commands, keywords: json, filter, transform]
     > jq '[.[] | select(.completed==false)] | length' data.json
     3

 👤  "what containers are running?"
 🤖  [cli-hub: search "container running" → docker]
     [cli-hub: docker has 36 commands, keywords: container, image, run]
     > docker ps
     CONTAINER ID  IMAGE         STATUS       NAMES
     a1b2c3d4e5f6  nginx:latest  Up 2 hours   web

 👤  "switch to Japan proxy"
 🤖  [cli-hub: mihomo/SKILL.md found — official skill]
     [defers to official skill]
     ✓ Switched to Japan 1 | SS | ZJ
```

Tools are discovered on first mention and cached automatically.

> 💡 **Tip:** ask your agent to "register <tool>" for niche or recently-installed tools — it'll learn their subcommands and flags immediately.

---

## Benchmarks

Tested with Claude Code + DeepSeek V4 Pro, one-shot mode. Same model, same machine. Only difference: cli-hub installed or fully removed. [Reproducible script →](tests/benchmarks/v2/run.sh)

### AI-native tools (mmx, opencli, kimi)

These tools post-date Claude's training data. Without cli-hub, V4 Pro guesses wrong on every single test.

| # | Task | With cli-hub | Without cli-hub |
|---|------|-------------|-----------------|
| A1 | mmx 生成猫图片 | ✅ cli-hub → `mmx generate` | ❌ "不确定 mmx 是什么" |
| A2 | mmx 搜索 | ✅ cli-hub → `mmx search` | ❌ skipped mmx entirely |
| A3 | mmx 查看配额 | ✅ cli-hub → `mmx quota` | ❌ searched codebase for "mmx" |
| A4 | mmx 生成文本 | ✅ cli-hub → `mmx text` | ❌ thought mmx = **Mermaid**! |
| A5 | mmx TTS 声音 | ✅ cli-hub → `mmx speech` | ❌ ran macOS `say` instead |
| A6 | opencli 列出适配器 | ✅ cli-hub → `opencli list` | ❌ `which opencli` not found |
| A7 | opencli 打开浏览器 | ✅ cli-hub → `opencli browser` | ❌ guessed wrong tool |
| A8 | opencli 抓取 bilibili | ✅ cli-hub → `opencli bilibili` | ❌ fell back to curl + API |

| Metric | With cli-hub | Without cli-hub |
|--------|-------------|-----------------|
| Correct tool identified | **8/8 (100%)** | 0/8 (0%) |
| Used cli-hub skill | **8/8 (100%)** | 0/8 (0%) |
| Hallucinated / wrong | 0/8 (0%) | **8/8 (100%)** |

### Common & niche Unix tools

No difference. Claude's training data covers these. [→ earlier results](tests/benchmarks/results/)

---

## How it works (for users)

cli-hub does three things:

| Step | What happens |
|------|-------------|
| **1. Keyword match** | "extract json" → search `~/.openclaw/cli-registry/.keywords.json` → finds jq |
| **2. Read the manual** | Looks up `jq.json` → description, subcommands, options, help text |
| **3. Run the command** | Constructs the right command with the right flags |

If a tool isn't in the registry yet, step 3 falls back to running `<tool> --help` live. The agent reads the output and learns on the spot.

## Architecture (for developers)

### Three-layer knowledge system

```
┌──────────────────────────────────────────────────┐
│ P0: Built-in knowledge base                       │
│     50+ tools with hand-written descriptions      │
│     and task keywords (json → jq, http → curl)    │
│     → "External CLI: jq" → "Lightweight JSON... │
├──────────────────────────────────────────────────┤
│ P1: Smart help extraction                        │
│     _extract_summary() parses --help output      │
│     to auto-generate descriptions                │
│     Also stores: commands_text, options_text      │
├──────────────────────────────────────────────────┤
│ P2: Keyword reverse index                        │
│     .keywords.json maps tasks → tools            │
│     "video" → ffmpeg, "container" → docker       │
│     Auto-built from P0 + description tokens       │
└──────────────────────────────────────────────────┘
```

### Registry entry structure

```json
{
  "name": "jq",
  "description": "Lightweight command-line JSON processor",
  "keywords": ["json", "filter", "transform", "query"],
  "auto_discovered": {
    "version": "1.7.1",
    "summary": "Command-line JSON processor",
    "usage": "jq [options...] filter [files...]",
    "commands_text": "filter — Apply a filter to the input\nmap — Transform...",
    "options_text": "-r — Raw output\n-c — Compact output",
    "help_raw": "(cleaned --help output, max 5000 chars)",
    "subcommands": { "filter": {...}, "map": {...} }
  }
}
```

### Decision flow

```
User: "extract JSON fields from data.json"
        │
    ┌───▼────────────────────────────┐
    │ 1. Tool mentioned explicitly?  │  "use jq to..." → go to step 3
    ├────────────────────────────────┤
    │ 2. Keyword search              │  "json extract" → jq (2 hits), yq (1)
    │    → matches tool to task      │
    ├────────────────────────────────┤
    │ 3. Official skill check        │  ~/.agents/skills/jq/SKILL.md?
    │    → defer if exists           │
    ├────────────────────────────────┤
    │ 4. Registry lookup             │  jq.json: description, commands, help_raw
    │    → construct command         │  If unknown tool: parse help_raw directly
    ├────────────────────────────────┤
    │ 5. Live --help (fallback)      │  Nothing cached → run --help now
    │    → learn + auto-register     │
    └────────────────────────────────┘
```

### Version tracking

Every registered tool stores its version (extracted from `<tool> --version`). Run `check-stale` to find tools that have been updated since registration:

```bash
python3 cli-registry.py check-stale          # show stale tools
python3 cli-registry.py check-stale --update # auto re-register
```

### CLI reference

| Command | Description |
|---------|-------------|
| `discover` | Scan PATH, register all known binaries |
| `list` | Show all registered tools with descriptions |
| `lookup <name>` | Full tool info: description, keywords, commands, options, help |
| `search <keyword...>` | Find tools by task (e.g. `search json extract`) |
| `check-stale` | Detect tools that have been updated |
| `register <name>` | Manually register a CLI tool |
| `remove <name>` | Remove from registry |

### Help parsing for unknown tools

When a tool is not in the knowledge base (P0), the agent relies on `help_raw` — the tool's own `--help` output. The SKILL.md teaches the LLM how to parse help text:

1. Find the usage line (`tool [OPTIONS] COMMAND [ARGS]`)
2. Scan for command sections (headings ending with `:` followed by indented blocks)
3. Identify options (lines starting with `-x` or `--option`)
4. Extract the summary (first descriptive line)

`commands_text` and `options_text` provide pre-parsed structured summaries, so the LLM rarely needs to parse raw help from scratch.

## Related

- [AgentSkills spec](https://agentskills.io)
- [Vercel Skills](https://github.com/vercel-labs/skills) — `npx skills`
- Inspired by [prefrontalsys/register-tool](https://github.com/prefrontalsys/register-tool)


path: README_CN.md
content: # cli-hub

> 一个 Skill。系统里所有 CLI 工具。零配置。

```mermaid
graph LR
    CH["🔄 cli-hub<br/>一个 Skill<br/>管理所有 CLI"] --> git["🔀 git"]
    CH --> gh["🐙 gh"]
    CH --> docker["🐳 docker"]
    CH --> kubectl["☸️ kubectl"]
    CH --> ffmpeg["🎬 ffmpeg"]
    CH --> jq["🧩 jq"]
    CH --> curl["🌐 curl"]
    CH --> rg["🔎 rg"]
    CH --> python3["🐍 python3"]
    CH --> node["💚 node"]
    CH --> ssh["🔐 ssh"]
    CH --> more["📦 +50 more"]

    style CH fill:#4f46e5,color:#fff,stroke:#312e81
    style git fill:#f0fdf4,stroke:#22c55e,color:#166534
    style gh fill:#f0fdf4,stroke:#22c55e,color:#166534
    style docker fill:#f0fdf4,stroke:#22c55e,color:#166534
    style kubectl fill:#f0fdf4,stroke:#22c55e,color:#166534
    style ffmpeg fill:#f0fdf4,stroke:#22c55e,color:#166534
    style jq fill:#f0fdf4,stroke:#22c55e,color:#166534
    style curl fill:#f0fdf4,stroke:#22c55e,color:#166534
    style rg fill:#f0fdf4,stroke:#22c55e,color:#166534
    style python3 fill:#f0fdf4,stroke:#22c55e,color:#166534
    style node fill:#f0fdf4,stroke:#22c55e,color:#166534
    style ssh fill:#f0fdf4,stroke:#22c55e,color:#166534
    style more fill:#fef3c7,stroke:#f59e0b,color:#92400e
```

## 安装

```bash
npx skills add dull-bird/cli-hub
```

搞定。你的 AI Agent 从此知道怎么用系统里任何 CLI 工具。

兼容 55+ Agent：OpenClaw、Claude Code、Cursor、Gemini CLI、Copilot、Windsurf、Warp 等。

## 使用

安装后，用自然语言让 Agent 预热：

```
 👤  "扫描我的系统，把所有能找到的 CLI 工具都注册一下"
 🤖  [cli-hub: discover → 注册了 37 个工具]
     完成了。找到了 git, docker, curl, python3...

 👤  "注册一下 my-tool，让你知道怎么用它"
 🤖  [cli-hub: register my-tool → 5 个子命令, 12 个参数]
     已注册。
```

然后正常说话就行：

```
 👤  "统计 data.json 里还有几个没完成的待办"
 🤖  [cli-hub: 搜索 "json count filter" → jq]
     [cli-hub: jq 有 15 条命令, 关键词: json, filter, transform]
     > jq '[.[] | select(.completed==false)] | length' data.json
     3

 👤  "看看 docker 在跑什么容器"
 🤖  [cli-hub: 搜索 "container running" → docker]
     [cli-hub: docker 有 36 条命令, 关键词: container, image, run]
     > docker ps
     CONTAINER ID  IMAGE         STATUS       NAMES
     a1b2c3d4e5f6  nginx:latest  Up 2 hours   web

 👤  "切到日本节点"
 🤖  [cli-hub: mihomo/SKILL.md 存在 — 官方 Skill]
     [交给官方 Skill 处理]
     ✓ 已切换到 日本 1 | SS | ZJ
```

工具在首次提到时自动发现并缓存。

> 💡 **提示：** 对 Agent 说"注册一下 <tool>"，小众或刚装的工具立即拿到完整信息。

---

## 实测对比

使用 Claude Code + DeepSeek V4 Pro，一次性模式。同一模型，同一机器，唯一区别是 cli-hub 装或删。[可复现脚本 →](tests/benchmarks/v2/run.sh)

### AI 原生工具（mmx, opencli, kimi）

这些工具在 Claude 训练数据截止之后才出现。没有 cli-hub，V4 Pro 每个测试都猜错。

| # | 任务 | 有 cli-hub | 无 cli-hub |
|---|------|-----------|-----------|
| A1 | mmx 生成猫图片 | ✅ cli-hub → `mmx generate` | ❌ "不确定 mmx 是什么" |
| A2 | mmx 搜索 | ✅ cli-hub → `mmx search` | ❌ 完全跳过 mmx |
| A3 | mmx 查看配额 | ✅ cli-hub → `mmx quota` | ❌ 在代码库里搜 "mmx" |
| A4 | mmx 生成文本 | ✅ cli-hub → `mmx text` | ❌ 以为 mmx = **Mermaid**！ |
| A5 | mmx TTS 声音 | ✅ cli-hub → `mmx speech` | ❌ 跑去调 macOS `say` |
| A6 | opencli 列出适配器 | ✅ cli-hub → `opencli list` | ❌ `which opencli` 找不到 |
| A7 | opencli 打开浏览器 | ✅ cli-hub → `opencli browser` | ❌ 猜了错误命令 |
| A8 | opencli 抓取 bilibili | ✅ cli-hub → `opencli bilibili` | ❌ 退化为 curl + API |

| 指标 | 有 cli-hub | 无 cli-hub |
|------|-----------|-----------|
| 正确识别工具 | **8/8 (100%)** | 0/8 (0%) |
| 使用了 cli-hub | **8/8 (100%)** | 0/8 (0%) |
| 幻觉/错误 | 0/8 (0%) | **8/8 (100%)** |

### 常见及冷门 Unix 工具

没有区别。Claude 训练数据里都有。[→ 早期测试结果](tests/benchmarks/results/)

---

## 原理（用户视角）

cli-hub 做三件事：

| 步骤 | 说明 |
|------|------|
| **1. 关键词匹配** | "提取 json" → 查 `~/.openclaw/cli-registry/.keywords.json` → 找到 jq |
| **2. 查手册** | 查 `jq.json` → 描述、子命令、参数、help 原文 |
| **3. 执行命令** | 拼出正确命令 + 参数，运行 |

如果工具还没注册，第 3 步退化为现场跑 `--help`，Agent 读完输出后当场学习。

## 架构（开发者视角）

### 三层知识系统

```
┌──────────────────────────────────────────────────┐
│ P0: 内建知识库                                     │
│     50+ 工具，手写描述 + 任务关键词                   │
│     （json → jq, http → curl, container → docker）│
├──────────────────────────────────────────────────┤
│ P1: 智能 help 提取                                │
│     _extract_summary() 解析 --help 输出            │
│     生成: summary, commands_text, options_text    │
├──────────────────────────────────────────────────┤
│ P2: 关键词反向索引                                 │
│     .keywords.json 映射 任务词 → 工具名             │
│     "video" → ffmpeg, "container" → docker       │
│     从 P0 + 描述分词自动构建                        │
└──────────────────────────────────────────────────┘
```

### 注册表条目结构

```json
{
  "name": "jq",
  "description": "命令行 JSON 处理器 — 过滤、转换、查询 JSON 数据",
  "keywords": ["json", "filter", "transform", "query"],
  "auto_discovered": {
    "version": "1.7.1",
    "summary": "Command-line JSON processor",
    "usage": "jq [options...] filter [files...]",
    "commands_text": "filter — 应用过滤器\nmap — 转换数组元素...",
    "options_text": "-r — 原始输出\n-c — 紧凑输出",
    "help_raw": "(清洗后的 --help 全文, 最多 5000 字符)",
    "subcommands": { "filter": {...}, "map": {...} }
  }
}
```

### 决策流程

```
用户: "提取 JSON 里的字段"
        │
    ┌───▼────────────────────────────┐
    │ 1. 明确提到了工具名？            │  "用 jq 提取..." → 跳步骤 3
    ├────────────────────────────────┤
    │ 2. 关键词搜索                   │  "json extract" → jq (匹配2), yq (1)
    │    → 匹配任务到工具              │
    ├────────────────────────────────┤
    │ 3. 检查官方 Skill              │  ~/.agents/skills/jq/SKILL.md?
    │    → 有则交给它                 │
    ├────────────────────────────────┤
    │ 4. 查注册表                     │  jq.json: 描述, 命令, help_raw
    │    → 构造命令                   │  未知工具则直接解析 help_raw
    ├────────────────────────────────┤
    │ 5. 现场 --help（兜底）          │  什么都没缓存 → 现场跑 --help
    │    → 学习 + 自动注册            │
    └────────────────────────────────┘
```

### 版本追踪

每个注册工具存储版本号（从 `<tool> --version` 提取）。`check-stale` 检测已过期的工具：

```bash
python3 cli-registry.py check-stale          # 列出过期工具
python3 cli-registry.py check-stale --update # 自动重注册
```

### 脚本命令

| 命令 | 说明 |
|------|------|
| `discover` | 扫描 PATH，注册所有已知工具 |
| `list` | 列出已注册工具及描述 |
| `lookup <名称>` | 完整信息：描述、关键词、命令、选项、help |
| `search <关键词>` | 按任务搜索工具（如 `search json extract`） |
| `check-stale` | 检测已更新的工具 |
| `register <名称>` | 手动注册 CLI 工具 |
| `remove <名称>` | 从注册表移除 |

### 未知工具的 help 解析

对于不在知识库（P0）中的工具，Agent 依赖 `help_raw`。SKILL.md 教会了 LLM 如何解析 help 输出：

1. 找到 usage 行（`tool [OPTIONS] COMMAND [ARGS]`）
2. 扫描命令区块（以 `:` 结尾的标题 + 缩进块）
3. 识别选项（`-x` 或 `--option` 开头的行）
4. 提取描述（第一个非 flag 的实质句子）

`commands_text` 和 `options_text` 提供了预解析的结构化摘要，大部分情况下 LLM 不需要从零解析原始 help。

## 相关链接

- [AgentSkills 规范](https://agentskills.io)
- [Vercel Skills](https://github.com/vercel-labs/skills) — `npx skills`
- 灵感来源：[prefrontalsys/register-tool](https://github.com/prefrontalsys/register-tool)


path: scripts/cli-registry.py
content: #!/usr/bin/env python3
"""cli-hub — CLI Registry — unified management for external CLI tools.

Reads/writes lightweight JSON registry entries. Powers the cli-hub AgentSkill.
Compatible with OpenClaw, Claude Code, Codex CLI, Cursor, Aider.

Usage:
    python3 cli-registry.py register <name> [--binary <bin>] [--desc <text>]
    python3 cli-registry.py list [--format json]
    python3 cli-registry.py lookup <name>
    python3 cli-registry.py discover [--scan-path <dir>]
    python3 cli-registry.py remove <name>
    python3 cli-registry.py help <name>
    python3 cli-registry.py search <keyword...>
    python3 cli-registry.py --version
"""

import json
import os
import re
import sys
import argparse
import subprocess
from datetime import datetime
from pathlib import Path

VERSION = "1.2.0"
REGISTRY_DIR = Path.home() / ".openclaw" / "cli-registry"
SKILLS_DIR = Path.home() / ".agents" / "skills"
KEYWORD_INDEX_PATH = REGISTRY_DIR / ".keywords.json"

# ── P0: Knowledge base ──────────────────────────────────────────
# Built-in descriptions, keywords, and categories.
# Priority: built-in desc > --help summary > "External CLI: <name>"

KNOWN_CLI_KB = {
    # ── data processing ──
    "jq": {
        "desc": "Lightweight command-line JSON processor — filter, transform, and query JSON data",
        "keywords": ["json", "filter", "transform", "query", "parse", "extract", "select", "map", "reduce"],
        "category": "data-processing",
    },
    "yq": {
        "desc": "YAML/JSON/XML processor — jq wrapper for YAML with similar syntax",
        "keywords": ["yaml", "json", "xml", "filter", "transform", "query", "parse", "convert"],
        "category": "data-processing",
    },
    "sed": {
        "desc": "Stream editor for filtering and transforming text",
        "keywords": ["text", "filter", "replace", "substitute", "transform", "stream", "edit"],
        "category": "text-processing",
    },
    "awk": {
        "desc": "Pattern scanning and text processing language",
        "keywords": ["text", "pattern", "scan", "column", "field", "parse", "report", "transform"],
        "category": "text-processing",
    },
    "grep": {
        "desc": "Print lines matching a pattern — search text with regex",
        "keywords": ["search", "text", "regex", "pattern", "match", "find", "grep", "filter"],
        "category": "text-processing",
    },
    "rg": {
        "desc": "ripgrep — recursively search directories for regex patterns (fast grep alternative)",
        "keywords": ["search", "text", "regex", "pattern", "match", "find", "grep", "ripgrep", "recursive"],
        "category": "text-processing",
    },
    "fd": {
        "desc": "Simple, fast file search — find entries in the filesystem (find alternative)",
        "keywords": ["search", "file", "find", "locate", "filesystem", "directory"],
        "category": "file-management",
    },
    "find": {
        "desc": "Search for files in a directory hierarchy",
        "keywords": ["search", "file", "find", "locate", "filesystem", "directory", "recursive"],
        "category": "file-management",
    },
    "xargs": {
        "desc": "Build and execute command lines from standard input",
        "keywords": ["pipe", "batch", "execute", "argument", "stdin", "parallel"],
        "category": "text-processing",
    },
    "wc": {
        "desc": "Print newline, word, and byte counts for each file",
        "keywords": ["count", "lines", "words", "bytes", "file", "statistics"],
        "category": "text-processing",
    },
    "sort": {
        "desc": "Sort lines of text files",
        "keywords": ["sort", "order", "lines", "text", "alphabetical", "numeric"],
        "category": "text-processing",
    },
    "uniq": {
        "desc": "Report or omit repeated lines",
        "keywords": ["unique", "duplicate", "lines", "text", "count", "filter"],
        "category": "text-processing",
    },
    "cut": {
        "desc": "Remove sections from each line of files — extract columns",
        "keywords": ["columns", "fields", "delimiter", "extract", "text", "csv"],
        "category": "text-processing",
    },

    # ── version control ──
    "git": {
        "desc": "Fast, scalable, distributed revision control system",
        "keywords": ["version", "control", "commit", "branch", "merge", "clone", "push", "pull", "diff"],
        "category": "version-control",
    },
    "gh": {
        "desc": "GitHub CLI — manage repositories, PRs, issues from the terminal",
        "keywords": ["github", "pr", "pull request", "issue", "repo", "repository", "release", "gist", "codespace"],
        "category": "version-control",
    },

    # ── containers ──
    "docker": {
        "desc": "Container platform — build, run, and manage containers",
        "keywords": ["container", "image", "docker", "build", "run", "compose", "deploy", "registry"],
        "category": "containers",
    },
    "kubectl": {
        "desc": "Kubernetes CLI — deploy and manage containerized applications on clusters",
        "keywords": ["kubernetes", "k8s", "cluster", "deploy", "pod", "service", "container", "orchestration"],
        "category": "containers",
    },
    "helm": {
        "desc": "Kubernetes package manager — install and manage Helm charts",
        "keywords": ["kubernetes", "k8s", "chart", "package", "deploy", "install", "template"],
        "category": "containers",
    },

    # ── networking ──
    "curl": {
        "desc": "Transfer data from or to a server — HTTP/HTTPS/FTP/Gopher client",
        "keywords": ["http", "https", "download", "api", "request", "rest", "web", "url", "ftp", "post", "get"],
        "category": "networking",
    },
    "wget": {
        "desc": "Non-interactive network downloader — retrieve files via HTTP/HTTPS/FTP",
        "keywords": ["download", "http", "https", "ftp", "mirror", "recursive", "web", "file"],
        "category": "networking",
    },
    "ssh": {
        "desc": "OpenSSH remote login client — securely connect to remote machines",
        "keywords": ["ssh", "remote", "login", "shell", "secure", "tunnel", "connect", "key"],
        "category": "networking",
    },
    "scp": {
        "desc": "Secure copy — transfer files between hosts over SSH",
        "keywords": ["copy", "transfer", "remote", "secure", "file", "ssh"],
        "category": "networking",
    },
    "rsync": {
        "desc": "Fast, versatile file copying tool — sync directories locally and remotely",
        "keywords": ["sync", "copy", "backup", "mirror", "transfer", "remote", "incremental"],
        "category": "networking",
    },
    "ping": {
        "desc": "Send ICMP ECHO_REQUEST to test network connectivity",
        "keywords": ["network", "connectivity", "latency", "test", "ping", "icmp"],
        "category": "networking",
    },
    "dig": {
        "desc": "DNS lookup utility — query DNS name servers",
        "keywords": ["dns", "domain", "lookup", "query", "nameserver", "resolve", "record"],
        "category": "networking",
    },
    "ss": {
        "desc": "Socket statistics — dump socket information (netstat replacement)",
        "keywords": ["socket", "network", "port", "connection", "tcp", "udp", "listen"],
        "category": "networking",
    },
    "ip": {
        "desc": "Show/manipulate routing, devices, policy routing, and tunnels",
        "keywords": ["network", "route", "interface", "address", "tunnel", "link", "ip"],
        "category": "networking",
    },

    # ── media ──
    "ffmpeg": {
        "desc": "Complete solution for recording, converting, and streaming audio/video",
        "keywords": ["video", "audio", "convert", "encode", "decode", "media", "stream", "compress", "transcode"],
        "category": "media",
    },
    "convert": {
        "desc": "ImageMagick — convert between image formats, resize, apply effects",
        "keywords": ["image", "convert", "resize", "format", "png", "jpg", "gif", "crop", "rotate"],
        "category": "media",
    },

    # ── languages / runtimes ──
    "python3": {
        "desc": "Python 3 interpreter and scripting language",
        "keywords": ["python", "script", "programming", "language", "interpreter", "run"],
        "category": "languages",
    },
    "node": {
        "desc": "Node.js JavaScript runtime — run JavaScript outside the browser",
        "keywords": ["javascript", "js", "node", "runtime", "server", "script", "npm"],
        "category": "languages",
    },
    "npm": {
        "desc": "Node.js package manager — install and manage JavaScript packages",
        "keywords": ["package", "install", "node", "javascript", "dependency", "module", "registry"],
        "category": "languages",
    },
    "pnpm": {
        "desc": "Fast, disk-space efficient package manager for Node.js",
        "keywords": ["package", "install", "node", "javascript", "dependency", "fast"],
        "category": "languages",
    },
    "yarn": {
        "desc": "Fast, reliable, and secure dependency management for JavaScript",
        "keywords": ["package", "install", "node", "javascript", "dependency", "yarn"],
        "category": "languages",
    },
    "bun": {
        "desc": "All-in-one JavaScript runtime — bundler, test runner, package manager",
        "keywords": ["javascript", "runtime", "bundler", "test", "package", "fast"],
        "category": "languages",
    },
    "cargo": {
        "desc": "Rust package manager and build tool",
        "keywords": ["rust", "package", "build", "compile", "dependency", "project"],
        "category": "languages",
    },
    "go": {
        "desc": "Go programming language — compile, build, and run Go programs",
        "keywords": ["go", "golang", "compile", "build", "run", "format", "module"],
        "category": "languages",
    },
    "rustc": {
        "desc": "Rust compiler — compile Rust source code",
        "keywords": ["rust", "compile", "build", "language", "binary"],
        "category": "languages",
    },
    "make": {
        "desc": "GNU make — build automation tool",
        "keywords": ["build", "compile", "automation", "makefile", "target", "dependency"],
        "category": "build",
    },
    "cmake": {
        "desc": "Cross-platform build system generator",
        "keywords": ["build", "compile", "cmake", "makefile", "project", "configure"],
        "category": "build",
    },
    "gcc": {
        "desc": "GNU C compiler — compile C programs",
        "keywords": ["c", "compile", "build", "gcc", "linker", "binary"],
        "category": "build",
    },

    # ── databases ──
    "sqlite3": {
        "desc": "SQLite CLI — manage SQLite databases from the command line",
        "keywords": ["sql", "database", "sqlite", "query", "table", "select", "insert", "db"],
        "category": "databases",
    },
    "psql": {
        "desc": "PostgreSQL interactive terminal — query and manage PostgreSQL databases",
        "keywords": ["sql", "database", "postgresql", "postgres", "query", "table", "select", "psql"],
        "category": "databases",
    },

    # ── cloud ──
    "aws": {
        "desc": "AWS CLI — manage Amazon Web Services from the command line",
        "keywords": ["aws", "amazon", "cloud", "s3", "ec2", "lambda", "iam", "deploy"],
        "category": "cloud",
    },
    "gcloud": {
        "desc": "Google Cloud CLI — manage Google Cloud Platform resources",
        "keywords": ["google", "cloud", "gcp", "compute", "storage", "deploy", "iam"],
        "category": "cloud",
    },
    "az": {
        "desc": "Azure CLI — manage Microsoft Azure resources",
        "keywords": ["azure", "microsoft", "cloud", "vm", "storage", "deploy", "resource"],
        "category": "cloud",
    },
    "terraform": {
        "desc": "Infrastructure as Code — define and provision cloud infrastructure",
        "keywords": ["infrastructure", "iac", "cloud", "provision", "deploy", "state", "plan", "apply"],
        "category": "cloud",
    },

    # ── system ──
    "systemctl": {
        "desc": "Control the systemd system and service manager",
        "keywords": ["systemd", "service", "daemon", "start", "stop", "restart", "status", "enable"],
        "category": "system",
    },
    "journalctl": {
        "desc": "Query the systemd journal — view and filter logs",
        "keywords": ["log", "journal", "systemd", "debug", "error", "service", "boot"],
        "category": "system",
    },
    "htop": {
        "desc": "Interactive process viewer — monitor system resources",
        "keywords": ["process", "cpu", "memory", "monitor", "system", "resource", "interactive"],
        "category": "system",
    },
    "ncdu": {
        "desc": "NCurses Disk Usage — interactive disk usage analyzer",
        "keywords": ["disk", "usage", "space", "file", "directory", "size", "interactive"],
        "category": "system",
    },

    # ── compression / archive ──
    "tar": {
        "desc": "Tape archiver — create and extract archive files",
        "keywords": ["archive", "compress", "extract", "tar", "gz", "backup", "bundle"],
        "category": "archives",
    },
    "gzip": {
        "desc": "Compress or decompress files using Lempel-Ziv coding",
        "keywords": ["compress", "decompress", "gz", "file", "zip"],
        "category": "archives",
    },
    "unzip": {
        "desc": "List, test, and extract compressed files in a ZIP archive",
        "keywords": ["unzip", "extract", "archive", "zip", "decompress"],
        "category": "archives",
    },
    "zip": {
        "desc": "Package and compress files into a ZIP archive",
        "keywords": ["zip", "compress", "archive", "package", "bundle"],
        "category": "archives",
    },

    # ── other tools ──
    "fzf": {
        "desc": "Command-line fuzzy finder — filter lists interactively",
        "keywords": ["fuzzy", "find", "filter", "interactive", "search", "select", "picker", "menu"],
        "category": "interactive",
    },
    "bat": {
        "desc": "cat clone with syntax highlighting and Git integration",
        "keywords": ["view", "cat", "syntax", "highlight", "file", "preview", "pager"],
        "category": "file-management",
    },
    "eza": {
        "desc": "Modern replacement for ls — list files with colors and icons",
        "keywords": ["list", "directory", "file", "ls", "tree", "color"],
        "category": "file-management",
    },
    "opencli": {
        "desc": "OpenClaw CLI — manage the OpenClaw AI agent gateway",
        "keywords": ["openclaw", "agent", "ai", "gateway", "skill", "plugin"],
        "category": "ai-tools",
    },
    "code": {
        "desc": "Visual Studio Code editor — open files, folders, and manage extensions",
        "keywords": ["editor", "code", "vscode", "ide", "file", "diff", "extension"],
        "category": "editors",
    },
    "mihomo": {
        "desc": "Clash meta kernel — proxy and network routing (official skill exists)",
        "keywords": ["proxy", "mihomo", "clash", "vpn", "node", "routing", "network", "switch"],
        "category": "networking",
    },
    "mmx": {
        "desc": "MiniMax multimodal AI toolkit — generate images, video, music, documents",
        "keywords": ["ai", "minimax", "generate", "image", "video", "music", "document", "multimodal"],
        "category": "ai-tools",
    },
}

# Common shell/Unix utilities to skip during PATH scanning
UNIX_BASICS = {
    "[", "alias", "apt", "apt-get", "arch", "bash", "bg", "bind", "builtin",
    "bunzip2", "bzcat", "bzip2", "cal", "case", "cat", "cd", "chgrp",
    "chmod", "chown", "clear", "cmp", "comm", "command", "compgen", "cp",
    "cpio", "csplit", "cut", "dash", "date", "dd", "declare", "df", "diff",
    "dir", "dircolors", "dirname", "dirs", "dmidecode", "dmesg", "done",
    "dpkg", "du", "echo", "ed", "egrep", "elif", "else", "enable", "env",
    "eval", "exec", "exit", "expand", "export", "expr", "factor", "false",
    "fc", "fg", "fgrep", "fi", "file", "findmnt", "fmt", "fold", "for",
    "free", "function", "getopt", "getopts", "groups", "gunzip", "gzip",
    "hash", "head", "history", "hostid", "hostname", "id", "if", "info",
    "install", "jobs", "join", "kill", "ld", "ldd", "less", "let", "link",
    "ln", "local", "locale", "logger", "login", "logname", "logout", "lp",
    "ls", "lscpu", "lsblk", "lsof", "man", "mapfile", "md5sum", "mkdir",
    "mkfifo", "mknod", "mktemp", "more", "mount", "mpstat", "mv", "namei",
    "newgrp", "nice", "nl", "nohup", "nproc", "od", "paste", "pip", "pip3",
    "popd", "pr", "printenv", "printf", "ps", "pushd", "pwd", "python",
    "read", "readarray", "readlink", "realpath", "rename", "renice", "rev",
    "rm", "rmdir", "run-parts", "sdiff", "select", "seq", "set", "sh",
    "sha1sum", "sha256sum", "shift", "shopt", "shred", "shuf", "sleep",
    "sort", "source", "split", "stat", "stdbuf", "strings", "strip", "stty",
    "su", "sudo", "sum", "suspend", "sync", "tac", "tail", "tee", "test",
    "time", "timeout", "times", "touch", "tput", "tr", "trap", "true",
    "truncate", "tset", "tsort", "tty", "type", "typeset", "ulimit",
    "umask", "umount", "unalias", "uname", "unexpand", "uniq", "unlink",
    "unset", "until", "updatedb", "uptime", "users", "vi", "vim", "wait",
    "wall", "watch", "w", "whatis", "whereis", "which", "while", "who",
    "whoami", "write", "xdg-open", "yes", "zcat", "zless", "zmore",
}

# Flattened set of known binary names (for quick lookup in discover)
_KNOWN_BINARIES = set(KNOWN_CLI_KB.keys())


# ── helpers ────────────────────────────────────────────────────

def _run(cmd, timeout=10):
    try:
        r = subprocess.run(
            cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
            universal_newlines=True, timeout=timeout
        )
        return r.returncode, r.stdout, r.stderr
    except FileNotFoundError:
        return -1, "", "binary not found: {}".format(cmd[0])
    except subprocess.TimeoutExpired:
        return -2, "", "timed out"


def _whereis(binary):
    return subprocess.call(["which", binary],
                           stdout=subprocess.DEVNULL,
                           stderr=subprocess.DEVNULL) == 0


def _find_official_skill(name):
    for root in [SKILLS_DIR, Path.home() / ".openclaw" / "skills"]:
        sp = root / name / "SKILL.md"
        if sp.is_file():
            with open(sp, encoding="utf-8") as fh:
                head = fh.read(500)
            if head.startswith("---") and "name:" in head:
                return str(sp)
    return None


# ── help extraction ─────────────────────────────────────────────

def _fetch_help_text(binary, subcommand=None):
    """Get help text trying --help, -h, help, man, bare invocation."""
    if subcommand:
        attempts = [
            [binary, subcommand, "-h"],
            [binary, subcommand, "--help"],
            [binary, "help", subcommand],
        ]
    else:
        attempts = [
            [binary, "--help"],
            [binary, "-h"],
            [binary, "help"],
        ]

    for cmd in attempts:
        code, out, err = _run(cmd, timeout=15)
        output = (out + err).strip()
        if len(output) > 80:
            return output

    # man page fallback
    code, out, err = _run(["man", binary], timeout=10)
    if len(out.strip()) > 80:
        return out.strip()[:16384]

    # bare invocation
    code, out, err = _run([binary], timeout=10)
    output = (out + err).strip()
    return output if output else ""


def _fetch_version(binary):
    """Extract version from --version output.

    Handles: "git version 2.17.1", "gh version 2.92.0",
    "curl 7.58.0", "v22.14.0", "Python 3.6.9", "mihomo-cli v2.8.1"
    """
    try:
        code, out, err = _run([binary, "--version"], timeout=5)
        output = (out + err).strip()
        if not output:
            code, out, err = _run([binary, "-V"], timeout=5)
            output = (out + err).strip()
        if output:
            m = re.search(r'\d+\.\d+(?:\.\d+)?(?:[a-z]\d*)?', output)
            if m:
                return m.group(0)
    except Exception:
        pass
    return None


# ── P1: Smart summary extraction ────────────────────────────────

def _extract_summary(text):
    """Extract a one-line tool description from --help output.

    Strategy (first to succeed wins):
      1. Find the NAME / DESCRIPTION section heading and grab the next line
      2. Find the first sentence after the usage block
      3. Find the first substantial non-flag line
    Returns None if nothing useful found.
    """
    if not text or len(text) < 20:
        return None

    lines = [l.strip() for l in text.split("\n")]

    # Strategy 1: NAME or DESCRIPTION section
    in_target = False
    for i, line in enumerate(lines):
        lo = line.lower().rstrip(":")
        if lo in ("name", "description", "overview"):
            in_target = True
            continue
        if in_target and len(line) > 15 and not line.startswith("-"):
            # Grab up to the next blank or heading
            parts = []
            for j in range(i, min(i + 8, len(lines))):
                l2 = lines[j].strip()
                if not l2 or l2.endswith(":") or l2.startswith("-"):
                    break
                parts.append(l2)
            if parts:
                return " ".join(parts)[:300]
            return line[:200]
        if in_target and (line == "" or line.startswith("-")):
            continue

    # Strategy 2: first sentence-like line after skipping usage/options blurb
    for i, line in enumerate(lines):
        if len(line) < 15:
            continue
        lo = line.lower()
        # Skip boilerplate
        if any(lo.startswith(w) for w in ("usage", "用法", "使い方", "usage:", "用法:", "使い方:")):
            continue
        if lo.startswith("-") and ("--" in line or "-" in line[:3]):
            continue
        # Look for a substantive line
        if re.match(r'^[A-Z]', line) and len(line) > 20:
            # Stop at next blank or flag line
            return line[:300]

    # Strategy 3: any substantial non-flag line
    for line in lines:
        if len(line) > 20 and not line.startswith("-") and not line.startswith("#"):
            return line[:200]

    return None


def _clean_help_raw(text):
    """Prepare help text for LLM consumption.

    - Strip version/copyright headers
    - Remove excessive blank lines
    - Keep first 5000 chars (the useful part is always at the top)
    """
    lines = text.split("\n")
    cleaned = []
    skip_header = True
    for line in lines:
        s = line.strip()
        # Skip version/copyright noise at the top
        if skip_header:
            if any(w in s.lower() for w in ("version", "copyright", "license",
                   "all rights reserved", "maintainer", "built on", "contributors")):
                continue
            if s and len(s) < 80 and not s.startswith("-"):
                skip_header = False
        cleaned.append(line)
    result = "\n".join(cleaned).strip()
    # Deduplicate blank lines
    import re as _re
    result = _re.sub(r'\n{3,}', '\n\n', result)
    return result[:5000]


def _format_commands_text(parsed_subs):
    """Build a compact command summary for LLM consumption."""
    if not parsed_subs:
        return ""
    lines = []
    for s in parsed_subs[:20]:
        name = s["name"]
        desc = s.get("desc", "")
        if desc:
            lines.append("{} — {}".format(name, desc))
        else:
            lines.append(name)
    return "\n".join(lines)


def _format_options_text(options):
    """Build a compact options summary for LLM consumption."""
    if not options:
        return ""
    lines = []
    for o in options[:15]:
        flag = o["flag"]
        aliases = ", ".join(o.get("aliases", []))
        label = "{} ({})".format(flag, aliases) if aliases else flag
        value = " <{}>".format(o["value"]) if o.get("value") else ""
        desc = o.get("desc", "")
        lines.append("{}{} — {}".format(label, value, desc))
    return "\n".join(lines)


def _extract_help(binary):
    """Extract structured help: usage, subcommands with options, global options."""
    text = _fetch_help_text(binary)
    if not text:
        return {"binary": binary, "help_raw": "{} — help unavailable".format(binary)}

    parsed_subs = _parse_subcommands(text)
    parsed_opts = _parse_options(text)

    result = {
        "binary": binary,
        "version": _fetch_version(binary),
        "help_raw": _clean_help_raw(text),
        "usage": _extract_usage(text),
        "summary": _extract_summary(text),
        "commands_text": _format_commands_text(parsed_subs),
        "options_text": _format_options_text(parsed_opts),
        "subcommands": {},
        "global_options": [],
    }

    top_subs = parsed_subs[:12]

    for s in top_subs:
        name = s["name"]
        sub_help = _fetch_help_text(binary, name)
        sub_info = {
            "desc": s["desc"],
            "usage": _extract_usage(sub_help) if sub_help else "",
            "options": _parse_options(sub_help) if sub_help else [],
        }
        result["subcommands"][name] = sub_info

    return result


def _extract_usage(text):
    """Extract usage line(s)."""
    lines = []
    in_usage = False
    for line in text.split("\n"):
        s = line.strip()
        if re.match(r'^(usage|用法|使用方式|使い方)\s*[:：]', s, re.I):
            in_usage = True
            continue
        if in_usage:
            if s == "" or re.match(r'^[A-Z][a-z]+(\s+[a-z]+)*\s*[:：]', s):
                break
            if len(s) > 5:
                lines.append(s)
    if not lines:
        for line in text.split("\n"):
            s = line.strip()
            if s and not s.startswith("#") and len(s) > 10:
                lines.append(s)
                break
    return "\n".join(lines[:5])


def _parse_subcommands(text):
    """Extract subcommands from indented help text."""
    subs = []
    seen = set()
    noise = {
        "or", "and", "the", "for", "see", "a", "an", "of", "to", "in",
        "usage", "options", "commands", "examples", "arguments", "flags",
        "help", "all", "on", "off", "yes", "no", "true", "false",
        "be", "is", "it", "if", "by", "at", "also", "note", "use",
        "after", "before", "with", "from", "into", "out", "up", "per",
    }
    for m in re.finditer(r'(?m)^\s{2,}([a-z][a-z0-9._-]{2,30})\s{2,}(.+)', text):
        name, desc = m.group(1), m.group(2).strip()[:150]
        name = name.rstrip("._")
        if name.lower() not in noise and name not in seen:
            seen.add(name)
            subs.append({"name": name, "desc": desc})
    return subs


def _parse_options(text):
    """Extract options/flags with types."""
    options = []
    seen = set()
    skip = {"-h", "--help", "-v", "--version", "-V"}
    for m in re.finditer(
        r'(?m)^\s{0,6}(-{1,2}[\w][\w-]*(?:,\s*-{1,2}[\w][\w-]*)?)'
        r'(?:\s+(?!-)\S+)?\s{2,}(.+)',
        text
    ):
        flag_group = m.group(1)
        desc = m.group(2).strip()[:120] if m.group(2) else ""
        flags = [f.strip() for f in flag_group.split(",")]
        primary = next((f for f in flags if f.startswith("--")), flags[0])
        if primary in skip:
            continue
        value = ""
        type_match = re.search(r'(?:<([^>]+)>|\[=?(.+?)\])', desc)
        if type_match:
            value = type_match.group(1) or type_match.group(2)
        if primary not in seen:
            seen.add(primary)
            options.append({
                "flag": primary,
                "aliases": [f for f in flags if f != primary],
                "value": value,
                "desc": desc[:100]
            })
    return options[:25]


# ── P2: Keyword index ───────────────────────────────────────────

def _build_keyword_index():
    """Build a reverse index: keyword → [tool names].

    Sources: built-in KNOWN_CLI_KB keywords, then fallback to description tokens.
    Stored as REGISTRY_DIR/.keywords.json.
    """
    index = {}

    # From knowledge base
    for name, kb in KNOWN_CLI_KB.items():
        for kw in kb.get("keywords", []):
            k = kw.lower()
            if k not in index:
                index[k] = []
            if name not in index[k]:
                index[k].append(name)

    # Supplement: extract keywords from registered tool descriptions
    for entry_path in sorted(REGISTRY_DIR.glob("*.json")):
        name = entry_path.stem
        if name.startswith("."):
            continue
        try:
            d = json.loads(entry_path.read_text(encoding="utf-8"))
        except Exception:
            continue
        desc = d.get("description", "")
        if not desc or desc.startswith("External CLI:"):
            continue
        # Tokenize description for additional keywords
        tokens = set(re.findall(r'[a-z0-9]{3,}', desc.lower()))
        noise = {"the", "and", "for", "with", "from", "into", "that", "this",
                 "tool", "command", "line", "cli", "external", "file"}
        for t in tokens - noise:
            if t not in index:
                index[t] = []
            if name not in index[t]:
                index[t].append(name)

    return index


def _save_keyword_index():
    """Save the keyword index to disk."""
    REGISTRY_DIR.mkdir(parents=True, exist_ok=True)
    index = _build_keyword_index()
    KEYWORD_INDEX_PATH.write_text(
        json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8")
    return len(index)


# ── commands ─────────────────────────────────────────────────────

def _resolve_description(name, help_text):
    """Resolve description with priority: KB > --help summary > fallback."""
    # P0: Built-in knowledge base
    kb = KNOWN_CLI_KB.get(name)
    if kb and kb.get("desc"):
        return kb["desc"]

    # P1: Smart extraction from --help
    summary = _extract_summary(help_text) if help_text else None
    if summary:
        return summary

    # Fallback
    return "External CLI: {}".format(name)


def _resolve_keywords(name):
    """Get keywords for a tool. Returns empty list if not in KB."""
    kb = KNOWN_CLI_KB.get(name)
    if kb:
        return kb.get("keywords", [])
    return []


def cmd_register(args):
    name = args.cli
    binary = args.binary or name
    desc = args.desc or ""

    if not args.force and not _whereis(binary):
        print("Warning: {} not found on PATH. Use --force to register anyway.".format(binary))

    official = _find_official_skill(name)
    info = _extract_help(binary)

    # P0+P1: smart description resolution
    description = desc or _resolve_description(name, info.get("help_raw", ""))

    # P0: keywords from knowledge base
    keywords = _resolve_keywords(name)

    entry = {
        "name": name,
        "binary": binary,
        "description": description,
        "official_skill": official,
        "registered_at": datetime.now().isoformat(),
        "keywords": keywords,
        "auto_discovered": info,
    }

    REGISTRY_DIR.mkdir(parents=True, exist_ok=True)
    path = REGISTRY_DIR / "{}.json".format(name)
    path.write_text(json.dumps(entry, indent=2, ensure_ascii=False), encoding="utf-8")

    print("Registered: {} -> {}".format(name, path))
    print("   {}".format(description[:80]))
    if official:
        print("   Official skill: {} (takes priority)".format(official))
    subs = info.get("subcommands", {})
    opts = info.get("global_options", [])
    print("   {} subcommands, {} options, {} keywords".format(
        len(subs), len(opts), len(keywords)))


def cmd_list(args):
    entries = [e for e in sorted(REGISTRY_DIR.glob("*.json")) if not e.name.startswith(".")]
    if not entries:
        print("No CLI tools registered.")
        print("Try: python3 cli-registry.py discover")
        return

    if args.format == "json":
        result = {}
        for e in entries:
            d = json.loads(e.read_text(encoding="utf-8"))
            ad = d.get("auto_discovered", {})
            result[d["name"]] = {
                "binary": d.get("binary"),
                "description": d.get("description"),
                "has_official_skill": bool(d.get("official_skill")),
                "subcommands": len(ad.get("subcommands", {})),
                "global_options": len(ad.get("global_options", [])),
                "keywords": d.get("keywords", []),
            }
        print(json.dumps(result, indent=2, ensure_ascii=False))
    else:
        header = "{:<18s} {:<13s} {:<8s} {:>4s} {:>4s} {:>3s}  {}".format(
            "NAME", "BINARY", "OFFICIAL", "SUBS", "OPTS", "KW", "DESCRIPTION")
        print(header)
        print("-" * 90)
        for e in entries:
            d = json.loads(e.read_text(encoding="utf-8"))
            official = "yes" if d.get("official_skill") else "-"
            ad = d.get("auto_discovered", {})
            subs = len(ad.get("subcommands", {}))
            opts = len(ad.get("global_options", []))
            kw_cnt = len(d.get("keywords", []))
            print("{:<18s} {:<13s} {:<8s} {:>4d} {:>4d} {:>3d}  {}".format(
                d["name"], d.get("binary", d["name"]),
                official, subs, opts, kw_cnt, d.get("description", "")[:35]))


def cmd_lookup(args):
    path = REGISTRY_DIR / "{}.json".format(args.cli)
    if not path.is_file():
        print("Not registered: {}".format(args.cli))
        print("  Try: python3 cli-registry.py register {}".format(args.cli))
        sys.exit(1)

    d = json.loads(path.read_text(encoding="utf-8"))

    print("# CLI: {}".format(d["name"]))
    print("Binary: {}".format(d.get("binary")))
    if d.get("description"):
        print("Description: {}".format(d["description"]))
    ad = d.get("auto_discovered", {})
    if ad.get("version"):
        print("Registered version: {}".format(ad["version"]))
        print("Official Skill: {} (takes priority)".format(d["official_skill"]))
    keywords = d.get("keywords", [])
    if keywords:
        print("Keywords: {}".format(", ".join(keywords)))

    ad = d.get("auto_discovered", {})

    summary = ad.get("summary")
    if summary:
        print("\n## Help Summary")
        print(summary)

    usage = ad.get("usage")
    if usage:
        print("\n## Usage")
        print("```")
        print(usage)
        print("```")

    globals_opts = ad.get("global_options", [])
    if globals_opts:
        print("\n## Global Options ({})".format(len(globals_opts)))
        for o in globals_opts[:10]:
            v = " <{}>".format(o["value"]) if o.get("value") else ""
            aliases = " ({})".format(", ".join(o["aliases"])) if o.get("aliases") else ""
            print("  {}{}  {} {}".format(o["flag"], v, o.get("desc", ""), aliases))

    subs = ad.get("subcommands", {})
    if subs:
        print("\n## Subcommands ({})".format(len(subs)))
        for name, info in subs.items():
            print("\n### {} — {}".format(name, info.get("desc", "")))
            sub_usage = info.get("usage")
            if sub_usage:
                print("  Usage: `{}`".format(sub_usage.split("\n")[0][:120]))
            sub_opts = info.get("options", [])
            if sub_opts:
                for o in sub_opts[:8]:
                    v = " <{}>".format(o["value"]) if o.get("value") else ""
                    print("    {} {}  {}".format(o["flag"], v, o.get("desc", "")[:60]))


def cmd_search(args):
    """Search registered tools by keyword."""
    keywords = [kw.lower() for kw in args.keyword]

    # Try to load the keyword index
    index = {}
    if KEYWORD_INDEX_PATH.is_file():
        try:
            index = json.loads(KEYWORD_INDEX_PATH.read_text(encoding="utf-8"))
        except Exception:
            pass

    # Also check individual registry entries
    hits = {}  # name → set of matched keywords

    # Search index
    for kw in keywords:
        matches = index.get(kw, [])
        for name in matches:
            if name not in hits:
                hits[name] = set()
            hits[name].add(kw)

    # Search entries directly (covers tools registered after last index build)
    for entry_path in sorted(REGISTRY_DIR.glob("*.json")):
        name = entry_path.stem
        if name.startswith("."):
            continue
        try:
            d = json.loads(entry_path.read_text(encoding="utf-8"))
        except Exception:
            continue

        # Check built-in keywords
        for kw in d.get("keywords", []):
            for qk in keywords:
                if qk in kw.lower():
                    if name not in hits:
                        hits[name] = set()
                    hits[name].add(qk)

        # Check description
        desc = d.get("description", "").lower()
        for qk in keywords:
            if qk in desc and name not in hits:
                hits[name] = set()
                hits[name].add(qk)

    if not hits:
        print("No tools found for: {}".format(", ".join(keywords)))
        print("Try: python3 cli-registry.py list")
        return

    # Sort by number of matched keywords (descending)
    sorted_hits = sorted(hits.items(), key=lambda x: len(x[1]), reverse=True)

    print("Search: {}".format(", ".join(keywords)))
    print("{:<18s} {:>4s}  {}".format("TOOL", "MATCH", "DESCRIPTION"))
    print("-" * 80)
    for name, matched in sorted_hits:
        entry_path = REGISTRY_DIR / "{}.json".format(name)
        desc = ""
        if entry_path.is_file():
            try:
                d = json.loads(entry_path.read_text(encoding="utf-8"))
                desc = d.get("description", "")[:55]
            except Exception:
                pass
        print("{:<18s} {:>4d}  {}".format(name, len(matched), desc))

    print("\n{} tools matched. Use 'lookup <name>' for details.".format(len(hits)))


def cmd_discover(args):
    count = 0

    # Phase 1: known list (from knowledge base)
    for binary in _KNOWN_BINARIES:
        if (REGISTRY_DIR / "{}.json".format(binary)).is_file():
            continue
        if _whereis(binary):
            print("Found: {} ...".format(binary), end=" ", flush=True)
            try:
                cmd_register(argparse.Namespace(
                    cli=binary, binary=binary, desc="", force=False))
                count += 1
            except Exception as exc:
                print("failed ({})".format(exc))

    # Phase 2: deep PATH scan
    if args.scan:
        print("\nScanning PATH for additional binaries...")
        seen = set(_KNOWN_BINARIES) | {
            e.stem for e in REGISTRY_DIR.glob("*.json")
        }
        for path_dir in os.environ.get("PATH", "").split(os.pathsep):
            d = Path(path_dir)
            if not d.is_dir():
                continue
            try:
                for entry in d.iterdir():
                    name = entry.name
                    if name in seen or name in UNIX_BASICS:
                        continue
                    if len(name) < 2 or len(name) > 30:
                        continue
                    if not re.match(r'^[a-z][a-z0-9._-]+$', name):
                        continue
                    if entry.is_file() and os.access(str(entry), os.X_OK):
                        seen.add(name)
                        print("Found: {} ...".format(name), end=" ", flush=True)
                        try:
                            cmd_register(argparse.Namespace(
                                cli=name, binary=str(entry), desc="", force=False))
                            count += 1
                        except Exception as exc:
                            print("failed ({})".format(exc))
            except PermissionError:
                continue

    # Phase 3: custom directory
    if args.scan_path:
        sp = Path(args.scan_path)
        if sp.is_dir():
            for f in sp.iterdir():
                if f.is_file() and os.access(str(f), os.X_OK):
                    name = f.name
                    if not (REGISTRY_DIR / "{}.json".format(name)).is_file():
                        print("Found: {} ...".format(name), end=" ", flush=True)
                        try:
                            cmd_register(argparse.Namespace(
                                cli=name, binary=str(f), desc="", force=False))
                            count += 1
                        except Exception as exc:
                            print("failed ({})".format(exc))

    # P2: Build keyword index after discovering
    if count > 0:
        kw_count = _save_keyword_index()
        print("\nKeyword index: {} terms".format(kw_count))

    print("\nRegistered {} new CLI tools.".format(count))


def cmd_remove(args):
    path = REGISTRY_DIR / "{}.json".format(args.cli)
    if path.is_file():
        path.unlink()
        print("Removed: {}".format(args.cli))
        # Rebuild keyword index
        _save_keyword_index()
    else:
        print("Not registered: {}".format(args.cli))


def cmd_help_cli(args):
    binary = args.cli
    p = REGISTRY_DIR / "{}.json".format(args.cli)
    if p.is_file():
        binary = json.loads(p.read_text(encoding="utf-8")).get("binary", binary)
    output = _fetch_help_text(binary)
    if output:
        print(output[:5000])
    else:
        print("No help output from {}".format(binary))


def cmd_check_stale(args):
    """Check for tools whose installed version differs from registry."""
    entries = [e for e in sorted(REGISTRY_DIR.glob("*.json")) if not e.name.startswith(".")]
    if not entries:
        print("No tools registered.")
        return

    stale = []
    current = 0
    for e in entries:
        try:
            d = json.loads(e.read_text(encoding="utf-8"))
        except Exception:
            continue
        name = d.get("name")
        binary = d.get("binary", name)
        ad = d.get("auto_discovered", {})
        registered_ver = ad.get("version")
        installed_ver = _fetch_version(binary)

        if installed_ver is None:
            continue

        current += 1
        if registered_ver and installed_ver != registered_ver:
            stale.append((name, registered_ver, installed_ver))

    print("{} tools with version info".format(current))
    if stale:
        print("{} stale (installed ≠ registered):\n".format(len(stale)))
        for name, old, new in stale:
            print("  {}: {} → {}".format(name, old, new))
        if args.update:
            print("\nRe-registering stale tools...")
            for name, old, new in stale:
                cmd_register(argparse.Namespace(
                    cli=name, binary=name, desc="", force=True))
            _save_keyword_index()
            print("Done.")
        else:
            print("\nDry run. Use --update to re-register.")
    else:
        print("All up to date.")

def main():
    parser = argparse.ArgumentParser(
        description="CLI Registry for cli-hub AgentSkill",
        prog="cli-registry")
    parser.add_argument("--version", action="version", version=VERSION)

    sub = parser.add_subparsers(dest="command")
    sub.required = True

    p = sub.add_parser("register", help="Register a CLI tool")
    p.add_argument("cli", help="CLI name")
    p.add_argument("--binary", help="Binary name if different")
    p.add_argument("--desc", help="Description (override auto-detection)")
    p.add_argument("--force", action="store_true",
                   help="Register even if binary not on PATH")

    p = sub.add_parser("list", help="List registered CLIs")
    p.add_argument("--format", default="table", choices=["table", "json"])

    p = sub.add_parser("lookup", help="Look up a CLI")
    p.add_argument("cli", help="CLI name")

    p = sub.add_parser("search", help="Search tools by keyword")
    p.add_argument("keyword", nargs="+", help="Keyword(s) to search for")

    p = sub.add_parser("discover", help="Auto-discover CLI binaries")
    p.add_argument("--scan", action="store_true",
                   help="Deep scan all PATH directories (slower, finds more)")
    p.add_argument("--scan-path", help="Extra directory to scan for executables")

    p = sub.add_parser("check-stale", help="Check for tools with updated versions")
    p.add_argument("--update", action="store_true",
                   help="Re-register tools whose version changed")

    p = sub.add_parser("remove", help="Remove from registry")
    p.add_argument("cli", help="CLI name")

    p = sub.add_parser("help", help="Fetch live --help output")
    p.add_argument("cli", help="CLI name or binary")

    args = parser.parse_args()
    {
        "register": cmd_register,
        "list": cmd_list,
        "lookup": cmd_lookup,
        "search": cmd_search,
        "discover": cmd_discover,
        "remove": cmd_remove,
        "help": cmd_help_cli,
        "check-stale": cmd_check_stale,
    }[args.command](args)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)