Skill: nemo4110/041-llm-wiki (version 1.4.1)

--- name: llm-wiki description: "Karpathy's llm-wiki pattern implementation — cumulative knowledge management for AI agents" version: 1.4.1 author: "@yourname" license: MIT repository: "https://github.com/Nemo4110/llm-wiki.git" # Supported platforms platforms: - claude-code - openclaw - generic-llm-agent # Required capabilities from host agent capabilities: - filesystem-read - filesystem-write - llm-completion # Entry points for different modes entryPoints: protocol: "CLAUDE.md" agent-guide: "AGENTS.md" agent-bridge: "scripts/agent-bridge.py" cli: "src/llm_wiki/commands.py" # Hooks (optional integration) hooks: available: false note: "Protocol mode requires no hooks. CLI mode available for scripting." # Dependencies dependencies: required: [] optional: - name: python version: ">=3.8" reason: "CLI mode only" - name: click version: ">=8.0.0" reason: "CLI framework" - name: pyyaml version: ">=6.0" reason: "YAML parsing" - name: pymupdf version: ">=1.25.0" reason: "PDF processing (recommended)" - name: numpy version: ">=1.24.0" reason: "Vector operations for embedding retrieval" - name: httpx version: ">=0.27.0" reason: "HTTP client for Ollama/local embedding services" - name: openai version: ">=1.0.0" reason: "OpenAI embedding API" - name: mcp version: ">=1.0.0" reason: "MCP SDK for remote embedding providers" # Installation methods installation: - method: uv command: "uv venv && uv pip install -r src/requirements.txt --python .venv/Scripts/python.exe" note: "Fastest, recommended if uv available" - method: conda command: "conda create -n llm-wiki python=3.11 && pip install -r src/requirements.txt" note: "For data science environments" - method: pip command: "python -m venv .venv && pip install -r src/requirements.txt" note: "Standard Python" - method: none command: null note: "Protocol mode requires no installation" # Core functions exposed to agent functions: ingest: description: "Ingest source material into wiki" trigger: "Please ingest material" inputs: - name: source_path type: string description: "Path to source file in sources/" workflow: - Read source content - Extract source time metadata: publication/release/post date, collection date, ingest date, and date precision when available - Extract key insights - Identify/create affected wiki pages - Dynamic linking: run `python scripts/agent-bridge.py link --source <new_page> --mode light` to discover related pages - For high-confidence relations (score >= 0.5), apply merge strategy to backward-update existing pages - Update cross-references - Preserve temporal relations in page frontmatter (`sources_meta`) and, when useful, a `## 时间线` / `## Timeline` section - Create stub pages for any new [[Dead Link]] introduced in the content - Append to log.md - For batch ingest (>=2 sources), run `python scripts/agent-bridge.py relink --since <date> --mode deep` link: description: "Discover and merge relationships between wiki pages" trigger: "Link wiki pages" inputs: - name: source type: string description: "Source page title" - name: target type: string description: "Target page title (optional, for merge execution)" - name: mode type: string description: "light or deep" workflow: - Run `python scripts/agent-bridge.py link --source <page> --mode light` to discover relations - Run `python scripts/agent-bridge.py link --source <page> --target <page> --strategy <strategy>` to merge - Review diff before applying relink: description: "Batch global relationship discovery for recent pages" trigger: "Global linking" inputs: - name: since type: string description: "Date cutoff (YYYY-MM-DD)" - name: mode type: string description: "light or deep" workflow: - Run `python scripts/agent-bridge.py relink --since <date> --mode deep` to batch-link all recent pages - Review the generated relation report for high-confidence connections - For each high-confidence pair, run `python scripts/agent-bridge.py link --source <new> --target <old> --strategy <strategy>` query: description: "Query wiki knowledge base" trigger: "Query wiki" inputs: - name: question type: string description: "User question about wiki content" workflow: - Read wiki/index.md - Navigate through [[links]] - Synthesize answer with citations - Optional: archive response lint: description: "Health check for wiki" trigger: "Check wiki health" checks: - orphan pages - dead links: link targets must be real wiki file stems - stale pages - empty pages - duplicate titles - non-canonical links - draft pages - contradictions # File structure structure: protocol: "CLAUDE.md" agent-guide: "AGENTS.md" specification: "SKILL.md" changelog: "log.md" agent-bridge: "scripts/agent-bridge.py" sources: "sources/" wiki: "wiki/" assets: "assets/" scripts: "scripts/" src: "src/" examples: "examples/" # Related resources related: - name: "Karpathy's llm-wiki gist" url: "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" - name: "Sage-Wiki" url: "https://github.com/xoai/sage-wiki" note: "Alternative full-featured implementation" --- # CLI Reference ## Temporal Metadata Requirement Agents should distinguish wiki maintenance dates from source/work dates: - `created` / `updated`: when the wiki page was created or edited. - `sources_meta[].published`: when the paper, post, release, documentation, or work appeared. - `sources_meta[].collected`: when the user saved, collected, or imported it. - `sources_meta[].ingested`: when llm-wiki processed it. - `sources_meta[].date_precision`: `day`, `month`, `year`, or `unknown`. When multiple works are discussed, preserve historical order in the prose or in a `## 时间线` / `## Timeline` section. Do not infer missing months or days. ### Visible Time Anchor Format Use native Markdown to make time visible without relying on HTML, colors, or wide tables: - Start `### 时间定位` / `### Temporal Position` with a blockquote summary: - `> **时间范围**：YYYY.MM-YYYY.MM` - `> **阶段判断**：one concise sentence about the historical stage` - For dated nodes, prefer list items that start with a bold bracketed anchor: - `- **[2025.09] Work or event name**: why it matters.` - `- **[2026.01] Follow-up work**: what changed.` - Use `**[YYYY.MM-YYYY.MM]**` for ranges and `**[YYYY]**` when only year precision is reliable. - Apply this especially in `### 时间定位`, `## 时间线`, `### 与已有知识的关系`, `## Related Pages`, and source-note summaries. - Prefer readable lists over wide Markdown tables when rows contain wiki links or long relationship text. Use tables only for short, dense, comparison-oriented metadata. ## Agent Bridge (Recommended for Agents) Use `scripts/agent-bridge.py` as the single entry point for all tool-assisted operations: ```bash # Environment check python scripts/agent-bridge.py check # Discover relations for a new page python scripts/agent-bridge.py link --source "NewPage" --mode light # Execute merge with diff review python scripts/agent-bridge.py link --source "NewPage" --target "OldPage" --strategy append_related # Batch global linking for recent pages python scripts/agent-bridge.py relink --since 2026-04-20 --mode deep # Health check python scripts/agent-bridge.py lint # Status overview python scripts/agent-bridge.py status ``` **Why Agent Bridge?** - Single obvious entry point — no guessing whether to use protocol mode or CLI mode - Structured Markdown output — human-readable and machine-parseable - Execution traceability — detailed logging with file:line references to stderr - Auto-detects Python environment (uv venv / conda / system) ## Protocol Mode (Natural Language) For tasks requiring LLM judgment (content extraction, synthesis, strategy selection): ``` "Please ingest sources/paper.pdf into wiki" "Query wiki: What is the difference between Transformer and RNN?" "Check wiki health" ``` ## Legacy CLI Mode (Optional) Direct library access for scripting or debugging: ```bash # Show wiki status overview python -m src.llm_wiki status # Run health check python -m src.llm_wiki lint # Show help python -m src.llm_wiki --help ``` **Note**: `ingest` and `query` commands in legacy CLI only provide auxiliary functions (like listing pages). Actual content processing requires natural language interaction with the agent. # LLM-Wiki Karpathy's llm-wiki pattern implementation — cumulative knowledge management for AI agents. > **Core Philosophy**: LLM as programmer, Wiki as codebase, User as product manager. ## Why SKILL Form? We chose the SKILL form because it brings these advantages: - **Zero deployment** — No services to run, no databases to configure; works the moment you clone the repository - **Native integration** — Direct command execution via Claude Code, no middleware or protocol translation needed - **Plain-text data** — Pure Markdown files, git-native, with no proprietary formats or vendor lock-in - **Editor freedom** — Use Obsidian, VS Code, or any text editor you prefer - **Minimal footprint** — ~500 lines of core protocol, keeping complexity low ## Features - **Protocol-driven**: Works with natural language (no installation required) - **Pure Markdown**: No database, no lock-in, git-native - **Wiki-style links**: `[[PageName]]` format with canonical page files; avoid duplicate shell pages - **Cumulative learning**: Every query can create new knowledge - **Temporal knowledge**: Preserve publication/release/collection dates so related works can be read in historical order - **Health checks**: Orphan pages, dead links, stale content detection - **Optional CLI**: Python scripts for automation and batch operations ## Quick Start ```bash # 1. Clone git clone https://github.com/Nemo4110/llm-wiki.git cd llm-wiki # 2. Add source material cp ~/Downloads/paper.pdf sources/ # 3. Tell your agent "Please ingest sources/paper.pdf into wiki" ``` ## Installation ### Protocol Mode (Recommended) No installation needed. Agent reads `CLAUDE.md` and operates directly. ### CLI Mode (Optional) #### Using uv (Fastest) ```bash # Create virtual environment and install dependencies uv venv uv pip install -r src/requirements.txt --python .venv/Scripts/python.exe # Activate environment (Windows) .venv\Scripts\activate # Or Linux/macOS source .venv/bin/activate ``` #### Using conda ```bash # Create environment conda create -n llm-wiki python=3.11 # Activate environment conda activate llm-wiki # Install dependencies pip install -r src/requirements.txt ``` #### Using pip ```bash # Create virtual environment python -m venv .venv # Activate environment source .venv/bin/activate # Linux/macOS .venv\Scripts\activate # Windows # Install dependencies pip install -r src/requirements.txt ``` #### Verify Installation ```bash python -c "from src.llm_wiki.core import WikiManager; print('✓ Installation successful')" ``` **Important Dependency Notes**: | Dependency | Version | Purpose | Notes | |------------|---------|---------|-------| | `click` | >=8.0.0 | CLI framework | - | | `pyyaml` | >=6.0 | YAML parsing | - | | `pymupdf` | >=1.25.0 | PDF processing | Primary PDF engine, best for CJK | **Optional dependencies** (for enhanced features): - `numpy >=1.24.0` — Vector operations for embedding retrieval - `httpx >=0.27.0` — HTTP client for Ollama/local services - `openai >=1.0.0` — OpenAI embedding API - `mcp >=1.0.0` — MCP SDK for remote embedding providers **Fallback PDF dependency**: - `pdfplumber >=0.11.8` — Table extraction fallback (security version required for CVE-2025-64512) - `pdfminer.six >=20251107` — PDF underlying library fallback ## Project Structure ``` llm-wiki/ ├── CLAUDE.md # ⭐ Core protocol: Agent behavior guidelines ├── AGENTS.md # Agent implementation guide (CLI usage) ├── SKILL.md # This file, machine-readable specification ├── log.md # Timeline log (append-only) ├── sources/ # Raw materials (user-managed + tool-fetched; Agent forbidden from writing LLM-generated content) │ └── README.md ├── wiki/ # Generated knowledge pages (Agent-managed) │ ├── index.md # Entry index │ └── *.md # Topic pages ├── assets/ # Templates and configuration │ ├── page_template.md │ └── ingest_rules.md ├── src/ # SKILL implementation (optional, for CLI) │ ├── llm_wiki/ │ └── requirements.txt ├── scripts/ # Auxiliary scripts ├── hooks/ # Platform hooks (optional) └── examples/ # Example wiki ``` **About `sources/`**: Excluded from git by default to avoid repository bloat. Wiki only retains extracted knowledge; original files are managed separately (cloud storage, Zotero, etc.). See `sources/README.md` for tracking specific files. ## How It Works ### Data Flow ``` +----------+ +--------------------+ +--------------+ | sources/ |---->| LLM Processing |---->| wiki/ | | (Raw) | | (Extract + Link) | | (Structured) | +----------+ +--------------------+ +--------------+ | v +----------+ | log.md | | (Record) | +----------+ ``` ### Key Design 1. **CLAUDE.md as Protocol**: Defines Agent behavior standards, anyone/any Agent can follow 2. **Pure Markdown**: No database, no lock-in, native git version control 3. **Bidirectional Links**: `[[PageName]]` format, compatible with Obsidian when the link target matches the canonical page file 4. **Cumulative Learning**: Each query can generate new wiki pages, knowledge continuously accumulates ## Query Mechanism ### Current Implementation: Symbolic Navigation + LLM Synthesis (Default) By default, this SKILL **does not require Embedding/vector retrieval**. Queries are completed through: ``` User asks question | v +-------------------------------+ | 1. Read index.md | <-- Human/Agent-maintained category index | Locate relevant topics | +-------------------------------+ | v +-------------------------------+ | 2. Read relevant pages | <-- Discover associations through [[links]] | and their link neighbors | +-------------------------------+ | v +-------------------------------+ | 3. LLM Synthesis | <-- Generate answers based on read content | Generate with citations | Citation format: [[PageName]] +-------------------------------+ ``` **Optional Enhancement**: After enabling `config.yaml` embedding settings, CLI `query --semantic` adds hybrid search (Keyword Match + Vector Search + Link Traversal) for faster, more accurate retrieval. **Example Flow**: User asks: "What is LoRA?" 1. **Agent reads** `wiki/index.md`, finds `[[LoRA]]` under "AI/ML" topic 2. **Agent reads** `wiki/LoRA.md`, discovers links to `[[Fine-tuning]]`, `[[Adapter]]` 3. **Agent synthesizes** answer: > LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method — see [[LoRA]]. > Compared to traditional [[Fine-tuning]], it only trains low-rank matrices... ### Why is Embedding Optional? | Consideration | Current Solution | Embedding Solution | |---------------|------------------|-------------------| | **Dependencies** | Zero external dependencies | Requires Embedding API or local model | | **Cost** | No additional fees | Charged per token/request | | **Privacy** | Data not uploaded | Must send content to external service | | **Accuracy** | Precise links, explainable | Approximate similarity, may retrieve irrelevant content | | **Scale** | Suitable for 0-500 pages | Essential for large scale (1000+ pages) | **Conclusion**: For personal/small team knowledge bases, maintaining `index.md` and page links is simpler and more effective than introducing Embedding. Embedding is available as an opt-in CLI enhancement when scale demands it. ### When to Use Embedding? Consider enabling `config.yaml` embedding settings when your wiki shows: - [ ] Page count > 500, manual indexing hard to maintain - [ ] Queries are mostly fuzzy semantic ("that paper about optimization...") - [ ] Need cross-topic associations (user didn't mention keywords, but semantically related) See [ROADMAP.md](ROADMAP.md) for details. ## Example Workflows ### Scenario 1: Researching New Domain ```bash # Day 1: Add materials sources/ ├── 2026-04-10-transformer-paper.pdf └── 2026-04-10-lora-blog.md ``` ``` User: Please ingest new materials Claude: Processed 2 materials - Created [[Transformer]] — Attention mechanism architecture - Created [[LoRA]] — Low-rank adaptation - Created [[Self-Attention]] — Self-attention details - Updated wiki/index.md ``` ### Scenario 2: Deep Exploration ``` User: What's the difference between LoRA and full fine-tuning? Claude (reads wiki/index.md → finds [[LoRA]] → synthesizes answer): Based on our wiki: **LoRA** — [[LoRA]] - Freezes pre-trained weights, only trains low-rank matrices - Parameter count reduced ~10,000x **Full Fine-tuning** - Updates all parameters - High computational cost Key differences: 1. Parameters: LoRA only trains 0.1%~1% of parameters [[Parameter Efficiency]] 2. Storage: LoRA only saves adapters, not full model 3. Composition: Multiple LoRAs can be stacked [[LoRA Composition]] Was this helpful? Should I archive this answer to wiki? ``` ### Scenario 3: Knowledge Accumulation ``` User: Yes, archive this answer Claude: Created [[LoRA vs Full Fine-tuning]] - Extracted comparison points from conversation - Linked to [[LoRA]] and [[Fine-tuning]] - Added to FAQ section in wiki/index.md ``` ## Using with Obsidian 1. Open `wiki/` directory in Obsidian 2. Enjoy graph view, quick navigation, beautiful rendering 3. Claude Code handles maintenance, Obsidian handles reading and thinking ## Comparison with Alternatives | Solution | Characteristics | Best For | |----------|----------------|----------| | **This SKILL** | Zero dependencies, pure text, Claude Code native | Personal knowledge management, research notes | | Sage-Wiki | Full-featured, multimodal, standalone app | Team knowledge base, enterprise deployment | | Obsidian + Plugins | Strong visualization, rich community | Existing Obsidian workflow | | Notion/Logseq | Collaborative, real-time sync | Multi-user collaboration, mobile access | ## Documentation - [CLAUDE.md](CLAUDE.md) — User-facing protocol (read this first) - [AGENTS.md](AGENTS.md) — Implementation guide for agent developers - [SKILL.md](SKILL.md) — This file, machine-readable specification - [ROADMAP.md](ROADMAP.md) — Future plans ## Contributing Issues and PRs welcome! ### Current TODO - [ ] MCP server wrapper (for other Agents) - [ ] Zotero MCP integration for literature discovery, ingest, metadata linking, and optional backlink sync - [ ] Temporal metadata and timeline views for source publication/release/collection order - [ ] Obsidian plugin (one-click sync) - [x] Incremental embedding for faster retrieval - [ ] Multi-language support ## License MIT — free to use, modify, and distribute. --- *Inspired by [Karpathy's llm-wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)*


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)