npm package: @neogp/agent-docs
Description: NotebookLM for coding agents — ingest API docs into agent-consumable markdown
Author: Pritam Neog
Latest version: 0.1.1

# agent-docs

**NotebookLM for coding agents.** Ingest API references and product guides from documentation websites into structured, token-optimized markdown that coding agents (Claude Code, Codex, Cursor) can reference across sessions.

## The Problem

Coding agents lose documentation context between sessions. Every new conversation starts from zero — the agent has to re-read docs, re-discover API shapes, and re-learn product conventions. Copy-pasting docs into prompts is fragile, eats tokens, and goes stale.

**agent-docs** solves this by creating a persistent, git-tracked `.agent-docs/` directory in your project. It fetches live documentation, transforms it into a canonical format optimized for agent consumption, and writes it as plain markdown files. Agents read these files like any other code — no special tooling, no databases, no infrastructure.

## Quick Start

```bash
# Install globally
npm install -g @neogp/agent-docs

# Or run without installing
npx @neogp/agent-docs ingest <url>

# Ingest an API reference (single-page docs with endpoint listings)
agent-docs ingest https://fintechprimitives.com/docs/api/ \
  --type api-reference --name fp-api

# Ingest product guides (multi-page docs with sidebar navigation)
agent-docs ingest https://docs.fintechprimitives.com/ \
  --type guide --name fp-guides

# Check what's been ingested
agent-docs status
```

This creates a `.agent-docs/` directory in your current working directory:

```
.agent-docs/
├── manifest.yaml           # Index of all ingested doc sets
├── fp-api/
│   ├── _meta.yaml          # API overview, auth pattern, resource index
│   └── api/
│       ├── purchases.md    # One file per resource group
│       ├── redemptions.md
│       └── ...
└── fp-guides/
    ├── _meta.yaml          # Guide overview, topic index
    └── guides/
        ├── going-live-checklist.md
        ├── identity-overview.md
        └── ...
```

## Usage

### Ingest documentation

```bash
agent-docs ingest <url> [options]
```

| Option | Description | Default |
|--------|-------------|---------|
| `-t, --type <type>` | `api-reference`, `guide`, or `auto` | `auto` |
| `-n, --name <name>` | Name for the doc source | Derived from URL |
| `-o, --output <dir>` | Output directory | `.agent-docs` |
| `-f, --force` | Overwrite existing output | `false` |
| `--dry-run` | Preview without writing files | `false` |
| `-v, --verbose` | Debug logging | `false` |

**Auto-detection** (`--type auto`) examines the page structure:
- Many anchor links (`#section`) and few sidebar page links → **single-page** → treated as API reference
- Sidebar with links to multiple pages → **multi-page** → treated as guide

### Check status

```bash
agent-docs status [--output .agent-docs]
```

Prints a table of all ingested sources with type, date, and stats. Sources older than 30 days are flagged as stale.

### Re-ingest

```bash
agent-docs ingest <url> --name <existing-name> --force
```

The `--force` flag removes the existing output for that source before re-ingesting. The manifest is updated in place — other sources are preserved.

### Use from any project

Run from inside your project directory. Docs land in `.agent-docs/` by default:

```bash
cd ~/my-project
agent-docs ingest https://api-docs.example.com/ --name example-api
# Creates ~/my-project/.agent-docs/example-api/
```

Or specify a custom output directory:

```bash
agent-docs ingest https://api-docs.example.com/ --output docs/.agent-docs
```

## Output Format

Every generated file includes YAML frontmatter so agents can quickly assess relevance:

**API endpoint file:**

```markdown
---
source: https://example.com/docs/api/
type: api-reference
resource: purchases
last_ingested: 2026-03-11
---

# Purchases

## POST /v2/purchases

Create a new purchase order.

### Parameters

| Param | Type | Required | Description |
|-------|------|----------|-------------|
| amount | number | yes | Amount in INR |

### Request

​```json
{ "amount": 5000 }
​```

### Response

​```json
{ "id": "pur_xxx", "status": "pending" }
​```
```

**Guide file:**

```markdown
---
source: https://docs.example.com/identity/overview/
type: guide
topic: identity-overview
last_ingested: 2026-03-11
---

# Identity Overview

Content in clean markdown...
```

**`_meta.yaml`** per source provides a structured index:

```yaml
name: fp-api
base_url: https://fintechprimitives.com
auth: Bearer token in Authorization header
resources:
  - name: Purchases
    file: api/purchases.md
  - name: Redemptions
    file: api/redemptions.md
```

**`manifest.yaml`** at the root lists all sources:

```yaml
version: "1.0"
note: >-
  The source of truth for each API or product guide is its official website
  (see source_url). However, coding agents MUST NOT fetch docs from the
  internet. Use the pre-ingested local files in this directory instead.
  Re-run agent-docs ingest to update.
sources:
  - name: fp-api
    type: api-reference
    path: fp-api
    source_url: https://fintechprimitives.com/docs/api/
    last_ingested: 2026-03-11
    endpoint_count: 245
  - name: fp-guides
    type: guide
    path: fp-guides
    source_url: https://docs.fintechprimitives.com/
    last_ingested: 2026-03-11
    guide_count: 78
```

## Pointing Your Agent at the Docs

Copy the documentation retrieval protocol into your project's `CLAUDE.md`, `.cursorrules`, or equivalent agent instructions file:

```bash
# The AGENT_DOCS_INSTRUCTIONS.md file is included in the npm package
cat node_modules/@neogp/agent-docs/AGENT_DOCS_INSTRUCTIONS.md >> CLAUDE.md
```

Or manually add:

```markdown
Reference `.agent-docs/manifest.yaml` for available API documentation.
Read the relevant files under `.agent-docs/` before making API calls.
Do NOT fetch documentation from the internet — use the local .agent-docs/ files only.
```

The agent will discover the manifest, find the right resource file, and use the structured endpoint definitions, parameter tables, and examples to write accurate code.

---

## How agent-docs Works

### Why it exists

Large language models are stateless between sessions. When you use a coding agent to build against an external API, every new session starts with zero knowledge of that API. The common workarounds all have problems:

- **Paste docs into the prompt** — manual, eats context window, goes stale
- **Let the agent fetch docs live** — slow, unreliable, agent may misparse the HTML
- **Summarize docs yourself** — time-consuming, loses detail, hard to maintain

agent-docs automates the transformation of live documentation into a format that is both human-readable and agent-optimized. The output is plain markdown files tracked in git — no runtime dependencies, no databases, no API keys required.

### Architecture

```
URL ──► Fetcher ──► Crawler ──► Parser ──► Extractor ──► Writer ──► .agent-docs/
```

**1. Fetcher** (`src/utils/fetcher.ts`)
HTTP client wrapping Node's native `fetch` with retry logic (3 attempts, exponential backoff) and concurrency control via `p-limit` (max 3 parallel requests). Handles transient failures without overwhelming the target server.

**2. Crawler** (`src/crawlers/`)
Two strategies based on auto-detected doc type:

- **Single-page crawler** — For API references where all endpoints live on one page (hash-anchored sections). Parses the DOM with Cheerio, finds all headings with `id` attributes, and splits the page into sections by collecting content between consecutive headings.

- **Multi-page crawler** — For product guides with sidebar navigation. Extracts all internal links from sidebar/nav elements, deduplicates them, then fetches each page in parallel (rate-limited).

**3. Parser** (`src/parsers/`)
Transforms raw HTML into clean, structured content:

- **HTML Cleaner** — Strips navigation, footers, scripts, styles, sidebars, and ad elements. Extracts the main content area (`<main>`, `<article>`, `.content`).
- **Markdown Converter** — Turndown with custom rules for code blocks (preserves language hints from `<pre>` class attributes), images (replaced with `[Image: alt]` placeholders), and tables.
- **Section Splitter** — Splits markdown on heading boundaries while respecting code fences (won't split inside ``` blocks).
- **Table Parser** — Detects parameter tables by column headers (Name/Type/Required/Description and variants), extracts structured parameter data.

**4. Extractor** (`src/extractors/`)
Derives structured data from parsed content:

- **Endpoint Extractor** — Scans sections for HTTP method + path patterns (`GET /v2/...` or `POST https://...`), extracts description, parameters, request/response JSON examples, and curl snippets. Groups endpoints by resource (derived from URL path segments).
- **Guide Extractor** — Extracts page title, converts content to clean markdown, pulls out code examples with language tags, identifies prerequisite sections, and replaces mermaid diagrams with placeholders.

**5. Writer** (`src/writers/`)
Writes the canonical output:

- **Canonical Writer** — Generates one `.md` file per resource group (API) or per page (guide), with YAML frontmatter and consistent heading structure.
- **Meta Writer** — Generates `_meta.yaml` per source with resource/guide index, detected auth pattern (from curl examples), and base URL.
- **Manifest Writer** — Creates or updates the top-level `manifest.yaml`, merging new entries without overwriting existing sources.

### Design decisions

**Heuristic-first, no API keys required.** The core pipeline uses DOM pattern matching (Cheerio) and regex-based extraction. It works out of the box without any LLM API calls. An optional `--enhance` pass (not yet implemented) can use an LLM to improve parameter descriptions and fill gaps.

**One file per resource group, not per endpoint.** An API with 150 endpoints produces ~15-80 resource files instead of 150 individual files. This keeps the file count manageable while letting agents read all endpoints for a resource in a single file read.

**Cheerio over Puppeteer/Playwright.** Both target doc sites are server-rendered HTML. No JavaScript execution needed. Cheerio is orders of magnitude faster, has zero system dependencies, and works in any environment.

**YAML frontmatter on every file.** Agents can scan frontmatter to assess relevance without reading full content. The `type`, `resource`/`topic`, and `source` fields enable targeted lookups.

**Git-tracked output.** The `.agent-docs/` directory is plain files. Commit it to your repo and every developer (and every agent session) has the same documentation baseline. Diffs show exactly what changed between re-ingests.

### Testing

77 tests across 15 test suites, using real HTML fixtures saved from the target documentation sites. Tests run against these snapshots — no network calls during testing.

```bash
npm test
```