npm package: @leibniz/docnom
Description: DocNom (doc + nom 🍴) - A powerful blog crawler that hungrily consumes and archives web content from blogs and documentation sites
Latest version: 1.0.2

# DocNom 🍴

**DocNom** = **doc** + **nom** (the sound of eating) 

A powerful blog crawler that hungrily "noms" through blogs and documentation sites, consuming and archiving web content into clean, organized Markdown files.

[![npm version](https://img.shields.io/npm/v/@leibniz/docnom.svg)](https://www.npmjs.com/package/@leibniz/docnom)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Features

- 🍽️ **Hungry for Content** - Devours blogs, documentation sites, and multi-page articles
- 🔌 **Plugin-Based Adapters** - Custom extractors for VictoriaMetrics, Cloudflare blogs, and more
- 📝 **Clean Markdown Output** - Converts HTML to readable Markdown with frontmatter
- 🖼️ **Image Archiving** - Downloads and localizes all images (with parallel downloads!)
- 💾 **Incremental Crawling** - Smart caching to avoid re-downloading known content
- 🎯 **Multiple Formats** - Export as Markdown and/or JSON
- 🗄️ **Database Support** - SQLite, PostgreSQL, or MySQL for content tracking
- ⚡ **High Performance** - Rate limiting, concurrency control, and retry logic

---

## Installation

### Global Installation (Recommended)

```bash
npm install -g @leibniz/docnom
```

### Using npx (No Installation Required)

```bash
npx @leibniz/docnom --help
```

### Local Project Installation

```bash
npm install @leibniz/docnom
```

---

## Quick Start

### Basic Usage

```bash
# Crawl a blog or documentation site
docnom crawl https://victoriametrics.com/blog/

# Crawl with options
docnom crawl https://blog.cloudflare.com/ \
  --max-posts 50 \
  --output ./my-archive \
  --formats markdown json

# Crawl specific post
docnom crawl https://zh.wikipedia.org/wiki/Euler
```

### Using npx

```bash
# No installation needed!
npx @leibniz/docnom crawl https://example.com/blog/

# With options
npx @leibniz/docnom crawl https://blog.example.com/ \
  --max-posts 20 \
  --formats markdown
```

---

## Command Line Options

### `crawl` Command

```bash
docnom crawl <url> [options]
```

#### Options

| Option | Description | Default |
|--------|-------------|---------|
| `-o, --output <dir>` | Output directory | `./output` |
| `-f, --formats <types>` | Export formats (markdown, json) | `markdown` |
| `--max-posts <number>` | Maximum posts to crawl | Unlimited |
| `--max-pages <number>` | Maximum list pages to scan | `10` |
| `-c, --concurrency <number>` | Concurrent requests | `3` |
| `--rate-limit <ms>` | Delay between requests (ms) | `1000` |
| `--retries <number>` | Retry attempts for failed requests | `3` |
| `--headless` | Run browser in headless mode | `true` |
| `--incremental` | Skip known URLs (requires database | `false` |
| `--stop-after-known <number>` | Stop after N consecutive known posts | `10` |
| `--db <type>` | Database type (sqlite, postgres, mysql) | `sqlite` |
| `--db-path <path>` | SQLite database file path | `./data/blog-crawler.db` |

### Examples

**Crawl with limits:**
```bash
docnom crawl https://blog.example.com/ \
  --max-posts 100 \
  --max-pages 5
```

**High-speed crawling:**
```bash
docnom crawl https://docs.example.com/ \
  --concurrency 10 \
  --rate-limit 500
```

**Incremental mode (skip existing):**
```bash
docnom crawl https://blog.example.com/ \
  --incremental \
  --stop-after-known 5
```

**Export both formats:**
```bash
docnom crawl https://example.com/blog/ \
  --formats markdown json
```

---

## Configuration File

Create a `.config.json` file in your project root:

```json
{
  "outputDir": "./archive",
  "formats": ["markdown", "json"],
  "maxPosts": 100,
  "concurrency": 5,
  "rateLimit": 1000,
  "retries": 3,
  "incremental": true,
  "stopAfterKnown": 10,
  "database": {
    "type": "sqlite",
    "path": "./data/archive.db"
  }
}
```

Then run:

```bash
docnom crawl https://blog.example.com/
```

---

## Output Structure

DocNom organizes content by domain:

```
output/
├── example.com/              # Domain-based directory
│   ├── index.md              # Index of all posts
│   ├── posts.json            # Metadata (if --formats json)
│   ├── post-title-1.md       # Individual post
│   ├── post-title-2.md
│   └── assets/               # Downloaded images
│       ├── post-title-1/
│       │   ├── image_0.jpg
│       │   └── image_1.png
│       └── post-title-2/
│           └── diagram.svg
```

### Markdown Format

Each `.md` file includes:

```markdown
---
title: "Post Title"
url: "https://example.com/post"
image: "https://example.com/cover.jpg"
wordCount: 1500
readingTime: 7
crawledAt: 2026-01-26T05:30:00.000Z
imagesCount: 5
---

# Post Title

Content with localized images:

![Alt text](./assets/post-title/image_0.jpg)

...
```

---

## Supported Sites

DocNom includes specialized adapters for:

### Built-in Adapters

- **VictoriaMetrics Blog** (`victoriametrics.com/blog`)
- **Cloudflare Blog** (`blog.cloudflare.com`)
- **Default Adapter** - Works with most blogs and documentation sites

### Custom Adapters

You can create your own adapter by implementing the `SiteAdapter` interface:

```typescript
import { SiteAdapter } from '@leibniz/docnom';

export class MyBlogAdapter implements SiteAdapter {
  name = 'my-blog';
  baseUrl = 'https://myblog.com';
  
  canHandle(url: string): boolean {
    return url.includes('myblog.com');
  }
  
  async getListPages(startUrl: string): Promise<string[]> {
    // Return URLs of list pages
  }
  
  async extractPosts(listPageHtml: string, listUrl: string): Promise<BlogPostPreview[]> {
    // Extract post previews from list page
  }
  
  async extractFullPost(html: string, url: string): Promise<BlogPost> {
    // Extract full post content
  }
}
```

---

## Programmatic Usage

```typescript
import { Crawler } from '@leibniz/docnom';
import { VictoriaMetricsAdapter } from '@leibniz/docnom/adapters';

const crawler = new Crawler(new VictoriaMetricsAdapter(), {
  outputDir: './output',
  formats: ['markdown', 'json'],
  maxPosts: 50,
  concurrency: 5,
});

const result = await crawler.crawl('https://victoriametrics.com/blog/');

console.log(`Crawled ${result.posts.length} posts in ${result.duration}ms`);
```

---

## Advanced Features

### Incremental Crawling

Save time by skipping already-crawled posts:

```bash
docnom crawl https://blog.example.com/ \
  --incremental \
  --stop-after-known 10
```

DocNom will:
1. Check URLs against the database
2. Skip known posts
3. Stop after encountering 10 consecutive known posts (configurable)

### Custom Databases

#### PostgreSQL
```bash
docnom crawl https://example.com/blog/ \
  --db postgres \
  --db-connection "postgresql://user:pass@localhost:5432/docnom"
```

#### MySQL
```bash
docnom crawl https://example.com/blog/ \
  --db mysql \
  --db-connection "mysql://user:pass@localhost:3306/docnom"
```

### Rate Limiting

Respect server resources:

```bash
docnom crawl https://example.com/ \
  --rate-limit 2000 \  # 2 seconds between requests
  --concurrency 1       # Sequential requests
```

---

## Troubleshooting

### Images Not Downloading

- Check network connectivity
- Verify image URLs are accessible
- Images are downloaded with 10 concurrent requests by default (configurable in `@leibniz/extractor`)

### Database Locked (SQLite)

- Ensure only one crawler instance is running
- Use PostgreSQL/MySQL for concurrent access

### Out of Memory

- Reduce `--concurrency`
- Crawl in smaller batches with `--max-posts`
- Use `--incremental` mode

---

## Architecture

DocNom is built on:

- **[@leibniz/extractor](https://www.npmjs.com/package/@leibniz/extractor)** - Content extraction and image downloading
- **Playwright** - JavaScript rendering for dynamic sites
- **Drizzle ORM** - Type-safe database operations
- **Turndown** - HTML to Markdown conversion

---

## Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new adapters
4. Submit a pull request

---

## License

MIT © 2026

---

## Related Projects

- [@leibniz/extractor](https://www.npmjs.com/package/@leibniz/extractor) - The extraction library powering DocNom
- [Readability](https://github.com/mozilla/readability) - Mozilla's content extraction algorithm (inspiration)

---

**Happy Nomming! 🍴📚**