npm package: @k0nf/crawler-detector
Description: High-speed crawler detection library using optimized local databases for IP and user-agent matching
Latest version: 1.3.0

# Crawler Detector

⚡ High-speed crawler detection library for JavaScript using optimized local databases. Detects web crawlers and bots using IP addresses and user-agent patterns with microsecond-level performance.

## Features

- 🚀 **Blazing Fast** - Optimized detection with early-exit logic (< 0.02ms per check)
- 🌐 **IPv4 & IPv6 Support** - Full support for both IPv4 and IPv6 addresses using BigInt
- 📦 **Zero Runtime Dependencies** - No external API calls, all data is local
- 🎯 **Dual Detection** - Identifies crawlers by IP address and user-agent patterns
- 🔄 **Auto-Updated Patterns** - Easy rebuild from latest crawler sources
- 📊 **Binary Search** - Efficient CIDR range lookup using integer comparisons
- 🔌 **Framework Middleware** - Built-in support for Express, Next.js, Remix, Koa, Fastify, Hapi
- ✅ **Well Tested** - Comprehensive test suite with 94 passing tests

## Coverage

**IP Database:**
- **IPv4:** 20.7 million addresses (11,057 exact IPs + 749 CIDR ranges)
- **IPv6:** 2.6 sextillion addresses (475 exact IPs + 777 CIDR blocks)
- **Sources:** Googlebot (official API), Yandex, Meta/Facebook, TikTok, 27 sources from GoodBots

**User-Agent Database:**
- **602 patterns** (503 substrings + 99 regex)
- **Source:** [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents)

## Installation

```bash
npm install @k0nf/crawler-detector
```

## Usage

### Basic Detection

```javascript
const { isCrawler } = require('@k0nf/crawler-detector');

// Check both IP and user-agent
const isBot = isCrawler(
  '66.249.64.1',
  'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
);
console.log(isBot); // true
```

### IP-Only Detection

```javascript
const { isCrawlerByIP } = require('@k0nf/crawler-detector');

const isFromCrawlerIP = isCrawlerByIP('66.249.64.1');
console.log(isFromCrawlerIP); // true (Googlebot IP range)
```

### User-Agent-Only Detection

```javascript
const { isCrawlerByUA } = require('@k0nf/crawler-detector');

const hasCrawlerUA = isCrawlerByUA('Mozilla/5.0 (compatible; Googlebot/2.1)');
console.log(hasCrawlerUA); // true
```

### IPv6 Support

```javascript
const { isCrawlerByIP } = require('@k0nf/crawler-detector');

// Detect Googlebot IPv6
const isGooglebot = isCrawlerByIP('2001:4860:4801:10::1');
console.log(isGooglebot); // true

// Detect Yandex IPv6
const isYandex = isCrawlerByIP('2a02:6b8::1');
console.log(isYandex); // true
```

## Framework Middleware

Built-in middleware for popular Node.js frameworks. All middleware support these options:

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `block` | boolean | `false` | Block crawlers with HTTP error |
| `blockStatusCode` | number | `403` | Status code for blocked requests |
| `blockMessage` | string | `'Forbidden'` | Response body for blocked requests |
| `onCrawlerDetected` | function | `null` | Custom callback when crawler detected |
| `ipHeaders` | string[] | `['x-forwarded-for', ...]` | Headers to check for client IP |
| `ipOnly` | boolean | `false` | Only check IP, skip user-agent |
| `uaOnly` | boolean | `false` | Only check user-agent, skip IP |

### Express / Connect

```javascript
const express = require('express');
const crawlerDetector = require('@k0nf/crawler-detector/middleware/express');

const app = express();

// Tag all requests with req.isCrawler (boolean)
app.use(crawlerDetector());

// Block crawlers from specific routes
app.use('/api', crawlerDetector({ block: true }));

// Custom handling
app.use(crawlerDetector({
  onCrawlerDetected: (req, res, next) => {
    console.log(`Bot detected: ${req.headers['user-agent']}`);
    next(); // continue processing
  }
}));

app.get('/', (req, res) => {
  if (req.isCrawler) {
    res.send('Hello bot!');
  } else {
    res.send('Hello human!');
  }
});
```

### Next.js

**API Routes (Pages Router):**

```javascript
// pages/api/hello.js
const { withCrawlerDetection } = require('@k0nf/crawler-detector/middleware/nextjs');

function handler(req, res) {
  res.json({ isCrawler: req.isCrawler });
}

module.exports = withCrawlerDetection(handler);

// Block crawlers
module.exports = withCrawlerDetection(handler, { block: true });
```

**getServerSideProps:**

```javascript
// pages/index.js
const { isCrawlerRequest } = require('@k0nf/crawler-detector/middleware/nextjs');

export async function getServerSideProps(context) {
  const isBot = isCrawlerRequest(context);
  return { props: { isBot } };
}
```

**App Router Route Handlers:**

```javascript
// app/api/hello/route.js
const { isCrawlerAppRoute } = require('@k0nf/crawler-detector/middleware/nextjs');

export const runtime = 'nodejs'; // Required - not compatible with Edge runtime

export async function GET(request) {
  const isBot = isCrawlerAppRoute(request);
  return Response.json({ isBot });
}
```

> **Note:** Requires Node.js runtime. Not compatible with Edge runtime since the detection databases are loaded from the filesystem.

### Remix / React Router v7

**In loaders and actions:**

```javascript
// app/routes/index.tsx
import { isCrawlerRequest } from '@k0nf/crawler-detector/middleware/remix';

export async function loader({ request }) {
  const isBot = isCrawlerRequest(request);

  if (isBot) {
    return json({ content: 'SEO-optimized content', isBot: true });
  }

  return json({ content: 'Full interactive content', isBot: false });
}
```

**Block crawlers with middleware:**

```javascript
import { createCrawlerMiddleware } from '@k0nf/crawler-detector/middleware/remix';

const crawlerMiddleware = createCrawlerMiddleware({ block: true });

export function middleware(request) {
  const blocked = crawlerMiddleware(request);
  if (blocked) return blocked;
}
```

### Koa

```javascript
const Koa = require('koa');
const crawlerDetector = require('@k0nf/crawler-detector/middleware/koa');

const app = new Koa();

// Tag all requests - sets ctx.state.isCrawler
app.use(crawlerDetector());

// Block crawlers
app.use(crawlerDetector({ block: true }));

app.use(async (ctx) => {
  if (ctx.state.isCrawler) {
    ctx.body = 'Hello bot!';
  } else {
    ctx.body = 'Hello human!';
  }
});
```

### Fastify

```javascript
const fastify = require('fastify')();
const crawlerDetectorPlugin = require('@k0nf/crawler-detector/middleware/fastify');

// Register plugin - decorates request.isCrawler
fastify.register(crawlerDetectorPlugin);

// With options
fastify.register(crawlerDetectorPlugin, { block: true });

fastify.get('/', (request, reply) => {
  if (request.isCrawler) {
    reply.send('Hello bot!');
  } else {
    reply.send('Hello human!');
  }
});
```

### Hapi

```javascript
const Hapi = require('@hapi/hapi');
const crawlerDetectorPlugin = require('@k0nf/crawler-detector/middleware/hapi');

const server = Hapi.server({ port: 3000 });

await server.register({
  plugin: crawlerDetectorPlugin,
  options: { block: false }
});

server.route({
  method: 'GET',
  path: '/',
  handler: (request, h) => {
    if (request.plugins.crawlerDetector.isCrawler) {
      return 'Hello bot!';
    }
    return 'Hello human!';
  }
});
```

### Raw Node.js HTTP

```javascript
const http = require('http');
const withCrawlerDetection = require('@k0nf/crawler-detector/middleware/http');

const server = http.createServer(
  withCrawlerDetection((req, res) => {
    if (req.isCrawler) {
      res.end('Hello bot!');
    } else {
      res.end('Hello human!');
    }
  })
);

// Block crawlers
const server = http.createServer(
  withCrawlerDetection(handler, { block: true })
);
```

## How It Works

The library uses two optimized local databases:

### 1. User-Agent Patterns
Patterns are sourced from **[monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents)** ([raw JSON](https://raw.githubusercontent.com/monperrus/crawler-user-agents/refs/heads/master/crawler-user-agents.json)).

During build time, patterns are:
- Downloaded from the GitHub repository
- Classified into **substrings** (fast path) or **regex patterns** (slower path)
- Normalized and lowercased for case-insensitive matching
- Deduplicated and optimized

Detection order (fastest first, early-exit):
1. Substring matching - simple `.includes()` checks
2. Regex pattern matching - pre-compiled patterns

### 2. IP Database
IP ranges are manually curated from known crawler sources and stored as:
- **Exact IPs** - Direct Set lookup (O(1))
- **CIDR ranges** - Converted to integer pairs and sorted for binary search (O(log n))

Detection order (fastest first, early-exit):
1. Exact IP match
2. CIDR range binary search

## Performance

Expected performance characteristics:

- **User-Agent Detection**: < 0.1ms average (substring match)
- **IP Detection**: < 0.01ms average (exact match) or < 0.1ms (CIDR range)
- **Combined Detection**: < 1ms total

**Actual test results** (10,000 iterations):
```
Ran 10,000 detections in 110ms
Average: 0.011ms per detection
✅ Well under 1ms target
```

## Updating Databases

### Rebuild All Databases

```bash
npm run build
```

This will:
1. Fetch the latest user-agent patterns from GitHub
2. Build the optimized user-agent database
3. Build the IP database from seed file
4. Validate all generated data

### Update Only User-Agent Patterns

```bash
npm run build:ua
```

### Update Only IP Database

```bash
npm run build:ip
```

### Validate Databases

```bash
npm run validate
```

## Adding Custom Crawler IPs

Edit `scripts/seed-crawler-ips.txt` and add IPs or CIDR ranges (one per line):

```
# Custom crawler IPs
203.0.113.5
198.51.100.0/24
```

Then rebuild:
```bash
npm run build:ip
```

## Testing

Run the test suite:

```bash
npm test
```

Tests cover:
- ✅ Known crawler detection (Googlebot, Bingbot, etc.)
- ✅ Regular browser exclusion (Chrome, Firefox, Safari)
- ✅ Edge cases (null, undefined, empty strings)
- ✅ IP conversion and binary search logic
- ✅ Performance benchmarks

## Architecture

```
crawler-detector/
├── index.js                      # Main API entry point
├── lib/
│   ├── ip-detector.js           # IP detection logic
│   └── ua-detector.js           # User-agent detection logic
├── data/
│   ├── crawler-ips.json         # Built IP database
│   └── crawler-ua-patterns.json # Built UA patterns database
├── scripts/
│   ├── fetch-ua-patterns.js     # Fetch patterns from GitHub
│   ├── build-ip-database.js     # Build IP database
│   ├── validate-data.js         # Validate databases
│   ├── build-all.js             # Build all databases
│   └── seed-crawler-ips.txt     # Source IP list
└── test/
    └── test.js                   # Test suite
```

### Detection Flow

```
isCrawler(ip, userAgent)
    │
    ├─> Check IP (if provided)
    │   ├─> Exact match? → return true
    │   └─> In CIDR range? → return true
    │
    └─> Check User-Agent (if provided)
        ├─> Substring match? → return true
        └─> Regex match? → return true
        
return false
```

## Data Sources

- **User-Agent Patterns**: [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
  - Direct JSON source: [crawler-user-agents.json](https://raw.githubusercontent.com/monperrus/crawler-user-agents/refs/heads/master/crawler-user-agents.json)
  - 602 patterns maintained by community
  
- **IP Ranges** (IPv4 & IPv6):
  - **Googlebot**: Official IPv4/IPv6 ranges from [Google API](https://developers.google.com/search/apis/ipranges/googlebot.json)
    - 142 IPv6 /64 blocks automatically fetched
  - **27 Bot Sources**: [GoodBots Repository](https://github.com/AnTheMaker/GoodBots)
    - Includes Bingbot, Yandex, Facebook, Twitter, Telegram, Ahrefs, and more
  - **Manual Additions**:
    - Yandex: 15 IPv4 ranges + `2a02:6b8::/29` IPv6
    - Meta/Facebook: 7 IPv4 ranges (AS32934)
    - TikTok: 67 IPv4 ranges (AS138699, AS137775, AS396986)

## API Reference

### `isCrawler(ip, userAgent)`
Detects if either the IP or user-agent belongs to a known crawler.

**Parameters:**
- `ip` (string|null) - IP address to check
- `userAgent` (string|null) - User-Agent string to check

**Returns:** `boolean` - true if crawler detected

### `isCrawlerByIP(ip)`
Detects if the IP address belongs to a known crawler.

**Parameters:**
- `ip` (string) - IP address to check

**Returns:** `boolean` - true if crawler IP detected

### `isCrawlerByUA(userAgent)`
Detects if the user-agent belongs to a known crawler.

**Parameters:**
- `userAgent` (string) - User-Agent string to check

**Returns:** `boolean` - true if crawler user-agent detected

## License

MIT

## Contributing

Contributions welcome! Please:
1. Add tests for new functionality
2. Update documentation
3. Run `npm test` before submitting

## Credits

Special thanks to [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents) for maintaining the comprehensive crawler user-agent database.