npm package: @operor/knowledge
Description: Knowledge base with vector search, retrieval pipeline, and document ingestors for Agent OS
Latest version: 0.5.0

# @operor/knowledge

Knowledge base system with vector search, retrieval pipeline, and document ingestors for Operor.

## Overview

`@operor/knowledge` provides a complete knowledge base solution for AI agents, enabling them to retrieve relevant context from documents, FAQs, and web content. The system uses vector embeddings for semantic search and includes specialized pipelines for ingestion and retrieval.

Inspired by [Astra Agent KB design](https://github.com/datastax/astra-assistants-api) and [OpenClaw's extensible plugin architecture](https://github.com/openclaw/openclaw), this package provides a modular, production-ready knowledge base with pluggable ingestors.

## Quick Start

### Installation

```bash
pnpm add @operor/knowledge
```

### Basic Usage

```typescript
import {
  SQLiteKnowledgeStore,
  EmbeddingService,
  TextChunker,
  IngestionPipeline,
  RetrievalPipeline,
} from '@operor/knowledge';

// 1. Initialize components
const store = new SQLiteKnowledgeStore('./knowledge.db', 1536);
await store.initialize();

const embedder = new EmbeddingService({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'text-embedding-3-small',
});

const chunker = new TextChunker({ chunkSize: 500, chunkOverlap: 50 });
const ingestion = new IngestionPipeline(store, embedder, chunker);
const retrieval = new RetrievalPipeline(store, embedder, 0.85);

// 2. Ingest documents
await ingestion.ingest({
  sourceType: 'url',
  content: 'Your document content here...',
  title: 'Getting Started Guide',
});

// 3. Query the knowledge base
const result = await retrieval.retrieve('How do I get started?', { limit: 5 });
console.log(result.context); // Formatted context for LLM injection
console.log(result.isFaqMatch); // true if FAQ fast-path matched
```

## Architecture

### Core Components

- **KnowledgeStore**: SQLite-backed storage with `sqlite-vec` for vector search
- **EmbeddingService**: Multi-provider embeddings via Vercel AI SDK (OpenAI, Google, Mistral, Cohere, Ollama)
- **TextChunker**: Document chunking using LangChain text splitters
- **IngestionPipeline**: Orchestrates document ingestion (chunk → embed → store)
- **RetrievalPipeline**: Query-time retrieval with FAQ fast-path optimization

### Ingestors (Extensible)

Following OpenClaw's plugin pattern, ingestors are modular and extensible:

- **UrlIngestor**: Web crawling with Readability extraction
- **FileIngestor**: PDF, DOCX, XLSX, CSV, TXT, MD, HTML parsing
- **WatiFaqSync**: FAQ extraction from WATI conversations via LLM

## Technology Stack

All dependencies use the latest stable versions:

- **Vector Search**: `sqlite-vec` (0.1.7-alpha.2) — Fast vector search in SQLite
- **Embeddings**: `ai` (6.x) + `@ai-sdk/*` (3.x) — Vercel AI SDK with multi-provider support
- **Text Splitting**: `@langchain/textsplitters` (1.x) — Recursive and markdown splitters
- **Document Parsing**:
  - PDF: `unpdf` (1.4.0)
  - DOCX: `mammoth` (1.11.0)
  - XLSX: `xlsx` (0.18.5)
  - HTML: `@mozilla/readability` (0.6.0) + `linkedom` (0.18.12)
- **Storage**: `better-sqlite3` (12.x) — Fast, synchronous SQLite

## Embedding Providers

The `EmbeddingService` supports multiple providers via the Vercel AI SDK:

| Provider | Model (default) | Dimensions | API Key Required |
|----------|----------------|------------|------------------|
| OpenAI | `text-embedding-3-small` | 1536 | Yes |
| Google | `text-embedding-004` | 768 | Yes |
| Mistral | `mistral-embed` | 1024 | Yes |
| Cohere | `embed-english-v3.0` | 1024 | Yes |
| Ollama | `nomic-embed-text` | 768 | No (local) |

**Important**: When switching embedding providers, you must re-ingest all documents. The vector dimensions must match what the store was initialized with, or search will fail.

### Provider Examples

```typescript
// OpenAI (default)
const embedder = new EmbeddingService({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'text-embedding-3-small', // optional
  dimensions: 1536, // optional, defaults per provider
});

// Google Gemini
const embedder = new EmbeddingService({
  provider: 'google',
  apiKey: process.env.GOOGLE_API_KEY,
  model: 'text-embedding-004',
});

// Mistral
const embedder = new EmbeddingService({
  provider: 'mistral',
  apiKey: process.env.MISTRAL_API_KEY,
});

// Cohere
const embedder = new EmbeddingService({
  provider: 'cohere',
  apiKey: process.env.COHERE_API_KEY,
});

// Ollama (local, no API key)
const embedder = new EmbeddingService({
  provider: 'ollama',
  model: 'nomic-embed-text',
  baseURL: 'http://localhost:11434/v1', // optional
});
```

## Usage Guide

### Ingesting Documents

#### From URL

```typescript
import { UrlIngestor } from '@operor/knowledge';

const urlIngestor = new UrlIngestor(ingestion);

// Single URL
await urlIngestor.ingestUrl('https://docs.example.com/guide');

// Sitemap (batch ingest)
await urlIngestor.ingestSitemap('https://example.com/sitemap.xml', {
  maxPages: 50,
});

// Crawl website (BFS traversal)
await urlIngestor.crawl('https://docs.example.com', {
  maxPages: 20,
  maxDepth: 2,
});
```

#### From File

```typescript
import { FileIngestor } from '@operor/knowledge';

const fileIngestor = new FileIngestor(ingestion);

// Supports: PDF, DOCX, XLSX, CSV, TXT, MD, HTML
await fileIngestor.ingestFile('./docs/manual.pdf', 'User Manual');
await fileIngestor.ingestFile('./data/products.xlsx', 'Product Catalog');
```

#### FAQ Entries

```typescript
// Manual FAQ entry
await ingestion.ingestFaq(
  'What is the return policy?',
  'You can return items within 30 days of purchase.'
);

// Batch FAQ sync from WATI conversations
import { WatiFaqSync } from '@operor/knowledge';

const faqSync = new WatiFaqSync(ingestion, async (conversation) => {
  // Your LLM extraction logic here
  return [
    { question: 'How do I reset my password?', answer: 'Click Forgot Password...' },
  ];
});

await faqSync.syncFromConversations(conversations, {
  minAnswerLength: 20,
  maxPairs: 100,
});
```

### Retrieving Context

```typescript
// Basic retrieval
const result = await retrieval.retrieve('What is the return policy?', {
  limit: 5,
  scoreThreshold: 0.7,
});

console.log(result.isFaqMatch); // true if FAQ fast-path matched
console.log(result.context); // Formatted context for LLM injection
console.log(result.results); // Raw search results with scores

// Filter by source type
const result = await retrieval.retrieve('pricing info', {
  limit: 3,
  sourceTypes: ['url', 'file'], // exclude FAQs
});
```

### FAQ Fast-Path

The retrieval pipeline includes an optimization for FAQs:

1. First searches FAQ documents only
2. If score ≥ 0.85 (configurable), returns immediately (fast-path)
3. Otherwise, searches full knowledge base

This ensures instant responses for common questions while maintaining comprehensive search for complex queries.

```typescript
// Adjust FAQ threshold (default: 0.85)
const retrieval = new RetrievalPipeline(store, embedder, 0.90);
```

## CLI Usage

The `@operor/cli` package provides commands for managing the knowledge base:

```bash
# Add documents
operor kb add-url https://docs.example.com/guide
operor kb add-file ./manual.pdf
operor kb add-faq "What are your hours?" "We are open 9-5 Mon-Fri"

# Search
operor kb search "return policy" -n 5

# List all documents
operor kb list

# Delete a document
operor kb delete <doc-id>

# Show statistics
operor kb stats
```

## API Reference

### EmbeddingService

```typescript
class EmbeddingService {
  constructor(config: EmbeddingServiceConfig);

  embed(text: string): Promise<number[]>;
  embedMany(texts: string[]): Promise<number[][]>;

  get dimensions(): number;
  get provider(): string;

  static defaultDimensions(provider: string, model?: string): number;
}

interface EmbeddingServiceConfig {
  provider: 'openai' | 'google' | 'mistral' | 'cohere' | 'ollama';
  apiKey?: string;
  model?: string;
  baseURL?: string;
  dimensions?: number;
}
```

### SQLiteKnowledgeStore

```typescript
class SQLiteKnowledgeStore implements KnowledgeStore {
  constructor(dbPath?: string, dimensions?: number);

  initialize(): Promise<void>;
  close(): Promise<void>;

  addDocument(doc: KBDocument): Promise<void>;
  getDocument(id: string): Promise<KBDocument | null>;
  listDocuments(): Promise<KBDocument[]>;
  deleteDocument(id: string): Promise<void>;

  addChunks(chunks: KBChunk[]): Promise<void>;
  search(query: string, embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;
  searchByEmbedding(embedding: number[], options?: KBSearchOptions): Promise<KBSearchResult[]>;

  getDimensions(): number;
}
```

### TextChunker

```typescript
class TextChunker {
  constructor(options?: ChunkOptions);

  chunk(text: string, options?: ChunkOptions): Promise<string[]>;
  chunkMarkdown(markdown: string, options?: ChunkOptions): Promise<string[]>;
}

interface ChunkOptions {
  chunkSize?: number; // default: 500
  chunkOverlap?: number; // default: 50
}
```

### IngestionPipeline

```typescript
class IngestionPipeline {
  constructor(store: KnowledgeStore, embedder: EmbeddingService, chunker: TextChunker);

  ingest(input: IngestInput): Promise<KBDocument>;
  ingestFaq(question: string, answer: string, metadata?: Record<string, any>): Promise<KBDocument>;
}

interface IngestInput {
  sourceType: 'url' | 'file' | 'faq' | 'annotation';
  content: string;
  title?: string;
  sourceUrl?: string;
  fileName?: string;
  metadata?: Record<string, any>;
}
```

### RetrievalPipeline

```typescript
class RetrievalPipeline {
  constructor(store: KnowledgeStore, embedder: EmbeddingService, faqThreshold?: number);

  retrieve(query: string, options?: KBSearchOptions): Promise<RetrievalResult>;
}

interface RetrievalResult {
  results: KBSearchResult[];
  context: string; // Formatted for LLM injection
  isFaqMatch: boolean; // true if FAQ fast-path matched
}
```

### Types

```typescript
interface KBDocument {
  id: string;
  sourceType: 'url' | 'file' | 'faq' | 'annotation';
  sourceUrl?: string;
  fileName?: string;
  title?: string;
  content: string;
  metadata?: Record<string, any>;
  createdAt: number;
  updatedAt: number;
}

interface KBChunk {
  id: string;
  documentId: string;
  content: string;
  chunkIndex: number;
  embedding?: number[];
  metadata?: Record<string, any>;
}

interface KBSearchResult {
  chunk: KBChunk;
  document: KBDocument;
  score: number; // 0-1, higher is better
  distance: number; // raw vector distance
}

interface KBSearchOptions {
  limit?: number;
  scoreThreshold?: number;
  sourceTypes?: ('url' | 'file' | 'faq' | 'annotation')[];
  metadata?: Record<string, any>;
}
```

## Extending with Custom Ingestors

Following OpenClaw's plugin architecture pattern, you can create custom ingestors:

```typescript
export class CustomIngestor {
  private pipeline: IngestionPipeline;

  constructor(pipeline: IngestionPipeline) {
    this.pipeline = pipeline;
  }

  async ingestCustomSource(source: string): Promise<KBDocument> {
    // 1. Extract content from your source
    const content = await this.extractContent(source);

    // 2. Use pipeline to ingest
    return this.pipeline.ingest({
      sourceType: 'url', // or 'file', 'annotation'
      content,
      title: 'Custom Source',
      metadata: { source: 'custom' },
    });
  }

  private async extractContent(source: string): Promise<string> {
    // Your extraction logic
    return 'extracted content';
  }
}
```

## Troubleshooting

### Dimension Mismatch Errors

If you see errors like "Dimension mismatch for inserted vector", it means:
- You switched embedding providers without re-ingesting documents
- The store was initialized with different dimensions than the embedder produces

**Solution**: Delete the KB database and re-ingest all documents with the new provider.

```bash
rm knowledge.db knowledge.db-shm knowledge.db-wal
operor kb add-url https://docs.example.com  # re-ingest
```

### Performance Tips

- Use batch ingestion for multiple documents (sitemap, crawl)
- Adjust chunk size based on your content (smaller for FAQs, larger for docs)
- Set appropriate `scoreThreshold` to filter low-quality results
- Use `sourceTypes` filter to narrow search scope

## License

MIT