npm package: @nexo-labs/payload-indexer
Description: Generic document indexing library for Payload CMS with support for multiple search backends, embedding providers, and chunking strategies.
Author: Zetesis - Rubén Garcia
Latest version: 1.15.1

# @nexo-labs/payload-indexer

Generic document indexing library for Payload CMS with support for multiple search backends, embedding providers, and chunking strategies.

## Installation

```bash
pnpm add @nexo-labs/payload-indexer
```

## Usage

```ts
import { createIndexerPlugin, OpenAIEmbeddingProvider } from '@nexo-labs/payload-indexer'

const indexerPlugin = createIndexerPlugin({
  collections: ['posts', 'pages'],
  adapter: mySearchAdapter,
  embedding: {
    provider: new OpenAIEmbeddingProvider({ apiKey: process.env.OPENAI_API_KEY }),
  },
})
```

## Entry Points

| Subpath | Description |
|---------|-------------|
| `.` | Main entry -- plugin factory, adapters, embedding, chunking, sync status services |
| `./client` | Client components for Payload admin UI (`SyncStatusCell`, `SyncStatusField`) |

## Exports

### Main (`@nexo-labs/payload-indexer`)

- **`createIndexerPlugin`** - Main plugin factory for Payload CMS
- **`IndexerAdapter`** - Abstract adapter interface for search backends
- **`OpenAIEmbeddingProvider`** / **`GeminiEmbeddingProvider`** - Embedding providers
- **`createEmbeddingService`** - Embedding service factory
- **`chunkMarkdown`** / **`chunkText`** - Text chunking strategies
- **`mapPayloadDocumentToIndex`** - Document mapping utilities
- **`DocumentSyncer`** - Sync utilities for document indexing
- **`Logger`** / **`createLogger`** - Configurable logging system
- **`createSyncStatusEndpoints`** - REST endpoints for sync status checks
- **`checkSyncStatus`** / **`checkBatchSyncStatus`** - Sync status comparison services

### Client (`@nexo-labs/payload-indexer/client`)

- **`SyncStatusCell`** - List view cell component showing sync status as a colored Pill
- **`SyncStatusField`** - Sidebar field component with status display and manual "Sync Now" button

## Architecture

### Key Classes

| Class / Function | File | Role |
|-----------------|------|------|
| `DocumentSyncer` | `src/plugin/sync/document-syncer.ts` | Orchestrates the full sync lifecycle: field mapping, chunking, embedding, and adapter upsert |
| `EmbeddingService` | `src/embedding/` | Batch embedding generation via OpenAI or Gemini providers |
| `createAfterChangeHook` | `src/plugin/sync/hooks.ts` | Payload lifecycle hook that triggers sync on document save |
| `createAfterDeleteHook` | `src/plugin/sync/hooks.ts` | Payload lifecycle hook that triggers deletion from the index |
| `checkSyncStatus` | `src/sync-status/sync-status-service.ts` | Compares SHA-256 content hashes between Payload and the index |
| `mapPayloadDocumentToIndex` | `src/document/field-mapper.ts` | Maps Payload document fields to the indexable schema |

### Sync Execution Flow

```
afterChange hook
  -> processTableConfigAfterChange
    -> DocumentSyncer.syncChunked
      -> mapPayloadDocumentToIndex (field mapping)
      -> chunkMarkdown / chunkText (chunking strategies)
      -> EmbeddingService.getEmbedding (OpenAI / Gemini)
      -> adapter.upsertDocument (backend-specific)
```

### Text Extraction Pipeline

For Lexical rich text fields, the plugin extracts plain text for indexing:

```
createSummarizeLexicalTransform
  -> extractLinkNodeText / extractHeadingNodeText / extractQuoteNodeText
```

## Features

- Multi-backend adapter pattern (Typesense, Algolia, or custom)
- Embedding providers for OpenAI and Google Gemini with batch processing
- Smart chunking for markdown and plain text with configurable size/overlap
- Automatic sync via Payload hooks (`afterChange`, `afterDelete`)
- Lexical editor content to markdown transformation
- Vector search with semantic embeddings
- Type-safe with full TypeScript support and schema inference
- **Sync status virtual field** -- automatically injected into indexed collections showing whether each document is synced, outdated, or not indexed
- **Manual sync trigger** -- "Sync Now" button in the document sidebar to re-index a single document on demand

### Sync Status

When `features.sync.enabled` is `true`, the plugin automatically:

1. **Injects a `_syncStatus` virtual field** into every indexed collection. This field appears in the admin sidebar and list view.
2. **Registers REST endpoints** for sync status checks:
   - `GET /api/sync-status/:collection` -- Batch check all documents in a collection (supports `page`, `limit`, `ids`, `where` params)
   - `GET /api/sync-status/:collection/:id` -- Check a single document
   - `POST /api/sync-status/:collection/:id/sync` -- Trigger re-indexing of a single document

The sync status is computed by comparing content hashes (SHA-256) between the Payload document and its indexed counterpart. Possible values: `synced`, `outdated`, `not-indexed`, `error`.

## Peer Dependencies

- `payload` ^3.75.0
- `@payloadcms/richtext-lexical` ^3.75.0 (optional)
- `@payloadcms/ui` ^3.75.0 (optional -- needed for client components)
- `react` ^19.0.0 (optional -- needed for client components)

## License

MIT