npm package: @nexo-labs/payload-typesense
Description: A powerful, production-ready search plugin that integrates Typesense with Payload CMS, providing lightning-fast, typo-tolerant search capabilities with real-time synchronization and a comprehensive theme system.
Author: Zetesis - Rubén Garcia
Latest version: 1.15.0

# @nexo-labs/payload-typesense

A powerful, production-ready search plugin that integrates [Typesense](https://typesense.org/) with [Payload CMS](https://payloadcms.com/). This plugin provides lightning-fast, typo-tolerant search capabilities with real-time synchronization, and a comprehensive RAG (Retrieval Augmented Generation) system for building AI-powered conversational agents.

## Features

*   **Real-time Sync**: Automatically syncs your Payload collections to Typesense.
*   **Flexible Indexing**: Map a single Payload collection to multiple Typesense indices (tables) with different configurations.
*   **Advanced Search**:
    *   **Semantic Search**: Vector-based search using embeddings.
    *   **Keyword Search**: Traditional text matching with typo tolerance.
    *   **Hybrid Search**: Combines vector and keyword search for optimal relevance.
*   **RAG & AI Agents**:
    *   Built-in support for conversational AI agents.
    *   Integrates with OpenAI and Gemini.
    *   Manages chat sessions and history.
    *   Streaming responses (SSE).
*   **Optimized Performance**: Configurable HNSW parameters for vector search and batch processing for sync.

## Installation

```bash
pnpm add @nexo-labs/payload-typesense
```

## Configuration

The plugin is configured in your `payload.config.ts`.

### Basic Setup

```typescript
import { buildConfig } from 'payload/config';
import { typesenseSearch } from '@nexo-labs/payload-typesense';

export default buildConfig({
  // ...
  plugins: [
    typesenseSearch({
      typesense: {
        apiKey: process.env.TYPESENSE_API_KEY,
        nodes: [
          {
            host: process.env.TYPESENSE_HOST,
            port: 443,
            protocol: 'https',
          },
        ],
      },
      features: {
        sync: { enabled: true },
        search: { enabled: true },
        embedding: {
          type: 'openai',
          apiKey: process.env.OPENAI_API_KEY,
          model: 'text-embedding-3-small',
        },
      },
      collections: {
        posts: [
          {
            enabled: true,
            tableSuffix: 'v1',
            searchFields: ['title', 'content'],
            chunkingStrategy: { type: 'markdown' },
          },
        ],
      },
    }),
  ],
});
```

### Sync Configuration

You can define how each collection is synced to Typesense. The `collections` object maps collection slugs to an array of `CollectionTableConfig`.

```typescript
collections: {
  posts: [
    {
      enabled: true,
      // Optional suffix for the Typesense collection name
      // Result: "posts_summary"
      tableSuffix: 'summary',
      // Fields to include in the Typesense document
      searchFields: ['title', 'excerpt'],
      // Chunking strategy for long text
      chunkingStrategy: { type: 'simple', chunkSize: 500 },
    },
    {
      enabled: true,
      // Explicitly naming the table overrides the default naming convention
      // Result: "my_custom_posts_index"
      tableName: 'my_custom_posts_index',
      searchFields: ['title', 'content'],
      chunkingStrategy: { type: 'markdown' },
    }
  ]
}
```

### RAG & Agents Configuration

Enable RAG to build chat interfaces on top of your data.

```typescript
features: {
  rag: {
    enabled: true,
    agents: [
      {
        slug: 'support-bot',
        name: 'Support Assistant',
        systemPrompt: 'You are a helpful support assistant.',
        llmModel: 'gpt-4o-mini',
        // Collections this agent can access
        searchCollections: ['posts_full_text'],
      }
    ]
  }
}
```

## Usage

### Search API

The plugin exposes endpoints for searching your data.

**Endpoint:** `/api/typesense/search`

```typescript
// Example using fetch
const response = await fetch('/api/typesense/search', {
  method: 'POST',
  body: JSON.stringify({
    query: 'how to install',
    collections: ['posts_full_text'],
  })
});
```

### RAG Chat API

**Endpoint:** `/api/typesense/rag/chat`

This endpoint supports Server-Sent Events (SSE) for streaming AI responses.

```typescript
// Example client-side code (simplified)
const eventSource = new EventSource('/api/typesense/rag/chat?message=hello');

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (data.type === 'token') {
    console.log(data.data); // Streamed text
  }
};
```

## Roadmap

We are actively working to improve the modularity and flexibility of the plugin.

### 1. Modular Embedding Strategies (High Priority)
**Goal:** Allow different embedding models for different collections (tables).
*   **Current State:** Global embedding configuration only.
*   **Future:**
    *   Move `embedding` config to `CollectionTableConfig`.
    *   Support "Table X" -> Gemini, "Table Y" -> OpenAI Large.
    *   Update RAG agents to dynamically select the correct embedding model based on the collection being queried.

### 2. Expanded Provider Support
*   **Embeddings:** Add support for Cohere, HuggingFace, and local models (via Ollama).
*   **LLMs:** Add support for Anthropic (Claude), Mistral, and local LLMs.

### 4. Admin UI
*   Dashboard within Payload Admin to view Typesense collection status.
*   Buttons to manually trigger re-sync or re-indexing.
*   Visual management of RAG agents.

### 5. Advanced Sync Controls
*   **Soft Deletes:** Option to mark documents as deleted in Typesense instead of removing them.
*   **Conditional Sync:** specialized `syncCondition` function to control which documents are synced based on their content (e.g., only sync `published` posts).