npm package: @se-studio/search
Description: AI-powered site search with Upstash Search for Next.js marketing sites
Latest version: 1.0.36

# @se-studio/search

AI-powered site search for Next.js marketing sites using [Upstash Search](https://upstash.com/docs/search/overall/whatisupstashsearch). Combines semantic and full-text search with zero infrastructure to manage.

## Overview

This package provides:

- **Search client** – typed wrapper around `@upstash/search` with automatic batch handling
- **Content indexing** – extracts searchable text from CMS content using `MarkdownConverter`
- **Webhook handler** – incremental index updates on Contentful publish/delete
- **Full rebuild** – enumerates all content and re-indexes in batches
- **API route factories** – drop-in Next.js route handlers for search and rebuild
- **Client hook** – `useSearch()` for building search UIs with debouncing

## Setup

### 1. Install

The package is a workspace dependency. Add it to your app's `package.json`:

```json
{
  "dependencies": {
    "@se-studio/search": "workspace:*"
  }
}
```

### 2. Create an Upstash Search database

Go to [console.upstash.com/search](https://console.upstash.com/search) and create a database. Copy the REST URL and token.

### 3. Environment variables

Add to your `.env.local`:

```bash
UPSTASH_SEARCH_REST_URL=https://your-search-url.upstash.io
UPSTASH_SEARCH_REST_TOKEN=your-token
```

### 4. Search config

Create `src/lib/search-config.ts`:

```typescript
import 'server-only';
import type { SearchIndexingConfig } from '@se-studio/search';

export const searchIndexingConfig: SearchIndexingConfig = {
  searchIndex: {
    connection: {
      url: process.env.UPSTASH_SEARCH_REST_URL ?? '',
      token: process.env.UPSTASH_SEARCH_REST_TOKEN ?? '',
    },
    publishedIndexName: 'published',
    previewIndexName: 'preview',
  },
  contentTypes: [
    { type: 'page', enabled: true },
    { type: 'article', enabled: true },
    { type: 'person', enabled: false },
  ],
  indexComponents: true,
  respectIndexedFlag: true,
  respectHiddenFlag: true,
};
```

### 5. Search API route

Create `src/app/api/search/route.ts`:

```typescript
import { createSearchApiHandler } from '@se-studio/search/api';
import { buildInformation } from '@/lib/converter-context';
import { searchIndexingConfig } from '@/lib/search-config';

export const GET = createSearchApiHandler({
  searchConfig: searchIndexingConfig.searchIndex.connection,
  publishedIndexName: searchIndexingConfig.searchIndex.publishedIndexName,
  previewIndexName: searchIndexingConfig.searchIndex.previewIndexName,
  isPreview: buildInformation.preview ?? false,
});
```

### 6. Rebuild API route

Create `src/app/api/search/rebuild/route.ts`:

```typescript
import { createSearchClient } from '@se-studio/search/client';
import { rebuildSearchIndex } from '@se-studio/search/indexing';
import { createRebuildApiHandler } from '@se-studio/search/api';
import { buildOptions, getContentfulConfig } from '@/lib/cms-server';
import { customerName, license } from '@/lib/constants';
import { buildInformation, converterContext } from '@/lib/converter-context';
import { searchIndexingConfig } from '@/lib/search-config';
import { baseUrl, revalidationSecret } from '@/lib/server-config';

const isPreview = buildInformation.preview ?? false;

export const POST = createRebuildApiHandler({
  rebuildSecret: revalidationSecret ?? '',
  isPreview,
  rebuildFn: () => {
    const config = getContentfulConfig(isPreview);
    const client = createSearchClient(searchIndexingConfig.searchIndex.connection);
    const indexName = isPreview
      ? searchIndexingConfig.searchIndex.previewIndexName
      : searchIndexingConfig.searchIndex.publishedIndexName;

    return rebuildSearchIndex({
      client,
      indexName,
      indexingConfig: searchIndexingConfig,
      converterContext,
      contentfulConfig: config,
      fetchOptions: buildOptions({ preview: isPreview }),
      urlCalculators: converterContext.urlCalculators,
      siteConfig: { canonicalBaseUrl: baseUrl, source: customerName, license },
    });
  },
});
```

### 7. Webhook integration

Update your `src/app/api/revalidate/route.ts` to call the search webhook handler after cache revalidation. See the example-brightline app for the full pattern.

### 8. Client-side search

```tsx
'use client';
import { useSearch } from '@se-studio/search/hooks';

export function SearchPage() {
  const { query, setQuery, results, isLoading, error, totalCount } = useSearch();

  return (
    <div>
      <input value={query} onChange={(e) => setQuery(e.target.value)} placeholder="Search..." />
      {isLoading && <p>Searching...</p>}
      {error && <p>Error: {error}</p>}
      {results.map((r) => (
        <a key={r.id} href={r.metadata.href}>
          <h3>{r.content.title}</h3>
          <p>{r.content.description}</p>
        </a>
      ))}
    </div>
  );
}
```

## Triggering a full rebuild

```bash
curl -X POST https://your-site.com/api/search/rebuild \
  -H "Authorization: Bearer YOUR_REVALIDATION_SECRET"
```

The index to rebuild (published vs preview) is determined by the app's own `isPreview` flag
set at initialization time in `createRebuildApiHandler`.

## Advanced: documentTransformer

`SearchIndexingConfig` accepts an optional `documentTransformer` callback that is invoked for every `SearchDocument` after it has been built — in both the full-rebuild and webhook-driven incremental-update paths. Use it to patch metadata, inject custom fields, or drop specific documents entirely.

```typescript
import type { ContentData } from '@se-studio/search'; // re-exported from @se-studio/markdown-renderer
import type { SearchIndexingConfig } from '@se-studio/search';

export const searchIndexingConfig: SearchIndexingConfig = {
  // ...
  documentTransformer: (doc, contentData) => {
    // Augment: add a custom field
    doc.metadata.myCustomField = 'value';

    // Drop: returning null removes the document from the index
    if (doc.metadata.slug === 'draft-preview') return null;

    return doc;
  },
};
```

**Signature:**

```typescript
documentTransformer?: (doc: SearchDocument, contentData: ContentData) => SearchDocument | null;
```

- `doc` – the built `SearchDocument` (safe to mutate in place or return a new object)
- `contentData` – the raw CMS content data for the entry, giving access to all fields
- Return the document (modified or not) to include it, or `null` to drop it

The transformer runs on every chunk of a multi-chunk entry, so a 3-chunk page will call it 3 times.

## Architecture

- **Single Upstash database, two indexes**: `published` and `preview`
- **Text extraction**: Uses `MarkdownConverter` to deeply extract text from page components, then strips markdown formatting
- **Content truncation**: Body text is truncated to ~4,000 chars (Upstash limit)
- **Batch upsert**: Documents are upserted in batches of 100 (Upstash API limit)
- **Webhook-driven**: Incremental updates on Contentful publish/delete events
- **Flags**: Respects `indexed` and `hidden` fields on content entries

## Subpath Exports

| Import | Purpose |
|--------|---------|
| `@se-studio/search` | Types only |
| `@se-studio/search/client` | `createSearchClient()` |
| `@se-studio/search/indexing` | `rebuildSearchIndex()`, `buildSearchDocument()` |
| `@se-studio/search/webhook` | `createSearchWebhookHandler()` |
| `@se-studio/search/api` | `createSearchApiHandler()`, `createRebuildApiHandler()` |
| `@se-studio/search/hooks` | `useSearch()` |