npm package: @matteuccimarco/slim-vector-db
Description: SLIM compression for vector database metadata (Pinecone, Chroma, Qdrant, Weaviate)
Author: Marco Matteucci
Latest version: 0.1.0

# @matteuccimarco/slim-vector-db

SLIM compression for vector database metadata. Reduce metadata storage by 40-50% in Pinecone, Chroma, and other vector databases.

## Installation

```bash
npm install @matteuccimarco/slim-vector-db
```

## Quick Start

### Pinecone

```typescript
import { Pinecone } from '@pinecone-database/pinecone';
import { SlimPinecone } from '@matteuccimarco/slim-vector-db/pinecone';

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });

const slim = new SlimPinecone({
  client: pinecone,
  indexName: 'my-index',
  preserveFields: ['category'], // Keep for filtering
});

// Upsert with SLIM-encoded metadata
await slim.upsert([
  {
    id: 'doc-1',
    values: [0.1, 0.2, 0.3, ...],
    metadata: {
      category: 'tech',           // Preserved for filtering
      content: 'Long text...',    // SLIM-encoded
      attributes: {               // SLIM-encoded
        author: 'Alice',
        tags: ['ai', 'ml'],
      },
    },
  },
]);

// Query with automatic decoding
const results = await slim.query({
  vector: [0.1, 0.2, 0.3, ...],
  topK: 10,
  filter: { category: 'tech' },
});

console.log(results[0].metadata);
// { category: 'tech', content: 'Long text...', attributes: { author: 'Alice', tags: ['ai', 'ml'] } }
```

### ChromaDB

```typescript
import { ChromaClient } from 'chromadb';
import { SlimChroma } from '@matteuccimarco/slim-vector-db/chroma';

const chroma = new ChromaClient();

const slim = new SlimChroma({
  client: chroma,
  preserveFields: ['source'],
});

const collection = await slim.getOrCreateCollection('documents');

// Add with SLIM-encoded metadata
await collection.add({
  ids: ['doc-1', 'doc-2'],
  embeddings: [[0.1, 0.2], [0.3, 0.4]],
  metadatas: [
    { source: 'file.pdf', content: { page: 1, text: '...' } },
    { source: 'file.pdf', content: { page: 2, text: '...' } },
  ],
});

// Query with automatic decoding
const results = await collection.query({
  queryEmbeddings: [[0.1, 0.2]],
  nResults: 5,
});
```

## How It Works

Metadata is stored with SLIM encoding:

```javascript
// Original metadata
{
  category: 'tech',
  attributes: { author: 'Alice', tags: ['ai', 'ml'], rating: 4.5 }
}

// Stored in vector DB (with SLIM)
{
  category: 'tech',  // Preserved for filtering
  attributes: '__slim__{author:Alice,tags:@[ai,ml],rating:#4.5}'  // ~40% smaller
}
```

## Options

```typescript
const slim = new SlimPinecone({
  client: pinecone,
  indexName: 'my-index',

  // Fields to compress (default: all object fields)
  slimFields: ['content', 'attributes'],

  // Fields to keep uncompressed (for filtering)
  preserveFields: ['category', 'timestamp'],

  // Disable SLIM (pass-through mode)
  enabled: true,
});
```

## API Reference

### SlimPinecone

```typescript
class SlimPinecone {
  upsert(vectors: VectorRecord[], namespace?: string): Promise<void>;
  query(options: QueryOptions): Promise<QueryResult[]>;
  fetch(ids: string[], namespace?: string): Promise<Record<string, VectorRecord>>;
  delete(ids: string[], namespace?: string): Promise<void>;
  deleteAll(namespace?: string): Promise<void>;
}
```

### SlimChroma

```typescript
class SlimChroma {
  getOrCreateCollection(name: string): Promise<SlimCollection>;
  getCollection(name: string): Promise<SlimCollection>;
  deleteCollection(name: string): Promise<void>;
  listCollections(): Promise<string[]>;
}

class SlimCollection {
  add(params: AddParams): Promise<void>;
  query(params: QueryParams): Promise<QueryResults>;
  get(params: GetParams): Promise<GetResults>;
  update(params: UpdateParams): Promise<void>;
  upsert(params: UpsertParams): Promise<void>;
  delete(params: DeleteParams): Promise<void>;
  count(): Promise<number>;
}
```

## Benchmarks

| Metadata Type | JSON Size | SLIM Size | Savings |
|---------------|-----------|-----------|---------|
| Document chunk | 500 bytes | 290 bytes | 42% |
| Product data | 2 KB | 1.1 KB | 45% |
| User profile | 1.5 KB | 850 bytes | 43% |

## Use Cases

- **RAG Applications**: Compress document metadata
- **E-commerce**: Compact product attributes
- **Recommendation Systems**: Reduce user profile storage
- **Knowledge Bases**: Efficient entity metadata

## License

MIT