Skill: mrlynn/voyageai-skill (version 1.4.0)

--- name: voyageai description: > Voyage AI embedding and reranking CLI integrated with MongoDB Atlas Vector Search. Use for: generating text embeddings, reranking search results, storing embeddings in Atlas, performing vector similarity search, creating vector search indexes, listing available models, comparing text similarity, bulk ingestion, interactive demos, and learning about AI concepts. Triggers: embed text, generate embeddings, vector search, rerank documents, voyage ai, semantic search, similarity search, store embeddings, atlas vector search, embedding models, cosine similarity, bulk ingest, explain embeddings. metadata: openclaw: emoji: "🧭" author: name: Michael Lynn url: https://mlynn.org github: mrlynn version: "1.4.0" license: MIT tags: - embeddings - vector-search - reranking - mongodb - atlas - semantic-search - rag - voyage-ai requires: bins: - vai env: - VOYAGE_API_KEY install: - id: npm kind: npm package: voyageai-cli global: true --- # 🧭 Voyage AI Skill Uses the `vai` CLI ([voyageai-cli](https://github.com/mrlynn/voyageai-cli)) for Voyage AI embeddings, reranking, and MongoDB Atlas Vector Search. Pure Node.js — no Python required. ## Setup ```bash npm install -g voyageai-cli ``` ### Environment Variables | Variable | Required For | Description | |----------|-------------|-------------| | `VOYAGE_API_KEY` | embed, rerank, store, search, similarity, ingest, ping | Model API key from MongoDB Atlas | | `MONGODB_URI` | store, search, index, ingest, ping (optional) | Atlas connection string | Get your API key: **MongoDB Atlas → AI Models → Create model API key** ## Command Reference (14 commands) ### embed — Generate embeddings ```bash vai embed "What is MongoDB?" vai embed "search query" --model voyage-4-large --input-type query --dimensions 512 vai embed --file document.txt --input-type document cat texts.txt | vai embed vai embed "hello" --output-format array ``` ### rerank — Rerank documents ```bash vai rerank --query "database performance" --documents "MongoDB is fast" "SQL is relational" vai rerank --query "best database" --documents-file candidates.json --top-k 3 ``` ### store — Embed and store in Atlas ```bash vai store --db mydb --collection docs --field embedding \ --text "MongoDB Atlas is a cloud database" \ --metadata '{"source": "docs"}' # Batch from JSONL vai store --db mydb --collection docs --field embedding --file documents.jsonl ``` ### search — Vector search ```bash vai search --query "cloud database" --db mydb --collection docs \ --index vector_index --field embedding # With pre-filter vai search --query "performance" --db mydb --collection docs \ --index vector_index --field embedding --filter '{"category": "guides"}' --limit 5 ``` ### index — Manage vector search indexes ```bash vai index create --db mydb --collection docs --field embedding \ --dimensions 1024 --similarity cosine --index-name my_index vai index list --db mydb --collection docs vai index delete --db mydb --collection docs --index-name my_index ``` ### models — List available models ```bash vai models vai models --type embedding vai models --type reranking vai models --json ``` ### ping — Test connectivity ```bash vai ping vai ping --json ``` ### config — Manage persistent configuration ```bash vai config set api-key "pa-your-key" echo "pa-your-key" | vai config set api-key --stdin vai config get vai config delete api-key vai config path vai config reset ``` ### demo — Interactive guided walkthrough ```bash vai demo vai demo --no-pause vai demo --skip-pipeline vai demo --keep ``` ### explain — Learn about AI concepts ```bash vai explain # List all topics vai explain embeddings vai explain reranking vai explain vector-search vai explain rag vai explain cosine-similarity vai explain two-stage-retrieval vai explain input-type vai explain models vai explain api-keys vai explain api-access vai explain batch-processing ``` ### similarity — Compare text similarity ```bash vai similarity "MongoDB is a document database" "MongoDB Atlas is a cloud database" vai similarity "database performance" --against "MongoDB is fast" "PostgreSQL is relational" vai similarity --file1 doc1.txt --file2 doc2.txt vai similarity "text A" "text B" --json ``` ### ingest — Bulk import with progress ```bash vai ingest --file corpus.jsonl --db myapp --collection docs --field embedding vai ingest --file data.csv --db myapp --collection docs --field embedding --text-column content vai ingest --file corpus.jsonl --db myapp --collection docs --field embedding \ --model voyage-4 --batch-size 100 --input-type document vai ingest --file corpus.jsonl --db myapp --collection docs --field embedding --dry-run ``` ### completions — Shell completion scripts ```bash vai completions bash # Output bash completion script vai completions zsh # Output zsh completion script # Install bash completions vai completions bash >> ~/.bashrc && source ~/.bashrc # Install zsh completions vai completions zsh > ~/.zsh/completions/_vai ``` ### help — Display help ```bash vai help vai help embed vai embed --help ``` ## Common Workflows ### Embed → Store → Search Pipeline ```bash # 1. Store documents vai store --db myapp --collection articles --field embedding \ --text "MongoDB Atlas provides a fully managed cloud database" \ --metadata '{"title": "Atlas Overview"}' # 2. Create index vai index create --db myapp --collection articles --field embedding \ --dimensions 1024 --similarity cosine --index-name article_search # 3. Search vai search --query "how does cloud database work" \ --db myapp --collection articles --index article_search --field embedding ``` ### Two-Stage Retrieval (Embed + Rerank) ```bash # 1. Get candidates via vector search vai search --query "database scaling" --db myapp --collection articles \ --index article_search --field embedding --limit 20 --json > candidates.json # 2. Rerank for precision vai rerank --query "database scaling" --documents-file candidates.json --top-k 5 ``` ### Bulk Ingest Pipeline ```bash # 1. Validate data (dry run) vai ingest --file corpus.jsonl --db myapp --collection docs --field embedding --dry-run # 2. Ingest with progress vai ingest --file corpus.jsonl --db myapp --collection docs --field embedding # 3. Create index vai index create --db myapp --collection docs --field embedding \ --dimensions 1024 --similarity cosine ``` ## Global Flags | Flag | Description | |------|-------------| | `--json` | Machine-readable JSON output | | `--quiet` | Suppress non-essential output | ## References - [Model Catalog](references/models.md) — All models with pricing and specs - [Vector Search Patterns](references/vector-search.md) — Atlas Vector Search integration guide

path: references/vector-search.md
content: # Atlas Vector Search Integration Patterns

## Index Definition

Atlas Vector Search indexes are defined as JSON and created via the MongoDB driver or Atlas UI.

### Basic Index Definition

```json
{
  "name": "vector_index",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding",
        "numDimensions": 1024,
        "similarity": "cosine"
      }
    ]
  }
}
```

### Index with Pre-filter Fields

```json
{
  "name": "vector_index",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding",
        "numDimensions": 1024,
        "similarity": "cosine"
      },
      {
        "type": "filter",
        "path": "category"
      },
      {
        "type": "filter",
        "path": "status"
      }
    ]
  }
}
```

### Similarity Functions

| Function | Best For | Notes |
|----------|----------|-------|
| `cosine` | General purpose | Normalized, direction-based. Default choice. |
| `dotProduct` | Pre-normalized vectors | Faster than cosine when vectors are unit length |
| `euclidean` | Absolute distance matters | Sensitive to vector magnitude |

**Recommendation:** Use `cosine` unless you have a specific reason not to.

## $vectorSearch Aggregation Stage

### Basic Vector Search

```javascript
db.collection.aggregate([
  {
    $vectorSearch: {
      index: "vector_index",
      path: "embedding",
      queryVector: [0.1, 0.2, ...],  // Your query embedding
      numCandidates: 150,
      limit: 10
    }
  },
  {
    $project: {
      _id: 1,
      text: 1,
      score: { $meta: "vectorSearchScore" }
    }
  }
])
```

### Vector Search with Pre-filters

```javascript
db.collection.aggregate([
  {
    $vectorSearch: {
      index: "vector_index",
      path: "embedding",
      queryVector: [0.1, 0.2, ...],
      numCandidates: 150,
      limit: 10,
      filter: {
        $and: [
          { category: { $eq: "technology" } },
          { status: { $eq: "published" } }
        ]
      }
    }
  },
  {
    $project: {
      text: 1,
      category: 1,
      score: { $meta: "vectorSearchScore" }
    }
  }
])
```

**Important:** Filter fields must be declared in the index definition with `"type": "filter"`.

### Supported Filter Operators

- `$eq`, `$ne`
- `$gt`, `$gte`, `$lt`, `$lte`
- `$in`, `$nin`
- `$and`, `$or`

## Hybrid Search (Vector + Full-Text)

Combine vector search with Atlas Search for hybrid retrieval using `$unionWith` or reciprocal rank fusion.

### Reciprocal Rank Fusion (RRF)

```javascript
// Stage 1: Vector search results
const vectorResults = await collection.aggregate([
  {
    $vectorSearch: {
      index: "vector_index",
      path: "embedding",
      queryVector: queryEmbedding,
      numCandidates: 100,
      limit: 20
    }
  },
  { $addFields: { vs_score: { $meta: "vectorSearchScore" } } },
  { $project: { text: 1, vs_score: 1 } }
]).toArray();

// Stage 2: Text search results
const textResults = await collection.aggregate([
  {
    $search: {
      index: "text_index",
      text: { query: "search terms", path: "text" }
    }
  },
  { $limit: 20 },
  { $addFields: { ts_score: { $meta: "searchScore" } } },
  { $project: { text: 1, ts_score: 1 } }
]).toArray();

// Combine with RRF
const k = 60; // RRF constant
const combined = {};
vectorResults.forEach((doc, i) => {
  const id = doc._id.toString();
  combined[id] = combined[id] || { ...doc, rrf: 0 };
  combined[id].rrf += 1 / (k + i + 1);
});
textResults.forEach((doc, i) => {
  const id = doc._id.toString();
  combined[id] = combined[id] || { ...doc, rrf: 0 };
  combined[id].rrf += 1 / (k + i + 1);
});
const finalResults = Object.values(combined).sort((a, b) => b.rrf - a.rrf);
```

## Best Practices

### 1. Always Use `input_type`

```bash
# For queries (what you're searching for)
voyageai embed "how to scale MongoDB" --input-type query

# For documents (what you're indexing)
voyageai embed --file article.txt --input-type document
```

This asymmetric encoding improves retrieval quality by 3-5% on average.

### 2. Dimension Selection

| Dimensions | Storage per Vector | Use Case |
|------------|-------------------|----------|
| 256 | 1 KB | Mobile/edge, very large collections |
| 512 | 2 KB | Good balance for large collections |
| 1024 | 4 KB | **Recommended default** |
| 2048 | 8 KB | Maximum quality, smaller collections |

### 3. numCandidates Tuning

- `numCandidates` should be 10-20× your `limit`
- Higher values → better recall, slower queries
- Start with `numCandidates: 150, limit: 10` and adjust

### 4. Batch Embedding for Storage

When storing many documents, batch your embeddings:
```bash
# Embed multiple texts at once (up to 128 per request)
cat documents.txt | voyageai embed --input-type document --json
```

### 5. Model Compatibility

All voyage-4 series models produce compatible embeddings. Strategy:
- Index documents with `voyage-4-large` (best quality)
- Query with `voyage-4-lite` (cheapest, fastest)
- Results are still high quality due to shared embedding space

## Example Pipeline: End-to-End

```bash
# 1. Create the collection index
voyageai index create \
  --db semanticapp \
  --collection knowledge \
  --field embedding \
  --dimensions 1024 \
  --similarity cosine \
  --index-name knowledge_vector

# 2. Ingest documents
for file in docs/*.txt; do
  voyageai store \
    --db semanticapp \
    --collection knowledge \
    --field embedding \
    --file "$file" \
    --model voyage-4-large \
    --metadata "{\"source\": \"$file\"}"
done

# 3. Search
voyageai search \
  --query "How does sharding work in MongoDB?" \
  --db semanticapp \
  --collection knowledge \
  --index knowledge_vector \
  --field embedding \
  --model voyage-4 \
  --limit 5

# 4. Optional: Rerank the results for better precision
voyageai search \
  --query "How does sharding work?" \
  --db semanticapp \
  --collection knowledge \
  --index knowledge_vector \
  --field embedding \
  --limit 20 --json | \
  jq '[.[].text]' | \
  voyageai rerank --query "How does sharding work?" --documents-file /dev/stdin --top-k 5
```


path: references/models.md
content: # Voyage AI Model Catalog

All models are accessed via the MongoDB AI API at `https://ai.mongodb.com/v1/`.

## Text Embedding Models

### General Purpose

| Model | Context | Dimensions | Price/1M tokens | Best For |
|-------|---------|------------|-----------------|----------|
| voyage-4-large | 32K | 1024 (default), 256, 512, 2048 | $0.12 | Best quality, multilingual |
| voyage-4 | 32K | 1024 (default), 256, 512, 2048 | $0.06 | Balanced quality & performance |
| voyage-4-lite | 32K | 1024 (default), 256, 512, 2048 | $0.02 | Lowest cost, high throughput |

### Domain-Specific

| Model | Context | Dimensions | Price/1M tokens | Best For |
|-------|---------|------------|-----------------|----------|
| voyage-code-3 | 32K | 1024 (default), 256, 512, 2048 | $0.18 | Code retrieval & search |
| voyage-finance-2 | 32K | 1024 | $0.12 | Financial documents |
| voyage-law-2 | 16K | 1024 | $0.12 | Legal documents |

### Specialized

| Model | Context | Dimensions | Price/1M tokens | Best For |
|-------|---------|------------|-----------------|----------|
| voyage-context-3 | 32K | 1024 (default), 256, 512, 2048 | $0.18 | Contextualized chunks (document + chunk) |
| voyage-multimodal-3.5 | 32K | 1024 (default), 256, 512, 2048 | $0.12/M tokens + $0.60/B pixels | Text + images + video frames |

## Reranking Models

| Model | Context | Price/1M tokens | Best For |
|-------|---------|-----------------|----------|
| rerank-2.5 | 32K | $0.05 | Best quality reranking |
| rerank-2.5-lite | 32K | $0.02 | Fast, cost-effective reranking |

## Key Notes

### Free Tier
- **200M tokens** free for most models
- **50M tokens** free for domain-specific models (code, finance, law)

### Embedding Space Compatibility
All **4-series models** (voyage-4-large, voyage-4, voyage-4-lite) share the same embedding space. This means:
- You can embed queries with `voyage-4-lite` and documents with `voyage-4-large` — they're compatible
- Mix and match based on cost/quality tradeoffs
- Great for asymmetric retrieval (cheap query embeddings, high-quality document embeddings)

### Dimension Selection
- Higher dimensions → better quality, more storage
- Lower dimensions → faster search, less storage
- **1024** is the sweet spot for most use cases
- **256** works well for constrained environments
- Use Matryoshka truncation: embed at full dim, truncate later

### Input Types
Always specify `input_type` for best results:
- `query` — for search queries (short text being searched for)
- `document` — for documents being indexed (text being searched over)

This asymmetric encoding significantly improves retrieval quality.

### Rate Limits
- Default: 300 RPM, 1M TPM
- Batch sizes: up to 128 texts per request for embedding models
- Reranking: up to 1000 documents per request


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (MEDIUM)