Skill: kaiyuelv/vector-db-toolkit (version 1.0.0)

--- name: vector-db-toolkit description: Vector database operations toolkit for AI/RAG applications. Supports Qdrant, Chroma, and in-memory vector stores. Use when working with embeddings, semantic search, similarity queries, vector collections, or RAG retrieval pipelines. Triggers on phrases like "vector database", "embedding search", "semantic similarity", "Qdrant", "Chroma", "vector store", "embedding retrieval". --- # Vector Database Toolkit Operations toolkit for vector databases used in AI and RAG systems. ## Supported Backends - **Qdrant** - Open-source vector DB (REST/gRPC API) - **Chroma** - Lightweight embedded vector DB - **InMemory** - Pure Python fallback for small datasets ## Quick Start ### Qdrant ```python from scripts.qdrant_client import VectorClient client = VectorClient(backend="qdrant", url="http://localhost:6333") client.create_collection("docs", dimension=768) client.upsert("docs", ids=["a", "b"], vectors=[[0.1, ...], [0.2, ...]], payloads=[{"title": "A"}, {"title": "B"}]) results = client.search("docs", vector=[0.1, ...], top_k=5) ``` ### Chroma ```python client = VectorClient(backend="chroma", path="/tmp/chroma") client.create_collection("docs") client.upsert("docs", ids=["a"], vectors=[[0.1, ...]], payloads=[{"title": "A"}]) results = client.search("docs", vector=[0.1, ...], top_k=5) ``` ## Scripts - `scripts/vector_client.py` - Unified client for all backends - `scripts/qdrant_client.py` - Qdrant-specific operations - `scripts/chroma_client.py` - Chroma-specific operations - `scripts/embedding_utils.py` - Text-to-embedding helpers (optional OpenAI, sentence-transformers) ## References - `references/qdrant_api.md` - Qdrant API patterns - `references/chroma_api.md` - Chroma API patterns

path: README.md
content: # Vector Database Toolkit

Vector database operations toolkit for AI/RAG applications.

## 功能 | Features

- 统一的向量数据库客户端 | Unified vector DB client
- 支持 Qdrant、Chroma、内存存储 | Supports Qdrant, Chroma, InMemory
- 嵌入向量工具 | Embedding utilities
- 语义搜索 | Semantic search
- 批量操作 | Batch operations

## 安装 | Installation

```bash
pip install -r requirements.txt
```

## 快速开始 | Quick Start

```python
from scripts.vector_client import VectorClient

# Qdrant
client = VectorClient(backend="qdrant", url="http://localhost:6333")
client.create_collection("articles", dimension=768)
client.upsert("articles", ids=["1"], vectors=[[0.1]*768], payloads=[{"title": "Hello"}])
results = client.search("articles", vector=[0.1]*768, top_k=5)
```

## 目录结构 | Structure

```
vector-db-toolkit/
├── SKILL.md
├── README.md
├── requirements.txt
├── scripts/
│   ├── vector_client.py
│   ├── qdrant_client.py
│   ├── chroma_client.py
│   └── embedding_utils.py
├── examples/
│   └── basic_usage.py
└── tests/
    └── test_vector_client.py
```


path: examples/basic_usage.py
content: #!/usr/bin/env python3
"""
Basic usage examples for vector-db-toolkit.
"""
import sys
sys.path.insert(0, "../scripts")

from vector_client import VectorClient
from embedding_utils import EmbeddingProvider

def demo_memory():
    client = VectorClient(backend="memory")
    client.create_collection("items", dimension=3)
    client.upsert("items", ids=["a", "b", "c"],
                  vectors=[[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.5, 0.5, 0.0]],
                  payloads=[{"name": "red"}, {"name": "green"}, {"name": "yellow"}])
    results = client.search("items", vector=[0.9, 0.1, 0.0], top_k=2)
    print("Memory search results:", results)

def demo_embeddings():
    provider = EmbeddingProvider(provider="fallback")
    texts = ["hello world", "vector database", "semantic search"]
    embeddings = provider.encode(texts)
    print(f"Generated {len(embeddings)} embeddings, dim={len(embeddings[0])}")

if __name__ == "__main__":
    demo_memory()
    demo_embeddings()


path: references/chroma_api.md
content: # Chroma API Patterns

## Collection Management

```python
client.create_collection("my-collection", dimension=768)
client.list_collections()
client.delete("my-collection")  # deletes entire collection
```

## Document Operations

```python
client.upsert("my-collection", ids=["doc1"], vectors=[[0.1]*768], payloads=[{"source": "web"}])
client.delete("my-collection", ids=["doc1"])
```

## Query Patterns

```python
# Simple query
results = client.search("my-collection", vector=[0.1]*768, top_k=10)

# Metadata filter
results = client.search("my-collection", vector=[0.1]*768, top_k=10, filter_dict={"source": "web"})
```

## Notes

- Chroma distances are L2 by default (not cosine). Use `collection.query` with `n_results`.
- PersistentClient stores data on disk; EphemeralClient is in-memory.
- HttpClient connects to Chroma server.


path: references/qdrant_api.md
content: # Qdrant API Patterns

## Collection Management

```python
client.create_collection("my-collection", dimension=768, distance="COSINE")
client.list_collections()
client.delete("my-collection")  # deletes entire collection
```

## Point Operations

```python
client.upsert("my-collection", ids=["id1"], vectors=[[0.1]*768], payloads=[{"key": "val"}])
client.delete("my-collection", ids=["id1"])
```

## Search Patterns

```python
# Simple search
results = client.search("my-collection", vector=[0.1]*768, top_k=10)

# Filtered search
results = client.search("my-collection", vector=[0.1]*768, top_k=10, filter_dict={"status": "active"})
```

## Payload Filtering Syntax

Qdrant uses FieldCondition + MatchValue for equality filters.
See Qdrant docs for range, geo, and keyword filters.


path: requirements.txt
content: qdrant-client>=1.7.0
chromadb>=0.4.18
numpy>=1.24.0
requests>=2.31.0
sentence-transformers>=2.2.2


path: scripts/chroma_client.py
content: """
Chroma-specific vector client wrapper.
"""
from typing import List, Dict, Any, Optional

try:
    import chromadb
except ImportError:
    chromadb = None

class ChromaVectorClient:
    def __init__(self, path: Optional[str] = None, host: Optional[str] = None, port: int = 8000):
        if chromadb is None:
            raise ImportError("chromadb not installed. Run: pip install chromadb")
        if host:
            self.client = chromadb.HttpClient(host=host, port=port)
        else:
            self.client = chromadb.PersistentClient(path=path or "/tmp/chroma")

    def create_collection(self, name: str, dimension: Optional[int] = None, **kwargs):
        self.client.create_collection(name=name, metadata={"dimension": str(dimension) if dimension else None})
        return True

    def upsert(self, collection: str, ids: List[str], vectors: List[List[float]], payloads: Optional[List[Dict]] = None):
        col = self.client.get_collection(name=collection)
        metadatas = payloads if payloads else [{}] * len(ids)
        col.upsert(ids=ids, embeddings=vectors, metadatas=metadatas)
        return True

    def search(self, collection: str, vector: List[float], top_k: int = 5, filter_dict: Optional[Dict] = None):
        col = self.client.get_collection(name=collection)
        where = filter_dict if filter_dict else None
        results = col.query(query_embeddings=[vector], n_results=top_k, where=where)
        out = []
        if results["ids"] and results["ids"][0]:
            for i, rid in enumerate(results["ids"][0]):
                out.append({
                    "id": rid,
                    "score": results["distances"][0][i] if results.get("distances") else 0.0,
                    "payload": results["metadatas"][0][i] if results.get("metadatas") else {},
                })
        return out

    def delete(self, collection: str, ids: Optional[List[str]] = None):
        col = self.client.get_collection(name=collection)
        if ids:
            col.delete(ids=ids)
        else:
            self.client.delete_collection(name=collection)
        return True

    def list_collections(self):
        return [c.name for c in self.client.list_collections()]


path: scripts/embedding_utils.py
content: """
Embedding utilities for text-to-vector conversion.
"""
from typing import List, Optional
import numpy as np

try:
    from sentence_transformers import SentenceTransformer
except ImportError:
    SentenceTransformer = None

class EmbeddingProvider:
    def __init__(self, model_name: str = "all-MiniLM-L6-v2", provider: str = "local"):
        self.provider = provider
        self.model_name = model_name
        self._model = None
        if provider == "local" and SentenceTransformer:
            self._model = SentenceTransformer(model_name)

    def encode(self, texts: List[str]) -> List[List[float]]:
        if self.provider == "local" and self._model:
            embeddings = self._model.encode(texts, convert_to_numpy=True)
            return embeddings.tolist()
        elif self.provider == "openai":
            return self._openai_encode(texts)
        else:
            # simple fallback: char-level frequency vector (for demo only)
            return [self._fallback(t) for t in texts]

    def _openai_encode(self, texts: List[str]) -> List[List[float]]:
        import os, requests
        [REDACTED_SECRET]"OPENAI_API_KEY")
        if not api_key:
            raise RuntimeError("OPENAI_API_KEY not set")
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
            json={"model": self.model_name or "text-embedding-ada-002", "input": texts},
        )
        resp.raise_for_status()
        data = resp.json()["data"]
        return [d["embedding"] for d in data]

    def _fallback(self, text: str, dim: int = 384) -> List[float]:
        vec = np.zeros(dim)
        for i, ch in enumerate(text):
            vec[i % dim] += ord(ch) % 10
        norm = np.linalg.norm(vec)
        return (vec / (norm + 1e-10)).tolist()


path: scripts/qdrant_client.py
content: """
Qdrant-specific vector client wrapper.
"""
from typing import List, Dict, Any, Optional

try:
    from qdrant_client import QdrantClient
except ImportError:
    QdrantClient = None

class QdrantVectorClient:
    def __init__(self, url: str = "http://localhost:6333", [REDACTED_SECRET] = None):
        if QdrantClient is None:
            raise ImportError("qdrant-client not installed. Run: pip install qdrant-client")
        self.client = QdrantClient(url=url, [REDACTED_SECRET])

    def create_collection(self, name: str, dimension: int, distance: str = "COSINE", **kwargs):
        from qdrant_client.models import Distance, VectorParams
        dist = Distance.COSINE if distance == "COSINE" else Distance.EUCLID
        self.client.create_collection(
            collection_name=name,
            vectors_config=VectorParams(size=dimension, distance=dist),
        )
        return True

    def upsert(self, collection: str, ids: List[str], vectors: List[List[float]], payloads: Optional[List[Dict]] = None):
        from qdrant_client.models import PointStruct
        points = [
            PointStruct(id=ids[i], vector=vectors[i], payload=payloads[i] if payloads else {})
            for i in range(len(ids))
        ]
        self.client.upsert(collection_name=collection, points=points)
        return True

    def search(self, collection: str, vector: List[float], top_k: int = 5, filter_dict: Optional[Dict] = None):
        kwargs = {"collection_name": collection, "vector": vector, "limit": top_k}
        if filter_dict:
            from qdrant_client.models import FieldCondition, MatchValue, Filter
            conditions = [FieldCondition(key=k, match=MatchValue(v=v)) for k, v in filter_dict.items()]
            kwargs["query_filter"] = Filter(must=conditions)
        results = self.client.search(**kwargs)
        return [{"id": r.id, "score": r.score, "payload": r.payload} for r in results]

    def delete(self, collection: str, ids: Optional[List[str]] = None):
        if ids:
            from qdrant_client.models import PointIdsList
            self.client.delete(collection_name=collection, points_selector=PointIdsList(points=ids))
        else:
            self.client.delete_collection(collection_name=collection)
        return True

    def list_collections(self):
        return [c.name for c in self.client.get_collections().collections]


path: scripts/vector_client.py
content: """
Unified vector database client supporting Qdrant, Chroma, and InMemory backends.
"""
from typing import List, Dict, Any, Optional
import numpy as np

class VectorClient:
    def __init__(self, backend: str = "memory", **kwargs):
        self.backend = backend
        self._client = None
        self._collections: Dict[str, Any] = {}
        if backend == "qdrant":
            from .qdrant_client import QdrantVectorClient
            self._client = QdrantVectorClient(**kwargs)
        elif backend == "chroma":
            from .chroma_client import ChromaVectorClient
            self._client = ChromaVectorClient(**kwargs)
        elif backend == "memory":
            self._client = None
        else:
            raise ValueError(f"Unknown backend: {backend}")

    def create_collection(self, name: str, dimension: Optional[int] = None, **kwargs):
        if self.backend == "memory":
            self._collections[name] = {"dimension": dimension, "vectors": [], "ids": [], "payloads": []}
            return True
        return self._client.create_collection(name, dimension=dimension, **kwargs)

    def upsert(self, collection: str, ids: List[str], vectors: List[List[float]], payloads: Optional[List[Dict]] = None):
        if self.backend == "memory":
            col = self._collections[collection]
            for i, vid in enumerate(ids):
                if vid in col["ids"]:
                    idx = col["ids"].index(vid)
                    col["vectors"][idx] = vectors[i]
                    col["payloads"][idx] = payloads[i] if payloads else {}
                else:
                    col["ids"].append(vid)
                    col["vectors"].append(vectors[i])
                    col["payloads"].append(payloads[i] if payloads else {})
            return True
        return self._client.upsert(collection, ids, vectors, payloads)

    def search(self, collection: str, vector: List[float], top_k: int = 5, filter_dict: Optional[Dict] = None) -> List[Dict]:
        if self.backend == "memory":
            col = self._collections[collection]
            if not col["vectors"]:
                return []
            v = np.array(vector)
            vecs = np.array(col["vectors"])
            # cosine similarity
            norms = np.linalg.norm(vecs, axis=1)
            vnorm = np.linalg.norm(v)
            if vnorm == 0:
                return []
            sims = np.dot(vecs, v) / (norms * vnorm + 1e-10)
            top_idx = np.argsort(sims)[::-1][:top_k]
            results = []
            for idx in top_idx:
                r = {"id": col["ids"][idx], "score": float(sims[idx]), "payload": col["payloads"][idx]}
                if filter_dict is None or all(r["payload"].get(k) == v for k, v in filter_dict.items()):
                    results.append(r)
            return results
        return self._client.search(collection, vector, top_k, filter_dict)

    def delete(self, collection: str, ids: Optional[List[str]] = None):
        if self.backend == "memory":
            col = self._collections[collection]
            if ids:
                for vid in ids:
                    if vid in col["ids"]:
                        idx = col["ids"].index(vid)
                        col["ids"].pop(idx)
                        col["vectors"].pop(idx)
                        col["payloads"].pop(idx)
            else:
                col["ids"], col["vectors"], col["payloads"] = [], [], []
            return True
        return self._client.delete(collection, ids)

    def list_collections(self):
        if self.backend == "memory":
            return list(self._collections.keys())
        return self._client.list_collections()


path: tests/test_vector_client.py
content: """
Unit tests for vector-db-toolkit.
"""
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../scripts"))

import unittest
from vector_client import VectorClient
from embedding_utils import EmbeddingProvider

class TestVectorClient(unittest.TestCase):
    def test_memory_create_collection(self):
        c = VectorClient(backend="memory")
        self.assertTrue(c.create_collection("test", dimension=10))
        self.assertIn("test", c.list_collections())

    def test_memory_upsert_and_search(self):
        c = VectorClient(backend="memory")
        c.create_collection("test", dimension=3)
        c.upsert("test", ids=["1", "2"], vectors=[[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])
        results = c.search("test", vector=[1.0, 0.0, 0.0], top_k=1)
        self.assertEqual(len(results), 1)
        self.assertEqual(results[0]["id"], "1")

    def test_memory_delete(self):
        c = VectorClient(backend="memory")
        c.create_collection("test", dimension=2)
        c.upsert("test", ids=["x"], vectors=[[1.0, 1.0]])
        c.delete("test", ids=["x"])
        results = c.search("test", vector=[1.0, 1.0], top_k=1)
        self.assertEqual(len(results), 0)

    def test_memory_payload_filter(self):
        c = VectorClient(backend="memory")
        c.create_collection("test", dimension=2)
        c.upsert("test", ids=["a", "b"], vectors=[[1.0, 0.0], [1.0, 0.0]], payloads=[{"tag": "x"}, {"tag": "y"}])
        results = c.search("test", vector=[1.0, 0.0], top_k=5, filter_dict={"tag": "x"})
        self.assertEqual(len(results), 1)
        self.assertEqual(results[0]["id"], "a")

class TestEmbeddingProvider(unittest.TestCase):
    def test_fallback_encode(self):
        p = EmbeddingProvider(provider="fallback")
        embs = p.encode(["hello", "world"])
        self.assertEqual(len(embs), 2)
        self.assertEqual(len(embs[0]), 384)

if __name__ == "__main__":
    unittest.main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)