Skill: nissan/vector-store-shootout (version 1.0.0)

--- name: vector-store-shootout version: 1.0.0 description: 8 vector store implementations behind a common interface — numpy, lancedb, qdrant, pgvector, weaviate, weaviate_hybrid, milvus, lightrag. Use when evaluating RAG backends, building vector search, or comparing embedding stores. Each backend is a drop-in replacement via the base class. metadata: {"openclaw": {"emoji": "🔍", "requires": {"bins": ["python3"], "env": []}, "primaryEnv": null, "network": {"outbound": false, "reason": "All backends run locally. Network calls depend on your deployment (e.g. managed Qdrant Cloud)."}}} --- # Vector Store Shootout Eight vector store backends with a common `VectorStore` interface. Swap backends by changing one line — the rest of your code stays the same. ## Backends | Backend | Type | Dependencies | Best For | |---|---|---|---| | **numpy** | In-memory | numpy only | Prototyping, small datasets | | **lancedb** | File-based | lancedb | Local persistence, Arrow-native | | **qdrant** | Client-server | qdrant-client | Production, filtering | | **pgvector** | Postgres extension | psycopg2 | Existing Postgres deployments | | **weaviate** | Client-server | weaviate-client | Hybrid search (BM25 + vector) | | **weaviate_hybrid** | Client-server | weaviate-client | BM25-heavy hybrid (alpha=0.1) | | **milvus** | Client-server | pymilvus | Large-scale, GPU-accelerated | | **lightrag** | Graph-enhanced | lightrag | Graph + vector RAG | ## Common Interface ```python from base import VectorStore class MyStore(VectorStore): async def add(self, texts, embeddings, metadatas): ... async def search(self, query_embedding, k=5): ... async def delete(self, ids): ... ``` ## Key Finding Weaviate hybrid search at alpha=0.1 (BM25-heavy) scored avg 0.9940 vs 0.9700 at default 0.5. For technical content with specific terminology, keyword matching matters more than semantic similarity. ## Files - `scripts/base.py` — Abstract base class - `scripts/numpy_store.py` through `scripts/lightrag_store.py` — All 8 implementations

path: scripts/base.py
content: """Abstract base class for vector store backends.

All backends implement the same interface so RAG tasks and tests
are backend-agnostic. Swap ``NumpyCosineStore`` for ``QdrantStore``
(or any future backend) without changing task or eval code.
"""
from __future__ import annotations

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Optional


@dataclass
class RetrievedChunk:
    """A document chunk returned by a similarity query."""
    text: str
    doc_id: str
    score: float           # cosine similarity in [0, 1]
    metadata: dict = field(default_factory=dict)

    def __repr__(self) -> str:
        return f"RetrievedChunk(doc_id={self.doc_id!r}, score={self.score:.3f})"


class BaseVectorStore(ABC):
    """Backend-agnostic interface for vector retrieval.

    Implementations:
        NumpyCosineStore  — Ollama nomic-embed-text + numpy in-process cosine similarity
        QdrantStore       — Qdrant HNSW index (self-hosted Docker)
    """

    @abstractmethod
    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        """Index a list of document strings."""
        ...

    @abstractmethod
    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        """Return the top-n most similar chunks to query_text."""
        ...

    @property
    @abstractmethod
    def size(self) -> int:
        """Number of indexed documents."""
        ...

    @property
    @abstractmethod
    def backend_name(self) -> str:
        """Human-readable backend identifier for logging/tracing."""
        ...


path: scripts/lancedb_store.py
content: """LanceDB vector store — embedded, serverless, Apache 2.0.

LanceDB uses the Lance columnar format for on-disk persistence.
No server or Docker required — runs in-process like VoyageNumpyStore
but with a persistent on-disk index and ANN search.

Key differences from VoyageNumpyStore:
  - Persistent across runs (files written to disk)
  - ANN index (IVF+PQ or brute-force for small corpora) vs exact cosine
  - Columnar storage — fast on large corpora

For comparative RAG evaluation each task run creates an isolated
temp table (UUID name) so results are not contaminated across runs.

License: Apache 2.0
Docs:    https://lancedb.github.io/lancedb/
"""
from __future__ import annotations

import logging
import tempfile
import uuid
from pathlib import Path
from typing import Optional

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

_OLLAMA_EMBED_URL   = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"
_VECTOR_SIZE        = 768  # nomic-embed-text output dimension


class LanceDBStore(BaseVectorStore):
    """LanceDB-backed vector store (embedded, no server required).

    Each instance creates an isolated table in a temp directory so
    concurrent eval runs do not interfere with each other.

    Args:
        ollama_url:     Ollama embed API URL.
        openai_api_key: Optional OpenAI key (text-embedding-3-small fallback).
        db_path:        Path to LanceDB database directory.
                        Defaults to a fresh temp directory per instance.
        table_name:     LanceDB table name. UUID-generated if not provided.
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        db_path: Optional[str] = None,
        table_name: Optional[str] = None,
    ) -> None:
        self._ollama_url   = ollama_url
        self._openai_key   = openai_api_key
        self._table_name   = table_name or f"rag_{uuid.uuid4().hex[:12]}"
        self._size         = 0
        self._table        = None

        # Use a fresh temp dir per instance so runs are isolated by default.
        # If db_path is provided (e.g. for benchmarks), reuse it.
        if db_path:
            self._db_path = Path(db_path)
            self._owned_tmpdir = None
        else:
            self._owned_tmpdir = tempfile.mkdtemp(prefix="lancedb_rag_")
            self._db_path = Path(self._owned_tmpdir)

    # ── Public API ──────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return self._size

    @property
    def backend_name(self) -> str:
        return "lancedb"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        import lancedb
        import pyarrow as pa

        embeddings = self._embed(texts)
        auto_ids   = ids or [f"doc_{i}" for i in range(len(texts))]
        auto_meta  = metadatas or [{} for _ in texts]

        rows = []
        for text, doc_id, meta, emb in zip(texts, auto_ids, auto_meta, embeddings):
            rows.append({
                "doc_id": doc_id,
                "text":   text,
                "vector": emb,
            })

        schema = pa.schema([
            pa.field("doc_id", pa.string()),
            pa.field("text",   pa.string()),
            pa.field("vector", pa.list_(pa.float32(), len(embeddings[0]))),
        ])

        db = lancedb.connect(str(self._db_path))
        if self._table is None:
            self._table = db.create_table(
                self._table_name, data=rows, schema=schema, mode="overwrite"
            )
        else:
            self._table.add(rows)

        self._size += len(texts)
        logger.debug(
            "LanceDB: added %d docs to table %s (total=%d)",
            len(texts), self._table_name, self._size,
        )

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if self._size == 0 or self._table is None:
            return []

        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        k = min(n_results, self._size)
        results = (
            self._table
            .search(q_emb[0])
            .metric("cosine")
            .limit(k)
            .to_list()
        )

        chunks = []
        for row in results:
            # LanceDB returns _distance (lower = closer for cosine distance)
            # Convert to similarity: similarity = 1 - distance
            dist  = float(row.get("_distance", 0.0))
            score = max(0.0, 1.0 - dist)
            chunks.append(RetrievedChunk(
                text   = row.get("text", ""),
                doc_id = row.get("doc_id", ""),
                score  = score,
            ))
        return chunks

    def cleanup(self) -> None:
        """Remove temp directory (if we own it)."""
        if self._owned_tmpdir:
            import shutil
            try:
                shutil.rmtree(self._owned_tmpdir, ignore_errors=True)
            except Exception as exc:
                logger.warning("LanceDB cleanup failed: %s", exc)

    # ── Private helpers ──────────────────────────────────────────────────

    def _embed(self, texts: list[str]) -> list[list[float]]:
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed: %s", exc)
        logger.warning("All providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"] for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(texts: list[str]) -> list[list[float]]:
        import math
        tokenised = [t.lower().split() for t in texts]
        vocab: dict[str, int] = {}
        for toks in tokenised:
            for w in toks:
                if w not in vocab:
                    vocab[w] = len(vocab)
        n_vocab = len(vocab) or 1
        vecs = []
        for toks in tokenised:
            tf: dict[str, float] = {}
            for w in toks:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log(2)
            vecs.append(vec)
        return vecs


path: scripts/lightrag_store.py
content: """LightRAG graph-enhanced vector store.

Combines cosine vector similarity with a lightweight entity-relationship
graph built at index time using regex-based NER. At query time, top cosine
matches are expanded via graph traversal to pull in related context that
pure vector search might miss.

Backend string: ``"lightrag"``

Graph model (v1):
    - **Nodes**: entities extracted via regex (capitalized phrases, numbers,
      technical terms like acronyms or hyphenated compounds).
    - **Edges**: co-occurrence within the same chunk. Edge weight = number
      of chunks where both entities appear together.

Query pipeline:
    1. Embed query → cosine similarity over corpus → top-k candidates
    2. Extract entities from top candidates
    3. Graph-expand: 1-hop neighbours of matched entities
    4. Retrieve chunks containing neighbour entities
    5. Deduplicate + blend (vector results first, then graph expansions)

No external API calls in the indexing path — all entity extraction is
regex-based for speed.
"""
from __future__ import annotations

import hashlib
import logging
import re
from collections import defaultdict
from typing import Callable, Optional

import networkx as nx

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

# Regex patterns for entity extraction
_CAPITALIZED_PHRASE = re.compile(r"\b([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\b")
_ACRONYM = re.compile(r"\b([A-Z]{2,})\b")
_TECHNICAL_TERM = re.compile(r"\b([a-zA-Z]+-[a-zA-Z]+(?:-[a-zA-Z]+)*)\b")
_NUMBER_WITH_UNIT = re.compile(r"\b(\d+(?:\.\d+)?(?:\s*(?:GB|MB|KB|TB|GHz|MHz|ms|km|kg|m|s|%)))\b")

# Ollama defaults (same as numpy_store)
_OLLAMA_EMBED_URL = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"


def _extract_entities(text: str) -> set[str]:
    """Extract entities from text using regex heuristics."""
    entities: set[str] = set()
    for pattern in (_CAPITALIZED_PHRASE, _ACRONYM, _TECHNICAL_TERM, _NUMBER_WITH_UNIT):
        for match in pattern.finditer(text):
            ent = match.group(1).strip()
            if len(ent) > 1:  # skip single chars
                entities.add(ent)
    return entities


class LightRAGStore(BaseVectorStore):
    """Graph-enhanced vector store using networkx + cosine similarity.

    Args:
        ollama_url:     Ollama embed API URL.
        openai_api_key: Optional OpenAI fallback key.
        embed_fn:       Test injection for embeddings.
        graph_hops:     Number of hops for graph expansion (default: 1).
        graph_boost:    Max extra chunks from graph expansion (default: 2).
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        embed_fn: Optional[Callable[[list[str]], list[list[float]]]] = None,
        graph_hops: int = 1,
        graph_boost: int = 2,
    ) -> None:
        self._ollama_url = ollama_url
        self._openai_key = openai_api_key
        self._embed_fn = embed_fn
        self._graph_hops = graph_hops
        self._graph_boost = graph_boost

        # Vector data
        self._texts: list[str] = []
        self._ids: list[str] = []
        self._metadatas: list[dict] = []
        self._embeddings: list[list[float]] = []

        # Graph data
        self._graph = nx.Graph()
        # entity → set of chunk indices
        self._entity_to_chunks: dict[str, set[int]] = defaultdict(set)

    @property
    def size(self) -> int:
        return len(self._texts)

    @property
    def backend_name(self) -> str:
        return "lightrag"

    @property
    def graph(self) -> nx.Graph:
        """Expose graph for inspection/testing."""
        return self._graph

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        auto_ids = ids or [hashlib.md5(t.encode()).hexdigest()[:8] for t in texts]
        auto_meta = metadatas or [{} for _ in texts]

        embeddings = self._embed(texts)

        base_idx = len(self._texts)
        self._texts.extend(texts)
        self._ids.extend(auto_ids)
        self._metadatas.extend(auto_meta)
        self._embeddings.extend(embeddings)

        # Build graph from new documents
        for i, text in enumerate(texts):
            chunk_idx = base_idx + i
            entities = _extract_entities(text)
            ent_list = list(entities)

            for ent in ent_list:
                self._graph.add_node(ent)
                self._entity_to_chunks[ent].add(chunk_idx)

            # Co-occurrence edges
            for a_idx in range(len(ent_list)):
                for b_idx in range(a_idx + 1, len(ent_list)):
                    a, b = ent_list[a_idx], ent_list[b_idx]
                    if self._graph.has_edge(a, b):
                        self._graph[a][b]["weight"] += 1
                    else:
                        self._graph.add_edge(a, b, weight=1)

        logger.debug(
            "LightRAG indexed %d docs, graph: %d nodes, %d edges",
            len(texts), self._graph.number_of_nodes(), self._graph.number_of_edges(),
        )

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if not self._embeddings:
            return []

        try:
            import numpy as np
        except ImportError:
            logger.error("numpy required for LightRAGStore")
            return []

        # Step 1: Cosine similarity
        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        q_vec = np.array(q_emb[0], dtype=float)
        corpus = np.array(self._embeddings, dtype=float)
        q_norm = q_vec / (np.linalg.norm(q_vec) + 1e-10)
        norms = np.linalg.norm(corpus, axis=1, keepdims=True) + 1e-10
        scores = (corpus / norms) @ q_norm

        # Get top-k vector results
        k = min(n_results, len(self._texts))
        top_idx = list(np.argsort(scores)[::-1][:k])

        # Step 2: Graph expansion
        # Extract entities from top vector results
        query_entities = _extract_entities(query_text)
        for idx in top_idx:
            query_entities |= _extract_entities(self._texts[idx])

        # Find graph neighbours
        expanded_chunks: set[int] = set()
        for ent in query_entities:
            if ent in self._graph:
                for hop in range(self._graph_hops):
                    neighbours = set()
                    targets = {ent} if hop == 0 else expanded_chunks.copy()
                    for target in targets:
                        if isinstance(target, str) and target in self._graph:
                            neighbours |= set(self._graph.neighbors(target))
                    for neighbour in neighbours:
                        expanded_chunks |= self._entity_to_chunks.get(neighbour, set())

        # Step 3: Blend results (vector first, then graph expansions)
        seen = set(top_idx)
        graph_additions = []
        for chunk_idx in expanded_chunks:
            if chunk_idx not in seen and len(graph_additions) < self._graph_boost:
                graph_additions.append(chunk_idx)
                seen.add(chunk_idx)

        result_indices = top_idx + graph_additions

        results = []
        for idx in result_indices:
            results.append(RetrievedChunk(
                text=self._texts[idx],
                doc_id=self._ids[idx],
                score=float(scores[idx]),
                metadata=self._metadatas[idx],
            ))

        return results[:n_results + self._graph_boost]

    # ── Private helpers ───────────────────────────────────────────────────

    def _embed(self, texts: list[str]) -> list[list[float]]:
        if self._embed_fn:
            return self._embed_fn(texts)
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed: %s", exc)
        # TF-IDF fallback
        return self._tfidf_embed(texts)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"]
                for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(texts: list[str]) -> list[list[float]]:
        import math
        tokenised = [t.lower().split() for t in texts]
        vocab: dict[str, int] = {}
        for tokens in tokenised:
            for w in tokens:
                if w not in vocab:
                    vocab[w] = len(vocab)
        n_vocab = len(vocab) or 1
        vecs: list[list[float]] = []
        for tokens in tokenised:
            tf: dict[str, float] = {}
            for w in tokens:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log((len(texts) + 1) / 1 + 1)
            vecs.append(vec)
        return vecs


path: scripts/milvus_store.py
content: """Milvus vector store — Apache 2.0, embedded (Lite) or Docker server.

Milvus is the most widely adopted open-source vector database, backed by
Zilliz.  It supports multiple index types (FLAT, IVF_FLAT, HNSW, etc.),
hybrid search (dense + sparse), and partitioning for multi-tenant use.

Two deployment modes:
  - **Milvus Lite** (default): embedded, no Docker, local file storage.
    Uses pymilvus + milvus-lite packages.  Great for evaluation & dev.
  - **Milvus Standalone**: full Docker deployment with etcd + MinIO.
    Connects via `uri="http://localhost:19530"`.

For our RAG eval comparison, we use FLAT index (exact search) to isolate
retrieval algorithm differences from approximate-NN trade-offs.  HNSW
mode is available via `index_type="HNSW"` for throughput comparison.

Docker setup (standalone, add to docker-compose.yml if needed):
    milvus-etcd:
        image: quay.io/coreos/etcd:v3.5.5
    milvus-minio:
        image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    milvus-standalone:
        image: milvusdb/milvus:v2.5.1
        ports: ["19530:19530", "9091:9091"]

License: Apache 2.0
Docs:    https://milvus.io/docs
"""
from __future__ import annotations

import logging
import tempfile
import uuid
from pathlib import Path
from typing import Optional

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

_OLLAMA_EMBED_URL   = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"
_VECTOR_SIZE        = 768


class MilvusStore(BaseVectorStore):
    """Milvus-backed vector store (Lite embedded or Docker server).

    Args:
        ollama_url:     Ollama embed API URL.
        openai_api_key: Optional OpenAI fallback key.
        uri:            Milvus connection URI.
            - File path (e.g. "/tmp/milvus.db") → Milvus Lite (embedded).
            - URL (e.g. "http://localhost:19530") → Milvus server.
            - None → creates a temp file for Milvus Lite.
        collection_name: Collection name. UUID-generated if not provided.
        index_type:     "FLAT" (exact, default) or "HNSW" (approximate).
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        uri: Optional[str] = None,
        collection_name: Optional[str] = None,
        index_type: str = "FLAT",
    ) -> None:
        self._ollama_url  = ollama_url
        self._openai_key  = openai_api_key
        self._index_type  = index_type
        self._size        = 0
        self._tmp_dir     = None
        self._client      = None

        # Milvus Lite: use temp file if no URI given
        if uri is None:
            self._tmp_dir = tempfile.mkdtemp(prefix="milvus_rag_")
            self._uri = str(Path(self._tmp_dir) / "milvus.db")
        else:
            self._uri = uri

        # Collection name: alphanumeric + underscore only
        raw = collection_name or f"rag_{uuid.uuid4().hex[:12]}"
        self._collection = raw.replace("-", "_")

    # ── Public API ──────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return self._size

    @property
    def backend_name(self) -> str:
        mode = "lite" if self._uri.endswith(".db") else "server"
        return f"milvus_{mode}"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        client = self._get_client()
        self._ensure_collection(client)

        embeddings = self._embed(texts)
        auto_ids   = ids or [f"doc_{self._size + i}" for i in range(len(texts))]

        data = [
            {
                "id": i,
                "doc_id": doc_id,
                "text": text,
                "embedding": emb,
            }
            for i, (text, doc_id, emb) in enumerate(
                zip(texts, auto_ids, embeddings), start=self._size
            )
        ]

        client.insert(collection_name=self._collection, data=data)
        self._size += len(texts)
        logger.debug(
            "Milvus: inserted %d docs into %s (total=%d, index=%s)",
            len(texts), self._collection, self._size, self._index_type,
        )

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if self._size == 0:
            return []

        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        client = self._get_client()
        k      = min(n_results, self._size)

        results = client.search(
            collection_name=self._collection,
            data=[q_emb[0]],
            limit=k,
            output_fields=["doc_id", "text"],
            anns_field="embedding",
        )

        chunks = []
        if results and len(results) > 0:
            for hit in results[0]:
                # Milvus returns distance; for IP/cosine, higher = more similar
                # For FLAT with COSINE metric, score is cosine similarity directly
                score = float(hit.get("distance", 0.0))
                entity = hit.get("entity", {})
                chunks.append(RetrievedChunk(
                    text   = entity.get("text", ""),
                    doc_id = entity.get("doc_id", ""),
                    score  = score,
                ))
        return chunks

    def cleanup(self) -> None:
        """Drop collection and close connection."""
        if self._client:
            try:
                self._client.drop_collection(self._collection)
                logger.debug("Milvus: dropped collection %s", self._collection)
            except Exception as exc:
                logger.warning("Milvus cleanup failed: %s", exc)
            finally:
                self._client.close()
                self._client = None

        if self._tmp_dir:
            import shutil
            try:
                shutil.rmtree(self._tmp_dir, ignore_errors=True)
            except Exception:
                pass

    # ── Private helpers ──────────────────────────────────────────────────

    def _get_client(self):
        if self._client is None:
            from pymilvus import MilvusClient
            self._client = MilvusClient(uri=self._uri)
        return self._client

    def _ensure_collection(self, client) -> None:
        """Create collection with schema if it doesn't exist."""
        if client.has_collection(self._collection):
            return

        from pymilvus import CollectionSchema, FieldSchema, DataType

        fields = [
            FieldSchema(name="id",        dtype=DataType.INT64,
                        is_primary=True, auto_id=False),
            FieldSchema(name="doc_id",     dtype=DataType.VARCHAR, max_length=256),
            FieldSchema(name="text",       dtype=DataType.VARCHAR, max_length=65535),
            FieldSchema(name="embedding",  dtype=DataType.FLOAT_VECTOR,
                        dim=_VECTOR_SIZE),
        ]
        schema = CollectionSchema(fields=fields, description="RAG eval store")

        # Index params
        if self._index_type == "HNSW":
            index_params = client.prepare_index_params()
            index_params.add_index(
                field_name="embedding",
                index_type="HNSW",
                metric_type="COSINE",
                params={"M": 16, "efConstruction": 200},
            )
        else:
            index_params = client.prepare_index_params()
            index_params.add_index(
                field_name="embedding",
                index_type="FLAT",
                metric_type="COSINE",
            )

        client.create_collection(
            collection_name=self._collection,
            schema=schema,
            index_params=index_params,
        )
        logger.debug(
            "Milvus: created collection %s (dim=%d, index=%s)",
            self._collection, _VECTOR_SIZE, self._index_type,
        )

    def _embed(self, texts: list[str]) -> list[list[float]]:
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed: %s", exc)
        logger.warning("All providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"] for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(texts: list[str]) -> list[list[float]]:
        import math
        tokenised = [t.lower().split() for t in texts]
        vocab: dict[str, int] = {}
        for toks in tokenised:
            for w in toks:
                if w not in vocab:
                    vocab[w] = len(vocab)
        n_vocab = len(vocab) or 1
        vecs = []
        for toks in tokenised:
            tf: dict[str, float] = {}
            for w in toks:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log(2)
            vecs.append(vec)
        return vecs


path: scripts/numpy_store.py
content: """In-process numpy cosine similarity vector store.

Uses ``nomic-embed-text`` via Ollama (local, free, 768-dim) for embeddings.
OpenAI ``text-embedding-3-small`` is an optional fallback (paid, ~$0.02/M).
TF-IDF is the final offline fallback (no external deps, no network).

History
-------
Formerly called ``VoyageNumpyStore`` / ``chroma_client.py`` — both names
were inherited from an earlier design that used Voyage AI embeddings and
ChromaDB.  Both have since been removed:
  - Chroma 1.5.1 is incompatible with Python 3.14 (Pydantic V1 issue).
  - Voyage AI free tier (3 RPM) contaminated RAG eval scores with TF-IDF
    fallback noise.  Replaced by Ollama nomic-embed-text (local, unlimited).

This store is the **in-process baseline**: exact cosine similarity over a
flat numpy matrix.  No server, no Docker, no persistence — ephemeral per
instance.  Use it as the reference to compare approximation quality in
Qdrant, pgvector, Milvus, etc.

Backend string: ``"numpy"``

Typical use::

    store = NumpyCosineStore()   # uses Ollama nomic-embed-text
    store.add_documents(["Paris is the capital of France.", "Berlin is in Germany."])
    chunks = store.query("What is the capital of France?", n_results=2)
    for c in chunks:
        print(c.score, c.text)
"""
from __future__ import annotations

import hashlib
import logging
from typing import Callable, Optional

from .base import BaseVectorStore, RetrievedChunk  # noqa: F401 — re-exported

logger = logging.getLogger(__name__)

# Primary: Ollama nomic-embed-text (local, free, 768-dim)
_OLLAMA_EMBED_URL   = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"

# Optional fallback: OpenAI text-embedding-3-small (paid, 1536-dim)
_OPENAI_EMBED_URL   = "https://api.openai.com/v1/embeddings"
_OPENAI_EMBED_MODEL = "text-embedding-3-small"


class NumpyCosineStore(BaseVectorStore):
    """Ollama embeddings + numpy cosine similarity (in-process, ephemeral).

    The simplest possible vector store: embeddings in a list, similarity via
    numpy dot product.  One instance = one collection.  Exact cosine — no
    approximation error, no index build time, no external dependencies beyond
    numpy.

    Backend string: ``"numpy"``

    Args:
        ollama_url:     Ollama embed API URL (default: http://localhost:11434/api/embed).
                        Set to None/empty to skip Ollama and fall through to
                        openai_api_key or TF-IDF.
        openai_api_key: Optional OpenAI key for text-embedding-3-small fallback.
                        Only used if Ollama is unavailable.
        embed_fn:       Test injection point. Pass a deterministic
                        ``(texts: list[str]) -> list[list[float]]`` function
                        to bypass all network calls in unit tests.
        voyage_api_key: Ignored. Kept so old callers don't crash.
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        embed_fn: Optional[Callable[[list[str]], list[list[float]]]] = None,
        voyage_api_key: Optional[str] = None,   # deprecated — ignored
    ) -> None:
        self._ollama_url   = ollama_url
        self._openai_key   = openai_api_key
        self._embed_fn     = embed_fn
        self._texts:      list[str]         = []
        self._ids:        list[str]         = []
        self._metadatas:  list[dict]        = []
        self._embeddings: list[list[float]] = []
        self._tfidf_vocab: dict[str, int]   = {}

        if voyage_api_key:
            logger.warning(
                "voyage_api_key is deprecated and ignored — "
                "NumpyCosineStore now uses Ollama nomic-embed-text."
            )

    # ── Public API ────────────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return len(self._texts)

    @property
    def backend_name(self) -> str:
        return "numpy"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        auto_ids  = ids or [hashlib.md5(t.encode()).hexdigest()[:8] for t in texts]
        auto_meta = metadatas or [{} for _ in texts]

        if not self._embed_fn and not self._ollama_url and not self._openai_key:
            self._build_tfidf_vocab(texts)

        embeddings = self._embed(texts)
        self._texts.extend(texts)
        self._ids.extend(auto_ids)
        self._metadatas.extend(auto_meta)
        self._embeddings.extend(embeddings)

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if not self._embeddings:
            return []
        try:
            import numpy as np
        except ImportError:
            logger.error("numpy required for NumpyCosineStore — install numpy")
            return []

        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        q_vec  = np.array(q_emb[0], dtype=float)
        corpus = np.array(self._embeddings, dtype=float)

        q_norm     = q_vec / (np.linalg.norm(q_vec) + 1e-10)
        norms      = np.linalg.norm(corpus, axis=1, keepdims=True) + 1e-10
        scores     = (corpus / norms) @ q_norm

        k       = min(n_results, len(self._texts))
        top_idx = np.argsort(scores)[::-1][:k]

        return [
            RetrievedChunk(
                text     = self._texts[i],
                doc_id   = self._ids[i],
                score    = float(scores[i]),
                metadata = self._metadatas[i],
            )
            for i in top_idx
        ]

    # ── Private helpers ───────────────────────────────────────────────────────

    def _embed(self, texts: list[str]) -> list[list[float]]:
        """Priority: embed_fn (tests) → Ollama → OpenAI → TF-IDF."""
        if self._embed_fn:
            return self._embed_fn(texts)
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed, trying OpenAI: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed, using TF-IDF: %s", exc)
        logger.warning("All embedding providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts, vocab=self._tfidf_vocab or None)

    def _build_tfidf_vocab(self, texts: list[str]) -> None:
        for text in texts:
            for w in text.lower().split():
                if w not in self._tfidf_vocab:
                    self._tfidf_vocab[w] = len(self._tfidf_vocab)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            _OPENAI_EMBED_URL,
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": _OPENAI_EMBED_MODEL},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"]
                for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(
        texts: list[str],
        vocab: dict[str, int] | None = None,
    ) -> list[list[float]]:
        """Minimal TF-IDF bag-of-words fallback — no external deps."""
        import math
        tokenised = [t.lower().split() for t in texts]
        if vocab is None:
            vocab = {}
            for tokens in tokenised:
                for w in tokens:
                    if w not in vocab:
                        vocab[w] = len(vocab)
        n_docs  = max(len(texts), 1)
        n_vocab = len(vocab) or 1
        vecs: list[list[float]] = []
        for tokens in tokenised:
            tf: dict[str, float] = {}
            for w in tokens:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log((n_docs + 1) / 1 + 1)
            vecs.append(vec)
        return vecs


# ── Backward-compatibility aliases ────────────────────────────────────────────
# Code that still imports VoyageNumpyStore or VectorStore will keep working.
VoyageNumpyStore = NumpyCosineStore   # deprecated name — use NumpyCosineStore
VectorStore      = NumpyCosineStore   # legacy alias


path: scripts/pgvector_store.py
content: """pgvector vector store — PostgreSQL + pgvector extension.

Uses the pgvector extension for PostgreSQL to store and query
768-dim embeddings with exact nearest-neighbour cosine search.

Key properties:
  - ACID-compliant storage (full PostgreSQL durability)
  - Exact cosine similarity via <=> operator (no approximation)
  - Hybrid search possible: combine SQL WHERE clauses with vector search
  - Persistent across runs (unlike VoyageNumpyStore / LanceDB ephemeral mode)
  - Each eval instance uses an isolated table (UUID-named) then drops it on cleanup

Docker setup (add to docker-compose.yml):
    pgvector:
        image: pgvector/pgvector:pg16
        ports: ["5432:5432"]
        environment:
            POSTGRES_USER: rag
            POSTGRES_PASSWORD: rag
            POSTGRES_DB: rag
        volumes:
            - pgvector_data:/var/lib/postgresql/data

Python deps: psycopg2-binary pgvector

License: MIT (pgvector), PostgreSQL License (PostgreSQL)
Docs:    https://github.com/pgvector/pgvector
"""
from __future__ import annotations

import logging
import uuid
from typing import Optional

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

_OLLAMA_EMBED_URL   = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"
_VECTOR_SIZE        = 768

_DEFAULT_DSN = "postgresql://rag:rag@localhost:5432/rag"


class PgvectorStore(BaseVectorStore):
    """PostgreSQL + pgvector exact cosine similarity store.

    Each instance creates an isolated temporary table in PostgreSQL.
    The table is dropped on cleanup() — no cross-run contamination.

    Args:
        ollama_url:     Ollama embed API URL.
        openai_api_key: Optional OpenAI fallback key.
        dsn:            PostgreSQL connection string.
                        Default: postgresql://rag:rag@localhost:5432/rag
        table_name:     Table name. UUID-generated if not provided.
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        dsn: str = _DEFAULT_DSN,
        table_name: Optional[str] = None,
    ) -> None:
        self._ollama_url = ollama_url
        self._openai_key = openai_api_key
        self._dsn        = dsn
        # Safe table name — UUIDs have hyphens, replace with underscores
        raw = table_name or f"rag_{uuid.uuid4().hex[:12]}"
        self._table_name = raw.replace("-", "_")
        self._conn       = None
        self._size       = 0
        self._created    = False

    # ── Public API ──────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return self._size

    @property
    def backend_name(self) -> str:
        return "pgvector"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        conn = self._get_conn()
        if not self._created:
            self._create_table(conn)

        embeddings = self._embed(texts)
        auto_ids   = ids or [f"doc_{self._size + i}" for i in range(len(texts))]

        with conn.cursor() as cur:
            for text, doc_id, emb in zip(texts, auto_ids, embeddings):
                # Explicit ::vector cast — no register_vector adapter needed
                vec_str = "[" + ",".join(str(x) for x in emb) + "]"
                cur.execute(
                    f"INSERT INTO {self._table_name} (doc_id, text, embedding)"
                    f" VALUES (%s, %s, %s::vector)",
                    (doc_id, text, vec_str),
                )
        conn.commit()
        self._size += len(texts)
        logger.debug(
            "pgvector: inserted %d docs into %s (total=%d)",
            len(texts), self._table_name, self._size,
        )

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if self._size == 0:
            return []

        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        k   = min(n_results, self._size)
        vec = q_emb[0]

        conn = self._get_conn()
        vec_str = "[" + ",".join(str(x) for x in vec) + "]"
        with conn.cursor() as cur:
            # <=> is cosine distance in pgvector; 1 - distance = similarity
            # Explicit ::vector casts — no register_vector adapter needed.
            cur.execute(
                f"""
                SELECT doc_id, text, 1 - (embedding <=> %s::vector) AS similarity
                FROM {self._table_name}
                ORDER BY embedding <=> %s::vector
                LIMIT %s
                """,
                (vec_str, vec_str, k),
            )
            rows = cur.fetchall()

        return [
            RetrievedChunk(
                text   = row[1],
                doc_id = row[0],
                score  = float(row[2]),
            )
            for row in rows
        ]

    def cleanup(self) -> None:
        """Drop the isolated table and close the connection."""
        if self._conn and self._created:
            try:
                with self._conn.cursor() as cur:
                    cur.execute(f"DROP TABLE IF EXISTS {self._table_name}")
                self._conn.commit()
                logger.debug("pgvector: dropped table %s", self._table_name)
            except Exception as exc:
                logger.warning("pgvector cleanup failed: %s", exc)
            finally:
                self._conn.close()
                self._conn = None

    # ── Private helpers ──────────────────────────────────────────────────

    def _get_conn(self):
        if self._conn is None:
            import psycopg2
            self._conn = psycopg2.connect(self._dsn)
            # NOTE: we intentionally do NOT call register_vector() here.
            # register_vector() runs a query (SELECT oid FROM pg_type WHERE
            # typname='vector') that opens an implicit transaction BEFORE the
            # extension is created, which then blocks the autocommit=True switch
            # needed by _create_table.  Instead we use explicit ::vector casts
            # in all SQL strings — no psycopg2 adapter registration required.
        return self._conn

    def _create_table(self, conn) -> None:
        # CREATE EXTENSION must be committed in its own transaction
        # before the vector type can be used in a subsequent DDL statement.
        old_autocommit = conn.autocommit
        conn.autocommit = True
        with conn.cursor() as cur:
            cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
        conn.autocommit = old_autocommit

        with conn.cursor() as cur:
            cur.execute(
                f"""
                CREATE TABLE IF NOT EXISTS {self._table_name} (
                    id        SERIAL PRIMARY KEY,
                    doc_id    TEXT NOT NULL,
                    text      TEXT NOT NULL,
                    embedding vector({_VECTOR_SIZE})
                )
                """
            )
        conn.commit()
        self._created = True
        logger.debug("pgvector: created table %s (dim=%d)", self._table_name, _VECTOR_SIZE)

    def _embed(self, texts: list[str]) -> list[list[float]]:
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed: %s", exc)
        logger.warning("All providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"] for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(texts: list[str]) -> list[list[float]]:
        import math
        tokenised = [t.lower().split() for t in texts]
        vocab: dict[str, int] = {}
        for toks in tokenised:
            for w in toks:
                if w not in vocab:
                    vocab[w] = len(vocab)
        n_vocab = len(vocab) or 1
        vecs = []
        for toks in tokenised:
            tf: dict[str, float] = {}
            for w in toks:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log(2)
            vecs.append(vec)
        return vecs


path: scripts/qdrant_store.py
content: """Qdrant-backed vector store for comparative RAG retrieval evaluation.

Uses the same ``nomic-embed-text`` embeddings (via Ollama) as ``VoyageNumpyStore``
so retrieval algorithm differences are isolated — both backends operate on
identical 768-dim embedding vectors, different indexing/search strategies.

VoyageNumpyStore: brute-force cosine similarity over a numpy matrix.
QdrantStore:      Qdrant HNSW (Hierarchical Navigable Small World) graph index.

At small corpus sizes (< 1,000 docs) quality differences are minimal.
At larger corpora HNSW should show better recall and sub-linear query time.
That gap is exactly what we're measuring in the comparative evaluation.

Connection modes:
  - Docker (production): QdrantStore(url="http://localhost:6333")
  - In-memory (tests):   QdrantStore(in_memory=True)

Voyage AI removed: 3 RPM free tier contaminated eval scores.  nomic-embed-text
is local, free, and has no rate limits.  Existing Qdrant collections (512-dim)
must be deleted and recreated at 768-dim on first use.

Typical use::

    store = QdrantStore(url="http://localhost:6333")
    store.add_documents(["Paris is the capital of France.", ...])
    chunks = store.query("What is the capital of France?", n_results=2)
"""
from __future__ import annotations

import hashlib
import logging
import uuid
from typing import Optional

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

# Primary: Ollama nomic-embed-text (local, free, 768-dim)
_OLLAMA_EMBED_URL = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"

# Optional fallback: OpenAI text-embedding-3-small (1536-dim)
_OPENAI_EMBED_URL = "https://api.openai.com/v1/embeddings"
_OPENAI_EMBED_MODEL = "text-embedding-3-small"

_DEFAULT_QDRANT_URL = "http://localhost:6333"
_VECTOR_SIZE = 768   # nomic-embed-text output dimension


class QdrantStore(BaseVectorStore):
    """Qdrant HNSW vector store for comparative RAG retrieval.

    Args:
        ollama_url:     Ollama base URL (default: http://localhost:11434/api/embed).
                        Set to None or empty to skip and fall through to OpenAI/TF-IDF.
        openai_api_key: Optional OpenAI key for text-embedding-3-small fallback.
        url:            Qdrant server URL (Docker). Ignored when in_memory=True.
        in_memory:      Use Qdrant's in-process in-memory mode (no server needed).
        collection:     Collection name. Auto-generated UUID if not provided.
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        url: str = _DEFAULT_QDRANT_URL,
        in_memory: bool = False,
        collection: Optional[str] = None,
        # Legacy parameter kept for backward compatibility — ignored
        voyage_api_key: Optional[str] = None,
    ) -> None:
        self._ollama_url = ollama_url
        self._openai_key = openai_api_key
        self._url = url
        self._in_memory = in_memory
        self._collection = collection or f"rag-{uuid.uuid4().hex[:12]}"
        self._client = None
        self._size = 0
        self._texts: list[str] = []
        self._ids: list[str] = []
        self._metadatas: list[dict] = []
        self._tfidf_vocab: dict[str, int] = {}

        if voyage_api_key:
            logger.warning(
                "voyage_api_key passed to QdrantStore — Voyage removed; "
                "using Ollama nomic-embed-text instead."
            )

    # ── Public API ────────────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return self._size

    @property
    def backend_name(self) -> str:
        return "qdrant"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        from qdrant_client import QdrantClient
        from qdrant_client.models import Distance, VectorParams, PointStruct

        auto_ids = ids or [hashlib.md5(t.encode()).hexdigest()[:8] for t in texts]
        auto_meta = metadatas or [{} for _ in texts]

        # Build TF-IDF vocab only if no embedding provider available
        if not self._ollama_url and not self._openai_key:
            for text in texts:
                for w in text.lower().split():
                    if w not in self._tfidf_vocab:
                        self._tfidf_vocab[w] = len(self._tfidf_vocab)

        embeddings = self._embed(texts)
        vector_size = len(embeddings[0]) if embeddings else _VECTOR_SIZE

        client = self._get_client()

        # Create collection if not already created
        existing = [c.name for c in client.get_collections().collections]
        if self._collection not in existing:
            client.create_collection(
                collection_name=self._collection,
                vectors_config=VectorParams(
                    size=vector_size,
                    distance=Distance.COSINE,
                ),
            )
            logger.debug("Created Qdrant collection: %s (dim=%d)", self._collection, vector_size)

        # Upsert points
        points = []
        for i, (text, doc_id, meta, emb) in enumerate(
            zip(texts, auto_ids, auto_meta, embeddings)
        ):
            points.append(PointStruct(
                id=self._size + i,       # sequential integer IDs
                vector=emb,
                payload={"text": text, "doc_id": doc_id, **meta},
            ))

        client.upsert(collection_name=self._collection, points=points)

        self._texts.extend(texts)
        self._ids.extend(auto_ids)
        self._metadatas.extend(auto_meta)
        self._size += len(texts)
        logger.debug("Upserted %d points into %s", len(points), self._collection)

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if self._size == 0:
            return []

        q_emb = self._embed([query_text])
        if not q_emb:
            return []

        client = self._get_client()
        k = min(n_results, self._size)

        # qdrant-client v1.x: query_points() replaces the old search()
        response = client.query_points(
            collection_name=self._collection,
            query=q_emb[0],
            limit=k,
            with_payload=True,
        )

        chunks = []
        for hit in response.points:
            payload = hit.payload or {}
            chunks.append(RetrievedChunk(
                text=payload.get("text", ""),
                doc_id=payload.get("doc_id", str(hit.id)),
                score=float(hit.score),
                metadata={k: v for k, v in payload.items() if k not in ("text", "doc_id")},
            ))
        return chunks

    def cleanup(self) -> None:
        """Delete the collection from Qdrant (call when done with this store)."""
        try:
            if self._client is not None:
                self._client.delete_collection(self._collection)
                logger.debug("Deleted Qdrant collection: %s", self._collection)
        except Exception as exc:
            logger.warning("Qdrant cleanup failed for %s: %s", self._collection, exc)

    # ── Private helpers ───────────────────────────────────────────────────────

    def _get_client(self):
        """Return a cached Qdrant client (lazy init)."""
        if self._client is None:
            from qdrant_client import QdrantClient
            if self._in_memory:
                self._client = QdrantClient(":memory:")
            else:
                self._client = QdrantClient(url=self._url)
        return self._client

    def _embed(self, texts: list[str]) -> list[list[float]]:
        """Embed texts. Priority: Ollama → OpenAI → TF-IDF."""
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed, trying OpenAI fallback: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed, using TF-IDF fallback: %s", exc)
        logger.warning("All embedding providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts, vocab=self._tfidf_vocab or None)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        """Embed via Ollama nomic-embed-text (local, free, 768-dim)."""
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        """Embed via OpenAI text-embedding-3-small (paid fallback, 1536-dim)."""
        import requests
        resp = requests.post(
            _OPENAI_EMBED_URL,
            headers={
                "Authorization": f"Bearer {self._openai_key}",
                "Content-Type": "application/json",
            },
            json={"input": texts, "model": _OPENAI_EMBED_MODEL},
            timeout=30,
        )
        resp.raise_for_status()
        items = sorted(resp.json()["data"], key=lambda x: x["index"])
        return [item["embedding"] for item in items]

    @staticmethod
    def _tfidf_embed(
        texts: list[str],
        vocab: dict[str, int] | None = None,
    ) -> list[list[float]]:
        """Minimal TF-IDF bag-of-words fallback — no external deps."""
        import math
        tokenised = [t.lower().split() for t in texts]
        if vocab is None:
            vocab = {}
            for tokens in tokenised:
                for w in tokens:
                    if w not in vocab:
                        vocab[w] = len(vocab)
        n_docs = max(len(texts), 1)
        n_vocab = len(vocab) or 1
        vecs: list[list[float]] = []
        for tokens in tokenised:
            tf: dict[str, float] = {}
            for w in tokens:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    idf = math.log((n_docs + 1) / 1 + 1)
                    vec[vocab[w]] = (count / max_tf) * idf
            vecs.append(vec)
        return vecs


path: scripts/weaviate_store.py
content: """Weaviate vector store — BSD 3-Clause, Docker self-hosted.

Weaviate is a purpose-built vector database with built-in BM25 hybrid
search (sparse + dense). Unlike pure-vector stores, it can combine
keyword relevance with embedding similarity — interesting for RAG tasks
where exact keyword matches matter (model names, version numbers, etc.).

Key differences from other backends:
  - Built-in BM25 hybrid search (alpha parameter blends sparse + dense)
  - GraphQL + REST API, persistent with Docker volume
  - Module system: text2vec-ollama module for native embedding support
  - Each eval instance creates an isolated collection (UUID-named)

Docker setup (add to docker-compose.yml):
    weaviate:
        image: semitechnologies/weaviate:1.27.0
        ports: ["8082:8080", "50051:50051"]
        environment:
            QUERY_DEFAULTS_LIMIT: 25
            AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true"
            PERSISTENCE_DATA_PATH: /var/lib/weaviate
            DEFAULT_VECTORIZER_MODULE: none
            ENABLE_MODULES: ""
            CLUSTER_HOSTNAME: node1
        volumes:
            - weaviate_data:/var/lib/weaviate

Note: we use BYOV (Bring Your Own Vector) mode — embeddings are
generated externally via Ollama and passed to Weaviate. This keeps
the same nomic-embed-text embeddings across all backends.

License: BSD 3-Clause
Docs:    https://weaviate.io/developers/weaviate
"""
from __future__ import annotations

import logging
import uuid
from typing import Optional

from .base import BaseVectorStore, RetrievedChunk

logger = logging.getLogger(__name__)

_OLLAMA_EMBED_URL   = "http://localhost:11434/api/embed"
_OLLAMA_EMBED_MODEL = "nomic-embed-text"
_VECTOR_SIZE        = 768

_DEFAULT_WEAVIATE_URL = "http://localhost:8082"


class WeaviateStore(BaseVectorStore):
    """Weaviate-backed vector store with optional BM25 hybrid search.

    Two query modes:
      - vector:  pure dense cosine similarity (default, matches other backends)
      - hybrid:  BM25 + cosine blend (alpha=0.5 by default)

    Args:
        ollama_url:     Ollama embed API URL.
        openai_api_key: Optional OpenAI fallback key.
        weaviate_url:   Weaviate server URL (default: http://localhost:8082)
        collection_name: Collection name. UUID-generated if not provided.
        query_mode:     "vector" (default) or "hybrid".
        hybrid_alpha:   Blend factor for hybrid mode. 1.0 = pure vector,
                        0.0 = pure BM25. Default 0.5.
    """

    def __init__(
        self,
        ollama_url: Optional[str] = _OLLAMA_EMBED_URL,
        openai_api_key: Optional[str] = None,
        weaviate_url: str = _DEFAULT_WEAVIATE_URL,
        collection_name: Optional[str] = None,
        query_mode: str = "vector",
        hybrid_alpha: float = 0.1,  # Sprint 16 alpha sweep: 0.1 optimal (avg 0.9940 vs 0.5 at 0.9700)
    ) -> None:
        self._ollama_url     = ollama_url
        self._openai_key     = openai_api_key
        self._weaviate_url   = weaviate_url
        # Weaviate class names must start with uppercase letter
        raw = collection_name or f"Rag{uuid.uuid4().hex[:10]}"
        self._collection     = raw[0].upper() + raw[1:]
        self._query_mode     = query_mode
        self._hybrid_alpha   = hybrid_alpha
        self._client         = None
        self._size           = 0

    # ── Public API ──────────────────────────────────────────────────────

    @property
    def size(self) -> int:
        return self._size

    @property
    def backend_name(self) -> str:
        mode = f"_{self._query_mode}" if self._query_mode == "hybrid" else ""
        return f"weaviate{mode}"

    def add_documents(
        self,
        texts: list[str],
        ids: Optional[list[str]] = None,
        metadatas: Optional[list[dict]] = None,
    ) -> None:
        if not texts:
            return

        client = self._get_client()
        self._ensure_collection(client)

        embeddings = self._embed(texts)
        auto_ids   = ids or [f"doc_{self._size + i}" for i in range(len(texts))]
        collection = client.collections.get(self._collection)

        from weaviate.classes.data import DataObject
        objects = []
        for text, doc_id, emb in zip(texts, auto_ids, embeddings):
            objects.append(DataObject(
                properties={"text": text, "doc_id": doc_id},
                vector=emb,
            ))

        collection.data.insert_many(objects)
        self._size += len(texts)
        logger.debug(
            "Weaviate: inserted %d docs into %s (total=%d, mode=%s)",
            len(texts), self._collection, self._size, self._query_mode,
        )

    def query(self, query_text: str, n_results: int = 3) -> list[RetrievedChunk]:
        if self._size == 0:
            return []

        client  = self._get_client()
        k       = min(n_results, self._size)
        coll    = client.collections.get(self._collection)

        from weaviate.classes.query import MetadataQuery

        if self._query_mode == "hybrid":
            q_emb = self._embed([query_text])
            if not q_emb:
                return []
            response = coll.query.hybrid(
                query=query_text,
                vector=q_emb[0],
                alpha=self._hybrid_alpha,
                limit=k,
                return_metadata=MetadataQuery(score=True),
            )
        else:
            # Pure vector
            q_emb = self._embed([query_text])
            if not q_emb:
                return []
            response = coll.query.near_vector(
                near_vector=q_emb[0],
                limit=k,
                return_metadata=MetadataQuery(distance=True),
            )

        chunks = []
        for obj in response.objects:
            props = obj.properties
            if self._query_mode == "hybrid":
                score = float(obj.metadata.score or 0.0)
            else:
                dist  = float(obj.metadata.distance or 0.0)
                score = max(0.0, 1.0 - dist)
            chunks.append(RetrievedChunk(
                text   = props.get("text", ""),
                doc_id = props.get("doc_id", ""),
                score  = score,
            ))
        return chunks

    def cleanup(self) -> None:
        """Delete the isolated collection from Weaviate."""
        if self._client:
            try:
                self._client.collections.delete(self._collection)
                logger.debug("Weaviate: deleted collection %s", self._collection)
            except Exception as exc:
                logger.warning("Weaviate cleanup failed: %s", exc)
            finally:
                self._client.close()
                self._client = None

    # ── Private helpers ──────────────────────────────────────────────────

    def _get_client(self):
        if self._client is None:
            import weaviate
            self._client = weaviate.connect_to_local(
                host=self._weaviate_url.replace("http://", "").split(":")[0],
                port=int(self._weaviate_url.rsplit(":", 1)[-1]) if ":" in self._weaviate_url.rsplit("/", 1)[-1] else 8082,
            )
        return self._client

    def _ensure_collection(self, client) -> None:
        """Create collection if it doesn't exist.

        We use BYOV (Bring Your Own Vector) mode — embeddings are generated
        externally via Ollama and passed directly.  The weaviate-client v4
        deprecated `vectorizer_config` in favour of `vector_config` with named
        vectors, but `Configure.Vectors.none()` doesn't exist in v4.20.1.
        `Configure.Vectorizer.none()` still works and triggers only a
        DeprecationWarning, which we suppress here until the API stabilises.
        """
        import warnings
        from weaviate.classes.config import Configure, Property, DataType
        if not client.collections.exists(self._collection):
            with warnings.catch_warnings():
                warnings.simplefilter("ignore", DeprecationWarning)
                client.collections.create(
                    name=self._collection,
                    vectorizer_config=Configure.Vectorizer.none(),  # BYOV
                    properties=[
                        Property(name="text",   data_type=DataType.TEXT),
                        Property(name="doc_id", data_type=DataType.TEXT),
                    ],
                )
            logger.debug("Weaviate: created collection %s", self._collection)

    def _embed(self, texts: list[str]) -> list[list[float]]:
        if self._ollama_url:
            try:
                return self._ollama_embed(texts)
            except Exception as exc:
                logger.warning("Ollama embed failed: %s", exc)
        if self._openai_key:
            try:
                return self._openai_embed(texts)
            except Exception as exc:
                logger.warning("OpenAI embed failed: %s", exc)
        logger.warning("All providers failed — using TF-IDF fallback")
        return self._tfidf_embed(texts)

    def _ollama_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            self._ollama_url,
            json={"model": _OLLAMA_EMBED_MODEL, "input": texts},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json()["embeddings"]

    def _openai_embed(self, texts: list[str]) -> list[list[float]]:
        import requests
        resp = requests.post(
            "https://api.openai.com/v1/embeddings",
            headers={"Authorization": f"Bearer {self._openai_key}"},
            json={"input": texts, "model": "text-embedding-3-small"},
            timeout=30,
        )
        resp.raise_for_status()
        return [item["embedding"] for item in sorted(resp.json()["data"], key=lambda x: x["index"])]

    @staticmethod
    def _tfidf_embed(texts: list[str]) -> list[list[float]]:
        import math
        tokenised = [t.lower().split() for t in texts]
        vocab: dict[str, int] = {}
        for toks in tokenised:
            for w in toks:
                if w not in vocab:
                    vocab[w] = len(vocab)
        n_vocab = len(vocab) or 1
        vecs = []
        for toks in tokenised:
            tf: dict[str, float] = {}
            for w in toks:
                tf[w] = tf.get(w, 0) + 1
            max_tf = max(tf.values(), default=1)
            vec = [0.0] * n_vocab
            for w, count in tf.items():
                if w in vocab:
                    vec[vocab[w]] = (count / max_tf) * math.log(2)
            vecs.append(vec)
        return vecs


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)