npm package: 0g-vector-search
Description: Unified Vector Search service for 0G Storage — decentralized HNSW index on 0G's sharded KV layer
Author: Sage-senpai / Dvyne
Latest version: 0.1.0

# ZeroG VectorSearch

> Unified Vector Search on 0G Storage — a decentralized HNSW index built on 0G's sharded KV layer.

Transforms 0G Labs from a Data Repository into a Reasoning Engine by implementing approximate nearest neighbor search directly on 0G's modular storage stack.

## Features

- **Decentralized HNSW Index** — O(log N) vector search with multi-layer graph traversal
- **Deterministic Sharding** — FNV-1a consistent hashing maps vectors to 0G shards
- **Hybrid Search** — BM25 keyword + semantic vector fusion (0gmem pattern)
- **0G KV Persistence** — Hex-encoded key structure for native KV layer storage
- **Security Layer** — PoRA verification, data availability guard, alignment verification
- **Drop-in SDK** — Feels like a native extension of `@0glabs/0g-ts-sdk`

## Quick Start

```bash
npm install 0g-vector-search @0glabs/0g-ts-sdk ethers
```

```typescript
import { VectorStore } from "0g-vector-search";

// Local development (no 0G network required)
const store = new VectorStore({ dimensions: 768 });
await store.initialize();

// Insert vectors
await store.upsert(
  "doc_001",
  embedding,                          // Float32Array[768]
  { source: "arxiv", topic: "ml" },   // metadata
  "Attention is all you need..."      // text for keyword search
);

// Semantic search
const results = await store.search(queryEmbedding, { topK: 10 });

// Hybrid search (recommended)
const results = await store.search(queryEmbedding, {
  topK: 10,
  hybrid: {
    enabled: true,
    semanticWeight: 0.7,
    query: "transformer architecture",
  },
});
```

### With 0G Network

```typescript
const store = new VectorStore({
  dimensions: 768,
  zeroG: {
    endpoint: "http://3.101.147.150:6789",
    streamId: "your-stream-id",
    privateKey: "0x...",
    flowContract: "0x...",
    kvNodeCount: 1,
  },
});
```

## Architecture

```
┌─────────────────────────────────────────────────┐
│                  VectorStore                     │
│  ┌──────────┐  ┌──────────┐  ┌───────────────┐ │
│  │  HNSW    │  │  BM25    │  │   Security    │ │
│  │  Index   │  │  Index   │  │   Layer       │ │
│  └────┬─────┘  └────┬─────┘  └───────┬───────┘ │
│       │              │               │          │
│  ┌────┴──────────────┴───────────────┴───────┐ │
│  │          Shard Router (FNV-1a)             │ │
│  └────────────────┬──────────────────────────┘ │
│                   │                             │
│  ┌────────────────┴──────────────────────────┐ │
│  │     0G KV Layer (via Batcher / KvClient)  │ │
│  └───────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘
```

### KV Key Structure

All data is stored in 0G's KV layer with hex-encoded keys:

| Key Pattern | Description |
|---|---|
| `vec:<shard>:<node>` | Raw vector data (Float32 buffer) |
| `idx:<shard>:<node>:<layer>` | HNSW graph connections per layer |
| `meta:<shard>:<node>` | Vector metadata (JSON) |
| `bm25:<shard>:<node>` | BM25 term frequencies |
| `cfg:global:index_state` | Index config, entry point, max level |

### Vector Upsert Flow

1. `ShardRouter.getShardId(id)` → FNV-1a hash determines target shard
2. `HNSWIndex.insert(id, vector)` → builds graph links across layers
3. `BM25Index.addDocument(id, content)` → indexes text for keyword search
4. `PoRAVerifier.generateCommitment()` → creates proof for 0G validators
5. `GraphPersistence.save*()` → writes to 0G KV via Batcher

### Vector Search Flow

1. `DataAvailabilityGuard` checks shard health
2. `HybridSearchEngine.search()` → HNSW + BM25 score fusion
3. `AlignmentVerifier.verifyResults()` → checks for injection attacks
4. Results returned sorted by fused score

## Configuration

| Parameter | Default | Description |
|---|---|---|
| `dimensions` | 768 | Vector dimensionality |
| `M` | 16 | Max connections per HNSW layer |
| `efConstruction` | 200 | Candidate list size during insertion |
| `efSearch` | 50 | Candidate list size during search |
| `metric` | cosine | Distance metric (cosine/euclidean/dot_product) |
| `numShards` | 4 | Number of 0G storage shards |

## Development

```bash
npm install        # Install dependencies
npm test           # Run test suite (21 tests)
npm run build      # TypeScript compilation
```

## Security

- **PoRA Compatibility**: Vector updates generate proof commitments for 0G validator verification
- **Data Availability**: Shard health monitoring with fallback routing for offline shards
- **Alignment Verification**: Detects suspicious result clustering indicating injection attacks

## License

MIT