npm package: @solixdb/core
Description: Core indexing engine for SolixDB
Latest version: 1.0.0

# @solixdb/core

Production-grade Solana blockchain indexer core engine.

## Features

**Multiple Indexing Modes**

- **Polling Mode**: Fetch historical blocks via RPC
- **Streaming Mode**: Real-time updates via WebSocket
- **Hybrid Mode**: Historical + live data (coming soon)

**High Performance**

- Adaptive batch sizing
- Concurrent block processing
- Efficient buffering and batching

**Flexible Querying**

- Fluent query builder API
- Filter by slot, time, program, account
- Pagination and streaming support

**Production Ready**

- Comprehensive error handling
- Automatic retry with exponential backoff
- Health checks and monitoring
- Detailed logging

## Installation

```bash
pnpm add @solixdb/core
```

## Quick Start

```typescript
import { SolanaIndexer } from "@solixdb/core";
import { JSONStorage } from "@solixdb/storage-adapters";
import { DefaultProvider } from "@solixdb/rpc-providers";

// 1. Create storage adapter
const storage = new JSONStorage({
  directory: "./data",
});

// 2. Create RPC provider
const rpc = new DefaultProvider({
  endpoint: "https://api.mainnet-beta.solana.com",
});

// 3. Create and configure indexer
const indexer = new SolanaIndexer(
  {
    mode: "polling",
    startSlot: 100000,
    endSlot: 100100,
    batchSize: 10,
    commitment: "confirmed",
  },
  storage,
  rpc
);

// 4. Start indexing
await indexer.start();

// 5. Query indexed data
const blocks = await indexer.query.getRecentBlocks(10);
console.log("Recent blocks:", blocks);

// 6. Stop when done
await indexer.stop();
```

## Configuration

```typescript
interface IndexerConfig {
  // Indexing mode
  mode: "polling" | "streaming" | "hybrid";

  // Slot range (optional)
  startSlot?: number;
  endSlot?: number;

  // Performance tuning
  batchSize?: number; // Default: 10
  pollingInterval?: number; // Default: 1000ms
  commitment?: "processed" | "confirmed" | "finalized";

  // Retry configuration
  maxRetries?: number; // Default: 3
  retryDelay?: number; // Default: 1000ms

  // Filtering
  programIds?: string[]; // Only index these programs
  accountFilters?: AccountFilter[]; // Custom account filters

  // Logging
  logLevel?: "debug" | "info" | "warn" | "error";
}
```

## Usage Examples

### Historical Indexing

Index a specific range of slots:

```typescript
const indexer = SolanaIndexer.forHistoricalData(
  100000, // start slot
  200000, // end slot
  storage,
  rpc,
  {
    batchSize: 20,
    logLevel: "info",
  }
);

await indexer.start();

// Wait for completion
await indexer.waitForCompletion();
```

### Live Indexing

Stream real-time blockchain data:

```typescript
const indexer = SolanaIndexer.forLiveData(storage, rpc, {
  mode: "streaming",
  logLevel: "info",
});

await indexer.start();

// Runs continuously, call stop() when done
```

### Program-Specific Indexing

Index only specific programs:

```typescript
const indexer = new SolanaIndexer(
  {
    mode: "polling",
    startSlot: 100000,
    programIds: [
      "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA", // SPL Token
      "JUP6LkbZbjS1jKKwapdHNy74zcZ3tLUZoi5QNyVTaV4", // Jupiter
    ],
  },
  storage,
  rpc
);

await indexer.start();
```

### Resume from Last Indexed Slot

Automatically resume from where you left off:

```typescript
const indexer = SolanaIndexer.fromLatest(storage, rpc);

// Starts from the last indexed slot in storage
await indexer.start();
```

## Querying Data

### Using Query Builder

```typescript
import { QueryBuilder } from "@solixdb/core";

// Get recent blocks
const query = QueryBuilder.create().orderBySlot("desc").limit(10).build();

const result = await indexer.query.queryBlocks(query);
```

### Quick Queries

```typescript
// Recent blocks
const blocks = await indexer.query.getRecentBlocks(10);

// Blocks in range
const rangeBlocks = await indexer.query.getBlocksInRange(100, 200);

// Transactions for a program
const txs = await indexer.query.getTransactionsForProgram(
  "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"
);

// Transactions for an account
const accountTxs = await indexer.query.getTransactionsForAccount(
  "YourAccountPublicKey"
);

// Single transaction
const tx = await indexer.query.getTransaction("signature...");

// Single block
const block = await indexer.query.getBlock(123456);
```

### Streaming Large Result Sets

```typescript
// Stream blocks (for large datasets)
for await (const block of indexer.query.streamBlocks(query)) {
  console.log("Block:", block.slot);
  // Process block...
}

// Stream transactions
for await (const tx of indexer.query.streamTransactions(query)) {
  console.log("Transaction:", tx.signature);
  // Process transaction...
}
```

## Monitoring

### Get Statistics

```typescript
const stats = indexer.getStats();
console.log({
  status: stats.status,
  currentSlot: stats.currentSlot,
  processedBlocks: stats.processedBlocks,
  processedTransactions: stats.processedTransactions,
  blocksPerSecond: stats.blocksPerSecond,
  transactionsPerSecond: stats.transactionsPerSecond,
});
```

### Progress Tracking (Historical Mode)

```typescript
setInterval(() => {
  const progress = indexer.getProgress();
  if (progress !== null) {
    console.log(`Progress: ${progress.toFixed(2)}%`);
  }
}, 5000);
```

### Health Checks

```typescript
const isHealthy = await indexer.healthCheck();
if (!isHealthy) {
  console.error("Indexer health check failed!");
}
```

## Advanced Usage

### Custom Processing

```typescript
import { BlockProcessor } from "@solixdb/core";

const processor = new BlockProcessor({
  includeTransactions: true,
  includeInstructions: true,
  includeAccounts: false,
  programFilters: ["YourProgramId"],
});

processor.configure({
  includeRewards: true,
});
```

### Custom Retry Logic

```typescript
import { retry } from "@solixdb/core";

const data = await retry(
  async () => {
    // Your async operation
    return await fetchSomeData();
  },
  {
    maxRetries: 5,
    initialDelay: 2000,
    maxDelay: 30000,
    backoffMultiplier: 2,
  },
  "fetchSomeData"
);
```

### Custom Logging

```typescript
import { createLogger } from "@solixdb/core";

const logger = createLogger({
  level: "debug",
  prefix: "MyIndexer",
  enableTimestamp: true,
  enableColors: true,
});

logger.info("Indexer started");
logger.debug("Processing block", { slot: 12345 });
logger.error("Error occurred", error);
```

## Architecture

```
SolanaIndexer
├── IndexerEngine (orchestrates everything)
│   ├── IndexerState (tracks progress)
│   ├── IndexerConfig (configuration)
│   ├── BlockProcessor (processes blocks)
│   │   ├── TransactionProcessor
│   │   └── AccountProcessor
│   ├── RPCPoller (polling mode)
│   │   └── PollingStrategy
│   └── StreamManager (streaming mode)
│       ├── WebSocketStream
│       └── StreamBuffer
├── Storage Adapter (persists data)
└── RPC Provider (fetches data)
```

## Error Handling

```typescript
import { IndexerError, RPCError, StorageError } from "@solixdb/core";

try {
  await indexer.start();
} catch (error) {
  if (error instanceof IndexerError) {
    console.error("Indexer error:", error.message, error.code);
  } else if (error instanceof RPCError) {
    console.error("RPC error:", error.message);
  } else if (error instanceof StorageError) {
    console.error("Storage error:", error.message);
  }
}
```

## Performance Tips

1. **Batch Size**: Larger batches = higher throughput, but more memory usage
2. **Commitment Level**: Use 'confirmed' for a balance of speed and finality
3. **Program Filters**: Filter early to reduce processing overhead
4. **Adaptive Polling**: Enable for automatic performance tuning
5. **Storage Choice**: Use PostgreSQL/ClickHouse for production workloads

## License

MIT

## Support

- Documentation: https://docs.solixdb.xyz
- GitHub Issues: https://github.com/SolixDB/solixdb/issues
<!-- - Discord: https://discord.gg/solixdb -->