npm package: @sf-bot/rag-loader
Description: Lazy-loading RAG utilities for fetching and caching chunks from GitHub data repos
Author: City of San Francisco
Latest version: 0.1.0

# @sf-bot/rag-loader

Lazy-loading RAG utilities for fetching and caching chunks from GitHub data repos.

## Installation

```bash
pnpm add @sf-bot/rag-loader
```

## Usage

### Basic Usage - Search GitHub Data

```typescript
import { RAGLoader } from '@sf-rag-utils/rag-loader';

// Create a loader for a specific data repo
const loader = new RAGLoader({
  baseUrl: 'https://raw.githubusercontent.com/city-of-san-francisco/housing-hub/main/packages/housing-sample-db'
});

// Search using keywords
const results = await loader.search(['eviction', 'tenant'], { limit: 10 });

// Or use natural language query
const results = await loader.query('What are my rights for eviction?');

// Load full content for results
const resultsWithContent = await loader.search(['eviction'], { loadContent: true });
```

### PostgreSQL Integration

```typescript
import { Pool } from 'pg';
import { RAGLoader, loadFromRemote, searchByKeywords, lazyLoadMissingShards } from '@sf-rag-utils/rag-loader';

const pool = new Pool();
const loader = new RAGLoader({
  baseUrl: 'https://raw.githubusercontent.com/org/repo/main/packages/data'
});

// Load chunks from GitHub into PostgreSQL
const result = await loadFromRemote(
  loader,
  pool,
  collectionId,
  ['eviction', 'adu', 'permits'],
  'sf://housing'
);
console.log(`Loaded ${result.loaded} chunks from ${result.filesLoaded.length} files`);

// Lazy load only missing shards (idempotent)
const lazyResult = await lazyLoadMissingShards(
  loader,
  pool,
  collectionId,
  ['eviction', 'tenant'],
  'sf://housing'
);

// Search chunks in PostgreSQL by keywords
const chunks = await searchByKeywords(pool, collectionId, ['eviction'], 10);
```

## API

### RAGLoader

The main class for fetching RAG data from GitHub.

#### Constructor Options

```typescript
interface LoaderOptions {
  baseUrl: string;        // Required: Base URL for raw GitHub content
  cacheTimeout?: number;  // Optional: Cache timeout in ms (default: 15 minutes)
}
```

#### Methods

- `loadManifest()` - Load the repo manifest
- `loadShardIndex()` - Load the shard index
- `loadShard(keyword)` - Load a keyword shard
- `loadShards(keywords)` - Load multiple shards in parallel
- `loadChunkFile(filename)` - Load a chunk file
- `loadChunk(chunkId, contentUrl)` - Load a specific chunk
- `search(keywords, options)` - Search for chunks by keywords
- `query(query, options)` - Search using natural language
- `tokenize(query)` - Tokenize a query string
- `getStats()` - Get database statistics
- `clearCache()` - Clear the cache
- `preload(keywords)` - Preload shards for faster queries
- `getChunkFilesForKeywords(keywords)` - Get chunk files referenced by keywords

### PostgreSQL Utilities

- `hashContent(content)` - Generate content hash for deduplication
- `isChunkFileLoaded(pool, collectionId, filename)` - Check if chunk file is loaded
- `getCachedKeywords(pool, collectionId)` - Get keywords with cached chunks
- `insertChunksToPostgres(pool, collectionId, filename, data, namespace)` - Insert chunks
- `loadFromRemote(loader, pool, collectionId, keywords, namespace)` - Load from GitHub
- `lazyLoadMissingShards(loader, pool, collectionId, keywords, namespace)` - Lazy load missing
- `searchByKeywords(pool, collectionId, keywords, limit)` - Search PostgreSQL

## Types

```typescript
interface ChunkResult {
  id: string;
  score: number;
  preview?: string;
  content_url?: string;
  content?: string;
  metadata?: Record<string, unknown>;
  keywords?: string[];
}

interface ShardData {
  keyword: string;
  chunk_count: number;
  chunks: Array<{
    id: string;
    score: number;
    preview?: string;
    content_url?: string;
    metadata?: Record<string, unknown>;
  }>;
}

interface ChunkFileData {
  shard_index: number;
  chunks: Record<string, {
    content: string;
    doc_uri: string;
    keywords: string[];
    metadata: Record<string, unknown>;
    token_count: number;
  }>;
}
```

## License

MIT