npm package: @holokai/holo-test
Description: Test SDK for Holo provider plugins
Author: Holokai
Latest version: 1.4.2

# @holokai/holo-test

Conformance testing framework for Holo provider plugins. Provides fixture-driven testers, vitest contracts, a fluent DSL, and a CLI.

## Installation

```bash
npm install @holokai/holo-test
```

Peer dependencies: `vitest ^3.0.0`, `tsyringe ^4.10.0`

## Quick Start

```typescript
import {describe, it} from 'vitest';
import {loadFixtures, runWireContract} from '@holokai/holo-test';

const fixtures = loadFixtures('./tests/fixtures', {plugin: 'openai'});

describe('openai wire conformance', () => {
    for (const fixture of fixtures) {
        it(fixture.name, async () => {
            await runWireContract('openai', fixture);
        });
    }
});
```

## Testing Modes

| Mode | What it Tests | Function |
|------|--------------|----------|
| **Wire** | Provider events serialize to correct HTTP wire chunks (SSE/JSON) | `testWire()` / `runWireContract()` |
| **Audit** | Worker requests produce correct `ProviderRequest`/`ProviderResponse` audit records | `testAudit()` / `runAuditContract()` |
| **Pipeline** | End-to-end: payload in, events out, text accumulated, done event emitted | `testPipeline()` / `runPipelineContract()` |
| **RoundTrip** | SDK request → HTTP fixture server → SDK response matches expected output | `testRoundTrip()` / `runRoundTripContract()` |

## Fixtures

Create `.fixture.ts` files that export a `FixtureScenario`:

```typescript
import type {FixtureScenario} from '@holokai/holo-test';

const fixture: FixtureScenario = {
    name: 'simple chat completion',
    plugin: 'openai',
    protocol: 'openai.chatCompletions',
    streaming: false,

    providerChunks: [/* mock provider response objects */],

    expectedText: 'Hello, world!',
    expectedWire: ['{"id":"chatcmpl-...}'],
    expectedStatus: 200,
    expectedHeaders: {'content-type': 'application/json'},

    expectedAudit: {
        access_model: 'gpt-4o',
        input_tokens: 10,
        output_tokens: 5,
        status: 'success',
    },

    tags: ['chat', 'non-streaming'],
};

export default fixture;
```

Load fixtures:

```typescript
import {loadFixtures} from '@holokai/holo-test';

// All fixtures
const all = loadFixtures('./tests/fixtures');

// Filtered
const streaming = loadFixtures('./tests/fixtures', {plugin: 'openai', tag: 'streaming'});
```

## Suite DSL

Fluent builder for running multiple test modes:

```typescript
import {suite} from '@holokai/holo-test';

await suite()
    .forPlugin('openai')
    .fixtureDir('./tests/fixtures')
    .wire()
    .audit()
    .pipeline()
    .verbose()
    .run();
```

## Vitest Contracts

Import contract functions directly into vitest test files:

```typescript
import {runWireContract, runAuditContract, runPipelineContract} from '@holokai/holo-test';

describe('openai conformance', () => {
    it('wire: simple chat', async () => {
        await runWireContract('openai', fixture);
    });

    it('audit: simple chat', async () => {
        await runAuditContract('openai', fixture);
    });

    it('pipeline: simple chat', async () => {
        await runPipelineContract('openai', fixture);
    });
});
```

## CLI

```bash
npx holo-test --fixture-dir ./tests/fixtures --plugin openai
npx holo-test --fixture-dir ./tests/fixtures --plugin claude --tag streaming --verbose
```

## Live Testing

Test against real provider APIs:

```typescript
import {runLiveTest, isProviderAvailable} from '@holokai/holo-test';

if (isProviderAvailable('openai')) {
    const results = await runLiveTest({
        families: ['openai'],
        timeout: 30000,
    });
    console.log(results);
}
```

Environment variables for live tests:
- `OPENAI_API_KEY`, `CLAUDE_API_KEY`, `GEMINI_API_KEY`, `OLLAMA_HOST`
- `OPENAI_MODEL`, `CLAUDE_MODEL`, etc. (override default test models)

## Helpers

```typescript
import {
    loadPlugin,                  // Load and initialize a plugin by family name
    parseSseBody,                // Parse SSE response body into frames
    parseNdjsonBody,             // Parse NDJSON response body
    createMockFetch,             // Mock fetch for unit tests
    createSseMockFetch,          // Mock SSE streaming responses
    collectStreamText,           // Extract all text from a stream
    collectStreamEvents,         // Collect all events from a stream
    HttpFixtureServer,           // Mock HTTP server serving fixture responses
    createTestWorkerRequest,     // Build a HoloWorkerRequest from a fixture
    createTestEnvelope,          // Build a WorkerResponseEnvelope from a fixture
    discoverProviders,           // Discover available providers via gateway
    assertTestResult,            // Assert a TestResult passed (throws on failure)
} from '@holokai/holo-test';
```

## License

MIT