npm package: @providerprotocol/ai
Description: UPP: Unified Provider Protocol for AI inference
Author: Provider Protocol
Latest version: 0.0.43

# @providerprotocol/ai

A unified TypeScript SDK for AI inference across multiple providers. One API for LLMs, embeddings, and image generation.

```bash
bun add @providerprotocol/ai
```

## Quick Start

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';

const claude = llm({ model: anthropic('claude-sonnet-4-20250514') });
const turn = await claude.generate('Hello!');
console.log(turn.response.text);
```

## Providers

| Provider | Import | LLM | Embedding | Image |
|----------|--------|:---:|:---------:|:-----:|
| Anthropic | `@providerprotocol/ai/anthropic` | ✓ | | |
| OpenAI | `@providerprotocol/ai/openai` | ✓ | ✓ | ✓ |
| Google | `@providerprotocol/ai/google` | ✓ | ✓ | ✓ |
| xAI | `@providerprotocol/ai/xai` | ✓ | | ✓ |
| Ollama | `@providerprotocol/ai/ollama` | ✓ | ✓ | |
| OpenRouter | `@providerprotocol/ai/openrouter` | ✓ | ✓ | ✓ |
| Groq | `@providerprotocol/ai/groq` | ✓ | | |
| Cerebras | `@providerprotocol/ai/cerebras` | ✓ | | |
| Moonshot | `@providerprotocol/ai/moonshot` | ✓ | | |
| OpenResponses | `@providerprotocol/ai/responses` | ✓ | | |

API keys are loaded automatically from environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GROQ_API_KEY`, `CEREBRAS_API_KEY`, `MOONSHOT_API_KEY`, etc.).

## LLM

### Streaming

```typescript
const stream = claude.stream('Count to 5');
for await (const event of stream) {
  if (event.type === 'text_delta') {
    process.stdout.write(event.delta.text);
  }
}
const turn = await stream.turn;
```

Stream results are PromiseLike, so you can also await the stream directly to auto-drain:

```typescript
const turn = await claude.stream('Count to 5');
```

**Stream Control:**

```typescript
const stream = claude.stream('Write a long story');

// Abort the stream at any time
setTimeout(() => stream.abort(), 5000);

for await (const event of stream) {
  // Process events until abort
}
```

**Stream Events:**

| Event | Description |
|-------|-------------|
| `text_delta` | Incremental text output |
| `reasoning_delta` | Incremental reasoning/thinking output |
| `object_delta` | Incremental structured output JSON |
| `tool_call_delta` | Tool call arguments being streamed |
| `tool_execution_start` | Tool execution has started |
| `tool_execution_end` | Tool execution has completed |
| `message_start` / `message_stop` | Message boundaries |
| `content_block_start` / `content_block_stop` | Content block boundaries |

### Multi-turn Conversations

```typescript
const history: Message[] = [];

const t1 = await claude.generate(history, 'My name is Alice');
history.push(...t1.messages);

const t2 = await claude.generate(history, 'What is my name?');
// Response: "Your name is Alice"
```

### System-Only Inference

Both `generate()` and `stream()` can be called with no arguments for system-prompt-only inference:

```typescript
const assistant = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  system: 'You are a haiku generator. Generate a haiku about coding.',
});

// No user input needed
const turn = await assistant.generate();
console.log(turn.response.text);
```

### Tools

```typescript
const turn = await claude.generate({
  tools: [{
    name: 'getWeather',
    description: 'Get weather for a location',
    parameters: {
      type: 'object',
      properties: { location: { type: 'string' } },
      required: ['location'],
    },
    run: async ({ location }) => ({ temp: 72, conditions: 'sunny' }),
  }],
}, 'What is the weather in Tokyo?');
```

#### Tools with Zod Parameters

Tool parameters also accept Zod schemas:

```typescript
import { z } from 'zod';

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  tools: [{
    name: 'get_weather',
    description: 'Get weather for a location',
    parameters: z.object({
      location: z.string().describe('City name'),
      units: z.enum(['celsius', 'fahrenheit']).optional(),
    }),
    run: async ({ location, units }) => fetchWeather(location, units),
  }],
});
```

### Structured Output

```typescript
import { llm } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const extractor = llm({
  model: openai('gpt-4o'),
  structure: {
    type: 'object',
    properties: {
      name: { type: 'string' },
      age: { type: 'number' },
    },
    required: ['name', 'age'],
  },
});

const turn = await extractor.generate('John is 30 years old');
console.log(turn.data); // { name: 'John', age: 30 }
```

#### Zod Schema Support

Structured output and tool parameters accept Zod schemas directly, with automatic conversion to JSON Schema:

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { z } from 'zod';

const extractor = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  structure: z.object({
    name: z.string(),
    age: z.number(),
    tags: z.array(z.string()),
  }),
});

const turn = await extractor.generate('Extract: John Doe, 30 years old, likes coding');
console.log(turn.data); // { name: "John Doe", age: 30, tags: ["coding"] }
```

**Requirements:**
- Zod schemas must be object schemas (`z.object()`). Non-object schemas will throw an error.
- Zod is an optional peer dependency - install only if using Zod schemas:

```bash
bun add zod                    # v4+ for native JSON Schema conversion
bun add zod zod-to-json-schema # v3 requires additional package
```

### Multimodal Input

```typescript
import { Image, Document, Audio, Video } from '@providerprotocol/ai';

// Images
const img = await Image.fromPath('./photo.png');
const turn = await claude.generate([img, 'What is in this image?']);

// Documents (PDF, text)
const doc = await Document.fromPath('./report.pdf', 'Annual Report');
const docTurn = await claude.generate([doc.toBlock(), 'Summarize this document']);

// Audio (Google, OpenRouter)
const audio = await Audio.fromPath('./recording.mp3');
const audioTurn = await gemini.generate([audio.toBlock(), 'Transcribe this audio']);

// Video (Google, OpenRouter)
const video = await Video.fromPath('./clip.mp4');
const videoTurn = await gemini.generate([video.toBlock(), 'Describe this video']);
```

**Multimodal Support by Provider:**

| Provider | Image | Document | Audio | Video |
|----------|:-----:|:--------:|:-----:|:-----:|
| Anthropic | ✓ | PDF, Text | | |
| OpenAI | ✓ | PDF, Text | | |
| Google | ✓ | PDF, Text | ✓ | ✓ |
| OpenRouter | ✓ | PDF, Text | ✓ | ✓ |
| xAI | ✓ | | | |
| Groq | ✓ | | | |
| Moonshot | ✓ | | | ✓* |

\* Moonshot video input is experimental.

## Anthropic Beta Features

Anthropic provides beta features through the `betas` export. Enable them at the model level:

```typescript
import { anthropic, betas } from '@providerprotocol/ai/anthropic';
import { llm } from '@providerprotocol/ai';

// Native structured outputs with guaranteed JSON schema conformance
const model = llm({
  model: anthropic('claude-sonnet-4-20250514', {
    betas: [betas.structuredOutputs],
  }),
  structure: {
    type: 'object',
    properties: { answer: { type: 'string' } },
    required: ['answer'],
  },
});

// Extended thinking with interleaved tool calls
const thinker = llm({
  model: anthropic('claude-sonnet-4-20250514', {
    betas: [betas.interleavedThinking],
  }),
  params: {
    thinking: { type: 'enabled', budget_tokens: 10000 },
  },
});
```

**Available Beta Features:**

| Beta | Description |
|------|-------------|
| `structuredOutputs` | Guaranteed JSON schema conformance for responses |
| `interleavedThinking` | Claude can think between tool calls |
| `devFullThinking` | Developer mode for full thinking visibility |
| `effort` | Control response thoroughness vs efficiency (Opus 4.5) |
| `computerUseLegacy` | Computer use for Claude 3.x models |
| `computerUse` | Mouse, keyboard, screenshot control (Claude 4) |
| `computerUseOpus` | Computer use with extra commands (Opus 4.5) |
| `codeExecution` | Python/Bash sandbox execution |
| `tokenEfficientTools` | Up to 70% token reduction for tool calls |
| `fineGrainedToolStreaming` | Stream tool args without buffering |
| `maxTokens35Sonnet` | 8,192 output tokens for Claude 3.5 Sonnet |
| `output128k` | 128K token output length |
| `context1m` | 1 million token context window (Sonnet 4) |
| `promptCaching` | Reduced latency and costs via caching |
| `extendedCacheTtl` | 1-hour cache TTL (vs 5-minute default) |
| `contextManagement` | Automatic tool call clearing for context |
| `modelContextWindowExceeded` | Handle exceeded context windows |
| `advancedToolUse` | Tool Search, Programmatic Tool Calling |
| `mcpClient` | Connect to remote MCP servers |
| `mcpClientLatest` | Updated MCP client |
| `filesApi` | Upload and manage files |
| `pdfs` | PDF document support |
| `tokenCounting` | Token counting endpoint |
| `messageBatches` | Async batch processing at 50% cost |
| `skills` | Agent Skills (PowerPoint, Excel, Word, PDF) |

## Anthropic Built-in Tools

Use Anthropic's built-in tools directly with the `tools` export:

```typescript
import { anthropic, betas, tools } from '@providerprotocol/ai/anthropic';
import { llm } from '@providerprotocol/ai';

// Web search with optional user location
const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  params: {
    tools: [tools.webSearch({ max_results: 5 })],
  },
});

// Computer use (requires beta)
const computerModel = llm({
  model: anthropic('claude-sonnet-4-20250514', {
    betas: [betas.computerUse],
  }),
  params: {
    tools: [tools.computer({ display_width: 1920, display_height: 1080, display_number: 1 })],
  },
});

// Code execution (requires beta)
const codeModel = llm({
  model: anthropic('claude-sonnet-4-20250514', {
    betas: [betas.codeExecution],
  }),
  params: {
    tools: [tools.codeExecution()],
  },
});
```

**Available Built-in Tools:**

| Tool | Description |
|------|-------------|
| `tools.webSearch()` | Search the web with optional max results and location |
| `tools.computer()` | Mouse, keyboard, and screenshot control |
| `tools.textEditor()` | Edit text files programmatically |
| `tools.bash()` | Execute bash commands |
| `tools.codeExecution()` | Run code in a sandboxed environment |
| `tools.toolSearch()` | Search through available tools |

## Reasoning / Extended Thinking

Access model reasoning and extended thinking across providers with a unified API.

### Anthropic

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';

const claude = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  params: {
    max_tokens: 16000,
    thinking: {
      type: 'enabled',
      budget_tokens: 5000,
    },
  },
});

const turn = await claude.generate('Solve this complex problem...');
console.log(turn.response.reasoning); // Reasoning blocks
```

### OpenAI

```typescript
import { llm } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const gpt = llm({
  model: openai('o3-mini'),
  params: {
    max_output_tokens: 4000,
    reasoning: {
      effort: 'medium',
      summary: 'detailed',
    },
  },
});
```

### Google Gemini

```typescript
import { llm } from '@providerprotocol/ai';
import { google } from '@providerprotocol/ai/google';

const gemini = llm({
  model: google('gemini-2.5-flash'),
  params: {
    maxOutputTokens: 4000,
    thinkingConfig: {
      thinkingBudget: -1, // Dynamic
      includeThoughts: true,
    },
  },
});
```

### Cerebras

```typescript
import { llm } from '@providerprotocol/ai';
import { cerebras } from '@providerprotocol/ai/cerebras';

const model = llm({
  model: cerebras('gpt-oss-120b'),
  params: {
    reasoning_effort: 'high',
    reasoning_format: 'parsed',
  },
});
```

### Streaming Reasoning

All providers emit `ReasoningDelta` events during streaming:

```typescript
for await (const event of stream) {
  if (event.type === 'reasoning_delta') {
    process.stdout.write(event.delta.text);
  }
}
```

## Embeddings

```typescript
import { embedding } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const embedder = embedding({ model: openai('text-embedding-3-small') });

// Single or batch
const result = await embedder.embed('Hello world');
const batch = await embedder.embed(['doc1', 'doc2', 'doc3']);

console.log(result.embeddings[0].vector);     // [0.123, -0.456, ...]
console.log(result.embeddings[0].dimensions); // 1536
```

### Chunked Processing

For large datasets with progress tracking:

```typescript
const stream = embedder.embed(documents, {
  chunked: true,
  batchSize: 100,
  concurrency: 2,
});

for await (const progress of stream) {
  console.log(`${progress.percent.toFixed(1)}% complete`);
}

const result = await stream.result;
```

## Image Generation

```typescript
import { image } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const dalle = image({ model: openai('dall-e-3') });
const result = await dalle.generate('A sunset over mountains');

console.log(result.images[0].image.toBase64());
```

### With Parameters

```typescript
const hd = image({
  model: openai('dall-e-3'),
  params: { size: '1792x1024', quality: 'hd', style: 'natural' },
});
```

### Image Editing

```typescript
import { image, Image } from '@providerprotocol/ai';

const editor = image({ model: openai('dall-e-2') });

const source = await Image.fromPath('./photo.png');
const mask = await Image.fromPath('./mask.png');

const result = await editor.edit({
  image: source,
  mask,
  prompt: 'Add a rainbow in the sky',
});
```

## Configuration

```typescript
import { llm, exponentialBackoff, roundRobinKeys } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const instance = llm({
  model: openai('gpt-4o'),
  config: {
    apiKey: roundRobinKeys(['sk-key1', 'sk-key2']),
    timeout: 30000,
    retryStrategy: exponentialBackoff({ maxAttempts: 3 }),
  },
  params: {
    temperature: 0.7,
    max_tokens: 1000,
  },
  system: 'You are a helpful assistant.',
});
```

### System Prompts

System prompts can be a simple string or a provider-specific array for advanced features:

```typescript
// Simple string (all providers)
const simple = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  system: 'You are a helpful assistant.',
});

// Anthropic cache_control format
import { anthropic, betas } from '@providerprotocol/ai/anthropic';

const cached = llm({
  model: anthropic('claude-sonnet-4-20250514', {
    betas: [betas.promptCaching],
  }),
  system: [
    { type: 'text', text: 'Large context document...', cache_control: { type: 'ephemeral' } },
    { type: 'text', text: 'Instructions...' },
  ],
});
```

### Provider Config Options

```typescript
interface ProviderConfig {
  apiKey?: string | (() => Promise<string>) | KeyStrategy; // API key, async getter, or strategy
  baseUrl?: string;                 // Custom API endpoint
  timeout?: number;                 // Per-attempt timeout (ms)
  retryStrategy?: RetryStrategy;    // Retry behavior
  headers?: Record<string, string>; // Custom headers (merged with provider defaults)
  fetch?: typeof fetch;             // Custom fetch implementation
  apiVersion?: string;              // API version override
  retryAfterMaxSeconds?: number;    // Cap for Retry-After header (default: 3600)
}
```

**Notes:**
- `timeout` applies per attempt; total time can exceed this with retries
- `headers` are merged with model-level headers (explicit config takes precedence)
- `retryAfterMaxSeconds` prevents honoring excessively long Retry-After values

### Key Strategies

```typescript
import { roundRobinKeys, weightedKeys, dynamicKey } from '@providerprotocol/ai/http';

// Cycle through keys evenly
roundRobinKeys(['sk-1', 'sk-2', 'sk-3'])

// Weighted selection (70% key1, 30% key2)
weightedKeys([
  { key: 'sk-1', weight: 70 },
  { key: 'sk-2', weight: 30 },
])

// Dynamic fetching (secrets manager, etc.)
dynamicKey(async () => fetchKeyFromVault())
```

### Retry Strategies

```typescript
import {
  exponentialBackoff,
  linearBackoff,
  noRetry,
  retryAfterStrategy,
} from '@providerprotocol/ai';

// Exponential: 1s, 2s, 4s...
exponentialBackoff({
  maxAttempts: 5,
  baseDelay: 1000,
  maxDelay: 30000,
  jitter: true,  // Randomize delays to prevent thundering herd (default: true)
})

// Linear: 1s, 2s, 3s...
linearBackoff({ maxAttempts: 3, delay: 1000 })

// Respect server Retry-After headers
retryAfterStrategy({ maxAttempts: 3, fallbackDelay: 5000 })

// No retries
noRetry()
```

**Retryable Errors:** `RATE_LIMITED`, `NETWORK_ERROR`, `TIMEOUT`, `PROVIDER_ERROR`

**Streaming Retry:** Retry strategies work with both `.generate()` and `.stream()`. During streaming, `stream_retry` events are emitted to notify consumers of retry attempts, and middleware can use the `onRetry` hook to reset accumulated state.

## Tool Execution Control

```typescript
const turn = await claude.generate({
  tools: [weatherTool, searchTool],
  toolStrategy: {
    maxIterations: 5,
    onBeforeCall: (tool, params) => {
      if (tool.name === 'dangerousTool') return false; // Block execution
      return true;
    },
    onAfterCall: (tool, params, result) => {
      console.log(`${tool.name} returned:`, result);
    },
    onError: (tool, params, error) => {
      console.error(`${tool.name} failed:`, error);
    },
  },
}, 'Search for recent news about AI');
```

## Thread Management

```typescript
import { Thread } from '@providerprotocol/ai';

const thread = new Thread();

thread.user('Hello!');
const turn = await claude.generate(thread.toMessages(), 'How are you?');
thread.append(turn);

// Serialize for storage
const json = thread.toJSON();
localStorage.setItem('conversation', JSON.stringify(json));

// Restore later
const restored = Thread.fromJSON(JSON.parse(localStorage.getItem('conversation')));
```

## Middleware

Compose request/response/stream transformations with the middleware system. Middleware is imported from dedicated entry points.

### Parsed Object Middleware

Automatically parse streaming JSON from structured output and tool call events:

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { parsedObjectMiddleware } from '@providerprotocol/ai/middleware/parsed-object';

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  structure: {
    type: 'object',
    properties: {
      city: { type: 'string' },
      country: { type: 'string' },
      population: { type: 'number' },
    },
    required: ['city', 'country', 'population'],
  },
  middleware: [parsedObjectMiddleware()],
});

for await (const event of model.stream('What is the capital of France?')) {
  if (event.type === 'object_delta') {
    // Access incrementally parsed structured data
    console.log(event.delta.parsed);
    // { city: "Par" } -> { city: "Paris" } -> { city: "Paris", country: "Fr" } -> ...
  }
}
```

### Logging Middleware

Add visibility into request lifecycle:

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { loggingMiddleware } from '@providerprotocol/ai/middleware/logging';

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  middleware: [loggingMiddleware({ level: 'debug' })],
});

// Logs: [PP] [anthropic] Starting llm request (streaming)
// Logs: [PP] [anthropic] Completed in 1234ms
const result = await model.generate('Hello');
```

### Persistence Middleware

Load and save conversation threads around LLM requests:

```typescript
import { llm, Thread } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { persistenceMiddleware, PersistenceAdapter } from '@providerprotocol/ai/middleware/persistence';

const adapter = new PersistenceAdapter({
  id: 'conversation-123',
  load: async (id) => loadThreadFromDatabase(id), // Thread | ThreadJSON | null
  save: async (id, thread, turn) => {
    await saveThreadToDatabase(id, thread);
    if (turn) {
      await saveTurnToDatabase(id, turn);
    }
  },
});

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  middleware: [persistenceMiddleware({ adapter })],
});

const turn = await model.generate('Hello!');
```

### Pub-Sub Middleware (Stream Resumption)

Enable reconnecting clients to catch up on missed events during active generation. The middleware buffers events, publishes them to subscribers, and removes streams on completion/abort/error.
If a stream never reaches those hooks (for example, a process crash), the adapter may retain the entry. Custom adapters should invoke `onComplete` when `remove()` runs so subscriber streams can terminate.
Streams are created lazily on first `append()` or `subscribe()` call.

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { pubsubMiddleware, memoryAdapter } from '@providerprotocol/ai/middleware/pubsub';
import { webapi } from '@providerprotocol/ai/middleware/pubsub/server';

// Create a shared adapter instance
const adapter = memoryAdapter({ maxStreams: 1000 });

// Server route handling both new requests and reconnections
Bun.serve({
  port: 3000,
  async fetch(req) {
    const { messages, streamId } = await req.json();
    const exists = await adapter.exists(streamId);

    if (!exists) {
      // Start background generation (fire and forget)
      // Stream is created lazily on first append()
      const model = llm({
        model: anthropic('claude-sonnet-4-20250514'),
        middleware: [pubsubMiddleware({ adapter, streamId })],
      });
      model.stream(messages).then(turn => { /* save to DB */ });
    }

    // Both new and reconnect: subscribe to events
    return new Response(webapi.createSubscriberStream(streamId, adapter), {
      headers: { 'Content-Type': 'text/event-stream' },
    });
  },
});
```

**Framework Adapters:**

```typescript
// Express
import { express } from '@providerprotocol/ai/middleware/pubsub/server';

app.post('/api/ai', async (req, res) => {
  const { messages, streamId } = req.body;

  // Guard: prevent duplicate generations on reconnect
  if (!await adapter.exists(streamId)) {
    const model = llm({
      model: anthropic('claude-sonnet-4-20250514'),
      middleware: [pubsubMiddleware({ adapter, streamId })],
    });
    model.stream(messages).then(turn => { /* save to DB */ });
  }

  express.streamSubscriber(streamId, adapter, res);
});

// Fastify
import { fastify } from '@providerprotocol/ai/middleware/pubsub/server';

app.post('/api/ai', async (request, reply) => {
  const { messages, streamId } = request.body;

  // Guard: prevent duplicate generations on reconnect
  if (!await adapter.exists(streamId)) {
    const model = llm({
      model: anthropic('claude-sonnet-4-20250514'),
      middleware: [pubsubMiddleware({ adapter, streamId })],
    });
    model.stream(messages).then(turn => { /* save to DB */ });
  }

  return fastify.streamSubscriber(streamId, adapter, reply);
});

// H3/Nuxt
import { sendStream, setHeader } from 'h3';
import { h3 } from '@providerprotocol/ai/middleware/pubsub/server';

export default defineEventHandler(async (event) => {
  const { messages, streamId } = await readBody(event);

  // Guard: prevent duplicate generations on reconnect
  if (!await adapter.exists(streamId)) {
    const model = llm({
      model: anthropic('claude-sonnet-4-20250514'),
      middleware: [pubsubMiddleware({ adapter, streamId })],
    });
    model.stream(messages).then(turn => { /* save to DB */ });
  }

  // Required: H3's sendStream does NOT set these headers
  setHeader(event, 'Content-Type', 'text/event-stream');
  setHeader(event, 'Cache-Control', 'no-cache');
  setHeader(event, 'Connection', 'keep-alive');
  setHeader(event, 'X-Accel-Buffering', 'no');

  return sendStream(event, h3.createSubscriberSSEStream(streamId, adapter));
});
```

**Custom Adapters:**

Implement `PubSubAdapter` for custom backends (Redis, etc.):

```typescript
import type { PubSubAdapter } from '@providerprotocol/ai/middleware/pubsub';

const redisAdapter: PubSubAdapter = {
  async exists(streamId) { /* check if stream exists */ },
  async append(streamId, event) { /* append event, create lazily */ },
  async getEvents(streamId) { /* return events or [] */ },
  subscribe(streamId, onEvent, onComplete, onFinalData) { /* subscribe to live events */ },
  publish(streamId, event) { /* broadcast to subscribers */ },
  setFinalData(streamId, data) { /* store final Turn data */ },
  async remove(streamId) { /* notify onFinalData, onComplete, then delete */ },
};
```

### Combining Middleware

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { loggingMiddleware } from '@providerprotocol/ai/middleware/logging';
import { parsedObjectMiddleware } from '@providerprotocol/ai/middleware/parsed-object';

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  structure: mySchema,
  middleware: [
    loggingMiddleware({ level: 'info' }),
    parsedObjectMiddleware(),
  ],
});
```

### Pipeline Middleware (Post-Turn Processing)

Run async tasks (image generation, embeddings, slug creation, etc.) after the LLM completes, with progress events streamed to connected clients:

```typescript
import { llm } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { pubsubMiddleware, memoryAdapter } from '@providerprotocol/ai/middleware/pubsub';
import { pipelineMiddleware, isPipelineStageEvent } from '@providerprotocol/ai/middleware/pipeline';

const adapter = memoryAdapter();

const model = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  structure: BlogPostSchema,
  middleware: [
    pubsubMiddleware({ adapter, streamId: postId }),
    pipelineMiddleware<BlogPost>({
      stages: [
        {
          type: 'slug',
          run: (turn, emit) => {
            const slug = turn.data!.title.toLowerCase().replace(/\s+/g, '-');
            (turn as { slug?: string }).slug = slug;
            emit({ slug });
          },
        },
        {
          type: 'embedding',
          run: async (turn, emit) => {
            await vectorize(turn.data!);
            emit({ embedded: true });
          },
        },
      ],
      parallel: false,        // Run stages sequentially (default)
      continueOnError: false, // Stop on first error (default)
      onStageError: ({ stage, error }) => {
        console.error(`Stage ${stage.type} failed:`, error);
      },
    }),
  ],
});

// Stages run after streaming completes
model.stream(prompt).then(turn => {
  const extended = turn as typeof turn & { slug?: string };
  console.log(extended.slug);
});
```

**Consuming Pipeline Events:**

```typescript
for await (const event of model.stream(prompt)) {
  if (isPipelineStageEvent(event)) {
    console.log(event.delta.stage, event.delta.payload);
    // 'slug' { slug: 'my-blog-post' }
    // 'embedding' { embedded: true }
  }
}
```

**Middleware Order:** Place `pipelineMiddleware` after `pubsubMiddleware` in the array:

```typescript
middleware: [
  pubsubMiddleware({ ... }),  // Setup runs first in onStart
  pipelineMiddleware({ ... }),    // Events run first in onTurn (reverse order)
]
```

This ensures pubsub sets up before pipeline stages execute, and pipeline events emit before pubsub cleanup.

**Pipeline Configuration:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `stages` | `PipelineStage[]` | required | Stages to run after turn completion |
| `parallel` | `boolean` | `false` | Run stages in parallel instead of sequential |
| `continueOnError` | `boolean` | `false` | Continue running subsequent stages if one fails |
| `onStageError` | `function` | - | Called when a stage throws an error |

## Error Handling

All errors are normalized to `UPPError` with consistent error codes:

```typescript
import { UPPError } from '@providerprotocol/ai';

try {
  await claude.generate('Hello');
} catch (error) {
  if (error instanceof UPPError) {
    console.log(error.code);       // 'RATE_LIMITED'
    console.log(error.provider);   // 'anthropic'
    console.log(error.modality);   // 'llm'
    console.log(error.statusCode); // 429
    console.log(error.cause);      // Original error (if any)

    switch (error.code) {
      case 'RATE_LIMITED':
        // Wait and retry
        break;
      case 'CONTEXT_LENGTH_EXCEEDED':
        // Reduce input size
        break;
      case 'AUTHENTICATION_FAILED':
        // Check API key
        break;
      case 'CONTENT_FILTERED':
        // Content policy violation
        break;
    }
  }
}
```

**Error Codes:** `AUTHENTICATION_FAILED`, `RATE_LIMITED`, `CONTEXT_LENGTH_EXCEEDED`, `MODEL_NOT_FOUND`, `INVALID_REQUEST`, `INVALID_RESPONSE`, `CONTENT_FILTERED`, `QUOTA_EXCEEDED`, `PROVIDER_ERROR`, `NETWORK_ERROR`, `TIMEOUT`, `CANCELLED`

## API Gateway / Proxy

Build AI API gateways with your own authentication. Users authenticate with your platform - AI provider keys stay hidden on the server.

> **Security Note:** The proxy works without any configuration, but this means **no authentication by default**. Always add your own auth layer in production - the examples below show how.

### Server (Bun/Deno/Cloudflare Workers)

```typescript
import { llm, exponentialBackoff, roundRobinKeys } from '@providerprotocol/ai';
import { anthropic } from '@providerprotocol/ai/anthropic';
import { parseBody, toJSON, toSSE, toError } from '@providerprotocol/ai/proxy';

// Server manages AI provider keys - users never see them
const claude = llm({
  model: anthropic('claude-sonnet-4-20250514'),
  config: {
    apiKey: roundRobinKeys([process.env.ANTHROPIC_KEY_1!, process.env.ANTHROPIC_KEY_2!]),
    retryStrategy: exponentialBackoff({ maxAttempts: 3 }),
  },
});

Bun.serve({
  port: 3000,
  async fetch(req) {
    // Authenticate with YOUR platform credentials
    const token = req.headers.get('Authorization')?.replace('Bearer ', '');
    const user = await validatePlatformToken(token ?? '');
    if (!user) return toError('Unauthorized', 401);

    // Rate limit, track usage, bill user, etc.
    await trackUsage(user.id);

    const { messages, system, params } = parseBody(await req.json());

    if (params?.stream) {
      return toSSE(claude.stream(messages, { system }));
    }
    return toJSON(await claude.generate(messages, { system }));
  },
});
```

### Client

Clients authenticate with your platform token. They get automatic retry on network failures to your proxy.

```typescript
import { llm, exponentialBackoff } from '@providerprotocol/ai';
import { proxy } from '@providerprotocol/ai/proxy';

const claude = llm({
  model: proxy('https://api.yourplatform.com/ai'),
  config: {
    headers: { 'Authorization': 'Bearer user-platform-token' },
    retryStrategy: exponentialBackoff({ maxAttempts: 3 }),
    timeout: 30000,
  },
});

const turn = await claude.generate('Hello!');
```

### Framework Adapters

Server adapters for Express, Fastify, and Nuxt/H3:

```typescript
// Express
import { express as expressAdapter, parseBody } from '@providerprotocol/ai/proxy';
app.post('/ai', authMiddleware, async (req, res) => {
  const { messages, system, params } = parseBody(req.body);
  if (params?.stream) {
    expressAdapter.streamSSE(claude.stream(messages, { system }), res);
  } else {
    expressAdapter.sendJSON(await claude.generate(messages, { system }), res);
  }
});

// Fastify
import { fastify as fastifyAdapter, parseBody } from '@providerprotocol/ai/proxy';
app.post('/ai', async (request, reply) => {
  const { messages, system, params } = parseBody(request.body);
  if (params?.stream) {
    return fastifyAdapter.streamSSE(claude.stream(messages, { system }), reply);
  }
  return fastifyAdapter.sendJSON(await claude.generate(messages, { system }), reply);
});

// Nuxt/H3 (server/api/ai.post.ts)
import { sendStream } from 'h3';
import { h3 as h3Adapter, parseBody } from '@providerprotocol/ai/proxy';
export default defineEventHandler(async (event) => {
  const { messages, system, params } = parseBody(await readBody(event));
  if (params?.stream) {
    return sendStream(event, h3Adapter.createSSEStream(claude.stream(messages, { system })));
  }
  return h3Adapter.sendJSON(await claude.generate(messages, { system }), event);
});
```

**What this enables:**
- Users auth with your platform credentials (JWT, API keys, sessions)
- You manage/rotate AI provider keys centrally
- Per-user rate limiting, usage tracking, billing
- Model access control (different users get different models)
- Request/response logging, content filtering
- Double-layer retry: client retries to proxy, server retries to AI provider

## OpenAI API Modes

OpenAI supports two API endpoints. The Responses API is the default and recommended approach:

```typescript
import { openai } from '@providerprotocol/ai/openai';

// Responses API (default, recommended)
openai('gpt-4o')

// Chat Completions API (legacy)
openai('gpt-4o', { api: 'completions' })
```

The Responses API supports built-in tools and stateful conversations. Use completions for backward compatibility.

## OpenAI Built-in Tools

With the Responses API, use OpenAI's built-in tools directly:

```typescript
import { llm } from '@providerprotocol/ai';
import { openai, tools } from '@providerprotocol/ai/openai';

// Web search
const model = llm({
  model: openai('gpt-4o'),
  params: {
    tools: [tools.webSearch()],
  },
});

// File search with vector stores
const researchModel = llm({
  model: openai('gpt-4o'),
  params: {
    tools: [tools.fileSearch({ vector_store_ids: ['vs_abc123'] })],
  },
});

// Code interpreter
const codeModel = llm({
  model: openai('gpt-4o'),
  params: {
    tools: [tools.codeInterpreter()],
  },
});

// Image generation
const creativeModel = llm({
  model: openai('gpt-4o'),
  params: {
    tools: [tools.imageGeneration()],
  },
});
```

**Available Built-in Tools:**

| Tool | Description |
|------|-------------|
| `tools.webSearch()` | Search the web with optional user location |
| `tools.fileSearch()` | Search uploaded files in vector stores |
| `tools.codeInterpreter()` | Execute code in a sandboxed environment |
| `tools.computer()` | Computer use with display configuration |
| `tools.imageGeneration()` | Generate images via DALL-E |
| `tools.mcp()` | Connect to MCP servers |

## xAI API Modes

xAI supports multiple API compatibility modes:

```typescript
import { xai } from '@providerprotocol/ai/xai';

// Chat Completions (OpenAI-compatible, default)
xai('grok-3-fast')

// Responses API (stateful)
xai('grok-3-fast', { api: 'responses' })

// Messages API (Anthropic-compatible)
xai('grok-3-fast', { api: 'messages' })
```

## Groq

Fast inference with Llama, Gemma, and Mixtral models:

```typescript
import { llm } from '@providerprotocol/ai';
import { groq } from '@providerprotocol/ai/groq';

const model = llm({
  model: groq('llama-3.3-70b-versatile'),
  params: { max_tokens: 1000 },
});

const turn = await model.generate('Hello!');
```

**With web search:**

```typescript
const searchModel = llm({
  model: groq('llama-3.3-70b-versatile'),
  params: {
    search_settings: { mode: 'auto' },
  },
});
```

**With RAG documents:**

```typescript
const ragModel = llm({
  model: groq('llama-3.3-70b-versatile'),
  params: {
    documents: [
      { title: 'Doc 1', content: 'Document content here...' },
      { title: 'Doc 2', content: 'More content...' },
    ],
    citation_options: { include: true },
  },
});
```

**Capabilities:** Streaming, tool calling, structured output, image input (Llama 4 preview), web search, RAG with citations.

**Environment:** `GROQ_API_KEY`

## Cerebras

Ultra-fast inference with Llama, Qwen, and GPT-OSS models:

```typescript
import { llm } from '@providerprotocol/ai';
import { cerebras } from '@providerprotocol/ai/cerebras';

const model = llm({
  model: cerebras('llama-3.3-70b'),
  params: { max_completion_tokens: 1000 },
});

const turn = await model.generate('Hello!');
```

**With reasoning (GPT-OSS):**

```typescript
const model = llm({
  model: cerebras('gpt-oss-120b'),
  params: {
    reasoning_effort: 'high',
    reasoning_format: 'parsed',
  },
});
```

**Capabilities:** Streaming, tool calling, structured output, reasoning parameters.

**Environment:** `CEREBRAS_API_KEY`

## Moonshot

Kimi K2.5 with 256K context, thinking mode, vision, and server-side builtin tools:

```typescript
import { llm } from '@providerprotocol/ai';
import { moonshot, tools } from '@providerprotocol/ai/moonshot';

const model = llm({
  model: moonshot('kimi-k2.5'),
  params: { max_tokens: 1000 },
});

const turn = await model.generate('Hello!');
```

**With thinking mode (default for K2.5):**

```typescript
const model = llm({
  model: moonshot('kimi-k2.5'),
  params: {
    max_tokens: 2000,
    temperature: 1.0,
    thinking: { type: 'enabled' },
  },
});

// Response includes reasoning in turn.response.reasoning
const turn = await model.generate('Solve step by step: 2x + 5 = 13');
```

**With instant mode (disabled thinking):**

```typescript
const model = llm({
  model: moonshot('kimi-k2.5'),
  params: {
    temperature: 0.6,
    thinking: { type: 'disabled' },
  },
});
```

**With builtin tools:**

```typescript
const model = llm({
  model: moonshot('kimi-k2.5'),
  params: {
    tools: [
      tools.webSearch(),
      tools.codeRunner(),
      tools.date(),
    ],
  },
});
```

**Available Builtin Tools:**

| Tool | Description |
|------|-------------|
| `tools.webSearch()` | Real-time internet search |
| `tools.codeRunner()` | Python code execution with matplotlib/pandas |
| `tools.quickjs()` | JavaScript execution via QuickJS engine |
| `tools.fetch()` | URL content fetching with markdown extraction |
| `tools.convert()` | Unit conversion (length, mass, temperature, currency) |
| `tools.date()` | Date/time processing and timezone conversion |
| `tools.base64Encode()` | Base64 encoding |
| `tools.base64Decode()` | Base64 decoding |
| `tools.memory()` | Memory storage and retrieval system |
| `tools.rethink()` | Intelligent reasoning/reflection tool |
| `tools.randomChoice()` | Random selection with optional weights |

**Capabilities:** Streaming, tool calling, structured output, thinking mode, image input, video input (experimental).

**Environment:** `MOONSHOT_API_KEY` or `KIMI_API_KEY`

## OpenResponses Provider

Connect to any server implementing the [OpenResponses specification](https://www.openresponses.org):

```typescript
import { llm } from '@providerprotocol/ai';
import { responses } from '@providerprotocol/ai/responses';

// Using with OpenAI
const model = llm({
  model: responses('gpt-5.2', {
    host: 'https://api.openai.com/v1',
    apiKeyEnv: 'OPENAI_API_KEY',
  }),
  params: { max_output_tokens: 1000 },
});

// Using with OpenRouter
const routerModel = llm({
  model: responses('openai/gpt-4o', {
    host: 'https://openrouter.ai/api/v1',
    apiKeyEnv: 'OPENROUTER_API_KEY',
  }),
});

// Using with self-hosted server
const localModel = llm({
  model: responses('llama-3.3-70b', {
    host: 'http://localhost:8080/v1',
  }),
});
```

**Capabilities:** Full multimodal support, streaming, tool calling, structured output, reasoning summaries.

## Alternative Import Style

Use the `ai` namespace for a grouped import style:

```typescript
import { ai } from '@providerprotocol/ai';
import { openai } from '@providerprotocol/ai/openai';

const model = ai.llm({ model: openai('gpt-4o') });
const embedder = ai.embedding({ model: openai('text-embedding-3-small') });
const dalle = ai.image({ model: openai('dall-e-3') });
```

## TypeScript

Full type safety with no `any` types. All provider parameters are typed:

```typescript
import type {
  // Core types
  Turn,
  TurnJSON,
  Message,
  Tool,
  ToolInput,
  TokenUsage,

  // Streaming
  StreamEvent,
  StreamResult,

  // Content blocks
  TextBlock,
  ImageBlock,
  ReasoningBlock,
  DocumentBlock,
  AudioBlock,
  VideoBlock,

  // Modality results
  EmbeddingResult,
  ImageResult,

  // Errors
  UPPError,
  ErrorCode,

  // Configuration
  ProviderConfig,
  KeyStrategy,
  RetryStrategy,
  LLMCapabilities,

  // Middleware
  Middleware,
  MiddlewareContext,
  StreamContext,

  // Schema types (Zod support)
  Structure,
  ZodLike,
} from '@providerprotocol/ai';
```

**Zod Utilities:**

```typescript
import {
  isZodSchema,
  isZodV4,
  zodToJSONSchema,
  zodToJSONSchemaSync,
  resolveStructure,
  resolveTools,
} from '@providerprotocol/ai/utils';

// Type guard for Zod schemas
if (isZodSchema(schema)) {
  const jsonSchema = zodToJSONSchemaSync(schema);
}
```

**Error & ID Utilities:**

```typescript
import {
  toError,
  isCancelledError,
  generateId,
  generateShortId,
} from '@providerprotocol/ai/utils';

// Convert unknown thrown values to Error instances
const error = toError(unknownValue);

// Check if an error is a cancellation/abort error
if (isCancelledError(error)) {
  console.log('Request was cancelled');
}

// Generate IDs
const uuid = generateId();                    // UUID v4: "a1b2c3d4-..."
const shortId = generateShortId('req');       // "req_abc123xyz789"
```

**Provider-Specific Types:**

```typescript
// OpenAI
import type { OpenAIHeaders, OpenAIImageParams } from '@providerprotocol/ai/openai';

// Google
import type { GoogleImagenParams } from '@providerprotocol/ai/google';

// Ollama
import type { OllamaHeaders } from '@providerprotocol/ai/ollama';

// OpenRouter
import type { OpenRouterProviderOptions } from '@providerprotocol/ai/openrouter';
```

**Type-Safe Enums:**

```typescript
import {
  StreamEventType,
  ErrorCode,
  ContentBlockType,
  MessageRole,
  ModalityType,
} from '@providerprotocol/ai';

// Use instead of magic strings
if (event.type === StreamEventType.TextDelta) { ... }
if (error.code === ErrorCode.RateLimited) { ... }
if (block.type === ContentBlockType.Text) { ... }
```

### Custom Providers

Build custom providers with `createProvider`:

```typescript
import { createProvider } from '@providerprotocol/ai';

const myProvider = createProvider({
  name: 'my-provider',
  version: '1.0.0',
  handlers: {
    llm: myLLMHandler,
    embedding: myEmbeddingHandler,
  },
});
```

## License

MIT