npm package: @loonylabs/ttv-middleware
Description: Provider-agnostic Text-to-Video middleware. Supports OpenAI Sora and Google Veo.
Author: loonylabs-dev
Latest version: 0.1.0

<div align="center">

# TTV Middleware

*Provider-agnostic Text-to-Video middleware with async polling, retry logic, and comprehensive error handling. Currently supports OpenAI Sora (Sora 2, Sora 2 Pro) and Google Veo (Veo 2, Veo 3, Veo 3.1). Features image-to-video, video extension, configurable progress callbacks, and optional auto-download.*

<!-- Horizontal Badge Navigation Bar -->
[![npm version](https://img.shields.io/npm/v/@loonylabs/ttv-middleware.svg?style=for-the-badge&logo=npm&logoColor=white)](https://www.npmjs.com/package/@loonylabs/ttv-middleware)
[![npm downloads](https://img.shields.io/npm/dm/@loonylabs/ttv-middleware.svg?style=for-the-badge&logo=npm&logoColor=white)](https://www.npmjs.com/package/@loonylabs/ttv-middleware)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.3+-blue.svg?style=for-the-badge&logo=typescript&logoColor=white)](#-features)
[![Node.js](https://img.shields.io/badge/Node.js-18+-339933?style=for-the-badge&logo=nodedotjs&logoColor=white)](#-prerequisites)
[![MIT License](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge&logo=opensource&logoColor=white)](#-license)
[![GitHub](https://img.shields.io/badge/GitHub-Repository-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/loonylabs-dev/ttv-middleware)

</div>

<!-- Table of Contents -->
<details>
<summary><strong>Table of Contents</strong></summary>

- [Features](#-features)
- [Quick Start](#-quick-start)
- [Prerequisites](#-prerequisites)
- [Configuration](#%EF%B8%8F-configuration)
- [Providers & Models](#-providers--models)
- [Image-to-Video](#-image-to-video)
- [Video Extension](#-video-extension)
- [API Reference](#-api-reference)
- [Advanced Features](#-advanced-features)
- [Testing](#-testing)
- [Contributing](#-contributing)
- [License](#-license)
- [Links](#-links)

</details>

---

## Features

- **Multi-Provider Architecture**: Unified API for all TTV providers
  - **OpenAI Sora**: Sora 2 & Sora 2 Pro with text-to-video and image-to-video
  - **Google Veo**: Veo 2, Veo 3, Veo 3.1 (+ fast variants) with video extension support
- **Async Polling**: Both APIs are asynchronous - the middleware handles polling internally with configurable intervals and exponential backoff
- **Image-to-Video**: Animate a still image into a video (both providers)
- **Video Extension**: Continue/extend an existing video (Google Veo)
- **Progress Callbacks**: Optional `onProgress` callback for real-time generation status updates
- **Auto-Download**: Optionally download generated videos to Buffer (`downloadToBuffer`)
- **Retry Logic**: Exponential backoff with jitter for transient errors (429, 408, 5xx, timeouts)
- **TypeScript First**: Full type safety with comprehensive interfaces
- **Logging Control**: Configurable log levels via environment or API
- **Debug Logging**: Markdown file logging for debugging prompts and responses
- **Error Handling**: Typed error classes including `PollingTimeoutError` and `ContentModeratedError`
- **Dry Mode**: Validate requests without API calls (no costs during development)

## Quick Start

### Installation

Install from npm:

```bash
npm install @loonylabs/ttv-middleware

# For OpenAI Sora provider:
npm install openai

# For Google Veo provider:
npm install google-auth-library
```

Or install directly from GitHub:

```bash
npm install github:loonylabs-dev/ttv-middleware
```

### Basic Usage

```typescript
import { TTVService, OpenAISoraProvider, TTVProvider } from '@loonylabs/ttv-middleware';

// Create service and register provider
const service = new TTVService();
service.registerProvider(new OpenAISoraProvider({
  apiKey: process.env.OPENAI_API_KEY,
}));

// Generate a video
const result = await service.generate({
  prompt: 'A cat sitting on a windowsill watching rain fall outside',
  model: 'sora-2',
  duration: 8,         // 8 seconds
  aspectRatio: '16:9',
});

console.log('Video URL:', result.videos[0].url);
console.log('Duration:', result.metadata.duration, 'ms (total)');
console.log('Generation time:', result.metadata.generationTime, 'ms');
```

<details>
<summary><strong>Using Google Veo</strong></summary>

```typescript
import { TTVService, GoogleVeoProvider, TTVProvider } from '@loonylabs/ttv-middleware';

const service = new TTVService();
service.registerProvider(new GoogleVeoProvider({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  location: 'us-central1',
}));

const result = await service.generate({
  prompt: 'A cinematic aerial shot of a mountain range at sunrise',
  model: 'veo-3.0-generate-001',
  duration: 8,
  aspectRatio: '16:9',
  resolution: '1080p',
  generateAudio: true,  // Veo 3+ generates audio natively
});

console.log('Video URL:', result.videos[0].url);
```

</details>

<details>
<summary><strong>With Progress Callback</strong></summary>

```typescript
const result = await service.generate({
  prompt: 'A futuristic city with flying cars',
  model: 'sora-2',
  duration: 12,
  onProgress: (progress) => {
    console.log(`Status: ${progress.status}`, progress.message || '');
    // Status: queued
    // Status: in_progress
    // Status: completed
  },
});
```

</details>

<details>
<summary><strong>Download to Buffer</strong></summary>

```typescript
import * as fs from 'fs';

const result = await service.generate({
  prompt: 'Ocean waves crashing on a rocky shore',
  model: 'sora-2',
  downloadToBuffer: true,  // Download video into memory
});

// Save to disk
fs.writeFileSync('output.mp4', result.videos[0].buffer!);
```

</details>

<details>
<summary><strong>Switching Providers</strong></summary>

```typescript
// Use OpenAI Sora
const soraResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'sora-2',
}, TTVProvider.OPENAI_SORA);

// Use Google Veo
const veoResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'veo-3.0-generate-001',
}, TTVProvider.GOOGLE_VEO);
```

</details>

## Prerequisites

<details>
<summary><strong>Required Dependencies</strong></summary>

- **Node.js** 18+
- **TypeScript** 5.3+

For OpenAI Sora provider:
```bash
npm install openai
```

For Google Veo provider:
```bash
npm install google-auth-library
```

</details>

## Configuration

<details>
<summary><strong>Environment Setup</strong></summary>

Create a `.env` file in your project root:

```env
# Default provider
TTV_DEFAULT_PROVIDER=openai-sora

# Logging level (debug, info, warn, error, silent)
TTV_LOG_LEVEL=info

# OpenAI Sora
OPENAI_API_KEY=sk-...

# Google Veo (Vertex AI)
GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=./service-account.json
GOOGLE_CLOUD_REGION=us-central1
```

</details>

## Providers & Models

### OpenAI Sora

| Model | ID | Duration | Resolutions | Audio | Image-to-Video |
|-------|-----|----------|-------------|-------|----------------|
| **Sora 2** | `sora-2` | 4, 8, 12s | 720p, 1080p | Yes | Yes |
| **Sora 2 Pro** | `sora-2-pro` | 10, 15, 25s | 720p, 1080p | Yes | Yes |

**Pricing:** ~$0.10-$0.50/sec depending on model and resolution.

### Google Veo

| Model | ID | Duration | Resolutions | Audio | Video Extension |
|-------|-----|----------|-------------|-------|-----------------|
| **Veo 2** | `veo-2.0-generate-001` | 5-8s | 720p | No | Yes |
| **Veo 3** | `veo-3.0-generate-001` | 4-8s | 720p, 1080p | Yes | Yes |
| **Veo 3 Fast** | `veo-3.0-fast-generate-001` | 4-8s | 720p, 1080p | Yes | Yes |
| **Veo 3.1** | `veo-3.1-generate-001` | 4-8s | 720p, 1080p, 4K | Yes | Yes |
| **Veo 3.1 Fast** | `veo-3.1-fast-generate-001` | 4-8s | 720p, 1080p, 4K | Yes | Yes |

**Pricing:** ~$0.15-$0.75/sec depending on model.

## Image-to-Video

Animate a still image into a video. Both providers support this:

```typescript
import * as fs from 'fs';

// Load a reference image
const imageBase64 = fs.readFileSync('character.png').toString('base64');

// OpenAI Sora: image becomes first frame
const result = await service.generate({
  prompt: 'She turns around and smiles, then slowly walks out of frame',
  model: 'sora-2',
  referenceImage: {
    base64: imageBase64,
    mimeType: 'image/png',
  },
  duration: 8,
});

// Google Veo: first frame + optional last frame for interpolation
const veoResult = await service.generate({
  prompt: 'Camera slowly zooms out revealing the landscape',
  model: 'veo-3.0-generate-001',
  referenceImage: {
    base64: firstFrameBase64,
    mimeType: 'image/png',
  },
  lastFrameImage: {         // Veo-specific: interpolate between two keyframes
    base64: lastFrameBase64,
    mimeType: 'image/png',
  },
}, TTVProvider.GOOGLE_VEO);
```

## Video Extension

Extend an existing video (Google Veo only):

```typescript
const extended = await service.extend({
  prompt: 'The camera continues to pan revealing a hidden waterfall',
  videoBuffer: existingVideoBuffer,
  videoMimeType: 'video/mp4',
  duration: 7,  // Extend by 7 seconds
  downloadToBuffer: true,
}, TTVProvider.GOOGLE_VEO);

fs.writeFileSync('extended.mp4', extended.videos[0].buffer!);
```

## API Reference

### TTVService

```typescript
class TTVService {
  registerProvider(provider: BaseTTVProvider): void;
  generate(request: TTVRequest, provider?: TTVProvider): Promise<TTVResponse>;
  extend(request: TTVExtendRequest, provider?: TTVProvider): Promise<TTVResponse>;
  getProvider(name: TTVProvider): BaseTTVProvider | undefined;
  getAvailableProviders(): TTVProvider[];
  listAllModels(): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
  findProvidersWithCapability(capability: keyof TTVCapabilities): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
}
```

### TTVRequest

```typescript
interface TTVRequest {
  prompt: string;
  model?: string;              // 'sora-2', 'veo-3.0-generate-001', etc.
  duration?: number;           // Desired duration in seconds
  aspectRatio?: string;        // '16:9', '9:16'
  resolution?: '720p' | '1080p' | '4k';
  n?: number;                  // Number of videos (default: 1)

  // Image-to-video
  referenceImage?: TTVReferenceImage;
  lastFrameImage?: TTVReferenceImage;  // Google Veo only

  // Audio & content
  generateAudio?: boolean;
  negativePrompt?: string;

  // Output control
  downloadToBuffer?: boolean;  // Download video to Buffer (default: false)
  onProgress?: TTVProgressCallback;

  // Retry & debug
  retry?: boolean | RetryOptions;
  dry?: boolean;

  providerOptions?: Record<string, unknown>;
}
```

### TTVResponse

```typescript
interface TTVResponse {
  videos: TTVVideo[];
  metadata: {
    provider: string;
    model: string;
    region?: string;
    duration: number;          // Total request time (ms)
    generationTime?: number;   // Polling time only (ms)
  };
  usage: {
    videosGenerated: number;
    totalDurationSeconds: number;
    modelId: string;
  };
  billing?: {
    cost: number;
    currency: string;
    source: 'provider' | 'estimated';
  };
}
```

### TTVVideo

```typescript
interface TTVVideo {
  url?: string;           // Video URL (Sora) or undefined (Veo returns buffer directly)
  buffer?: Buffer;        // Video data (if downloadToBuffer)
  contentType: string;    // 'video/mp4'
  duration?: number;      // Duration in seconds
}
```

## Advanced Features

<details>
<summary><strong>Polling Configuration</strong></summary>

Video generation is asynchronous. The middleware polls for completion automatically. You can configure the polling behavior:

```typescript
import { GoogleVeoProvider } from '@loonylabs/ttv-middleware';

const provider = new GoogleVeoProvider({
  projectId: 'my-project',
  polling: {
    intervalMs: 10000,       // Start polling every 10s (default)
    maxIntervalMs: 30000,    // Cap at 30s between polls
    backoffMultiplier: 1.5,  // Increase interval by 1.5x each time
    timeoutMs: 600000,       // Give up after 10 minutes (default)
  },
});
```

| Option | Default | Description |
|--------|---------|-------------|
| `intervalMs` | 10000 | Initial polling interval (ms) |
| `maxIntervalMs` | 30000 | Maximum polling interval (ms) |
| `backoffMultiplier` | 1.5 | Multiplier per poll attempt |
| `timeoutMs` | 600000 | Maximum wait time (10 minutes) |

</details>

<details>
<summary><strong>Retry Configuration</strong></summary>

Automatic retry with **exponential backoff and jitter** for transient errors (429, 408, 5xx, network timeouts):

```typescript
// Default: 3 retries, exponential backoff (1s -> 2s -> 4s), jitter enabled
const result = await service.generate({
  prompt: 'A sunset over mountains',
  // retry: true (default)
});

// Custom retry configuration
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: {
    maxRetries: 5,
    delayMs: 1000,
    backoffMultiplier: 2.0,
    maxDelayMs: 30000,
    jitter: true,
  },
});

// Disable retry
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: false,
});
```

**Retryable errors:** 429, 408, 500, 502, 503, 504, timeouts, ECONNRESET, ECONNREFUSED, socket hang up
**Not retried:** 400, 401, 403, and other client errors

</details>

<details>
<summary><strong>Logging Configuration</strong></summary>

Control logging via environment variable or API:

```typescript
import { setLogLevel } from '@loonylabs/ttv-middleware';

// Set log level programmatically
setLogLevel('warn');  // Only show warnings and errors

// Or via environment variable
// TTV_LOG_LEVEL=error
```

Available levels: `debug`, `info`, `warn`, `error`, `silent`

</details>

<details>
<summary><strong>Debug Logging (Markdown Files)</strong></summary>

Log all TTV requests and responses to markdown files for debugging:

```typescript
import { TTVDebugger } from '@loonylabs/ttv-middleware';

// Enable via environment variable
// DEBUG_TTV_REQUESTS=true

// Or programmatically
TTVDebugger.setEnabled(true);
TTVDebugger.setLogsDir('./logs/ttv/requests');

TTVDebugger.configure({
  enabled: true,
  logsDir: './logs/ttv/requests',
  consoleLog: true,
});
```

</details>

<details>
<summary><strong>Error Handling</strong></summary>

Typed error classes for precise error handling:

```typescript
import {
  TTVError,
  InvalidConfigError,
  QuotaExceededError,
  ProviderUnavailableError,
  GenerationFailedError,
  NetworkError,
  CapabilityNotSupportedError,
  PollingTimeoutError,
  ContentModeratedError,
} from '@loonylabs/ttv-middleware';

try {
  const result = await service.generate({ prompt: 'test', duration: 8 });
} catch (error) {
  if (error instanceof PollingTimeoutError) {
    console.log('Video generation timed out - try again or increase timeout');
  } else if (error instanceof ContentModeratedError) {
    console.log('Content was blocked by safety filters');
  } else if (error instanceof QuotaExceededError) {
    console.log('Rate limit hit, try again later');
  } else if (error instanceof CapabilityNotSupportedError) {
    console.log('Model does not support this feature');
  } else if (error instanceof TTVError) {
    console.log(`TTV Error [${error.code}]: ${error.message}`);
  }
}
```

</details>

<details>
<summary><strong>Dry Mode</strong></summary>

Test your integration without making API calls or incurring costs:

```typescript
const result = await service.generate({
  prompt: 'A test video',
  duration: 8,
  dry: true,  // No API call, returns placeholder response
});

console.log(result.videos.length);       // 1
console.log(result.metadata.duration);   // 0 (no actual generation)
```

</details>

<details>
<summary><strong>Provider-Specific Options</strong></summary>

Use `providerOptions` as an escape hatch for provider-specific features:

```typescript
// Google Veo: seed for deterministic output
const result = await service.generate({
  prompt: 'A sunset timelapse',
  model: 'veo-3.0-generate-001',
  providerOptions: {
    seed: 42,
    personGeneration: 'allow_adult',
    enhancePrompt: true,
    storageUri: 'gs://my-bucket/output/',  // Direct output to GCS
  },
}, TTVProvider.GOOGLE_VEO);

// OpenAI Sora: remix (reinterpret an existing video)
// Use the Sora API directly for remix via providerOptions
```

</details>

## Testing

```bash
# Run all tests
npm test

# Unit tests only
npm run test:unit

# Unit tests with watch mode
npm run test:unit:watch

# Unit tests with coverage report
npm run test:unit:coverage

# Integration tests (requires TTV_INTEGRATION_TESTS=true)
npm run test:integration

# CI/CD mode
npm run test:ci
```

### Manual Smoke Tests

The `scripts/` directory contains manual smoke tests for real API calls:

```bash
# Text-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo.ts

# Image-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo-i2v.ts
```

Both scripts use `veo-3.0-fast-generate-001` (cheapest model), generate short 4s/720p videos, and save output to the `output/` directory. Requires Google Veo credentials in `.env`.

### Integration Tests

Integration tests make real API calls and **cost money**. They are **skipped by default**.

```bash
# Enable and run integration tests
TTV_INTEGRATION_TESTS=true npm run test:integration
```

**Prerequisites:**
- `OPENAI_API_KEY` for Sora tests
- `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS` for Veo tests

## Contributing

We welcome contributions! Please ensure:

1. **Tests:** Add tests for new features
2. **Linting:** Run `npm run lint` before committing
3. **Conventions:** Follow the existing project structure

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Links

- [Documentation](https://github.com/loonylabs-dev/ttv-middleware/docs)
- [Issues](https://github.com/loonylabs-dev/ttv-middleware/issues)
- [NPM Package](https://www.npmjs.com/package/@loonylabs/ttv-middleware)

---

<div align="center">

**Made with care by the LoonyLabs Team**

[![GitHub stars](https://img.shields.io/github/stars/loonylabs-dev/ttv-middleware?style=social)](https://github.com/loonylabs-dev/ttv-middleware/stargazers)
[![Follow on GitHub](https://img.shields.io/github/followers/loonylabs-dev?style=social&label=Follow)](https://github.com/loonylabs-dev)

</div>