npm package: @mux/ai
Description: AI library for Mux
Author: Mux
Latest version: 0.15.1

# `@mux/ai`

[![npm version](https://badge.fury.io/js/@mux%2Fai.svg)](https://www.npmjs.com/package/@mux/ai)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

Easy to use, purpose-driven, cost effective, configurable **_workflow functions_** in a TypeScript SDK for building AI-powered video and audio workflows on the server, powered by [Mux](https://www.mux.com), with support for popular AI/LLM providers (OpenAI, Anthropic, Google).

- **Examples:** [`getSummaryAndTags`](#video-summarization), [`getModerationScores`](#content-moderation), [`hasBurnedInCaptions`](#burned-in-caption-detection), [`generateChapters`](#chapter-generation), [`generateEmbeddings`](#search-with-embeddings), [`generateEngagementInsights`](#engagement-insights), [`translateCaptions`](#caption-translation), [`editCaptions`](#caption-editing), [`translateAudio`](#audio-dubbing)
- Workflows automatically ship with `"use workflow"` [compatability with Workflow DevKit](#compatability-with-workflow-devkit)

Turn your Mux video and audio assets into structured, actionable data — summaries, chapters, moderation scores, translations, embeddings, and more — with a single function call. `@mux/ai` handles fetching media data from Mux, formatting it for AI providers, and returning typed results so you can focus on building your product instead of wrangling prompts and media pipelines.

## Quick Start

### Install

```bash
npm install @mux/ai
```

### Configure

Add your credentials to a `.env` file (we support [dotenv](https://www.npmjs.com/package/dotenv)):

```bash
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
OPENAI_API_KEY=your_openai_api_key          # or ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY
```

You only need credentials for the AI provider you're using. See the [Credentials guide](./docs/CREDENTIALS.md) for full setup details including signed playback, S3 storage, and all supported providers.

For multi-tenant apps or cases where you need to provide API keys at runtime rather than through environment variables, every workflow accepts a `credentials` option. You can also register a global credentials provider with `setWorkflowCredentialsProvider()` for dynamic key resolution (e.g. per-tenant secrets). When using [Workflow DevKit](https://useworkflow.dev), credentials can be [encrypted](./docs/WORKFLOW-ENCRYPTION.md) before crossing workflow boundaries so plaintext secrets never appear in serialized payloads.

### Run Your First Workflow

```ts
import { getSummaryAndTags } from "@mux/ai/workflows";

const result = await getSummaryAndTags("your-asset-id", {
  provider: "openai",
  tone: "professional",
  includeTranscript: true
});

console.log(result.title);        // "Getting Started with TypeScript"
console.log(result.description);  // "A comprehensive guide to..."
console.log(result.tags);         // ["typescript", "tutorial", "programming"]
```

> **⚠️ Note:** Many workflows rely on transcripts for best results. Consider enabling [auto-generated captions](https://www.mux.com/docs/guides/add-autogenerated-captions-and-use-transcripts) on your Mux assets to unlock the full potential of transcript-based workflows like summarization, chapters, and embeddings. This applies to both video and audio-only assets.

## Why `@mux/ai`?

- **Pre-built workflows for media AI tasks.** Common multi-step operations (transcript access, frame analysis, LLM calls, and structured parsing) are available as high-level functions.
- **Support for video and audio assets.** The same workflows work with video and [audio-only assets](./docs/AUDIO-ONLY.md), including summarization, moderation, chaptering, and more.
- **Provider-flexible API.** Choose OpenAI, Anthropic, or Google through workflow options while keeping the same workflow interface.
- **Published evaluation coverage.** Workflows include [evals](./docs/EVALS.md) for quality, latency, and cost, with results [published publicly](https://evaluating-mux-ai.vercel.app/) on pushes to `main`.
- **Sensible default models.** Defaults (`gpt-5.1`, `claude-sonnet-4-5`, `gemini-3-flash-preview`) are selected to balance output quality and runtime cost.
- **Typed end-to-end.** Workflow inputs, options, and outputs are fully typed in TypeScript.
- **Operational defaults included.** Retry handling, error handling, signed playback support, and [Workflow DevKit](https://useworkflow.dev) compatibility are built in.
- **Prompt customization support.** Use `promptOverrides` to adjust sections of workflow prompts for your domain or product requirements.
- **Composable abstractions.** Start with full workflows and drop down to lower-level primitives when you need more control.

## Workflows

Workflows are high-level functions that handle complete media AI tasks end-to-end — fetching data from Mux, calling AI providers, and returning structured results. Most workflows support both video and audio-only assets.

| Workflow | What it does | Providers | Audio-only |
| --- | --- | --- | :---: |
| [`getSummaryAndTags`](./docs/WORKFLOWS.md#video-summarization) | Generate titles, descriptions, and tags | OpenAI, Anthropic, Google | Yes |
| [`getModerationScores`](./docs/WORKFLOWS.md#content-moderation) | Detect inappropriate content | OpenAI, Hive | Yes |
| [`hasBurnedInCaptions`](./docs/WORKFLOWS.md#burned-in-caption-detection) | Detect hardcoded subtitles in video frames | OpenAI, Anthropic, Google | — |
| [`askQuestions`](./docs/WORKFLOWS.md#ask-questions) | Answer yes/no questions about asset content | OpenAI, Anthropic, Google | Yes |
| [`generateChapters`](./docs/WORKFLOWS.md#chapter-generation) | Create chapter markers from transcripts | OpenAI, Anthropic, Google | Yes |
| [`generateEmbeddings`](./docs/WORKFLOWS.md#embeddings) | Generate vector embeddings for semantic search | OpenAI, Google | Yes |
| [`translateCaptions`](./docs/WORKFLOWS.md#caption-translation) | Translate captions into other languages | OpenAI, Anthropic, Google | Yes |
| [`editCaptions`](./docs/WORKFLOWS.md#caption-editing) | Edit captions: censor profanity and/or apply static replacements | OpenAI, Anthropic, Google | Yes |
| [`translateAudio`](./docs/WORKFLOWS.md#audio-dubbing) | Create AI-dubbed audio tracks | ElevenLabs | Yes |

See the [Workflows guide](./docs/WORKFLOWS.md) for detailed documentation, options, and examples for each workflow. See the [API Reference](./docs/API.md) for complete parameter and return type details.

### Quick Examples

**Content moderation:**

```ts
import { getModerationScores } from "@mux/ai/workflows";

const result = await getModerationScores("your-asset-id", {
  provider: "openai",
  thresholds: { sexual: 0.7, violence: 0.8 }
});

if (result.exceedsThreshold) {
  console.log("Content flagged for review");
}
```

**Chapter generation:**

```ts
import { generateChapters } from "@mux/ai/workflows";

const result = await generateChapters("your-asset-id", {
  provider: "anthropic"
});

// [{ startTime: 0, title: "Introduction" }, { startTime: 45, title: "Main Content" }, ...]
```

**Caption editing:**

```ts
import { editCaptions } from "@mux/ai/workflows";

const result = await editCaptions("your-asset-id", "track-id", {
  provider: "anthropic",
  autoCensorProfanity: {
    mode: "blank", // "blank" ([____]), "remove", or "mask" (????)
    alwaysCensor: ["brandname"],
    neverCensor: ["damn"],
  },
  replacements: [
    { find: "Mucks", replace: "Mux" },
  ],
});

console.log(result.totalReplacementCount); // 8
console.log(result.autoCensorProfanity?.replacements); // [{cueStartTime: 5, before: "shit", after: "[____]"}, ...]
```

**Semantic search embeddings:**

```ts
import { generateEmbeddings } from "@mux/ai/workflows";

const result = await generateEmbeddings("your-asset-id", {
  provider: "openai",
  chunkingStrategy: { type: "token", maxTokens: 500, overlap: 100 }
});

for (const chunk of result.chunks) {
  await vectorDB.insert({ embedding: chunk.embedding, startTime: chunk.metadata.startTime });
}
```

## Prompt Customization

Every workflow prompt is built from a structured template of named sections. The `promptOverrides` option lets you swap out individual sections with your own instructions while keeping the battle-tested defaults for everything else — no need to rewrite entire prompts.

```ts
const result = await getSummaryAndTags(assetId, {
  provider: "openai",
  promptOverrides: {
    title: "Create a search-optimized title (50-60 chars) with the primary keyword front-loaded.",
    keywords: "Focus on high search volume terms and long-tail keyword phrases.",
    // task, description, qualityGuidelines → keep defaults
  },
});
```

This works with `getSummaryAndTags`, `generateChapters`, and `hasBurnedInCaptions`. The [Prompt Customization guide](./docs/PROMPT-CUSTOMIZATION.md) has ready-to-use presets for SEO, social media, e-commerce, and technical analysis, along with tips for writing effective overrides.

## Evaluations

Choosing between OpenAI, Anthropic, and Google for a given workflow isn't guesswork. Every workflow in `@mux/ai` ships with eval coverage that benchmarks providers and models against three dimensions:

- **Efficacy** — Does it produce accurate, high-quality results?
- **Efficiency** — How fast is it and how many tokens does it consume?
- **Expense** — What does each request cost?

Evals run automatically on every push to `main` and results are published to a **[public dashboard](https://evaluating-mux-ai.vercel.app/)** so you can compare providers side-by-side before choosing one for your use case.

You can also run evals locally against your own assets:

```bash
npm run test:eval
```

See the [Evaluations guide](./docs/EVALS.md) for details on the 3 E's framework, adding your own evals, and cross-provider testing.

## Primitives

Primitives are low-level building blocks that give you direct access to Mux media data — transcripts, storyboards, thumbnails, and text chunking utilities. Use them when you need full control over your AI prompts or want to build custom workflows.

```ts
import { fetchTranscriptForAsset, getStoryboardUrl } from "@mux/ai/primitives";

const transcript = await fetchTranscriptForAsset(asset, playbackId, { languageCode: "en" });
const storyboard = getStoryboardUrl(playbackId, 640);
```

All pre-built workflows are composed from these primitives internally, so you can always drop down a level when you need to customize behavior.

See the [Primitives guide](./docs/PRIMITIVES.md) for the full list of available functions and examples of building custom workflows.

## Package Structure

```ts
// Import specific workflows
import { getSummaryAndTags, generateChapters } from "@mux/ai/workflows";

// Import specific primitives
import { fetchTranscriptForAsset, getStoryboardUrl } from "@mux/ai/primitives";

// Or import everything via namespace
import { workflows, primitives } from "@mux/ai";
```

## Prerequisites

- [Node.js](https://nodejs.org/en/download) (>= 21.0.0)
- A [Mux](https://dashboard.mux.com/signup) account (free to sign up)
- An API key for at least one supported AI provider

## Documentation

| Guide | Description |
| --- | --- |
| [Workflows](./docs/WORKFLOWS.md) | Detailed guide for each pre-built workflow with examples and options |
| [API Reference](./docs/API.md) | Complete API docs — all function signatures, parameters, and return types |
| [Primitives](./docs/PRIMITIVES.md) | Low-level building blocks for custom workflows |
| [Prompt Customization](./docs/PROMPT-CUSTOMIZATION.md) | Overriding prompt sections with `promptOverrides` for custom use cases |
| [Credentials](./docs/CREDENTIALS.md) | Setting up Mux, AI provider, and cloud storage credentials |
| [Workflow DevKit](./docs/WORKFLOW-DEVKIT.md) | Integration with Workflow DevKit for observability and orchestration |
| [Workflow Encryption](./docs/WORKFLOW-ENCRYPTION.md) | Encrypting credentials across Workflow DevKit boundaries |
| [Storage Adapters](./docs/STORAGE-ADAPTERS.md) | Using custom storage SDKs (AWS, Cloudflare R2, MinIO) |
| [Audio-Only Workflows](./docs/AUDIO-ONLY.md) | Working with audio-only assets (no video track) |
| [Evaluations](./docs/EVALS.md) | AI eval testing with the 3 E's framework — [public dashboard](https://evaluating-mux-ai.vercel.app/) |
| [Examples](./docs/EXAMPLES.md) | Running the example scripts from the repository |

### Additional Resources

- [Mux Video API Docs](https://docs.mux.com/guides/video) — Learn about Mux Video features
- [Auto-generated Captions](https://www.mux.com/docs/guides/add-autogenerated-captions-and-use-transcripts) — Enable transcripts for your assets
- [npm Package](https://www.npmjs.com/package/@mux/ai) — Package page and version history

## Contributing

We welcome contributions! Please see the [Contributing Guide](./CONTRIBUTING.md) for details on setting up your development environment, running tests, and submitting pull requests.

For questions or discussions, feel free to [open an issue](https://github.com/muxinc/ai/issues).

## License

[Apache 2.0](LICENSE)