npm package: @sanity-labs/resound
Description: Audio production for video. Track-based, markdown-driven, Remotion-ready.
Author: Sanity Labs
Latest version: 0.1.0

# Resound

Audio production for video. Track-based, markdown-driven, Remotion-ready.

> **Requires an [ElevenLabs](https://elevenlabs.io) API key.** Set `ELEVENLABS_API_KEY` in your environment or add `api_key` to your config file.

## What is Resound?

Resound is a CLI tool that generates audio for video projects. You define tracks in markdown files, and Resound generates audio files with timing data that integrates seamlessly with [Remotion](https://remotion.dev).

**Key features:**
- **Track-based workflow** — Separate files for dialog, sound effects, and music
- **Markdown-driven** — Write dialog naturally, define sounds with prompts
- **Incremental generation** — Only regenerate what changed
- **Remotion integration** — Outputs manifest.json with markers and timing
- **Self-documenting** — Run `resound` for full help, `resound --help <topic>` for details

## Quick Start

```bash
# In a Remotion project (auto-detects and configures)
npx create-video@latest my-video
cd my-video
npm install @sanity-labs/resound
npx resound init      # Creates config, tracks/, and src/audio/
npx resound generate  # Generate audio
```

`resound init` auto-detects Remotion projects and scaffolds everything:
- `resound.config.yaml` with output to `public/audio/`
- `src/audio/` with pre-loaded manifest and ready-to-use components

## Project Structure

```
my-video/
├── resound.config.yaml       # Project config
├── tracks/
│   ├── 01-dialog.md          # Multi-speaker dialog
│   ├── 02-sfx.md             # Sound effects
│   └── 03-music.md           # Background music
└── public/audio/             # Generated output
    ├── manifest.json         # Timing + markers for Remotion
    ├── dialog.mp3
    ├── dialog.timing.json    # Word-level timestamps
    ├── sfx.mp3
    └── music.mp3
```

## Track Types

### Dialog

Multi-speaker conversation with natural turn-taking:

```markdown
---
type: dialog
voices:
  sarah: EXAVITQu4vr4xnSDxMaL
  bob: iP95p4xoKVk53GoZ742B
---

@sarah: Welcome to the demo! {#intro}

@bob: [excited] This looks amazing!

<!-- pause: 0:00.5 -->

@sarah: Let me show you how it works. {#demo-start}
```

### Sound Effects

Define sounds once, place them by name:

```markdown
---
type: sfx
sounds:
  # Simple: just a prompt
  whoosh: "smooth digital transition sound"
  pop: "soft notification pop"
  
  # With duration: for ambiance, typing, loops
  typing:
    prompt: "keyboard typing, mechanical keys"
    duration: 5.0
---

0:00           whoosh
intro+0.5      pop
demo-begins    typing    # 5 seconds of typing
demo-start     whoosh
```

### Music

Background music with fades and ducking:

```markdown
---
type: music
volume: 0.3
fade_in: 0:02
fade_out: 0:03
duck_to: 0.15
---

Gentle ambient electronic, soft pads, optimistic mood.
```

## Remotion Integration

`resound init` scaffolds `src/audio/` with everything pre-configured:

```tsx
// Simple usage - everything pre-loaded
import { Audio, manifest, useTiming } from './audio';
import { loadTiming } from 'resound/remotion';
import dialogTimingJson from '../public/audio/dialog.timing.json';

const dialogTiming = loadTiming(dialogTimingJson);

export const MyVideo = () => {
  const { currentLine, visibleLines } = useTiming(dialogTiming);
  
  return (
    <>
      <Audio />
      {/* Your visuals synced to currentLine */}
    </>
  );
};
```

### Available Hooks

| Hook | Description |
|------|-------------|
| `useMarker(manifest, name)` | Get a marker's frame number |
| `useMarkers(manifest)` | Get all markers as `{ name: frame }` |
| `useTiming(timing)` | Word-level timing for streaming text effects |
| `useDucking(manifest, options)` | Calculate volume with auto-ducking |
| `useIsDucking(manifest)` | Check if currently in a ducking range |

### Type-Safe Loaders

```tsx
import { loadManifest, loadTiming } from 'resound/remotion';
import manifestJson from '../public/audio/manifest.json';
import dialogTimingJson from '../public/audio/dialog.timing.json';

const manifest = loadManifest(manifestJson);
const timing = loadTiming(dialogTimingJson);
```

## Commands

```bash
resound                       # Show help
resound --help <topic>        # Detailed help (dialog, sfx, music, markers, remotion)

resound init                  # Create new project
resound new <type> <name>     # Create track from template
resound status                # Show project state
resound generate              # Generate audio (incremental)
resound generate --force      # Regenerate everything
resound markers               # List all markers
resound voices                # List available voices
```

## Configuration

```yaml
# resound.config.yaml
name: my-project
fps: 30
output: public/audio

voices:
  sarah: EXAVITQu4vr4xnSDxMaL
  bob: iP95p4xoKVk53GoZ742B

# Or use ELEVENLABS_API_KEY env var
# api_key: sk_...
```

## Requirements

- Node.js 18+
- ElevenLabs API key

## License

MIT