Skill: fridaylifan/senseaudio-video-gen (version 1.0.0)

--- name: senseaudio-video-gen description: "Use when the user asks to create, inspect, render, or repair an HTML-authored video from a brief, website, Markdown/text file, or GitHub repository; needs captions, voiceover, background music, generated images/video clips, or a HyperFrames-like local video pipeline using SenseAudio media APIs and AudioClaw LLM planning." --- # SenseAudio Video Gen Author videos as HTML compositions, preview them in a browser, render them locally through Chrome screenshots plus FFmpeg, plan scripts/storyboards with AudioClaw by default, and generate supporting media through SenseAudio APIs. Treat HTML as the editable source of truth, SenseAudio as the media engine, and AudioClaw as the default LLM route. ## Default Setup On a new machine, configure the media API and LLM API separately: ```bash export SENSEAUDIO_API_KEY="..." ``` `SENSEAUDIO_API_KEY` powers only SenseAudio media APIs: TTS, ASR, image, video, and music. AudioClaw LLM planning uses a separate OpenAI-compatible route. If running inside AudioClaw, no extra LLM env is needed when the local AudioClaw config file exists. Otherwise set the LLM env explicitly: ```bash export AUDIOCLAW_CONFIG_PATH="config/audioclaw.json" export AUDIOCLAW_LLM_MODEL="doubao-seed-2-0-pro-260215" export AUDIOCLAW_LLM_BASE_URL="https://platform.senseaudio.cn/v1" export AUDIOCLAW_LLM_API_KEY="..." ``` LLM config precedence is CLI flags, then `AUDIOCLAW_LLM_*`, then `AUDIOCLAW_CONFIG_PATH` or the local AudioClaw config file. The CLI deliberately does not reuse `SENSEAUDIO_API_KEY` as an AudioClaw LLM key. Use `--llm none` for deterministic heuristic planning, or `--offline` to skip live media calls. ## Core Loop Start from a brief when the user wants a complete project shell: ```bash python3 scripts/senseaudio_video_gen.py compose \ --project my-video \ --brief "Make a premium launch film for a new AI research assistant." \ --duration 12 \ --style-preset executive-film ``` `compose` is the general video path for product launch films, feature explainers, report summaries, technical walkthroughs, title cards, social cuts, and branded motion pieces. It defaults to `executive-film`, a restrained cinematic style with large typography, letterbox framing, low ornament, and non-web chapter IDs. Use `--offline` when drafting without live API calls, and add `--render` when the project should be rendered immediately. `compose` now defaults to `--llm audioclaw`. If the default LLM route is unavailable, `--llm-fallback` keeps the project moving with heuristic planning and records a warning in `senseframe.json`. Pass `--no-llm-fallback` when an LLM failure should stop the run. For the closest HyperFrames-style website workflow, prefer the one-pass `site-video` pipeline: ```bash python3 scripts/senseaudio_video_gen.py site-video \ --url https://www.anthropic.com/ \ --project anthropic-site-video \ --brief "用中文介绍 Anthropic 官网的 Claude、安全 AI、研究与企业能力。" \ --duration 14 \ --fps 30 \ --llm audioclaw ``` `site-video` defaults to `editorial-pro`, layered beats, cinematic motion, GSAP-compatible timing, real website screenshots, AudioClaw LLM planning, SenseAudio narration/ASR when live media is enabled, audio-reactive data, local rendering, inspect frames, local frame-quality audit, and motion audits. If LLM planning fails, `--llm-fallback` retries with heuristic planning and records the warning. Use `--offline --no-render` for a safe draft that writes the same editable project structure and `pipeline-report.json`. Add `--music --music-poll` when the site video should request a SenseAudio music bed, download it, mix it under narration as `assets/final-audio.m4a`, and render with that mixed track. If SenseAudio accepts the task but does not return `audio_url` in time, `--music-fallback` creates a local ambient bed so the video still ships with background music while preserving the task manifest. Use `--music-dry-run` or `--offline` to inspect the `/music/song/create` payload without spending credits. Add `--auto-repair` when the project should run a second pass after motion/vision audits, tighten real screenshot crops, damp busy overlays, and rerender the repaired composition. Website capture follows the useful parts of the HyperFrames loop: warm the live page, dismiss common cookie/modals, scroll to trigger lazy assets, record `assets/site-capture-quality.json`, capture `renders/inspect` frames, and write `renders/inspect/contact-sheet.html` for review. Use `--vision-audit` when a live VL model should judge the rendered frames; the local `frame-quality-audit` still runs by default when rendering. For gated, cookie-sensitive, or region-personalized sites, keep browser state explicit: ```bash python3 scripts/senseaudio_video_gen.py site-video \ --url https://example.com/ \ --project example-site-video \ --browser-profile profiles/example-capture \ --cookie-file cookies/example.json ``` Cookies often make screenshots closer to what a real user sees, but the clean temporary browser remains the default to avoid leaking private account pages into generated videos. For URL-to-video work, `site-ingest` classifies real page material into semantic roles such as hero, product, research, safety, developer, enterprise, customer, pricing, and CTA. These roles drive `story_evidence`, shot choice, composition mode, camera path, and `data-material-role` markers in the rendered HTML. Use `source-ingest` for first-stage non-web inputs. It converts local Markdown/text files or a GitHub repository README into the same `site-profile.json` shape used by website projects, so `compose --site-file <profile.json>` can reuse storyboard, narration, semantic role, and production-spec logic without a separate document pipeline: ```bash python3 scripts/senseaudio_video_gen.py source-ingest \ --file product-notes.md \ --output product-notes.site.json \ --json python3 scripts/senseaudio_video_gen.py source-ingest \ --github-url heygen-com/hyperframes \ --output hyperframes-readme.site.json ``` Use `site-vision-plan` when screenshot crops need to be planned before rendering. The default `heuristic` provider derives crop center, zoom, pan, and focus from DOM highlights and semantic roles. `--provider openrouter` builds an OpenRouter-compatible vision request so a VL model can inspect screenshots first; keep `--fallback` enabled so rendering degrades to deterministic crops if the model route is unavailable. Use the music and repair commands directly when tuning an existing project: ```bash python3 scripts/senseaudio_video_gen.py music-create \ --prompt "Instrumental premium website explainer bed, subtle pulse, no vocals" \ --duration 16 \ --poll \ --download my-video/assets/background-music.mp3 \ --project my-video python3 scripts/senseaudio_video_gen.py mix-audio \ --project my-video \ --voice my-video/assets/narration.mp3 \ --music my-video/assets/background-music.mp3 \ --output my-video/assets/final-audio.m4a \ --duration 16 python3 scripts/senseaudio_video_gen.py repair --project my-video --json ``` Use the default AudioClaw route for creative plans when the brief needs LLM-written copy and storyboard: ```bash python3 scripts/senseaudio_video_gen.py llm-plan \ --brief "Make a concise webpage intro for SenseAudio's sound library." \ --duration 9 \ --output my-plan.json python3 scripts/senseaudio_video_gen.py compose \ --project my-video \ --brief "Make a concise webpage intro for SenseAudio's sound library." \ --generate-images \ --generate-broll \ --asset-dry-run \ --offline ``` `llm-plan` defaults to `--provider audioclaw`. The skill strips LiteLLM-style provider prefixes such as `volcengine/` for `platform.senseaudio.cn` and retries without `response_format` for models that do not support JSON-mode requests. DeepSeek remains available with `--provider deepseek` or `--llm deepseek`; set `DEEPSEEK_API_KEY`, `DEEPSEEK_MODEL`, or `DEEPSEEK_BASE_URL` when using it. If the AudioClaw configured model is not strong enough for dense product research, switch planning to OpenRouter with `--provider openrouter` or `--llm openrouter`, and choose a capable model via `--model`, `--llm-model`, `OPENROUTER_LLM_MODEL`, or `OPENROUTER_MODEL`. Build an existing project as a local pipeline: ```bash python3 scripts/senseaudio_video_gen.py build --project my-video --dry-run python3 scripts/senseaudio_video_gen.py build --project my-video --output my-video/renders/final.mp4 ``` Or scaffold a blank composition: ```bash python3 scripts/senseaudio_video_gen.py init my-video --duration 6 --fps 24 cd my-video python3 ../scripts/senseaudio_video_gen.py preview . python3 ../scripts/senseaudio_video_gen.py inspect . --samples 5 python3 ../scripts/senseaudio_video_gen.py render . --output renders/final.mp4 ``` Use SenseAudio assets inside the same project: ```bash python3 scripts/senseaudio_video_gen.py tts \ --text "让声音、字幕和画面在一个视频项目里完成。" \ --voice-id male_0028_a \ --output my-video/assets/narration.mp3 python3 scripts/senseaudio_video_gen.py asr \ --file my-video/assets/narration.mp3 \ --timestamps word \ --output my-video/assets/transcript.json python3 scripts/senseaudio_video_gen.py captions \ --project my-video \ --transcript my-video/assets/transcript.json \ --output my-video/assets/captions.json python3 scripts/senseaudio_video_gen.py captions-export \ --captions my-video/assets/captions.json \ --format srt \ --output my-video/renders/final.srt python3 scripts/senseaudio_video_gen.py render my-video \ --audio my-video/assets/narration.mp3 \ --parallel 4 \ --resume \ --output my-video/renders/final-with-voice.mp4 python3 scripts/senseaudio_video_gen.py lint --project my-video --json python3 scripts/senseaudio_video_gen.py asset-report --project my-video --json python3 scripts/senseaudio_video_gen.py generate-assets \ --project my-video \ --image-prompt "clean product UI hero image for a sound library" \ --video-prompt "short b-roll of creators choosing voices" \ --dry-run python3 scripts/senseaudio_video_gen.py timeline \ --project my-video \ --preset cinematic ``` ## Composition Contract - Root scene uses `data-composition-id`, `data-width`, `data-height`, `data-duration`. - Timed clips use `data-start`, `data-duration`, optional `data-media-start`, and optional `data-scene`. - Timeline DSL uses `assets/timeline.json`, `data-timeline-source`, and optional `data-effect` presets such as `fade-up`, `slide-left`, `zoom-in`, `spotlight`, and `parallax`. - Visual styles are selected through the local registry with `styles` and `compose --style-preset`; tokens are embedded as CSS variables, written to `assets/style-preset.json`, and recorded in `senseframe.json`. - The optional `gsap-compat` timeline engine writes `labels` and `tracks` and uses a local `createGsapCompatTimeline` adapter; never load external GSAP/CDN code for deterministic renders. - Transition presets use `transition_preset` plus `transitions[]` in `assets/timeline.json`; supported presets include `editorial`, `glass`, `ribbon`, `iris`, and `luma`. - Storyboard ids must bind to real DOM scenes: `compose` maps each storyboard item to matching `data-scene` and `data-timeline-id` elements rather than fixed template beats. - Beat composition uses `assets/beats.json`, `.beat-layer`, and `data-beat` markers. `compose --beat-mode layered` splits each storyboard scene into hook/proof/detail/cta overlays so a single scene can carry multiple timed visual arguments. - Beat pacing is readability-first: defaults use 3 requested beats per scene, automatically clamp short scenes to keep each beat around one second or longer, and `motion-map` reports flashiness risk when beat/transition rates are too high. - Website explainers rotate dedicated shot layouts (`hero-overview`, `nav-scan`, `feature-zoom`, `trust-message`, `cta-summary`) so adjacent scenes do not reuse the same visual structure. - Composition templates use `data-composition-mode` values such as `full-bleed`, `split-scan`, `zoom-callout`, `evidence-board`, and `cta-lockup`; every website scene should also set a `data-camera-path` so the renderer can apply distinct camera motion instead of a repeated left/right card. - Brand extraction uses `brand-extract` or `compose --brand-url` to create `assets/brand.json`; brand name, description, nav labels, colors, logos/icons, social images, typography, keywords, and inferred voice should influence website explainer shots. - Site ingestion uses `site-ingest` or `compose --site-url` to create `assets/site-profile.json`; headings, sections, CTA labels, and evidence snippets should drive storyboard scenes and visual cards before generic brief fallback. - Source ingestion uses `source-ingest --file <notes.md|notes.txt>` or `source-ingest --github-url <owner/repo>` to create the same site-profile shape from Markdown, plain text, or GitHub README content. - Website screenshots use `site-capture` or `compose --site-screenshots` to capture scroll positions into `assets/site-screenshots/`; these screenshots should appear in website explainer shots as visual evidence with deterministic pan/zoom and evidence highlight boxes, not decorative stock media. - Video copy defaults to Chinese (`zh-CN`) for narration, captions, beat text, and storyboard intent unless the user explicitly requests another language. - Seekable motion uses `window.__timelines["main"]`; the runtime seeks registered timelines during frame capture so entrances, breathing motion, exits, chips, waveforms, and focus highlights render deterministically. - Audio-reactive motion uses smoothed `assets/audio-data.json` from `audio-data`; the runtime loads `data-audio-source` and maps RMS/bands to local mesh intensity, card glow, waveform motion, and transition light. Do not drive the global camera directly from raw audio. - Caption containers use `data-caption-source="./assets/captions.json"` or inline `window.__sfCaptions`; `captions --include-words` enables active word highlighting with `.sf-word[data-sf-active="true"]`. - Word captions use kinetic karaoke states: active words scale, emphasized terms receive `sf-word-emphasis`, and styling must remain deterministic with no CSS animation loops. - Final renders are silent unless `render --audio <file>` is provided or `build` finds a registered audio asset such as `narration`; `lint` warns when narration text exists but no audio asset is registered. - Put deterministic frame logic in `window.renderFrame(time)`. - Run `motion-audit --project <dir> --strict` after composing to catch storyboard/DOM/timeline mismatches and legacy fixed-template markers. - Run `motion-map --project <dir> --strict` before expensive renders to score motion density, scene coverage, low-motion zones, transitions, and audio-reactive binding. - The runtime sets `window.__senseframes.time`, CSS variable `--sf-time`, `data-sf-active`, `data-scene-active`, and dispatches `sf-seek`. - Use CSS for final layout first, then animate from `renderFrame(time)`; avoid wall-clock animation for rendered output. ## Workflow 1. **Plan** — define aspect ratio, duration, audience, scenes, copy, voice, and output target. 2. **Scaffold** — run `compose` for brief-to-project or `init` for a blank source; use `--beat-mode layered` when the video needs dense HyperFrames-style scene internals. 3. **Generate media** — use `voices`, `tts`, `asr`, `captions`, `image-sync`, or `video-create` to produce assets. 4. **Register assets** — use `asset-add` or command manifests so `assets/asset-manifest.json` and `senseframe.json` stay current. 5. **Lint and audit** — run `lint`, `motion-audit`, and `motion-map` to catch missing assets, mismatched scenes, and flat motion. 6. **Inspect frames** — run `inspect` before rendering to catch layout, legibility, and timing issues. 7. **Render** — run `render` or `build`; mux narration with `--audio` when needed. 8. **Deliver** — return the MP4, render report JSON, subtitles, `senseframe.json`, transcripts, prompts, and asset manifests. ## Command Map | Task | Command | Purpose | | --- | --- | --- | | One-pass website video | `site-video --url <site>` | Ingest, plan, capture, narrate, bind audio data, render, and audit in one pipeline | | LLM plan | `llm-plan` / `llm-plan --provider openrouter` | Generate title, narration, visual style, and storyboard JSON; defaults to AudioClaw | | Brief to project | `compose --project <dir> --brief ...` | Create storyboard, narration script, caption scaffold, HTML, and manifests; defaults to AudioClaw with heuristic fallback | | Brand extraction | `brand-extract --url <site>` | Extract brand identity, colors, nav, logos/icons, typography, keywords, and voice | | Site ingestion | `site-ingest --url <site>` | Extract real headings, sections, CTA labels, and evidence snippets for URL-to-video | | Source ingestion | `source-ingest --file <md/txt>` / `--github-url <repo>` | Convert Markdown, text, or GitHub README content into a reusable `site-profile.json` | | Site screenshots | `site-capture --url <site>` | Capture real scroll screenshots with Chrome, warm lazy content, clean overlays, and register visual evidence | | Frame quality | `frame-quality-audit --project <dir>` | Check inspect/site frames for blank captures and leaked planning copy | | Visual crop plan | `site-vision-plan --project <dir>` | Plan screenshot crop, zoom, pan, and focus before rendering | | Beat layers | `beats --project <dir>` | Split storyboard scenes into hook/proof/detail/cta timed overlays | | Local pipeline | `build --project <dir>` | Run lint, create captions when a transcript exists, and render | | Generated assets | `generate-assets --project <dir>` | Plan or call SenseAudio image/video generation and register results | | Project validation | `lint --project <dir>` | Check entry HTML, runtime, caption sources, timing, and asset existence | | Style registry | `styles --json` | List built-in visual presets and recommended motion defaults | | Motion audit | `motion-audit --project <dir>` | Check storyboard scene binding, beat layers, transition layer, audio-reactive hooks, timeline registry, and legacy markers | | Motion map | `motion-map --project <dir>` | Score motion density, scene/beat coverage, flashiness risk, transition coverage, dead zones, and audio-reactive binding | | Audio data | `audio-data --audio <file> --output assets/audio-data.json` | Extract frame-level RMS/band data and bind it with `data-audio-source` | | Scaffold | `init <dir>` | Create `index.html`, runtime, manifest, assets, renders | | Preview | `preview <dir>` | Serve project for browser review | | Inspect | `inspect <dir>` | Capture timestamped sample frames | | Timeline | `timeline --project <dir> --timeline-engine gsap-compat` | Generate animation tracks, labels, transitions, and bind them to the runtime | | Render | `render <dir>` | Convert HTML frames to MP4 locally with optional `--parallel`, `--resume`, and `--frame-dir` | | Voiceover | `tts` | Generate narration from SenseAudio TTS | | Transcript | `asr --timestamps word` | Produce transcript timing for captions | | Captions | `captions --transcript ...` | Convert ASR JSON into `assets/captions.json` | | Subtitle files | `captions-export` | Export captions JSON to `.srt` or `.vtt` | | Asset registry | `asset-add` | Register local/generated assets in the project manifest | | Asset inventory | `asset-report` | List registered assets and missing files | | Still assets | `image-sync` | Generate first frames, backdrops, thumbnails | | Model clips | `video-create` / `video-status` | Generate AI video clips through SenseAudio | | Voices | `voices --voice-type all` | Discover usable `voice_id` values | ## Non-Negotiable Rules - Never rely on CSS `animation`, `setInterval`, or wall-clock playback for final renders; tie motion to `time`. - Do not use remote video generation when an HTML composition can express the exact UI/layout; use SenseAudio video generation for generative inserts, references, or b-roll. - Do not invent voice IDs. Query `voices` or use a user-provided voice. - Keep `senseframe.json` updated with generated assets, task IDs, transcripts, and final output paths. - For subtitles, use `captions` to convert ASR words into grouped caption cues before authoring caption elements. - If a local audio/video file must guide SenseAudio model video generation, upload it somewhere first; model video content fields require URLs. ## References - `references/authoring.md` — HTML composition patterns and timing rules. - `references/renderer.md` — local renderer requirements and troubleshooting. - `references/media-pipeline.md` — SenseAudio asset pipeline. - `references/api.md` — endpoint and model parameter summary. - `examples/starter-html-video` — minimal editable composition project.

path: requirements.txt
content: # No third-party Python packages are required.
# Runtime tools: Python 3.9+, Chrome or Chromium, and FFmpeg.


path: references/workflows.md
content: # Production Workflows

## Brief To HTML Video Project

1. For LLM-written copy and storyboard, run `llm-plan --provider deepseek --brief "..." --output plan.json`.
2. Run `styles --json` and choose a `--style-preset`, then run `compose --project <dir> --brief "..." --plan-file plan.json --beat-mode layered --offline` to create the editable project shell, or use `compose --llm deepseek` to call DeepSeek inline.
3. Review `index.html`, `assets/llm-plan.json`, `assets/narration.txt`, `assets/storyboard.json`, `assets/beats.json`, and `assets/captions.json`.
4. Add generated media slots with `compose --generate-images --generate-broll --asset-dry-run`, or run `generate-assets --dry-run` after composing.
5. Remove `--offline` and provide `--voice-id` when live SenseAudio TTS/ASR should generate final narration and captions.
6. Remove `--asset-dry-run` or run `generate-assets` without `--dry-run` when image/video credits should be used.
7. Run `beats --project <dir>` after storyboard edits, then run `timeline --project <dir> --preset cinematic --transition-preset editorial --timeline-engine gsap-compat` when scene boundaries and element tracks need stronger authored motion.
8. Run `lint --project <dir> --json` to catch missing assets or caption sources.
9. Run `motion-audit --project <dir> --strict` and `motion-map --project <dir> --strict` to catch mismatches, dead zones, and weak motion coverage.
10. Run `inspect` to catch visual timing issues.
11. Run `build --project <dir>` for the local pipeline, or `render --audio assets/narration.mp3` for manual control.
12. Export external subtitles with `captions-export --format srt` or `--format vtt` when the target platform needs a sidecar file.

## DeepSeek Planning

- Default base URL: `https://api.deepseek.com`
- Default model: `deepseek-v4-pro`
- Auth env var: `DEEPSEEK_API_KEY`
- Override with `DEEPSEEK_BASE_URL`, `DEEPSEEK_MODEL`, `--base-url`, `--model`, `--llm-base-url`, or `--llm-model`.

The planner asks for strict JSON with `title`, `headline`, `narration`, `visual_style`, and `storyboard`. Video copy defaults to Chinese (`zh-CN`) unless the brief explicitly asks for another language. Save plans with `llm-plan --output plan.json` when you want reproducible drafts.

## Text Prompt To Finished Video

1. Expand the user's idea into a concise visual prompt with subject, camera motion, lighting, style, and ending frame.
2. Run `video-create --prompt ... --poll --manifest manifest.json`.
3. If the result is too loose, generate a still with `image-sync`, then rerun as first-frame video.

## Narrated Short Video

1. Write a short narration script. Keep voiceover shorter than the target duration.
2. Run `tts` and save `narration.mp3`.
3. Run `asr --timestamps word --output transcript.json` to create caption timing.
4. Run `captions --transcript transcript.json --output captions.json`.
5. Use local HTML for controlled UI/layout, or `video-create` for generative motion inserts. If background music or narration should influence generation, provide a public `--audio-url`.
6. Return `asset-manifest.json`, `narration.mp3`, `transcript.json`, `captions.json`, and `result.mp4` or `video_url`.

## QA And Delivery

1. Run `lint --strict` before final render.
2. Run `asset-report --json` and fix missing files.
3. Render with `render` or `build`; both write a render report JSON next to the MP4 unless `--report` is set.
4. Return the MP4, render report, SRT/VTT if created, and `assets/asset-manifest.json`.

## Image To Video

1. Use an existing image URL or generate one with `image-sync`.
2. For controlled motion from one image, use `--image-role first_frame`.
3. For interpolation between two stills, pass first and last images with `Seedance-Pro-1.5` or a supported Seedance mode.
4. Keep prompts explicit about camera movement; otherwise generated motion may drift.

## Fallbacks

- If a video task is slow, return the task ID and manifest so the user can resume polling later.
- If local media cannot be accepted as a video/audio reference, ask for a public URL or upload the file to the user's chosen storage.
- If ASR lacks useful timestamps, retry with `senseaudio-asr-pro-1.5-260319` and `timestamp_granularities[]=word`.


path: references/api.md
content: # SenseAudio API Reference

Use `SENSEAUDIO_API_KEY` as the Bearer token for SenseAudio media APIs. If it is not exported, the CLI also checks AudioClaw's local workspace credential files for `SENSEAUDIO_API_KEY`.

## LLM Planning

Use `DEEPSEEK_API_KEY` as a Bearer token for OpenAI-compatible chat completions.

- Default base URL: `https://api.deepseek.com`
- Chat path: `/chat/completions`
- Default model: `deepseek-v4-pro`
- Override model/base URL with env vars or command flags.

AudioClaw planning is a separate OpenAI-compatible LLM route. It uses `AUDIOCLAW_LLM_API_KEY`, `AUDIOCLAW_LLM_BASE_URL`, and `AUDIOCLAW_LLM_MODEL`, or the local AudioClaw config file when `--provider audioclaw` or `--llm audioclaw` is selected. The config entry supplies `api_base`, `api_key`, and `model`; LiteLLM-style prefixes such as `volcengine/` are removed for `platform.senseaudio.cn`. Do not reuse `SENSEAUDIO_API_KEY` as the AudioClaw LLM key.

The skill uses the selected LLM only for creative planning: title, headline, narration, visual style, and storyboard JSON. Generated video copy defaults to Chinese (`zh-CN`) unless the brief explicitly requests another language. SenseAudio remains the media provider for TTS, ASR, images, and model video.

## One-Pass Website Pipeline

`site-video` is the recommended website-to-video path. It orchestrates URL/site evidence ingestion, AudioClaw or DeepSeek planning, real screenshot capture, optional SenseAudio TTS/ASR, audio-reactive data extraction, local render, `motion-audit`, and `motion-map`. `--llm-fallback` is enabled by default so transient model/API failures degrade to heuristic planning instead of aborting the whole run. Add `--vision-audit` when an OpenRouter/OpenAI-compatible vision model should review inspect frames.

`site-ingest` adds `semantic_sections`, `primary_roles`, and role-tagged evidence. Supported roles include `hero`, `product`, `research`, `safety`, `developer`, `enterprise`, `customer`, `pricing`, and `cta`. `compose` uses these roles to avoid repetitive generic sections and to choose richer shot grammar.

`site-vision-plan` adds `visual_plan` crop directives before rendering. Each item includes `screenshot_id`, `role`, `crop.x`, `crop.y`, `crop.zoom`, and `crop.pan`; `compose` writes these into screenshot CSS variables such as `--crop-x`, `--crop-y`, and `--crop-zoom`. The `heuristic` provider is deterministic; `openrouter` can be used to ask a VL model for the same crop plan when screenshots are available.

## Video Generation

- Create: `POST https://api.senseaudio.cn/v1/video/create`
- Status: `GET https://api.senseaudio.cn/v1/video/status?id=<task_id>`
- Models: `Seedance-2.0-Fast`, `Seedance-2.0`, `Seedance-Pro-1.5`
- Status values: `pending`, `processing`, `completed`, `failed`

Core request fields: `model`, `content`, `duration`, `resolution`, `ratio`, `watermark`, `provider_specific`.

Content item types:

- `{"type":"text","text":"..."}`
- `{"type":"image","url":"https://... or data:image/...","role":"first_frame|last_frame|reference"}`
- `{"type":"audio","audio_url":"https://..."}`
- `{"type":"video","video_url":"https://..."}`

Model guardrails:

- `Seedance-Pro-1.5`: `ratio` is `16:9` or `9:16`; `resolution` is `720p` or `1080p`; `duration` is 2–12 seconds; supports text and first/last-frame images.
- `Seedance-2.0` and `Seedance-2.0-Fast`: `ratio` is `16:9`, `4:3`, `1:1`, `3:4`, or `9:16`; `resolution` is `480p` or `720p`; `duration` is 4–15 seconds; supports first/last-frame mode or reference-material mode.
- Do not mix first/last-frame images with reference images in the same `Seedance-2.0` request.

## Image Generation

- Sync: `POST https://api.senseaudio.cn/v1/image/sync`
- Async: `POST https://api.senseaudio.cn/v1/image/async`
- Models: `senseaudio-image-1.0-260319`, `doubao-seedream-5-0-260128`

Fields: `model`, `prompt`, optional `reference`, optional `seed`, optional `size`.

Prefer sync when a downstream video needs a first frame immediately.

## Music Generation

- Create: `POST https://api.senseaudio.cn/v1/music/song/create`
- Status: `GET https://api.senseaudio.cn/v1/music/song/pending/<task_id>`
- Model: `senseaudio-music-1.0-260319`
- Auth: `[REDACTED_SECRET]`

Fields: `model`, `lyrics`, optional `custom_mode`, optional `instrumental`, optional `negative_tags`, optional `style`, optional `style_weight`, optional `title`, optional `vocal_gender`.

For stable instrumental beds, prefer structured music tags rather than free-form prompt text:

```text
[intro-medium] ; [inst-medium] ; [outro-short]
```

The official status response nests the generated audio at `response.data[].audio_url`. If the task remains `PENDING` without an `audio_url`, `site-video --music-fallback` can create a local ambient bed and continue the render while preserving the SenseAudio task manifest for later inspection.

## Text To Speech

- Endpoint: `POST https://api.senseaudio.cn/v1/t2a_v2`
- Model: `senseaudio-tts-1.5-260319`
- Output: `data.audio` is hex-encoded audio bytes.

`voice_setting`: `voice_id`, `speed` `[0.5,2.0]`, `vol` `[0.01,10.0]`, `pitch` `[-12,12]`, `latex_read`.

`audio_setting`: `format` `mp3|wav|pcm|flac`, `sample_rate`, `bitrate`, `channel`.

## Speech Recognition

- Endpoint: `POST https://api.senseaudio.cn/v1/audio/transcriptions`
- Content-Type: `multipart/form-data`
- Models: `senseaudio-asr-lite-1.5-260319`, `senseaudio-asr-1.5-260319`, `senseaudio-asr-pro-1.5-260319`, `senseaudio-asr-deepthink-1.5-260319`

For captions, use `senseaudio-asr-1.5-260319` or `senseaudio-asr-pro-1.5-260319` with `response_format=verbose_json` and `timestamp_granularities[]=word`.

## Voice Listing

- Endpoint: `POST https://api.senseaudio.cn/v1/get_voice`
- Body: `{"voice_type":"all"}`

Use this before picking a voice when the user has not provided a known `voice_id`.


path: references/authoring.md
content: # HTML Authoring Guide

## Source Of Truth

The `index.html` file is the video. Do not hide the important composition logic in prompts or generated manifests. The manifest records assets and parameters; HTML controls layout, timing, and render behavior.

## Required Structure

```html
<div data-composition-id="main" data-start="0" data-duration="6" data-width="1280" data-height="720">
  <section class="story-scene" data-scene="intro" data-timeline-id="intro" data-start="0" data-duration="2.5">Scene one</section>
  <section class="story-scene" data-scene="detail" data-timeline-id="detail" data-start="2.5" data-duration="3.5">Scene two</section>
  <div class="caption" data-caption-source="./assets/captions.json" data-caption-target></div>
</div>
<script src="./senseframe-runtime.js"></script>
<script>
  window.__timelines = window.__timelines || {};
  window.__timelines["main"] = { seek: function (time) { /* deterministic timeline state */ } };
  window.renderFrame = function (time) {
    // optional deterministic visual state for this exact timestamp
  };
</script>
```

## Timing

- Use seconds.
- `data-start` and `data-duration` control visibility for clips and scenes.
- `data-scene` marks named beats and receives `data-scene-active="true|false"` at render time.
- `data-caption-source` loads grouped captions from JSON and writes the active cue into the element.
- Use `window.__timelines["main"].seek(time)` for scene choreography; the runtime seeks registered timelines before calling `window.renderFrame(time)`.
- Use `window.renderFrame(time)` for small transforms, counters, progress bars, and custom highlights that do not conflict with the main timeline.
- For media elements, add `data-media-start` to trim into the source.

## Layout

- Build the final still frame first.
- Animate from or through that layout in the seekable timeline or `renderFrame`.
- Keep text large enough for video: 56px+ headlines, 24px+ body, 18px+ captions.
- Use `inspect` to sample frames before rendering.
- Use `motion-audit --project <dir> --strict` to catch storyboard scenes that are not represented by DOM scenes or timelines.
- Use `motion-map --project <dir> --strict` to find dead zones, weak scene coverage, missing transitions, or unbound audio-reactive motion before spending render time.

## Captions

Run `captions --project <dir> --transcript assets/transcript.json --output assets/captions.json` after ASR. The output shape is:

```json
{"captions":[{"id":"c0","text":"搜索音色","start":0.0,"end":0.8}]}
```

Use short cues. Prefer 18–32 Chinese characters per cue for a 720p short video.

For word highlighting, run `captions --include-words`. Style active words with:

```css
.sf-word { opacity: .45; transition: none; }
.sf-word[data-sf-active="true"] { opacity: 1; color: #ffd08a; }
.sf-word-emphasis[data-sf-active="true"] { text-shadow: 0 0 22px rgba(255,214,140,.66); }
```

The runtime assigns `sf-word-emphasis` to brand/action words and scales active words inline from their timestamps. Do not add wall-clock CSS animations to caption words.

## Audio-Reactive Motion

Generate frame-level voice/music data and bind it to the root composition:

```bash
python3 scripts/senseaudio_video_gen.py audio-data \
  --project my-video \
  --audio my-video/assets/narration.mp3 \
  --output my-video/assets/audio-data.json \
  --fps 24 \
  --bands 8
```

The runtime loads `data-audio-source="./assets/audio-data.json"` and exposes `window.__senseframes.audioData`. The `audio-data` command writes smoothed RMS plus `rawRms`; use smoothed values for visuals. The default composed template maps RMS/bands to mesh intensity, card glow, waveform bars, and transition light, but keeps the global camera independent from audio to avoid speech jitter. Keep text motion subtle: 3–6% scale for captions, stronger motion only for background/card layers.

## Beat Composition

Use layered beats when a scene should contain multiple authored reveals rather than one static text block:

```bash
python3 scripts/senseaudio_video_gen.py compose \
  --project my-video \
  --brief "..." \
  --beat-mode layered \
  --beats-per-scene 3 \
  --timeline-engine gsap-compat \
  --offline
```

`compose --beat-mode layered` writes `assets/beats.json`, inserts `.beat-layer` elements with `data-beat`, and adds local timeline tracks for every beat. The requested beat count is clamped on short scenes so each beat remains readable instead of flashing past. To regenerate only the beat plan after editing `assets/storyboard.json`, run:

```bash
python3 scripts/senseaudio_video_gen.py beats --project my-video --beats-per-scene 3 --json
```

Each beat has `scene_id`, `role`, `start`, `duration`, `text`, and `emphasis`. Keep roles short and visual: `hook` names the promise, `proof` shows the reason to believe, `detail` reveals a concrete interaction, and `cta` lands the action. `motion-map --strict` reports beat coverage plus flashiness risk so both flat sections and overly fast sections are caught before render.

## Brand Extraction

For website explainers, extract brand metadata before composing:

```bash
python3 scripts/senseaudio_video_gen.py brand-extract --url https://www.anthropic.com/ --output brand.json --json
python3 scripts/senseaudio_video_gen.py compose --project my-video --brief "..." --brand-file brand.json --beat-mode layered --offline
```

`compose --brand-url <site>` can fetch and apply the brand in one pass. The resulting `assets/brand.json` records `name`, `description`, `nav`, `colors`, `logos`, `assets.icons`, `assets.social_images`, `typography`, `keywords`, and inferred `voice`. Website shots should use this data for navigation chips, hero copy, palette accents, brand marks, tone labels, and brand-specific wording rather than generic filler UI.

## Site Ingestion

For URL-to-video work, extract real page evidence before composing:

```bash
python3 scripts/senseaudio_video_gen.py site-ingest --url https://www.anthropic.com/ --output site.json --json
python3 scripts/senseaudio_video_gen.py compose --project my-video --brief "介绍这个官网" --site-file site.json --brand-file brand.json --beat-mode layered --offline
```

`compose --site-url <site>` can fetch site evidence in one pass. The resulting `assets/site-profile.json` records headings, sections, CTA labels, evidence snippets, and text samples. Storyboard scenes should prefer these evidence items over generic brief-derived copy, so website explainers stay anchored to actual page content.

## Site Screenshots

Use real screenshots when the video needs to show the actual website, not just a redesigned card:

```bash
python3 scripts/senseaudio_video_gen.py site-capture --url https://www.anthropic.com/ --project my-video --count 3 --json
python3 scripts/senseaudio_video_gen.py compose --project my-video --brief "介绍这个官网" --site-url https://www.anthropic.com/ --site-screenshots --beat-mode layered --offline
```

Screenshots are captured with local Chrome through DevTools, saved under `assets/site-screenshots/`, registered as `website-screenshot` assets, and referenced from `assets/site-profile.json`. Rendered website shots apply deterministic pan/zoom plus a `.site-scan-highlight` evidence box to guide the viewer through real pixels. Use this when visual accuracy matters; DeepSeek can structure the story, but Chrome provides the actual page pixels.

## Composition Templates

Website explainers should vary scene structure instead of repeating one left-copy/right-card layout. `compose` assigns `data-composition-mode` and `data-camera-path` to each scene:

- `full-bleed` + `hero-push` for opening full-page website context.
- `split-scan` + `lateral-scan` for navigation or scroll-path explanation.
- `zoom-callout` + `macro-zoom` for product/detail emphasis.
- `evidence-board` + `board-orbit` for proof, policy, or research evidence.
- `cta-lockup` + `lockup-dolly` for final action and brand close.

The renderer reads these attributes to change layout, visual-card transforms, screenshot pan/zoom, and copy placement.

## Style Registry

Use `styles` to inspect built-in visual presets before composing:

```bash
python3 scripts/senseaudio_video_gen.py styles --json
python3 scripts/senseaudio_video_gen.py compose --project my-video --brief "..." --style-preset neon-console --offline
```

The registry currently includes `product-glass`, `neon-console`, `editorial-cream`, and `midnight-lab`. A composed project records the selected preset in `senseframe.json`, writes `assets/style-preset.json`, and embeds tokens such as `--accent`, `--ember`, `--aqua`, stage background, card gradient, and mesh gradient into `index.html`.

## Timeline DSL

Generate a timeline from the project storyboard:

```bash
python3 scripts/senseaudio_video_gen.py timeline --project my-video --preset cinematic
```

The runtime loads `data-timeline-source="./assets/timeline.json"` and applies scene timing plus effects. Timeline items target matching `data-scene` or `data-timeline-id` values:

```json
{"preset":"cinematic","items":[{"id":"intro","start":0,"end":2.5,"effect":"fade-up"}]}
```

Built-in effects: `fade-up`, `slide-left`, `zoom-in`, `spotlight`, `parallax`.

`compose --animation-preset ...` also creates a seekable `window.__timelines["main"]` bridge with build/breathe/resolve phases for each storyboard scene. Keep storyboard ids stable because they are used for `data-scene`, `data-timeline-id`, nav state, and audit checks.

Transition presets are authored as timeline data, not ad-hoc CSS loops:

```bash
python3 scripts/senseaudio_video_gen.py timeline --project my-video --preset cinematic --transition-preset editorial
```

This writes `transitions[]` entries with `from`, `to`, `at`, `duration`, `kind`, and `intensity`, then patches the inline runtime `transitionPlan`. Pick `glass` for polished UI reveals, `ribbon` for fast explainers, `iris` for feature focus, and `luma` for energetic sweeps.

For HyperFrames-style timeline authoring without external dependencies, enable the local GSAP-compatible engine:

```bash
python3 scripts/senseaudio_video_gen.py timeline --project my-video --preset cinematic --timeline-engine gsap-compat
```

The timeline JSON gains `labels` and `tracks`, and the HTML runtime exposes `window.__senseframes.gsapCompat`. Tracks support `set`, `to`, and `fromTo` with `opacity`, `x`, `y`, `z`, `scale`, `rotate`, `rotateX`, `rotateY`, `filter`, `background`, and `mixBlendMode`.

## Asset Registry

Every reusable file should be registered:

```bash
python3 scripts/senseaudio_video_gen.py asset-add \
  --project my-video \
  --id narration \
  --type audio \
  --path my-video/assets/narration.mp3 \
  --role voiceover
```

The command updates both `assets/asset-manifest.json` and `senseframe.json`.

## Generated Media Slots

Use `data-asset` to mark where generated assets should bind:

```html
<img data-asset="hero-image" src="" alt="Generated hero" />
<video data-asset="broll-video" src="" muted playsinline></video>
```

`generate-assets` records image/video requests in the manifest. When a generated file is downloaded and registered, the helper can update matching `src` values.


path: references/renderer.md
content: # Local Renderer

The renderer serves the project on a local HTTP server, opens Chrome headless once per frame with `?t=<seconds>`, captures PNG screenshots, and assembles them with FFmpeg.

## Commands

```bash
python3 scripts/senseaudio_video_gen.py compose --project my-video --brief "..." --offline
python3 scripts/senseaudio_video_gen.py styles --json
python3 scripts/senseaudio_video_gen.py init my-video
python3 scripts/senseaudio_video_gen.py lint --project my-video --strict
python3 scripts/senseaudio_video_gen.py beats --project my-video --json
python3 scripts/senseaudio_video_gen.py timeline --project my-video --preset cinematic --transition-preset editorial --timeline-engine gsap-compat
python3 scripts/senseaudio_video_gen.py audio-data --project my-video --audio my-video/assets/narration.mp3 --output my-video/assets/audio-data.json
python3 scripts/senseaudio_video_gen.py motion-audit --project my-video --strict
python3 scripts/senseaudio_video_gen.py motion-map --project my-video --strict --samples 32
python3 scripts/senseaudio_video_gen.py inspect my-video --samples 7
python3 scripts/senseaudio_video_gen.py render my-video --fps 24 --output my-video/renders/final.mp4
python3 scripts/senseaudio_video_gen.py render my-video --audio my-video/assets/narration.mp3 --output final.mp4
python3 scripts/senseaudio_video_gen.py render my-video --parallel 4 --frame-dir my-video/renders/frames --resume --output final.mp4
python3 scripts/senseaudio_video_gen.py build --project my-video --output my-video/renders/final.mp4
```

`render` writes a JSON report next to the MP4 by default, and registers the rendered video in the project asset manifest when the project has `senseframe.json`.

`beats` writes `assets/beats.json` from the storyboard. Layered projects use these entries to bind `.beat-layer[data-beat]` overlays and matching local timeline tracks.

`motion-audit` should run before expensive renders. It verifies storyboard ids are present as DOM scenes, checks beat-layer binding, checks the seekable timeline registry, confirms transition/audio-reactive hooks, and warns if legacy fixed-template markers remain.

`motion-map` is the second-pass visual audit. It samples the whole duration, scores active scene motion, beat coverage, flashiness risk, timeline effects, entrance/exit phases, transition boundaries, and audio-reactive hooks, then reports dead zones and practical recommendations.

`timeline --transition-preset` writes authored transition boundaries into `assets/timeline.json` and patches the local runtime plan. Use `editorial` for mixed wipes/flashes, `glass` for luminous product UI, `ribbon` for fast explainers, `iris` for focused feature reveals, and `luma` for energetic sweeps.

`timeline --timeline-engine gsap-compat` adds local GSAP-like `labels` and `tracks` to the timeline JSON and binds them to `createGsapCompatTimeline`. The adapter supports the small render-safe subset this skill needs: `addLabel`, `set`, `to`, `fromTo`, and deterministic `seek(time)`. Do not load external GSAP from npm or CDN inside generated projects.

## Faster Renders

- `--parallel N` captures frames concurrently with separate headless Chrome processes.
- `--frame-dir <dir>` stores PNG frames in a stable directory.
- `--resume` skips existing non-empty frames, useful after interrupted renders.
- `--keep-frames` keeps generated frames for debugging or later reuse.

## Requirements

- Chrome or Chromium available in `PATH`, discoverable on macOS, or configured through `CHROME_BIN`.
- FFmpeg available in `PATH`.

## Troubleshooting

- If Chrome is not found, set `CHROME_BIN` to the browser executable.
- If output is black or incomplete, increase `--virtual-time-budget`.
- If render is slow, lower `--fps` for drafts and render final at 24 or 30 fps.
- If fonts differ between machines, use local web fonts in the project.
- If captions do not appear, confirm the page is served through `preview`, not opened as a raw `file://` URL; caption JSON is fetched by the runtime.


path: references/media-pipeline.md
content: # SenseAudio Media Pipeline

Use SenseAudio for media creation around the local HTML renderer.

## Narration

1. Query voices with `voices --voice-type all`.
2. Generate narration with `tts`.
3. Add the resulting audio to `assets/`.
4. Register it with `asset-add --type audio --role voiceover`.
4. Render with `--audio`.

## Captions

1. Run `asr --timestamps word`.
2. Run `captions --transcript assets/transcript.json --output assets/captions.json`.
3. Add a caption element with `data-caption-source="./assets/captions.json"`.
4. Use `renderFrame(time)` only for extra effects such as word highlighting.
5. Export sidecar subtitles with `captions-export --format srt` or `--format vtt` when publishing platforms need uploads.

## Images

Use `image-sync` to generate first frames, backdrops, thumbnails, or product mood images. Download them into `assets/` for stable local renders.

Register downloaded images with `asset-add --type image`; this keeps later renders reproducible even when prompt history is long.

For storyboard-driven projects, prefer:

```bash
python3 scripts/senseaudio_video_gen.py generate-assets \
  --project my-video \
  --image-prompt "clean product UI hero image" \
  --video-prompt "short creator b-roll for voice selection" \
  --dry-run
```

Remove `--dry-run` to call SenseAudio. `compose --generate-images --generate-broll --asset-dry-run` creates the same HTML slots and manifest requests while drafting.

## Manifests And Reports

- `assets/asset-manifest.json` is the inventory for local and generated assets.
- `asset-report --json` checks what exists and what is missing.
- `lint --strict` blocks final builds when required files are missing.
- `render` writes a render report JSON and registers the MP4 as `final-video` by default.

## Generated Video Clips

Use `video-create` when the project needs generative b-roll or first-frame/reference-material motion. Download the completed clip and place it as a timed `<video>` element in the HTML project.

Use the local HTML renderer for controlled UI, typography, layout, and product walkthroughs. Use SenseAudio model video for b-roll, atmospheric shots, or motion inserts that do not need pixel-level UI fidelity.

Generated video tasks are registered as planned/submitted assets. Use `generate-assets --poll` to poll and download automatically, or poll with `video-status`, download the result, then update the asset with `asset-add --type video --role broll`.


path: agents/openai.yaml
content: interface:
  display_name: "SenseAudio Video Gen"
  short_description: "Create local HTML videos with AudioClaw planning and SenseAudio media"
  default_prompt: "Create a short HTML-authored video from a brief, website, or GitHub/Markdown source using AudioClaw LLM planning, SenseAudio narration or music when configured, registered assets, audits, and a local MP4 render"


path: README.md
content: # senseaudio-video-gen

HTML-authored video framework powered by SenseAudio media APIs and AudioClaw LLM planning.

## Capabilities

- Compose a project from a brief with storyboard, narration script, caption scaffold, HTML, runtime, and manifests.
- Default general `compose` videos to `executive-film`, a mature launch-film style with letterbox framing, large typography, restrained motion, and fewer toy-like UI ornaments.
- Generate creative plans through AudioClaw by default, with DeepSeek and OpenRouter available as explicit alternatives.
- Default video copy to Chinese (`zh-CN`) unless the brief explicitly asks for another language.
- Run a local build pipeline that validates, prepares captions, renders, and records output metadata.
- Run a one-command `site-video` pipeline that ingests a URL, plans with AudioClaw, captures screenshots, creates narration/audio-reactive data, renders, and audits motion.
- Generate SenseAudio music beds, mix them under narration, and render with the registered `final-audio` track by default.
- Run an automatic second-pass repair that uses motion/vision audit reports to tighten screenshot evidence, stabilize overlays, and rerender.
- Capture timestamped inspect frames and run a local frame-quality gate by default for `site-video`, catching blank/washed frames and leaked planning copy before delivery.
- Add storyboard-driven scene layers, seekable timeline motion, deterministic animation effects, and word-level caption highlighting.
- Split scenes into timed hook/proof/detail/cta beat layers with matching HTML markers and timeline tracks.
- Clamp beat density for readability and report flashiness risk when cuts or overlays move too quickly.
- Rotate website explainer shots across hero overview, navigation scan, feature zoom, trust message, and CTA summary layouts.
- Extract website brand profiles and apply brand colors, navigation labels, descriptions, logos/icons, typography, keywords, and inferred voice to generated shots.
- Ingest real website evidence from headings, sections, CTAs, and page text so URL-to-video projects explain actual page content instead of generic brand templates.
- Ingest local Markdown/text files and GitHub repository READMEs into the same reusable site-profile format for fast document-to-video drafts.
- Classify real website materials into semantic roles such as hero, product, research, safety, developer, enterprise, customer, pricing, and CTA, then use those roles to choose shots, composition modes, and camera paths.
- Plan screenshot crops and focus points before rendering with `site-vision-plan`, using a deterministic heuristic by default and an OpenRouter-compatible vision route when requested.
- Capture real website scroll screenshots with local Chrome and place them into explainer shots as visual evidence.
- Warm live pages before capture, dismiss common cookie/modals, record `site-capture-quality.json`, and optionally load cookies or a dedicated browser profile for gated/region-specific pages.
- Use a cleaner screenshot-first website mode that reduces overlays and keeps one evidence layer per scene.
- Use `--style-preset editorial-pro` for more mature website explainers with fine annotations, smaller type, restrained motion, and less toy-like UI.
- Use `--style-preset executive-film` for non-web product launches, report summaries, technical explainers, title cards, and premium brand films.
- Audit rendered frames with an OpenRouter/OpenAI-compatible vision model through `site-vision-audit`.
- Generate `renders/inspect/contact-sheet.html` for quick frame-by-frame review, inspired by the HyperFrames snapshot/contact-sheet loop.
- Choose visual direction from a local style registry with reusable CSS tokens and recommended motion defaults.
- Add a local GSAP-compatible timeline adapter with `labels`, `tracks`, `set`, `to`, `fromTo`, and deterministic `seek`.
- Add authored transition presets such as editorial wipes, glass flashes, iris focus, ribbon sweeps, and luma sweeps.
- Add pre-extracted audio-reactive motion data for camera, glow, waveform, and transition intensity.
- Audit storyboard/DOM/timeline/transition/audio-reactive alignment with `motion-audit`.
- Map motion density, scene coverage, dead zones, and recommendations with `motion-map`.
- Plan or generate SenseAudio images and b-roll videos as registered project assets.
- Scaffold HTML composition projects.
- Preview HTML videos in a browser.
- Render deterministic frames locally with Chrome and FFmpeg, defaulting to one persistent Chrome session for lower CPU overhead.
- Generate narration, transcripts, captions, images, and model-video clips through SenseAudio.
- Register every generated/local asset in `assets/asset-manifest.json`.
- Export subtitles as SRT or WebVTT for external upload workflows.

## Requirements

- Python 3.9+
- Chrome or Chromium (`CHROME_BIN` can override discovery)
- FFmpeg in `PATH`
- `SENSEAUDIO_API_KEY` for live SenseAudio media API calls
- Optional local AudioClaw config file for automatic AudioClaw LLM model discovery
- Optional `AUDIOCLAW_LLM_API_KEY`, `AUDIOCLAW_LLM_BASE_URL`, and `AUDIOCLAW_LLM_MODEL` for env-only LLM routing on a new machine
- Optional `DEEPSEEK_API_KEY` for live LLM planning through DeepSeek
- `OPENROUTER_API_KEY` or `VL_API_KEY` for OpenRouter LLM planning and live visual quality audits

No third-party Python package is required.

## Default Configuration

For a fresh install, configure SenseAudio media first:

```bash
export SENSEAUDIO_API_KEY="..."
```

That key is used for SenseAudio TTS, ASR, image, video, and music APIs.

Configure AudioClaw LLM separately, either through the local AudioClaw config file or env:

```bash
export AUDIOCLAW_LLM_API_KEY="..."
export AUDIOCLAW_LLM_BASE_URL="https://platform.senseaudio.cn/v1"
export AUDIOCLAW_LLM_MODEL="doubao-seed-2-0-pro-260215"
```

When running inside AudioClaw, the CLI automatically reads the local AudioClaw config file and uses the configured default model. Override order for LLM planning is:

1. CLI flags: `--llm-model`, `--llm-base-url`, `--model`, `--base-url`
2. Env vars: `AUDIOCLAW_LLM_MODEL`, `AUDIOCLAW_LLM_BASE_URL`, `AUDIOCLAW_LLM_API_KEY`
3. AudioClaw config: `AUDIOCLAW_CONFIG_PATH` or the local AudioClaw config file

`llm-plan`, `compose`, and `site-video` default to AudioClaw. They do not reuse `SENSEAUDIO_API_KEY` as an LLM key. If AudioClaw LLM is not configured, `compose` and `site-video` can continue through `--llm-fallback` with heuristic planning; use `--llm none` or `--provider deepseek/openrouter` when you intentionally want a different route.

## Smoke Test

```bash
python3 scripts/senseaudio_video_gen.py --help
python3 scripts/senseaudio_video_gen.py styles --json
python3 scripts/senseaudio_video_gen.py llm-plan --brief "Introduce a sound library webpage" --dry-run
python3 scripts/senseaudio_video_gen.py llm-plan --brief "Introduce a sound library webpage" --provider deepseek --dry-run
python3 scripts/senseaudio_video_gen.py llm-plan --brief "Introduce a sound library webpage" --provider openrouter --dry-run
python3 scripts/senseaudio_video_gen.py brand-extract --url https://www.anthropic.com/ --json
python3 scripts/senseaudio_video_gen.py site-ingest --url https://www.anthropic.com/ --json
python3 scripts/senseaudio_video_gen.py source-ingest --file README.md --output readme-site.json --json
python3 scripts/senseaudio_video_gen.py source-ingest --github-url heygen-com/hyperframes --output hyperframes-readme-site.json --json
python3 scripts/senseaudio_video_gen.py site-capture --url https://www.anthropic.com/ --output-dir demo-compose/assets/site-screenshots --json
python3 scripts/senseaudio_video_gen.py frame-quality-audit --project demo-compose --json
python3 scripts/senseaudio_video_gen.py site-vision-plan --project demo-compose --json
python3 scripts/senseaudio_video_gen.py site-vision-audit --project demo-compose --dry-run --json
python3 scripts/senseaudio_video_gen.py site-video --url https://www.anthropic.com/ --project anthropic-site-video --duration 14 --offline --no-render --music --music-dry-run --auto-repair
python3 scripts/senseaudio_video_gen.py compose --project executive-demo --brief "Make a premium launch film for an AI research assistant." --duration 12 --style-preset executive-film --offline --render
python3 scripts/senseaudio_video_gen.py compose --project demo-compose --brief "Introduce a sound library webpage" --style-preset neon-console --beat-mode layered --offline
python3 scripts/senseaudio_video_gen.py compose --project editorial-demo --brief "Introduce this website professionally" --site-url https://www.anthropic.com/ --site-screenshots --style-preset editorial-pro --offline
python3 scripts/senseaudio_video_gen.py beats --project demo-compose --json
python3 scripts/senseaudio_video_gen.py lint --project demo-compose --json
python3 scripts/senseaudio_video_gen.py timeline --project demo-compose --preset cinematic --transition-preset editorial --timeline-engine gsap-compat
python3 scripts/senseaudio_video_gen.py audio-data --project demo-compose --audio demo-compose/assets/narration.mp3 --output demo-compose/assets/audio-data.json --dry-run
python3 scripts/senseaudio_video_gen.py motion-audit --project demo-compose --json
python3 scripts/senseaudio_video_gen.py motion-map --project demo-compose --json
python3 scripts/senseaudio_video_gen.py music-create --prompt "Instrumental premium website explainer bed, subtle pulse, no vocals" --duration 16 --dry-run
python3 scripts/senseaudio_video_gen.py mix-audio --project demo-compose --voice demo-compose/assets/narration.mp3 --music demo-compose/assets/background-music.mp3 --output demo-compose/assets/final-audio.m4a --duration 16 --dry-run --json
python3 scripts/senseaudio_video_gen.py repair --project demo-compose --dry-run --json
python3 scripts/senseaudio_video_gen.py build --project demo-compose --dry-run
python3 scripts/senseaudio_video_gen.py generate-assets --project demo-compose --image-prompt "product UI hero" --video-prompt "creator b-roll" --dry-run
python3 scripts/senseaudio_video_gen.py init demo-video --duration 1 --fps 2
python3 scripts/senseaudio_video_gen.py render demo-video --output demo-video/renders/demo.mp4 --quiet
python3 scripts/senseaudio_video_gen.py render demo-video --output demo-video/renders/demo-fast.mp4 --parallel 2 --resume --quiet
python3 tests/test_cli.py
```

## One-Pass Project

```bash
export SENSEAUDIO_API_KEY="..."
python3 scripts/senseaudio_video_gen.py site-video \
  --url https://www.anthropic.com/ \
  --project anthropic-site-video \
  --brief "用中文介绍 Anthropic 官网的 Claude、安全 AI、研究与企业能力。" \
  --duration 14 \
  --fps 30 \
  --llm audioclaw \
  --music \
  --music-poll \
  --auto-repair
```

`site-video` is the recommended HyperFrames-like path for website explainers. It defaults to `editorial-pro`, layered beats, cinematic motion, GSAP-compatible timing, real website screenshots, AudioClaw LLM planning, SenseAudio TTS/ASR when not offline, audio-reactive data, render, `inspect`, `frame-quality-audit`, `motion-audit`, and `motion-map`. Add `--music --music-poll` to create and mix a SenseAudio music bed; `--music-fallback` is enabled by default so a local ambient bed is mixed when SenseAudio accepts the task but does not return `audio_url` in time. Add `--auto-repair` to run one repaired rerender after audits. If the selected LLM route fails, `--llm-fallback` keeps the pipeline moving with heuristic planning and records the warning in `pipeline-report.json`. If the AudioClaw configured model is weak for a research-heavy product film, use `--llm openrouter --llm-model <capable-model>`.

For sites where a clean browser sees a login wall, wrong region, consent overlay, or anti-bot interstitial, use a dedicated capture profile or exported cookies:

```bash
python3 scripts/senseaudio_video_gen.py site-video \
  --url https://example.com/ \
  --project example-site-video \
  --browser-profile profiles/example-capture \
  --cookie-file cookies/example.json
```

Cookies can improve screenshot fidelity because Chrome sees the same region/session state as a real visitor. Keep this explicit: clean capture is still the default so videos do not accidentally include personalized or private account content.

Use `--offline --no-render` to create the same editable project and pipeline report without live SenseAudio calls or frame rendering.

```bash
export SENSEAUDIO_API_KEY="..."
python3 scripts/senseaudio_video_gen.py compose \
  --project webpage-intro \
  --brief "Show the SenseAudio sound library and explain search, filters, voice clone, and one-click use." \
  --duration 9 \
  --voice-id male_0028_a \
  --generate-images \
  --generate-broll \
  --render
```

Use `--offline` to create the same editable HTML project without calling SenseAudio.
`compose` defaults to `--llm audioclaw`; if the default LLM route is unavailable, `--llm-fallback` keeps the draft moving with heuristic planning and records the warning in `senseframe.json`.
Use `--asset-dry-run` to add image/video planned assets and manifest requests without spending image/video credits.
Use `generate-assets --poll` to submit b-roll video generation, poll until completion, download the MP4, and bind matching `data-asset` slots.
Live `compose` generates SenseAudio TTS by default when `SENSEAUDIO_API_KEY` is available and `--offline` is not set. `render` also auto-muxes a registered `narration` audio asset when `--audio` is omitted; pass `--audio` to override it. Use `render --capture-mode process` only when you explicitly want the older one-Chrome-process-per-frame behavior.

`render` prefers a registered `final-audio` asset before `narration`, so `mix-audio` projects keep music and voice together without passing `--audio` every time.

## LLM Planning

```bash
python3 scripts/senseaudio_video_gen.py llm-plan \
  --brief "Introduce the SenseAudio sound library with search, filters, cloning, and one-click use." \
  --duration 9 \
  --output plan.json

python3 scripts/senseaudio_video_gen.py compose \
  --project webpage-intro \
  --brief "Introduce the SenseAudio sound library with search, filters, cloning, and one-click use." \
  --plan-file plan.json \
  --offline
```

`llm-plan` defaults to AudioClaw and can run from either `AUDIOCLAW_LLM_*` env vars or the local AudioClaw config file.

DeepSeek remains available when explicitly requested:

```bash
export DEEPSEEK_API_KEY="..."
python3 scripts/senseaudio_video_gen.py llm-plan \
  --brief "Introduce the SenseAudio sound library." \
  --provider deepseek \
  --duration 9 \
  --output plan.json
```

DeepSeek defaults to `deepseek-v4-pro`. Set `DEEPSEEK_MODEL`, `DEEPSEEK_BASE_URL`, `--llm-model`, or `--llm-base-url` when using a custom OpenAI-compatible endpoint.

OpenRouter can replace AudioClaw for stronger planning when needed:

```bash
export OPENROUTER_API_KEY="..."
python3 scripts/senseaudio_video_gen.py llm-plan \
  --brief "Introduce Claude Code as a concrete developer workflow product." \
  --provider openrouter \
  --model google/gemini-3.1-flash-lite \
  --duration 12 \
  --output plan.json
```

For `compose` or `site-video`, use `--llm openrouter --llm-model <capable-model>`. The defaults read `OPENROUTER_LLM_MODEL`, then `OPENROUTER_MODEL`, then `google/gemini-3.1-flash-lite`.

Inside AudioClaw, the configured model is used automatically. You can still be explicit:

```bash
python3 scripts/senseaudio_video_gen.py llm-plan \
  --brief "Introduce Anthropic's website in Chinese." \
  --provider audioclaw \
  --duration 10 \
  --output plan.json

python3 scripts/senseaudio_video_gen.py compose \
  --project webpage-intro \
  --brief "Introduce Anthropic's website in Chinese." \
  --site-url https://www.anthropic.com/ \
  --site-screenshots \
  --llm audioclaw \
  --offline
```

`--provider audioclaw` reads the local AudioClaw config file by default. Set `AUDIOCLAW_CONFIG_PATH`, `AUDIOCLAW_LLM_MODEL`, `AUDIOCLAW_LLM_BASE_URL`, or `AUDIOCLAW_LLM_API_KEY` to override it.


path: examples/site_explainer_request.json
content: {
  "goal": "website-to-video explainer",
  "input": {
    "url": "https://www.anthropic.com/",
    "brief": "用中文介绍官网中的产品、研究、安全和企业能力。"
  },
  "video": {
    "duration": 14,
    "fps": 24,
    "style_preset": "editorial-pro",
    "render": true,
    "inspect": true,
    "frame_quality_audit": true
  },
  "planning": {
    "llm": "audioclaw",
    "env": "AUDIOCLAW_LLM_API_KEY",
    "config": "local AudioClaw config file",
    "fallback": "heuristic"
  },
  "media": {
    "provider": "senseaudio",
    "env": "SENSEAUDIO_API_KEY",
    "narration": true,
    "music": true,
    "music_fallback": "local ambient bed if SenseAudio task has no audio_url"
  },
  "outputs": [
    "index.html",
    "senseframe.json",
    "assets/site-profile.json",
    "assets/narration.mp3",
    "assets/final-audio.m4a",
    "renders/final.mp4",
    "pipeline-report.json"
  ]
}


path: examples/starter-html-video/senseframe.json
content: {
  "name": "starter-html-video",
  "entry": "index.html",
  "width": 1280,
  "height": 720,
  "duration": 3.0,
  "fps": 12,
  "assets": {},
  "senseaudio": {
    "voices": [],
    "image_tasks": [],
    "video_tasks": [],
    "transcripts": []
  }
}

path: examples/starter-html-video/assets/asset-manifest.json
content: {
  "project": "starter-html-video",
  "assets": {}
}


path: examples/product_promo_request.json
content: {
  "goal": "10-second product promo",
  "video": {
    "model": "Seedance-2.0-Fast",
    "prompt": "A cinematic 10-second product promo for wireless earbuds on a glossy black surface. Slow orbit camera, blue rim lights, subtle particles, premium tech mood, final frame holds on the product and tagline.",
    "ratio": "16:9",
    "resolution": "720p",
    "duration": 10,
    "watermark": false
  },
  "narration": {
    "text": "清晰入耳，灵感随行。让每一次表达，都更有画面感。",
    "voice_id": "male_0004_a",
    "format": "mp3"
  },
  "outputs": [
    "manifest.json",
    "narration.mp3",
    "transcript.json",
    "result.mp4"
  ]
}


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal:  | SkillSpector: suspicious (CRITICAL)