Skill: sdk-team/alibabacloud-avatar-video (version 0.0.1)

--- name: alibabacloud-avatar-video description: Use Alibaba Cloud DashScope API and LingMou to generate AI video and speech. Seven capabilities — (1) LivePortrait talking-head (image + audio → video, two-step), (2) EMO talking-head, (3) AA/AnimateAnyone full-body animation (three-step), (4) T2I text-to-image (Wan 2.x, default wan2.2-t2i-flash), (5) I2V image-to-video (Wan 2.x, default wan2.7-i2v-flash, supports T2I→I2V pipeline), (6) Qwen TTS (auto model/voice by scene, default qwen3-tts-vd-realtime-2026-01-15), (7) LingMou digital-human template video with random template, public-template copy, and script confirmation. Trigger when the user needs talking-head, portrait, full-body animation, text-to-image, text-to-video, or speech synthesis. metadata: { "openclaw": { "emoji": "🎭", "requires": { "bins": ["ffmpeg", "ffprobe"], "env": [ "DASHSCOPE_API_KEY", "ALIBABA_CLOUD_ACCESS_KEY_ID", "ALIBABA_CLOUD_ACCESS_KEY_SECRET", "OSS_BUCKET", "OSS_ENDPOINT" ] } } } --- # Human Avatar — Alibaba Cloud AI Video & Speech ## Capabilities overview | Capability | Script | Model / API | Region | Summary | |------|------|---------|--------|------| | **LivePortrait** | `live_portrait.py` | `liveportrait` | cn-beijing | Portrait + audio/video → talking video, two steps | | **EMO** | `portrait_animate.py` | `emo-v1` | cn-beijing | Portrait + audio → talking head, detect + generate | | **AA** (AnimateAnyone) | `animate_anyone.py` | `animate-anyone-gen2` | cn-beijing | Full-body animation: detect → motion template → video | | **T2I** | `text_to_image.py` | `wan2.x-t2i` | Multi-region | Text → image, default wan2.2-t2i-flash | | **I2V** | `image_to_video.py` | `wan2.x-i2v` | Multi-region | Image → video; T2I→I2V pipeline supported; default wan2.7-i2v-flash | | **Qwen TTS** | `qwen_tts.py` | `qwen3-tts-*` | cn-beijing / Singapore | Text → speech; auto model/voice by scene | | **LingMou** | `avatar_video.py` | LingMou SDK | cn-beijing | Template-based digital-human broadcast video | --- ## Quick selection guide ``` Talking head (have audio/video already) → LivePortrait Talking head (no audio; synthesize first) → Qwen TTS → LivePortrait Full-body dance / motion → AA (AnimateAnyone) Text → image → T2I (text_to_image) Image → video → I2V (image_to_video) Text → video end-to-end → T2I → I2V (image_to_video --t2i-prompt) Enterprise digital human / template news → LingMou (avatar_video) ``` --- ## Environment setup ```bash pip install requests==2.33.1 dashscope==1.25.15 oss2==2.19.1 numpy==1.26.4 # LingMou additionally: pip install alibabacloud-lingmou20250527==1.7.0 alibabacloud-tea-openapi==0.4.4 ``` ```bash export DASHSCOPE_API_KEY=sk-xxxx # Beijing-region API key export ALIBABA_CLOUD_ACCESS_KEY_ID=xxx # OSS upload export ALIBABA_CLOUD_ACCESS_KEY_SECRET=xxx export OSS_BUCKET=your-bucket export OSS_ENDPOINT=oss-cn-beijing.aliyuncs.com ``` > ⚠️ API keys for `cn-beijing` and **Singapore are not interchangeable**; use the key for the correct region. > `OSS_ENDPOINT` may include or omit the `https://` prefix; scripts normalize it. --- ## 1. LivePortrait — talking-head video **When to use**: You have a portrait photo + speech and want a talking-head video quickly. **Flow**: ``` Step 1: liveportrait-detect (sync) → pass=true ↓ Step 2: liveportrait (async) → video_url ``` **Image**: Single person, front-facing portrait, clear face, no occlusion **Audio**: wav/mp3, < 15MB, 1s–3min **Video input**: Audio extracted automatically (ffmpeg) ```bash # Image + audio file python scripts/live_portrait.py \ --image ./portrait.jpg \ --audio ./speech.mp3 \ --template normal --download # Image + video (extract audio) python scripts/live_portrait.py \ --image ./portrait.jpg \ --video ./speech_video.mp4 \ --template active --download # Public URLs python scripts/live_portrait.py \ --image-url "https://..." \ --audio-url "https://..." \ --mouth-strength 1.2 --download ``` **Motion templates**: - `normal` (default, moderate motion) - `calm` (calm; news / storytelling) - `active` (lively; singing / hosting) --- ## 2. Qwen TTS — text to speech **When to use**: Generate speech files from text (for LivePortrait, EMO, etc.). **Default model**: [REDACTED_SECRET] ### Auto model selection by scene | Scene `--scene` | Suggested model | Suggested voice | |---------------|---------|---------| | `default` / `brand` | [REDACTED_SECRET] | Cherry | | `news` / `documentary` / `advertising` | [REDACTED_SECRET] | Serena / Ethan | | `audiobook` / `drama` | [REDACTED_SECRET] | Cherry / Dylan | | `customer_service` / `chatbot` / `education` | `qwen3-tts-flash-realtime` | Anna / Ethan | | `ecommerce` / `short_video` | `qwen3-tts-flash-realtime` | Cherry / Chelsie | ### Available voices | Voice | Character | |------|------| | `Cherry` | Bright, sweet female; ads / audiobooks / dubbing | | `Serena` | Mature, intellectual female; news / explainers / corporate | | `Ethan` | Steady, warm male; education / documentary / training | | `Dylan` | Expressive male; radio drama / game VO | | `Anna` | Gentle, friendly female; support / assistant / daily | | `Chelsie` | Young, fresh female; short video / e-commerce | | `Thomas` | Deep, magnetic male; brand / ads | | `Luna` | Warm, soft female; meditation / storytelling | ```bash # Default (qwen3-tts-vd-realtime + Cherry) python scripts/qwen_tts.py --text "Hello, welcome to Qwen TTS." --download # Match by scene python scripts/qwen_tts.py --text "Today's market..." --scene news --download python scripts/qwen_tts.py --text "Once upon a time..." --scene audiobook --download # Style via instructions python scripts/qwen_tts.py \ --text "Dear students..." \ --model qwen3-tts-instruct-flash-realtime \ --instructions "Warm tone, steady pace, suitable for teaching" \ --download # List options python scripts/qwen_tts.py --list-voices python scripts/qwen_tts.py --list-models ``` --- ## 3. T2I — Wan 2.x text-to-image **When to use**: Generate images from text (optionally feed into I2V). ```bash # Default model (wan2.2-t2i-flash, fast) python scripts/text_to_image.py \ --prompt "A woman in Hanfu in a peach blossom forest, cinematic, 4K, soft light" \ --size 960*1696 --download # Higher quality python scripts/text_to_image.py \ --prompt "..." --model wan2.2-t2i-plus --size 1280*1280 --download # Latest (Wan 2.6) python scripts/text_to_image.py \ --prompt "..." --model wan2.6-t2i --size 1280*1280 --n 1 --download ``` **Models**: - `wan2.2-t2i-flash` (default, fast, good for tests) - `wan2.2-t2i-plus` (higher quality) - `wan2.6-t2i` (latest; more aspect ratios; sync call) **Common sizes**: `1280*1280` (1:1) / `960*1696` (9:16) / `1696*960` (16:9) --- ## 4. I2V — Wan 2.x image-to-video **When to use**: Turn an image into motion video; supports text-to-video via T2I first. ```bash # Local image → video python scripts/image_to_video.py \ --image ./portrait.jpg \ --prompt "She turns slowly and smiles; dress and petals drift gently" \ --model wan2.7-i2v \ --resolution 720P --duration 5 --download # Pipeline: text → image → video python scripts/image_to_video.py \ --t2i-prompt "A woman in Hanfu in a peach blossom forest" \ --prompt "She turns slowly; petals fall; poetic mood" \ --download --output result.mp4 # With background music python scripts/image_to_video.py \ --image ./portrait.jpg \ --audio-url "https://..." \ --prompt "..." --download ``` **Models**: - `wan2.7-i2v` (default; includes sound; 5s/10s) - `wan2.5-i2v-preview` (high-quality preview) - `wan2.2-i2v-plus` (no built-in audio; faster) --- ## 5. AA AnimateAnyone — full-body animation **When to use**: Full-body photo + reference motion video → dance / motion video. **Requirements**: - Image: Single person, full body front, head to toe, aspect ratio 0.5–2.0 - Video: Full body in frame from first frame; mp4/avi/mov; fps ≥ 24; 2–60s **Three steps**: ``` Step 1: animate-anyone-detect-gen2 (sync) → check_pass=true ↓ Step 2: animate-anyone-template-gen2 (async) → template_id (~3–5 min) ↓ Step 3: animate-anyone-gen2 (async) → video_url (~3–5 min) ``` ```bash # Local files (auto convert + OSS upload) python scripts/animate_anyone.py \ --image ./portrait_fullbody.jpg \ --video ./dance.mp4 \ --download --output result.mp4 # Use image as background python scripts/animate_anyone.py \ --image ./portrait.jpg --video ./dance.mp4 \ --use-ref-img-bg --video-ratio 9:16 --download # Skip Step 2 (existing template_id) python scripts/animate_anyone.py \ --image ./portrait.jpg \ --template-id "AACT.xxx.xxx" --download ``` > Auto conversion: video webm/mkv/flv → mp4; image webp/heic → jpg; if fps is under 24, normalize to 24 fps --- ## 6. EMO — talking head (legacy) **Note**: Prefer LivePortrait; EMO suits cases that need stricter lip-sync. ```bash python scripts/portrait_animate.py \ --image ./portrait.jpg \ --audio ./speech.mp3 \ --download ``` --- ## 7. LingMou — enterprise template video **When to use**: Corporate digital-human news, template-based broadcasts, scripted reads with optional character images. ### New workflow (prefer no `template_id`) - If the user **provides `template_id`**: use that template to generate. - If **no `template_id`**: 1. List existing broadcast templates for the account. 2. If any exist, **pick one at random** for creation. 3. If none, fetch public templates and **copy up to 3** into the account. 4. Pick one at random from the copy results and continue. - **Caveat**: After a public template is copied, the copy may not yet be a fully “ready-to-render” template; some copies are still drafts and may lack clips, assets, or variable bindings—complete them in LingMou. - If the user only gives an image and “make a talking video” **without a script**: confirm the spoken copy before generating. ### What `scripts/avatar_video.py` supports - `--list-templates`: list account templates - `--list-public-templates`: list public templates (SDK 1.7.0+) - `--copy-public-templates`: copy up to 3 public templates (SDK 1.7.0+) - Omit `--template-id`: random existing template - When local templates are empty: auto try public-template copy as fallback - `--show-template-detail`: template detail and replaceable variables - Fills input text into template text variables (prefers `text_content` / `test_text`) - If generation fails right after copying a public template, surfaces a clear error that the template may still need completion (no silent failure) ```bash # List templates python scripts/avatar_video.py --list-templates # Public templates (SDK 1.7.0+) python scripts/avatar_video.py --list-public-templates # Copy up to 3 public templates (SDK 1.7.0+) python scripts/avatar_video.py --copy-public-templates # No template_id — random existing template python scripts/avatar_video.py \ --text "Hello, welcome to today's tech news." \ --download # Specific template_id python scripts/avatar_video.py \ --template-id "BS1b2WNnRMu4ouRzT4clY9Jhg" \ --text "Hello, welcome to today's tech news." \ --download # Detail for randomly chosen template python scripts/avatar_video.py \ --show-template-detail \ --text "This is a test script for broadcast." ``` ### Conversational usage When the user says things like: - “Make a talking video from this image” - “Digital-human broadcast for me” - “Upload image and make a news read” Do this: 1. Check whether they already gave copy/script ready to read. 2. If not, ask: **“What is the exact script to read? You can give bullet points and I can turn them into broadcast-ready copy.”** 3. With script in hand, run LingMou: prefer random existing template; if none locally, try public copy. 4. If they uploaded a portrait but the template API does not use it, explain: this path is template-driven; for image-driven talking head, use LivePortrait or EMO. --- ## API reference links - **LivePortrait**: https://help.aliyun.com/zh/model-studio/liveportrait-api - **EMO** (emo-detect + emo-v1): [references/emo-api.md](references/emo-api.md) - **AA** (Animate Anyone): [references/aa-api.md](references/aa-api.md) - **T2I** (text-to-image v2): https://help.aliyun.com/zh/model-studio/text-to-image-v2-api-reference - **I2V** (image-to-video): https://help.aliyun.com/zh/model-studio/image-to-video-api-reference/ - **Qwen TTS**: https://help.aliyun.com/zh/model-studio/qwen-tts-realtime - **LingMou**: [references/lingmou-api.md](references/lingmou-api.md) - **OSS upload**: [references/oss-upload.md](references/oss-upload.md)

path: references/emo-api.md
content: # EMO API (DashScope)

Official docs:
- EMO detect: `https://help.aliyun.com/zh/model-studio/emo-detect-api`
- EMO generate: `https://help.aliyun.com/zh/model-studio/emo-api`

## Auth and region

- Auth: `Authorization: Bearer $DASHSCOPE_API_KEY`
- Region: **Beijing** (`dashscope.aliyuncs.com`)
- Set header: `X-DashScope-Async: enable`

## 1) Portrait detection (required first)

`POST https://dashscope.aliyuncs.com/api/v1/services/aigc/image2video/face-detect`

Body:

```json
{
  "model": "emo-detect-v1",
  "input": {"image_url": "https://.../portrait.png"},
  "parameters": {"ratio": "1:1"}
}
```

Key success fields:

- `output.check_pass`
- `output.face_bbox`
- `output.ext_bbox`

## 2) Submit video generation

`POST https://dashscope.aliyuncs.com/api/v1/services/aigc/image2video/video-synthesis`

Body:

```json
{
  "model": "emo-v1",
  "input": {
    "image_url": "https://.../portrait.png",
    "audio_url": "https://.../speech.mp3",
    "face_bbox": [302,286,610,593],
    "ext_bbox": [71,9,840,778]
  },
  "parameters": {"style_level": "normal"}
}
```

Returns: `output.task_id`

## 3) Poll task

`GET https://dashscope.aliyuncs.com/api/v1/tasks/{task_id}`

States: `PENDING` → `RUNNING` → `SUCCEEDED` / `FAILED`

On success:

- `output.results.video_url`

## Limits

- Image: min edge ≥ 400, max edge ≤ 7000
- Audio: wav/mp3, ≤ 15MB, ≤ 60 seconds
- URLs must be public http/https


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)