Skill: zxkane/zxkane-audio-transcriber-funasr (version 1.7.1)

--- name: audio-transcribe version: 1.7.1 description: > This skill should be used when the user explicitly asks to "transcribe a meeting", "transcribe audio", "transcribe a meeting recording", "convert audio to text", "generate meeting minutes from audio", "do speech-to-text", "transcribe with speaker diarization", "identify speakers in audio", "transcribe Chinese audio", "transcribe English audio", "transcribe Japanese audio", "multi-speaker transcription", "transcribe a podcast", "transcribe podcast episode", "transcribe an interview", "convert podcast to text", "podcast to transcript", or mentions FunASR, Paraformer, SenseVoice, Whisper, MiMo, MiMo-V2.5-ASR, meeting transcription, podcast transcription, or speaker diarization. Supports multi-speaker meeting and podcast transcription in Chinese, English, Japanese, Korean, Cantonese, and 99 languages (via Whisper), plus Xiaomi MiMo-V2.5-ASR (8B, local GPU) for stronger proper-noun and code-switching accuracy. Automatic speaker diarization via CAM++, hotword biasing (FunASR path), LLM cleanup. FunASR works on GPU and CPU; MiMo requires a local CUDA GPU with >=20GB VRAM. metadata: openclaw: requires: bins: ["python3", "ffmpeg"] env_vars: - name: AWS_REGION required: false description: "AWS region for Bedrock LLM cleanup (default: us-west-2). Bedrock uses the standard AWS credential chain (IAM role, SSO, ~/.aws/credentials, env vars) — no explicit keys needed." - name: ANTHROPIC_API_KEY required: false description: "API key for Anthropic Claude LLM cleanup" - name: OPENAI_API_KEY required: false description: "API key for OpenAI-compatible LLM cleanup" - name: OPENAI_BASE_URL required: false description: "Base URL for OpenAI-compatible API (vLLM, Ollama, etc.)" emoji: "🎙️" homepage: "https://github.com/zxkane/audio-transcriber" --- # Meeting & Podcast Transcription (FunASR + MiMo) Transcribe multi-speaker audio into structured Markdown with automatic speaker diarization, hotword biasing, and optional LLM cleanup. Two ASR engine families are available: **FunASR** (Paraformer / SenseVoice / Whisper — fast, cheap, GPU or CPU, 99 languages) and **MiMo-V2.5-ASR** (Xiaomi's 8B model, local GPU only, stronger on proper nouns and code-switching). Both share the same VAD + speaker-clustering stack. All scripts run directly from the plugin directory — no copying needed. Define this shorthand at the start of every session: ```bash SCRIPTS=${CLAUDE_PLUGIN_ROOT}/skills/audio-transcribe/scripts ``` ## Supported Languages | `--lang` | Model | Languages | Hotword | |----------|-------|-----------|---------| | `zh` (default) | SeACo-Paraformer | Chinese (CER 1.95%) | Yes | | `zh-basic` | Paraformer-large | Chinese | No | | `en` | Paraformer-en | English | No | | `auto` | SenseVoiceSmall | Auto-detect: zh/en/ja/ko/yue | No | | `whisper` | Whisper-large-v3-turbo | 99 languages | No | | `mimo` | MiMo-V2.5-ASR (local 8B, GPU-only) | zh/en/code-switch/dialects | No | All presets include **speaker diarization** (CAM++) and **VAD** (FSMN). `mimo` reuses the FSMN VAD + CAM++ stack around MiMo's text output. > **Diarization caveat:** `auto` and `whisper` do not output per-sentence timestamps, > so speaker diarization does not work with these presets. Use `zh`, `zh-basic`, > `en`, or `mimo` when speaker identification is needed (e.g., podcasts, meetings). ## Workflow Before starting transcription, **always ask the user**: 1. **Audio file** — path to the recording (required) 2. **Type** — meeting, podcast, or interview? (affects defaults) 3. **Language** — what language is spoken? (default: Chinese) 4. **Number of speakers** — how many participants? (improves diarization) 5. **Speaker names** — for podcasts: host + guest names; for meetings: attendee list 6. **Supporting files** — ask: > "Do you have any of the following to improve accuracy?" > - **Attendee / guest list** — for hotwords and speaker mapping > - **Meeting agenda or episode topic** — for hotwords (terms, names) > - **Reference documents** (show notes, prior notes) — for speaker identification and ASR correction **Adapt defaults by recording type:** - **Meeting**: default `--lang zh`, ask about supporting files - **Podcast / interview**: default `--lang zh`, `--num-speakers 2`, always ask for host + guest names, suggest `--speaker-context` for roles (do NOT use `--lang auto` — it lacks timestamps for speaker diarization) > **⚠️ `--speakers` must use the speaker's real name, not a podcast alias.** > The value passed to `--speakers` is used verbatim as the speaker label in the > output transcript. Always derive it from the host/guest's actual name (e.g. > from a shownotes "Host:" field), not from the podcast feed name or title. > > Example: if shownotes lists "Host: 张三（张三的播客）", pass `--speakers '张三'` > — not the alias "张三的播客". Add both the real name and the alias to > `hotwords.txt` so ASR can recognise both forms. > > When both `--speakers` and `--reference` are supplied, the script detects > this mistake at startup and prints an `ACTION REQUIRED` block naming the > suggested real name. **If you see that block, stop the run and re-invoke > with the corrected `--speakers` value before Phase 3** — the warning does > not abort the pipeline. If the user provides supporting materials: - Extract participant names and key terms → create `hotwords.txt` (include both real name and alias) - Extract per-person context → create `speaker-context.json` - Pass original reference document with `--reference` - Use all three together for best results ## Quick Start ### 1. Environment Setup ```bash AUTO_YES=1 bash $SCRIPTS/setup_env.sh # Or force CPU: AUTO_YES=1 bash $SCRIPTS/setup_env.sh cpu ``` The setup script patches FunASR's spectral clustering for O(N²·k) performance. Without this, recordings over ~1 hour hang for hours during speaker clustering. ### 2. Run Transcription Output files are written to the current working directory. **LLM cleanup (Phase 3) is opt-in.** By default, transcription runs locally without contacting any external service. To enable LLM-powered ASR correction and speaker name refinement, pass `--model <model-id>`. Use LLM cleanup when: - The raw transcript has many ASR errors (names, technical terms) - You need polished, publication-ready output - Speaker names need to be refined from context > **⚠️ Data Privacy:** When LLM cleanup is enabled via `--model`, transcript > excerpts are sent to external LLM providers (AWS Bedrock, Anthropic, or > OpenAI depending on the model ID). Use `--skip-llm` or omit `--model` to > keep all data local. For Bedrock, boto3 uses the standard AWS credential > chain (IAM role, SSO, `~/.aws/credentials`, env vars). ```bash # Chinese meeting with hotwords (local-only, no LLM) python3 $SCRIPTS/transcribe.py meeting.wav \ --lang zh --num-speakers 9 --hotwords hotwords.txt # English meeting with speaker names python3 $SCRIPTS/transcribe.py meeting.wav \ --lang en --speakers "Alice,Bob,Carol,Dave" # Auto-detect language (zh/en/ja/ko/yue) python3 $SCRIPTS/transcribe.py meeting.wav \ --lang auto --num-speakers 6 # Whisper for any language python3 $SCRIPTS/transcribe.py meeting.wav \ --lang whisper --num-speakers 4 # Enable LLM cleanup for polished output (requires --model) # Bedrock (uses AWS credential chain: IAM role, SSO, ~/.aws/credentials) python3 $SCRIPTS/transcribe.py meeting.wav \ --lang zh --num-speakers 9 --hotwords hotwords.txt \ --provider bedrock --model us.anthropic.claude-sonnet-4-6 # Bedrock "global" cross-region profile (recent AWS deployments) python3 $SCRIPTS/transcribe.py meeting.wav \ --provider bedrock --model global.anthropic.claude-sonnet-4-6 # Bedrock via litellm-style wrapper (supported; prefix is stripped for boto3) python3 $SCRIPTS/transcribe.py meeting.wav \ --provider bedrock --model amazon-bedrock/global.anthropic.claude-sonnet-4-6 # Anthropic API (requires ANTHROPIC_API_KEY env var) python3 $SCRIPTS/transcribe.py meeting.wav \ --provider anthropic --model claude-sonnet-4-6 # OpenAI-compatible API (requires OPENAI_API_KEY env var) python3 $SCRIPTS/transcribe.py meeting.wav \ --provider openai --model gpt-4o # Full pipeline with all supporting files + LLM (best quality) python3 $SCRIPTS/transcribe.py episode.m4a \ --lang zh --num-speakers 2 \ --hotwords hotwords.txt \ --speakers "关羽,张飞" \ --speaker-context speaker-context.json \ --reference show-notes.md \ --model us.anthropic.claude-sonnet-4-6 # Resume interrupted LLM cleanup python3 $SCRIPTS/transcribe.py meeting.wav \ --skip-transcribe --model us.anthropic.claude-sonnet-4-6 ``` ### 3. Verify Speaker Labels If the transcript has swapped speaker labels (common with podcasts), the verification script can detect and fix mismatches using LLM analysis: ```bash # Dry-run: check if host/guest are swapped python3 $SCRIPTS/verify_speakers.py podcast_raw_transcript.json \ --speakers "关羽,张飞" \ --speaker-context speaker-context.json # Apply the fix python3 $SCRIPTS/verify_speakers.py podcast_raw_transcript.json \ --speakers "关羽,张飞" \ --speaker-context speaker-context.json --fix # Multi-speaker meeting: full reassignment python3 $SCRIPTS/verify_speakers.py meeting_raw_transcript.json \ --speakers "Alice,Bob,Carol,Dave" \ --speaker-context speaker-context.json --fix # Then regenerate the markdown with corrected labels python3 $SCRIPTS/transcribe.py original.m4a \ --skip-transcribe --clean-cache ``` The script analyzes the first 5 minutes (configurable with `--minutes`) and auto-detects podcast (2 speakers, swap detection) vs meeting (N speakers, full reassignment). ## Audio Preprocessing The script automatically converts input audio to 16kHz mono FLAC and validates that no audio is lost (detects silent truncation). | Format | 4h14m meeting | Quality | Recommendation | |--------|--------------|---------|----------------| | **FLAC** | **219MB** | Lossless | **Default, safest** | | Opus | 55MB | Lossy | Risk of truncation on long files | | WAV | 465MB | Lossless | Works but larger | | Original M4A | 173MB | Source | Also works directly | **Do NOT split long recordings** — splitting breaks speaker ID consistency. ## MiMo-V2.5-ASR (optional, GPU-only) `--lang mimo` runs Xiaomi's [MiMo-V2.5-ASR](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR) locally on a CUDA GPU. Use it when: - You want to evaluate MiMo against Paraformer on Chinese audio. - The recording has heavy code-switching, dialects (Wu, Cantonese, Hokkien, Sichuanese), lyrics, or rare proper nouns that other presets mis-transcribe. **Requirements:** - CUDA ≥12.0 and **≥20 GB VRAM** (16 GB cards OOM during inference). - Python 3.12 (enforced by `setup_env.sh`). - ~20 GB weight download (one-time) and `flash-attn==2.7.4.post1` compile (needs `nvcc` from the CUDA toolkit, takes 10–30 min). **Install (opt-in):** ```bash # One-time: install MiMo on top of the standard environment AUTO_YES=1 INSTALL_MIMO=1 \ MIMO_WEIGHTS_PATH=/mnt/models/hf \ bash $SCRIPTS/setup_env.sh ``` **Run:** ```bash python3 $SCRIPTS/transcribe.py podcast.m4a \ --lang mimo --num-speakers 2 \ --mimo-weights-path /mnt/models/hf ``` **Resume after failure:** ```bash python3 $SCRIPTS/transcribe.py podcast.m4a \ --lang mimo --resume-mimo --mimo-weights-path /mnt/models/hf ``` **Limitations:** - No hotword biasing (MiMo has no API for it — `--hotwords` is ignored). - No CPU fallback. - Inference is slower than Paraformer on the same GPU (8B model vs ~0.3B); expect RTF around 0.1–0.2 on an A100. ## Key Flags | Flag | Purpose | |------|---------| | `--lang` | `zh` (default), `zh-basic`, `en`, `auto`, `whisper` | | `--hotwords` | Hotword file or string — biases ASR (zh only) | | `--reference F` | Reference file for LLM ASR correction | | `--num-speakers N` | Expected speaker count (improves diarization) | | `--speakers "A,B,C"` | Assign real names by first-appearance order | | `--speaker-context F` | JSON with per-speaker roles for LLM | | `--no-detect-gender` | Disable automatic speaker gender detection (CAM++ gender classifier) | | `--speaker-genders "A:female,B:male"` | Override per-speaker gender (also accepts positional `female,male`) | | `--audio-format` | `flac` (default), `opus`, `wav` | | `--device cpu` | Force CPU mode | | `--batch-size N` | Adjust for memory (60 for CPU, 100 if GPU OOM) | | `--phase1-only` | Exit after Phase 1 (VAD + ASR + diarization), skip Phase 2 + 3 | | `--json-out PATH` | Write raw transcript JSON to explicit path (overrides default naming) | | `--skip-transcribe` | Resume from saved `*_raw_transcript.json` | | `--skip-llm` | Skip LLM cleanup (default when `--model` is omitted) | | `--model ID` | Enable LLM cleanup with this model (auto-detects Bedrock/Anthropic/OpenAI) | | `--title "..."` | Output document title | | `--clean-cache` | Delete LLM chunk cache after completion | | `--output PATH` | Custom output file path | | `--model-cache-dir` | ModelScope model cache directory (~3GB, default: `~/.cache/modelscope/`) | | `--mimo-audio-tag` | MiMo language hint: `<chinese>` (default), `<english>`, `<auto>` | | `--mimo-batch N` | Concurrent VAD segments per MiMo call (default 1; H100/80GB can go higher) | | `--mimo-weights-path DIR` | Cache dir for MiMo weights (default: `$HF_HOME` → `~/.cache/huggingface`) | | `--resume-mimo` | Resume MiMo Phase 1 from `*_mimo_partial.json` after a mid-run failure | ## Outputs - `<stem>-transcript.md` — Final Markdown with speaker labels and timestamps - `<stem>_raw_transcript.json` — Raw Phase 1 output (for resume/analysis) ## Speaker Diarization Tips FunASR's CAM++ may merge acoustically similar speakers. To improve: 1. **`--num-speakers N`** — Hint expected count 2. **`--hotwords`** — Include participant names (Chinese names work best) 3. **`--speaker-context`** — Provide per-person keywords for LLM splitting 4. **Keyword matching** — Search `*_raw_transcript.json` for unique phrases ### Speaker gender Enabled by default: each detected speaker is classified as `male` / `female` via 3D-Speaker's CAM++ gender classifier ([REDACTED_SECRET]). The result appears next to each name in the **Speaker List** table and is injected into the LLM cleanup prompt so pronouns (他/她, he/she) get corrected. Precedence when combined: 1. `--speaker-genders "Alice:female,Bob:male"` (explicit CLI) — always wins 2. Reference text hints like `主播（女）：韩梅梅` or `Host (male): Alice` — override auto 3. CAM++ auto-detection — fallback Disable with `--no-detect-gender` if you don't need gender and want to save the ~500 MB model download and extra inference time. ## CPU-only / Low-Memory Machines Long recordings on resource-constrained machines may hit exec timeouts or OOM kills. See `references/pipeline-details.md` for workarounds: - Detach from agent timeouts with `systemd-run` or `nohup` - Prevent OOM via swap and/or `--lang zh-basic` (lighter model) ## Additional Resources - **`references/pipeline-details.md`** — Architecture, model specs, benchmarks, speaker role verification, hotword effectiveness, clustering patch - **`scripts/transcribe.py`** — Main transcription pipeline - **`scripts/verify_speakers.py`** — Speaker label verification & fix - **`scripts/llm_utils.py`** — Shared LLM infrastructure (Bedrock/Anthropic/OpenAI) - **`scripts/setup_env.sh`** — Environment setup (venv + deps + patch)

path: references/pipeline-details.md
content: # FunASR Meeting Transcription Pipeline — Technical Details

## Architecture

```
Audio File (.m4a/.mp3/.wav)
  │
  ├─ [ffmpeg] ──► 16kHz mono WAV
  │
  ├─ [Phase 1: FunASR] ──► raw_transcript.json
  │   ├─ FSMN-VAD: segment speech vs silence
  │   ├─ ASR model (language-dependent, see below)
  │   ├─ (Optional) Hotword biasing (SeACo-Paraformer only)
  │   ├─ Punctuation restoration (model-dependent)
  │   └─ CAM++: speaker embeddings → spectral clustering
  │
  ├─ [Phase 2: Post-process]
  │   ├─ Merge consecutive same-speaker utterances (<2s gap)
  │   ├─ Map speaker IDs to names (if provided)
  │   └─ Auto-verify via self-introduction detection
  │
  └─ [Phase 3: LLM cleanup] ──► transcript.md
      ├─ LLM speaker role verification (if --speaker-context provided)
      ├─ Remove fillers (um, uh, 嗯, 啊, etc.)
      ├─ Fix ASR errors (homophones, context-based)
      ├─ Polish grammar while preserving meaning
      └─ (Optional) Identify merged speakers via context
```

## Language Presets & Models

### `--lang zh` (Chinese, default) — SeACo-Paraformer with hotword support

| Component | Model ID | Params | Role |
|-----------|----------|--------|------|
| ASR | [REDACTED_SECRET] | 220M | Chinese ASR (CER 1.95%), hotword-customizable |
| VAD | [REDACTED_SECRET] | 0.4M | Voice activity detection |
| Punctuation | [REDACTED_SECRET] | 290M | Punctuation restoration |
| Speaker | [REDACTED_SECRET] | 7.2M | Speaker diarization |

SeACo-Paraformer accepts a `--hotwords` parameter (space-separated string or .txt file)
to bias recognition toward specific terms. See [Hotword Biasing](#hotword-biasing) below.

### `--lang zh-basic` (Chinese, no hotword)

Same as `zh` but uses the base Paraformer-large without hotword support.
Use when hotword biasing is unnecessary or causing issues with English terms.

| Component | Model ID | Params | Role |
|-----------|----------|--------|------|
| ASR | [REDACTED_SECRET] | 220M | Chinese ASR (CER 1.95%) |

### `--lang en` (English)

| Component | Model ID | Params | Role |
|-----------|----------|--------|------|
| ASR | [REDACTED_SECRET] | 220M | English ASR |

### `--lang auto` (Auto-detect: zh/en/ja/ko/yue)

| Component | Model ID | Params | Role |
|-----------|----------|--------|------|
| ASR | `iic/SenseVoiceSmall` | 234M | Multi-language ASR with auto language detection |

SenseVoiceSmall includes built-in punctuation and supports emotion detection.

### `--lang whisper` (Multilingual, 99 languages)

| Component | Model ID | Params | Role |
|-----------|----------|--------|------|
| ASR | `iic/Whisper-large-v3-turbo` | 809M | OpenAI Whisper via FunASR, broadest language coverage |

All presets share the same VAD (`fsmn-vad`) and speaker diarization (`cam++`) models.
Models are auto-downloaded from ModelScope on first run.

## Hotword Biasing

SeACo-Paraformer (`--lang zh`) supports hotword customization to improve recognition
of specific terms — particularly useful for participant names, project names, and
domain-specific jargon in meetings.

### How to provide hotwords

```bash
# Space-separated string
python3 transcribe.py meeting.wav --lang zh --hotwords "张三 李四 ClawCon Rebase"

# Text file (one word per line)
python3 transcribe.py meeting.wav --lang zh --hotwords hotwords.txt
```

### What to include in hotwords

For meeting transcription, a good hotwords file includes:
- **Participant names** (full names in the meeting's language)
- **Project / product names** (internal codenames, product brands)
- **Domain-specific Chinese terms** (technical jargon, acronyms in Chinese)
- **Organization names** (company, team, department names)

### Effectiveness — empirical results

Tested on a 4h14m, 9-speaker Chinese meeting with 27 hotwords:

| Term | Without hotwords | With hotwords | Change |
|------|-----------------|---------------|--------|
| 龙虾 (lobechat) | 28 | 42 | **+50%** |
| 高琦 (person name) | 0 | 7 | **0 → 7** |
| 搬瓦工 (BandwagonHost) | 0 | 1 | **0 → 1** |
| 谢锐 (person name) | 0 | 1 | improved |
| 鲲鹏 (org name) | 6 | 7 | slight improvement |
| Rebase (English) | 5 | 0 | **regression** |
| Tailwind (English) | 3 | 1 | **regression** |

**Key findings:**
1. **Chinese terms benefit significantly** — names, brands, and Chinese jargon
   see clear improvement (龙虾 +50%, 高琦 from zero)
2. **English terms may regress** — SeACo's hotword biasing operates on Chinese
   token vocabulary; English loanwords can be disrupted
3. **Person names have limited uplift** — meeting participants rarely say full
   names aloud; hotwords help only when names do appear in speech

**Recommendation:** Include Chinese terms and names in hotwords. For English
technical terms, rely on Phase 3 LLM cleanup rather than hotword biasing.
If English term accuracy is critical and hotword biasing causes regressions,
use `--lang zh-basic` instead.

## Performance Benchmarks

Tested on a 4h14m, 9-speaker Chinese meeting recording (GPU: L40S 46GB):

| Metric | Paraformer-large | SeACo + Hotword | Notes |
|--------|-----------------|-----------------|-------|
| Model load | 14s | 14s (422s first run, downloading 944MB model) | Cached after first run |
| Transcription | 169s | 168s | Virtually identical |
| Raw sentences | 6672 | 6725 | Comparable |
| Merged segments | 1695 | 1724 | Comparable |
| Speakers detected | 7 (of 9) | 7 (of 9) | Same diarization result |

| Metric | GPU (L40S 46GB) | CPU (estimated) |
|--------|-----------------|-----------------|
| Model load | 14s | ~30s |
| Transcription | 169s | ~30-60 min |
| Speaker clustering | ~10s (patched) | ~2-5 min (patched) |
| LLM cleanup (17 chunks) | ~35 min | ~35 min (network-bound) |
| Total | ~38 min | ~70-100 min |

**Without the clustering patch**, the original `scipy.linalg.eigh()` on the full Laplacian
matrix was O(N^3) and took **10+ hours** on this recording. The patch reduces it to O(N^2*k)
via `scipy.sparse.linalg.eigsh()`.

## Clustering Patch (Critical for Long Meetings)

FunASR's `SpectralCluster.get_spec_embs()` uses `scipy.linalg.eigh(L)` which computes
ALL eigenvalues of the NxN Laplacian. For a 4-hour recording, N can be 6000+, making
this O(N^3) operation take hours.

The patch (`scripts/patch_clustering.py`) replaces this with:
- `scipy.sparse.linalg.eigsh(L_sparse, k=num_speakers, which='SM')` — only computes
  the k smallest eigenvalues needed, reducing complexity to O(N^2 * k)
- Vectorized `p_pruning()` — replaces Python loop with numpy broadcasting

**Always run the patch before processing meetings longer than ~1 hour.**

## Speaker Role Verification

Speaker names are assigned by first-appearance order in the audio, which may
swap host/guest labels (especially in podcasts). Two layers of automatic
verification, plus a standalone post-hoc tool:

1. **Phase 2 — self-introduction detection**: scans the first 5 minutes for
   explicit self-introductions ("我是X", "I'm X") and swaps labels if mismatched
2. **Phase 3 — LLM role verification**: when `--speaker-context` is provided,
   the LLM analyzes the first chunk (up to 15 minutes) before cleanup begins.
   For 2 speakers: binary CORRECT/SWAP detection. For 3+ speakers: full
   JSON-based reassignment matching each label to the correct person.
3. **Post-hoc — `verify_speakers.py`**: standalone script that verifies any
   existing `*_raw_transcript.json`. Same two modes (2-speaker swap, N-speaker
   reassignment) with dry-run support. See SKILL.md § Verify Speaker Labels.

For podcasts, always provide `--speaker-context` describing host/guest roles.

## Speaker Diarization Limitations

FunASR's CAM++ speaker diarization may merge acoustically similar speakers into one ID.
In the tested 9-person meeting, only 7 unique IDs were detected (two pairs merged).

Workarounds:
1. **Provide `--num-speakers N`** to hint expected count (uses `preset_spk_num`)
2. **Post-hoc keyword matching**: use reference documents (meeting agendas, attendee notes)
   to identify which speaker ID maps to which person
3. **LLM-assisted splitting**: provide `--speaker-context` with per-person keywords;
   the LLM can then split merged speakers when context is clear (~73% success rate)

## Supporting Files for Better Results

Prepare these files before transcription for best results:

| File | Used in | Purpose |
|------|---------|---------|
| `hotwords.txt` | Phase 1 (`--hotwords`) | Bias ASR toward names and terms |
| `speaker-context.json` | Phase 3 (`--speaker-context`) | Help LLM identify and split speakers |
| Meeting agenda | Manual reference | Identify meeting phases for post-analysis |
| Attendee list | Build hotwords + speaker names | Map speaker IDs to real names |

### Example: preparing supporting files from a meeting invite

```bash
# 1. Create hotwords.txt from attendee list and agenda
cat > hotwords.txt << 'EOF'
Alice
Bob
Carol
ProjectAlpha
Sprint Review
Q2 OKR
EOF

# 2. Create speaker-context.json from attendee roles
cat > speaker-context.json << 'EOF'
{
  "Alice": "Engineering manager, discusses sprint velocity and tech debt",
  "Bob": "Product manager, presents roadmap and customer feedback",
  "Carol": "Designer, shows mockups, mentions Figma and user testing"
}
EOF

# 3. Run with both
python3 transcribe.py meeting.wav \
  --lang zh --num-speakers 3 \
  --speakers "Alice,Bob,Carol" \
  --hotwords hotwords.txt \
  --speaker-context speaker-context.json
```

## Audio Preprocessing

FunASR works best with 16kHz mono audio. **FLAC is recommended** over WAV — lossless
quality at ~50% the file size, and FunASR reads it natively via soundfile.

```bash
# Recommended: FLAC (lossless, compact)
ffmpeg -i recording.m4a -ar 16000 -ac 1 -sample_fmt s16 meeting.flac

# Alternative: WAV (lossless, larger)
ffmpeg -i recording.m4a -ar 16000 -ac 1 meeting.wav
```

**Important:** Use `-sample_fmt s16` when converting to FLAC — without it, ffmpeg
may output 24-bit samples (s32/24bit) which doubles the file size with no ASR benefit.

### Format comparison (4h14m meeting)

| Format | Size | Quality | FunASR support |
|--------|------|---------|---------------|
| **FLAC (16kHz mono s16)** | **219MB** | Lossless | Native (soundfile) |
| WAV (16kHz mono) | 465MB | Lossless | Native (soundfile) |
| Opus (32kbps) | 54MB | Lossy | Native (soundfile) |
| M4A/AAC (original 48kHz) | 173MB | Source | Via librosa |
| M4A/AAC (16kHz 32kbps) | 60MB | Lossy | Via librosa |

FunASR accepts all common audio formats. FLAC offers the best trade-off: lossless
quality, reasonable size, and native reader support without librosa fallback.

For long recordings, do NOT split the audio — FunASR handles arbitrarily long files
and splitting breaks speaker consistency across segments.

## Resume / Checkpoint Support

The pipeline supports resuming interrupted runs:
- **Phase 1 output**: `<stem>_raw_transcript.json` — use `--skip-transcribe` to skip ASR
- **Phase 3 cache**: `<stem>_llm_cache/chunk_NNN.txt` — already-cleaned chunks are reused
  (kept by default; add `--clean-cache` to delete after completion)

## Model Caching

FunASR models (~3 GB for the `zh` preset) are downloaded from ModelScope on first run
and cached in `~/.cache/modelscope/hub/`. On ephemeral instances (EC2, cloud VMs), the
cache is lost when the instance is replaced, requiring a ~2 minute re-download.

To persist the cache on durable storage (e.g., an EBS data volume):

```bash
# Via CLI flag (recommended)
python3 transcribe.py meeting.flac --model-cache-dir /data/modelscope-cache ...

# Via environment variable
MODELSCOPE_CACHE=/data/modelscope-cache python3 transcribe.py meeting.flac ...
```

The `systemd-run` examples below include `-E MODELSCOPE_CACHE=...` for this reason.

## Speaker Context JSON Format

The `--speaker-context` file helps the LLM identify speakers and fix ASR errors:

```json
{
  "Alice": "Discussed Q1 revenue targets, mentioned Chicago office relocation",
  "Bob": "Presented the new CI/CD pipeline, uses Terraform and ArgoCD",
  "Carol": "HR updates, mentioned hiring freeze and new PTO policy"
}
```

The context is injected into the LLM system prompt for each cleanup chunk.

## Running on CPU-only / Low-Memory Machines

Long recordings (2+ hours) on resource-constrained machines (CPU-only, ≤8 GB RAM)
face two common failure modes. Both are silent — the process is killed mid-run
with no output files saved.

### Problem 1: Process killed by execution timeout

AI coding agents (Claude Code, OpenClaw, Cursor, etc.) impose execution timeouts
on shell commands — typically 2–10 minutes. On a 4-hour recording, CPU transcription
takes 1.5–2 hours, well past any agent timeout. The process is silently killed.

**Fix — detach the ASR phase from the agent's process supervision.** Use `--skip-llm`
for the detached run because Phase 1 (ASR) is the CPU-intensive bottleneck; Phase 3
(LLM cleanup) is network-bound and fast — run it afterward via `--skip-transcribe`
under the normal agent session.

Option A: `systemd-run` (preferred on systemd hosts):

```bash
systemd-run --user --unit=transcribe-job \
  --working-directory=/tmp \
  -E MODELSCOPE_CACHE=/data/modelscope-cache \
  bash -c 'source /path/to/.venv/bin/activate && \
    python3 /path/to/transcribe.py /tmp/meeting.flac \
    --lang zh --num-speakers 9 --skip-llm > /tmp/transcribe.log 2>&1'

# Monitor progress
systemctl --user status transcribe-job.service
tail -f /tmp/transcribe.log

# Check result
ls -lh /tmp/*-transcript.md /tmp/*_raw_transcript.json
```

`systemd-run` creates a transient systemd service fully independent of the agent
session — it survives session resets, context pruning, and exec timeouts.

> **Warning:** `systemd-run` creates an isolated mount namespace. FUSE mounts
> (rclone, sshfs, Google Drive, etc.) from the parent session are NOT visible
> to the transient service. Copy all dependency files (audio, hotwords,
> speaker-context, reference documents) to a local path (e.g., `/tmp`) before
> launching. Always use `--working-directory=/tmp` or another real filesystem path.

> **Note:** `systemd-run --user` requires a user-level systemd instance. It may not
> work in Docker containers or cloud VMs without `loginctl enable-linger`.

Option B: `nohup` (works everywhere):

```bash
nohup bash -c 'source .venv/bin/activate && python3 transcribe.py meeting.flac \
  --lang zh --num-speakers 9 --skip-llm' > transcribe.log 2>&1 &

echo $!  # Save PID for monitoring
tail -f transcribe.log
```

### Problem 2: OOM kill on machines with ≤8 GB RAM

The `zh` preset loads 4 model components simultaneously (SeACo-Paraformer + VAD +
Punctuation + CAM++ speaker). Peak RSS can exceed 7 GB on a 4-hour recording. On
machines without swap, the OOM killer terminates the process silently.

**Fix A — add swap before running** (requires root):

```bash
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# After transcription, optionally remove swap
sudo swapoff /swapfile && sudo rm /swapfile
```

**Fix B — use `zh-basic` instead of `zh`:**

`zh-basic` (Paraformer-large) loads one fewer model component than `zh`
(SeACo-Paraformer), reducing peak RSS by ~1–1.5 GB. Accuracy is slightly lower
(no hotword biasing) but sufficient for most meetings:

```bash
python3 transcribe.py meeting.flac --lang zh-basic --num-speakers 9 --skip-llm
```

**Combining both fixes** (swap + `zh-basic`) reliably handles 4+ hour recordings on
machines with as little as 8 GB RAM + 4 GB swap.

### Recommended CPU workflow for long recordings

```bash
# 1. Add swap if RAM ≤ 8 GB
sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile \
  && sudo mkswap /swapfile && sudo swapon /swapfile

# 2. Launch transcription detached from agent timeout
nohup bash -c 'source .venv/bin/activate && python3 transcribe.py meeting.flac \
  --lang zh-basic --num-speakers 9 --skip-llm' > transcribe.log 2>&1 &

# 3. Monitor
tail -f transcribe.log

# 4. When done, resume with LLM cleanup (network-bound, runs fine under agent)
python3 transcribe.py meeting.flac --skip-transcribe
```

## Podcast Transcription

The pipeline handles podcasts and interviews with the same engine, but the workflow
differs from meetings:

### Key differences from meetings

| Aspect | Meeting | Podcast / Interview |
|--------|---------|---------------------|
| Speakers | 3–15+, often unknown | 2–3, usually known (host + guests) |
| Language | Usually single | May mix languages (bilingual hosts) |
| Hotwords | Participant names + terms | Show name, guest name, topic terms |
| Speaker context | Role-based keywords | Host asks questions, guest answers |
| Diarization | Critical | Easier (fewer, distinct voices) |

### Recommended settings

```bash
# English podcast (2 speakers, host + guest)
python3 transcribe.py episode.flac --lang en --num-speakers 2 \
  --speakers "Host,Guest"

# Bilingual podcast (auto-detect language switches)
python3 transcribe.py episode.flac --lang auto --num-speakers 2 \
  --speakers "Alice,Bob"

# Chinese podcast with topic hotwords
python3 transcribe.py episode.flac --lang zh --num-speakers 3 \
  --speakers "主持人,嘉宾A,嘉宾B" \
  --hotwords "播客名 嘉宾全名 讨论主题关键词"

# Multi-language podcast (e.g., Spanish + English)
python3 transcribe.py episode.flac --lang whisper --num-speakers 2 \
  --speakers "Host,Guest"
```

### Tips for podcast transcription

1. **Always provide `--num-speakers`** — podcasts have a known, fixed speaker count;
   this dramatically improves diarization accuracy with only 2–3 voices
2. **Always provide `--speakers`** — host/guest names are known upfront
3. **`--lang auto`** works for bilingual transcript-only output (no speaker labels) —
   SenseVoiceSmall handles intra-utterance language switching (zh/en/ja/ko/yue) but
   does not output timestamps, so **speaker diarization is not supported**.
   Use `--lang zh` for Chinese podcasts that need speaker identification.
4. **`--lang whisper`** for any other language or heavy code-switching (also lacks
   timestamp support for diarization — transcript-only)
5. **Hotwords** — for Chinese podcasts, include the show name and guest's full name;
   for English podcasts, hotwords are usually unnecessary
6. **`--speaker-context`** — describe the host/guest dynamic:
   ```json
   {
     "Alice": "Host, asks questions, introduces topics, wraps up segments",
     "Bob": "Guest, expert on topic X, shares personal anecdotes"
   }
   ```
7. **Audio quality** — podcasts are typically studio-recorded with better SNR than
   meetings; diarization accuracy is correspondingly higher
8. **Audio source matters** — mobile app downloads may be truncated (trial/preview
   versions). Download from the web interface for complete files. The script's
   Phase 0 duration validation catches conversion truncation but cannot detect
   a source file that is already incomplete.

## `--lang mimo` — Xiaomi MiMo-V2.5-ASR (local, GPU)

MiMo is an 8B-parameter LLM-based ASR model from Xiaomi. It outputs plain text
with no per-sentence timestamps and no speaker labels. The `audio-transcribe`
skill wraps it in a VAD + speaker-clustering sandwich so output format matches
the FunASR presets:

```
Phase 1a  FSMN VAD           → [(start_ms, end_ms), ...]
Phase 1b  MiMo asr_sft()     → text per VAD segment
Phase 1c  CAM++ + KMeans     → speaker ID per VAD segment
```

Files: `scripts/mimo_asr.py` (orchestrator), `scripts/setup_mimo.sh` (installer).

### Expected RTF

On a single A100 (40 GB), 4h audio → ~24 min wall clock for Phase 1 (RTF ≈
0.1). Compare to `--lang zh` on the same GPU at RTF ≈ 0.02–0.05. Trade speed
for reported accuracy gains on dialects, code-switching, and lyrics.

### Resume (`--resume-mimo`)

Segment-level failures (OOM, CUDA error) retry 3× with
[0.5s, 2s, 5s] backoff after `gc.collect()` + `torch.cuda.empty_cache()`. If all
retries fail, a `*_mimo_partial.json` file captures VAD segments, completed
transcriptions, and the failed index. `--resume-mimo` picks up from the failed
segment, verifying audio SHA256 + `--mimo-audio-tag` match before continuing.


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)