Skill: charantejmandali18/voice-assistant (version 0.1.0)

--- name: voice-assistant description: Real-time voice assistant for OpenClaw. Streams mic audio through configurable STT (Deepgram or ElevenLabs) into your OpenClaw agent, then speaks the response via configurable TTS (Deepgram Aura or ElevenLabs). Sub-2s time-to-first-audio with full streaming at every stage. metadata: openclaw: emoji: "🎙️" requires: bins: ["uv"] env: [] primaryEnv: "VOICE_STT_PROVIDER" install: - id: "uv" kind: "brew" formula: "uv" bins: ["uv"] label: "Install uv (brew)" --- # Voice Assistant Real-time voice interface for your OpenClaw agent. Talk to your agent and hear it respond — with configurable STT and TTS providers, full streaming at every stage, and sub-2 second time-to-first-audio. ## Architecture ``` Browser Mic → WebSocket → STT (Deepgram / ElevenLabs) → Text → OpenClaw Gateway (/v1/chat/completions, streaming) → Response Text → TTS (Deepgram Aura / ElevenLabs) → Audio chunks → Browser Speaker ``` The voice interface connects to your running OpenClaw gateway's OpenAI-compatible endpoint. It's the same agent with all its context, tools, and memory — just with a voice. ## Quick Start ```bash cd {baseDir} cp .env.example .env # Fill in your API keys and gateway URL uv run scripts/server.py # Open http://localhost:7860 and click the mic ``` ## Supported Providers ### STT (Speech-to-Text) | Provider | Model | Latency | Notes | |-----------|------------------|----------|------------------------------| | Deepgram | nova-2 (streaming) | ~200-300ms | WebSocket streaming, best accuracy/speed | | ElevenLabs | Scribe v1 | ~300-500ms | REST-based, good multilingual | ### TTS (Text-to-Speech) | Provider | Model | Latency | Notes | |------------|--------------|----------|--------------------------------| | Deepgram | aura-2 | ~200ms | WebSocket streaming, low cost | | ElevenLabs | Turbo v2.5 | ~300ms | Best voice quality, streaming | ## Configuration All configuration is via environment variables in `.env`: ```bash # === Required === OPENCLAW_GATEWAY_URL=http://localhost:4141/v1 # Your OpenClaw gateway OPENCLAW_MODEL=claude-sonnet-4-5-20250929 # Model your gateway routes to # === STT Provider (pick one) === VOICE_STT_PROVIDER=deepgram # "deepgram" or "elevenlabs" DEEPGRAM_API_KEY=your-key-here # Required if STT=deepgram [REDACTED_SECRET] # Required if STT=elevenlabs # === TTS Provider (pick one) === VOICE_TTS_PROVIDER=elevenlabs # "deepgram" or "elevenlabs" # Uses the same API keys as above # === Optional Tuning === VOICE_TTS_VOICE=rachel # ElevenLabs voice name/ID VOICE_TTS_VOICE_DG=aura-2-theia-en # Deepgram Aura voice VOICE_VAD_SILENCE_MS=400 # Silence before end-of-turn (ms) VOICE_SAMPLE_RATE=16000 # Audio sample rate VOICE_SERVER_PORT=7860 # Server port VOICE_SYSTEM_PROMPT="" # Optional system prompt override ``` ## Provider Combinations | Setup | Best For | |------------------------------------|---------------------------------| | Deepgram STT + ElevenLabs TTS | Best quality voice output | | Deepgram STT + Deepgram TTS | Lowest latency, single vendor | | ElevenLabs STT + ElevenLabs TTS | Best multilingual support | ## How It Works 1. **Browser captures mic audio** via Web Audio API and streams raw PCM over a WebSocket 2. **Server receives audio** and pipes it to the configured STT provider's streaming endpoint 3. **STT returns partial transcripts** in real-time; on end-of-utterance the full text is sent to the OpenClaw gateway 4. **OpenClaw gateway streams** the LLM response token-by-token via SSE (Server-Sent Events) 5. **Tokens are accumulated** into sentence-sized chunks and streamed to the TTS provider 6. **TTS returns audio chunks** that are immediately forwarded to the browser over the same WebSocket 7. **Browser plays audio** using the Web Audio API with a jitter buffer for smooth playback ## Interruption Handling (Barge-In) When the user starts speaking while the agent is still talking: - Current TTS audio is immediately cancelled - The agent stops its current response - New STT session begins capturing the user's interruption ## Usage Examples ``` User: "Hey, set up my voice assistant" → OpenClaw runs: cd {baseDir} && cp .env.example .env → Opens .env for the user to fill in API keys → Runs: uv run scripts/server.py User: "Start a voice chat" → Opens http://localhost:7860 in the browser User: "Switch TTS to Deepgram" → Updates VOICE_TTS_PROVIDER=deepgram in .env → Restarts the server ``` ## Troubleshooting - **No audio output?** Check that your TTS API key is valid and the provider is set correctly - **High latency?** Use Deepgram for both STT and TTS; ensure your gateway is on the same network - **Cuts off speech?** Increase `VOICE_VAD_SILENCE_MS` to 600-800ms - **Echo/feedback?** Use headphones, or enable the built-in echo cancellation in the browser UI ## Latency Budget | Stage | Target | Actual (typical) | |-------------------------|-----------|------------------| | Audio capture + VAD | <200ms | ~100-150ms | | STT transcription | <400ms | ~200-400ms | | OpenClaw LLM first token| <1500ms | ~500-1500ms | | TTS first audio chunk | <400ms | ~200-400ms | | **Total first audio** | **<2.5s** | **~1.0-2.5s** |

path: _meta.json
content: {
  "slug": "voice-assistant",
  "name": "Voice Assistant",
  "version": "0.1.0",
  "description": "Real-time voice interface for OpenClaw with configurable STT (Deepgram/ElevenLabs) and TTS (Deepgram/ElevenLabs). Sub-2s latency with full streaming.",
  "tags": [
    "voice",
    "speech",
    "stt",
    "tts",
    "realtime",
    "deepgram",
    "elevenlabs",
    "audio",
    "assistant"
  ],
  "author": "charan.mandali",
  "license": "MIT",
  "repository": "",
  "homepage": ""
}


path: pyproject.toml
content: [project]
name = "voice-assistant"
version = "0.1.0"
description = "Real-time voice assistant skill for OpenClaw"
readme = "SKILL.md"
requires-python = ">=3.10"
license = { text = "MIT" }

dependencies = [
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.34.0",
    "websockets>=14.0",
    "httpx>=0.28.0",
    "python-dotenv>=1.0.0",
]

[project.optional-dependencies]
dev = [
    "ruff>=0.8.0",
    "pytest>=8.0",
    "pytest-asyncio>=0.24.0",
]

[tool.uv]
dev-dependencies = [
    "ruff>=0.8.0",
]

[tool.uv.scripts]
server = "python scripts/server.py"

[tool.ruff]
target-version = "py310"
line-length = 100


path: scripts/server.py
content: #!/usr/bin/env python3
"""
Voice Assistant Server for OpenClaw
====================================
Real-time voice pipeline: Browser Mic → STT → OpenClaw Gateway → TTS → Browser Speaker

Supports configurable STT (Deepgram / ElevenLabs) and TTS (Deepgram / ElevenLabs)
with full streaming at every stage for sub-2s time-to-first-audio.

Usage:
    uv run scripts/server.py
    # Then open http://localhost:7860
"""

import asyncio
import base64
import json
import logging
import os
import struct
import sys
import time
from pathlib import Path
from typing import AsyncGenerator, Optional

import httpx
import uvicorn
from dotenv import load_dotenv
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from fastapi.responses import FileResponse
from fastapi.staticfiles import StaticFiles

# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------

load_dotenv(Path(__file__).parent.parent / ".env")

GATEWAY_URL = os.getenv("OPENCLAW_GATEWAY_URL", "http://localhost:4141/v1")
GATEWAY_MODEL = os.getenv("OPENCLAW_MODEL", "claude-sonnet-4-5-20250929")
STT_PROVIDER = os.getenv("VOICE_STT_PROVIDER", "deepgram").lower()
TTS_PROVIDER = os.getenv("VOICE_TTS_PROVIDER", "elevenlabs").lower()
DEEPGRAM_KEY = os.getenv("DEEPGRAM_API_KEY", "")
ELEVENLABS_KEY = os.getenv("ELEVENLABS_API_KEY", "")
TTS_VOICE_EL = os.getenv("VOICE_TTS_VOICE", "rachel")
TTS_VOICE_DG = os.getenv("VOICE_TTS_VOICE_DG", "aura-2-theia-en")
VAD_SILENCE_MS = int(os.getenv("VOICE_VAD_SILENCE_MS", "400"))
SAMPLE_RATE = int(os.getenv("VOICE_SAMPLE_RATE", "16000"))
SERVER_PORT = int(os.getenv("VOICE_SERVER_PORT", "7860"))
SYSTEM_PROMPT = os.getenv(
    "VOICE_SYSTEM_PROMPT",
    "You are a helpful voice assistant. Keep responses concise and conversational — "
    "aim for 1-3 sentences unless the user asks for detail. "
    "Never use markdown formatting, bullet points, or code blocks in your responses "
    "since they will be spoken aloud.",
)

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
log = logging.getLogger("voice-assistant")

# ---------------------------------------------------------------------------
# STT Providers
# ---------------------------------------------------------------------------


class DeepgramSTT:
    """Streaming STT via Deepgram Nova-2 WebSocket API."""

    WS_URL = "wss://api.deepgram.com/v1/listen"

    def __init__(self):
        self._ws = None
        self._transcript_queue: asyncio.Queue[str] = asyncio.Queue()
        self._listener_task: Optional[asyncio.Task] = None

    async def connect(self):
        import websockets

        params = (
            f"?encoding=linear16&sample_rate={SAMPLE_RATE}&channels=1"
            f"&model=nova-2&punctuate=true&interim_results=false"
            f"&endpointing={VAD_SILENCE_MS}&smart_format=true"
            f"&vad_events=true"
        )
        headers = {"Authorization": f"Token {DEEPGRAM_KEY}"}
        self._ws = await websockets.connect(
            self.WS_URL + params, additional_headers=headers
        )
        self._listener_task = asyncio.create_task(self._listen())
        log.info("Deepgram STT connected")

    async def _listen(self):
        try:
            async for msg in self._ws:
                data = json.loads(msg)
                msg_type = data.get("type", "")

                if msg_type == "Results":
                    channel = data.get("channel", {})
                    alternatives = channel.get("alternatives", [{}])
                    transcript = alternatives[0].get("transcript", "").strip()
                    is_final = data.get("is_final", False)
                    speech_final = data.get("speech_final", False)

                    if transcript and (is_final or speech_final):
                        log.info(f"STT final: {transcript}")
                        await self._transcript_queue.put(transcript)

        except Exception as e:
            log.error(f"Deepgram listener error: {e}")

    async def send_audio(self, audio_bytes: bytes):
        if self._ws:
            await self._ws.send(audio_bytes)

    async def get_transcript(self) -> str:
        return await self._transcript_queue.get()

    async def close(self):
        if self._ws:
            try:
                await self._ws.send(json.dumps({"type": "CloseStream"}))
                await self._ws.close()
            except Exception:
                pass
        if self._listener_task:
            self._listener_task.cancel()


class ElevenLabsSTT:
    """STT via ElevenLabs Scribe API (REST-based, non-streaming)."""

    API_URL = "https://api.elevenlabs.io/v1/speech-to-text"

    def __init__(self):
        self._audio_buffer = bytearray()
        self._transcript_queue: asyncio.Queue[str] = asyncio.Queue()
        self._silence_counter = 0
        self._has_speech = False

    async def connect(self):
        self._audio_buffer.clear()
        self._silence_counter = 0
        self._has_speech = False
        log.info("ElevenLabs STT initialized")

    async def send_audio(self, audio_bytes: bytes):
        self._audio_buffer.extend(audio_bytes)

        # Simple energy-based VAD
        if len(audio_bytes) >= 2:
            samples = struct.unpack(f"<{len(audio_bytes)//2}h", audio_bytes)
            energy = sum(abs(s) for s in samples) / len(samples)

            if energy > 500:
                self._has_speech = True
                self._silence_counter = 0
            elif self._has_speech:
                self._silence_counter += len(audio_bytes) / (SAMPLE_RATE * 2) * 1000

                if self._silence_counter >= VAD_SILENCE_MS:
                    await self._transcribe()

    async def _transcribe(self):
        if len(self._audio_buffer) < SAMPLE_RATE:  # Less than 0.5s
            self._audio_buffer.clear()
            self._has_speech = False
            self._silence_counter = 0
            return

        # Build WAV in memory
        audio_data = bytes(self._audio_buffer)
        wav = self._build_wav(audio_data)

        self._audio_buffer.clear()
        self._has_speech = False
        self._silence_counter = 0

        try:
            async with httpx.AsyncClient(timeout=10.0) as client:
                resp = await client.post(
                    self.API_URL,
                    headers={"xi-api-key": ELEVENLABS_KEY},
                    files={"file": ("audio.wav", wav, "audio/wav")},
                    data={"model_id": "scribe_v1"},
                )
                resp.raise_for_status()
                result = resp.json()
                text = result.get("text", "").strip()
                if text:
                    log.info(f"STT final: {text}")
                    await self._transcript_queue.put(text)
        except Exception as e:
            log.error(f"ElevenLabs STT error: {e}")

    def _build_wav(self, pcm_data: bytes) -> bytes:
        """Build a minimal WAV header around raw PCM data."""
        data_size = len(pcm_data)
        header = struct.pack(
            "<4sI4s4sIHHIIHH4sI",
            b"RIFF", 36 + data_size, b"WAVE",
            b"fmt ", 16, 1, 1, SAMPLE_RATE, SAMPLE_RATE * 2, 2, 16,
            b"data", data_size,
        )
        return header + pcm_data

    async def get_transcript(self) -> str:
        return await self._transcript_queue.get()

    async def close(self):
        self._audio_buffer.clear()


# ---------------------------------------------------------------------------
# TTS Providers
# ---------------------------------------------------------------------------


class ElevenLabsTTS:
    """Streaming TTS via ElevenLabs WebSocket API."""

    def __init__(self):
        self._voice_id: Optional[str] = None

    async def _resolve_voice_id(self):
        if self._voice_id:
            return
        # Try to resolve voice name to ID
        try:
            async with httpx.AsyncClient(timeout=10.0) as client:
                resp = await client.get(
                    "https://api.elevenlabs.io/v1/voices",
                    headers={"xi-api-key": ELEVENLABS_KEY},
                )
                resp.raise_for_status()
                voices = resp.json().get("voices", [])
                for v in voices:
                    if v["name"].lower() == TTS_VOICE_EL.lower():
                        self._voice_id = v["voice_id"]
                        log.info(f"Resolved ElevenLabs voice '{TTS_VOICE_EL}' → {self._voice_id}")
                        return
                # If not found by name, assume it's already an ID
                self._voice_id = TTS_VOICE_EL
        except Exception as e:
            log.warning(f"Could not resolve voice, using as ID: {e}")
            self._voice_id = TTS_VOICE_EL

    async def synthesize_stream(self, text: str) -> AsyncGenerator[bytes, None]:
        """Stream audio bytes from ElevenLabs for a given text chunk."""
        await self._resolve_voice_id()

        url = (
            f"https://api.elevenlabs.io/v1/text-to-speech/{self._voice_id}/stream"
            f"?output_format=pcm_16000"
        )
        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                async with client.stream(
                    "POST",
                    url,
                    headers={
                        "xi-api-key": ELEVENLABS_KEY,
                        "Content-Type": "application/json",
                    },
                    json={
                        "text": text,
                        "model_id": "eleven_turbo_v2_5",
                        "voice_settings": {
                            "stability": 0.5,
                            "similarity_boost": 0.75,
                        },
                    },
                ) as resp:
                    resp.raise_for_status()
                    async for chunk in resp.aiter_bytes(chunk_size=4096):
                        yield chunk
        except Exception as e:
            log.error(f"ElevenLabs TTS error: {e}")


class DeepgramTTS:
    """Streaming TTS via Deepgram Aura REST API."""

    def __init__(self):
        pass

    async def synthesize_stream(self, text: str) -> AsyncGenerator[bytes, None]:
        """Stream audio bytes from Deepgram Aura for a given text chunk."""
        url = f"https://api.deepgram.com/v1/speak?model={TTS_VOICE_DG}&encoding=linear16&sample_rate={SAMPLE_RATE}"
        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                async with client.stream(
                    "POST",
                    url,
                    headers={
                        "Authorization": f"Token {DEEPGRAM_KEY}",
                        "Content-Type": "application/json",
                    },
                    json={"text": text},
                ) as resp:
                    resp.raise_for_status()
                    async for chunk in resp.aiter_bytes(chunk_size=4096):
                        yield chunk
        except Exception as e:
            log.error(f"Deepgram TTS error: {e}")


# ---------------------------------------------------------------------------
# OpenClaw Gateway Client (OpenAI-compatible streaming)
# ---------------------------------------------------------------------------


async def stream_llm_response(
    messages: list[dict], http_client: httpx.AsyncClient
) -> AsyncGenerator[str, None]:
    """Stream text tokens from the OpenClaw gateway."""
    url = f"{GATEWAY_URL}/chat/completions"
    payload = {
        "model": GATEWAY_MODEL,
        "messages": messages,
        "stream": True,
        "temperature": 0.7,
        "max_tokens": 512,
    }

    try:
        async with http_client.stream(
            "POST",
            url,
            json=payload,
            headers={"Content-Type": "application/json"},
            timeout=60.0,
        ) as resp:
            resp.raise_for_status()
            async for line in resp.aiter_lines():
                if not line.startswith("data: "):
                    continue
                data = line[6:]
                if data.strip() == "[DONE]":
                    break
                try:
                    chunk = json.loads(data)
                    delta = chunk.get("choices", [{}])[0].get("delta", {})
                    content = delta.get("content", "")
                    if content:
                        yield content
                except json.JSONDecodeError:
                    continue
    except Exception as e:
        log.error(f"LLM stream error: {e}")
        yield "Sorry, I had trouble processing that. Could you try again?"


# ---------------------------------------------------------------------------
# Sentence Chunker — accumulates tokens into speakable sentence chunks
# ---------------------------------------------------------------------------


class SentenceChunker:
    """Accumulates streaming tokens and yields sentence-sized chunks for TTS."""

    SENTENCE_ENDERS = {".", "!", "?", ":", ";"}

    def __init__(self):
        self._buffer = ""

    def add(self, token: str) -> Optional[str]:
        self._buffer += token
        # Check if we have a complete sentence
        for i in range(len(self._buffer) - 1, -1, -1):
            if self._buffer[i] in self.SENTENCE_ENDERS:
                chunk = self._buffer[: i + 1].strip()
                self._buffer = self._buffer[i + 1 :]
                if len(chunk) > 3:  # Skip very short fragments
                    return chunk
        # Flush if buffer gets long without punctuation
        if len(self._buffer) > 150:
            chunk = self._buffer.strip()
            self._buffer = ""
            return chunk if chunk else None
        return None

    def flush(self) -> Optional[str]:
        chunk = self._buffer.strip()
        self._buffer = ""
        return chunk if chunk else None


# ---------------------------------------------------------------------------
# Voice Session — manages one WebSocket connection's full pipeline
# ---------------------------------------------------------------------------


class VoiceSession:
    """Manages a single voice conversation session."""

    def __init__(self, ws: WebSocket):
        self.ws = ws
        self.messages: list[dict] = []
        if SYSTEM_PROMPT:
            self.messages.append({"role": "system", "content": SYSTEM_PROMPT})

        self.stt = DeepgramSTT() if STT_PROVIDER == "deepgram" else ElevenLabsSTT()
        self.tts = ElevenLabsTTS() if TTS_PROVIDER == "elevenlabs" else DeepgramTTS()
        self.http_client = httpx.AsyncClient()
        self._cancelled = False
        self._speaking = False
        self._tasks: list[asyncio.Task] = []

    async def start(self):
        await self.stt.connect()
        log.info(f"Session started: STT={STT_PROVIDER}, TTS={TTS_PROVIDER}")

        # Run audio receiver and conversation loop concurrently
        recv_task = asyncio.create_task(self._receive_audio())
        conv_task = asyncio.create_task(self._conversation_loop())
        self._tasks = [recv_task, conv_task]

        try:
            await asyncio.gather(*self._tasks)
        except asyncio.CancelledError:
            pass
        finally:
            await self.cleanup()

    async def _receive_audio(self):
        """Receive audio from the browser WebSocket and forward to STT."""
        try:
            while True:
                data = await self.ws.receive()
                if data.get("type") == "websocket.disconnect":
                    break

                if "bytes" in data:
                    audio = data["bytes"]
                    # If agent is speaking and user starts talking, barge-in
                    if self._speaking:
                        if self._detect_speech(audio):
                            log.info("Barge-in detected, cancelling TTS")
                            self._cancelled = True
                    await self.stt.send_audio(audio)

                elif "text" in data:
                    msg = json.loads(data["text"])
                    if msg.get("type") == "stop":
                        break

        except WebSocketDisconnect:
            log.info("Client disconnected")
        except Exception as e:
            log.error(f"Receive error: {e}")

    def _detect_speech(self, audio_bytes: bytes) -> bool:
        """Simple energy-based speech detection for barge-in."""
        if len(audio_bytes) < 2:
            return False
        samples = struct.unpack(f"<{len(audio_bytes) // 2}h", audio_bytes)
        energy = sum(abs(s) for s in samples) / len(samples)
        return energy > 800

    async def _conversation_loop(self):
        """Main loop: wait for STT → call LLM → stream TTS."""
        try:
            while True:
                # Wait for user utterance
                transcript = await self.stt.get_transcript()
                if not transcript:
                    continue

                log.info(f"User: {transcript}")
                self.messages.append({"role": "user", "content": transcript})

                # Send transcript + status to browser
                await self._send_json({"type": "transcript", "role": "user", "text": transcript})
                await self._send_json({"type": "status", "status": "thinking"})

                # Stream LLM response and TTS
                self._cancelled = False
                self._speaking = True
                t0 = time.monotonic()
                full_response = ""

                chunker = SentenceChunker()
                first_audio = True

                async for token in stream_llm_response(self.messages, self.http_client):
                    if self._cancelled:
                        break

                    full_response += token
                    chunk = chunker.add(token)

                    if chunk:
                        if first_audio:
                            latency = (time.monotonic() - t0) * 1000
                            log.info(f"Time to first TTS chunk: {latency:.0f}ms")
                            first_audio = False
                        await self._speak_chunk(chunk)

                # Flush remaining text
                if not self._cancelled:
                    remaining = chunker.flush()
                    if remaining:
                        await self._speak_chunk(remaining)

                self._speaking = False
                self.messages.append({"role": "assistant", "content": full_response})

                # Send assistant transcript + status to browser
                if full_response.strip():
                    await self._send_json(
                        {"type": "transcript", "role": "assistant", "text": full_response}
                    )
                await self._send_json({"type": "status", "status": "listening"})
                log.info(f"Assistant: {full_response[:100]}...")

        except asyncio.CancelledError:
            pass
        except Exception as e:
            log.error(f"Conversation loop error: {e}")

    async def _speak_chunk(self, text: str):
        """Convert a text chunk to audio and send to the browser."""
        try:
            async for audio_chunk in self.tts.synthesize_stream(text):
                if self._cancelled:
                    break
                b64 = base64.b64encode(audio_chunk).decode("ascii")
                await self._send_json({"type": "audio", "data": b64})
        except Exception as e:
            log.error(f"TTS streaming error: {e}")

    async def _send_json(self, data: dict):
        try:
            await self.ws.send_json(data)
        except Exception:
            pass

    async def cleanup(self):
        await self.stt.close()
        await self.http_client.aclose()
        for t in self._tasks:
            if not t.done():
                t.cancel()
        # Await cancelled tasks to ensure clean shutdown
        await asyncio.gather(*self._tasks, return_exceptions=True)
        log.info("Session cleaned up")


# ---------------------------------------------------------------------------
# FastAPI App
# ---------------------------------------------------------------------------

app = FastAPI(title="OpenClaw Voice Assistant")

STATIC_DIR = Path(__file__).parent.parent / "static"
if STATIC_DIR.exists():
    app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")


@app.get("/")
async def index():
    return FileResponse(str(STATIC_DIR / "index.html"))


@app.get("/health")
async def health():
    return {
        "status": "ok",
        "stt_provider": STT_PROVIDER,
        "tts_provider": TTS_PROVIDER,
        "gateway": GATEWAY_URL,
    }


@app.websocket("/ws/voice")
async def voice_ws(ws: WebSocket):
    await ws.accept()
    session = VoiceSession(ws)
    await session.start()


# ---------------------------------------------------------------------------
# Entry Point
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    # Validate configuration
    errors = []
    if STT_PROVIDER == "deepgram" and not DEEPGRAM_KEY:
        errors.append("DEEPGRAM_API_KEY is required when VOICE_STT_PROVIDER=deepgram")
    if TTS_PROVIDER == "deepgram" and not DEEPGRAM_KEY:
        errors.append("DEEPGRAM_API_KEY is required when VOICE_TTS_PROVIDER=deepgram")
    if STT_PROVIDER == "elevenlabs" and not ELEVENLABS_KEY:
        errors.append("ELEVENLABS_API_KEY is required when VOICE_STT_PROVIDER=elevenlabs")
    if TTS_PROVIDER == "elevenlabs" and not ELEVENLABS_KEY:
        errors.append("ELEVENLABS_API_KEY is required when VOICE_TTS_PROVIDER=elevenlabs")

    if errors:
        for e in errors:
            log.error(f"Config error: {e}")
        sys.exit(1)

    log.info(f"Starting voice server on port {SERVER_PORT}")
    log.info(f"STT: {STT_PROVIDER} | TTS: {TTS_PROVIDER} | Gateway: {GATEWAY_URL}")

    uvicorn.run(app, host="0.0.0.0", port=SERVER_PORT, log_level="info")


path: static/index.html
content: <!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>OpenClaw Voice Assistant</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }

        body {
            font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
            background: #0a0a0f;
            color: #e0e0e0;
            height: 100vh;
            display: flex;
            flex-direction: column;
            align-items: center;
            justify-content: center;
            overflow: hidden;
        }

        .container {
            text-align: center;
            max-width: 480px;
            padding: 2rem;
        }

        h1 {
            font-size: 1.4rem;
            font-weight: 500;
            color: #ffffff;
            margin-bottom: 0.5rem;
            letter-spacing: -0.02em;
        }

        .subtitle {
            font-size: 0.85rem;
            color: #666;
            margin-bottom: 3rem;
        }

        /* Mic Button */
        .mic-container {
            position: relative;
            width: 140px;
            height: 140px;
            margin: 0 auto 2rem;
        }

        .mic-ring {
            position: absolute;
            inset: 0;
            border-radius: 50%;
            border: 2px solid #222;
            transition: all 0.3s ease;
        }

        .mic-ring.active {
            border-color: #ef4444;
            box-shadow: 0 0 40px rgba(239, 68, 68, 0.2);
        }

        .mic-ring.thinking {
            border-color: #f59e0b;
            animation: pulse-think 1.5s ease-in-out infinite;
        }

        .mic-ring.speaking {
            border-color: #22c55e;
            animation: pulse-speak 0.8s ease-in-out infinite;
        }

        @keyframes pulse-think {
            0%, 100% { box-shadow: 0 0 20px rgba(245, 158, 11, 0.1); }
            50% { box-shadow: 0 0 50px rgba(245, 158, 11, 0.3); }
        }

        @keyframes pulse-speak {
            0%, 100% { box-shadow: 0 0 20px rgba(34, 197, 94, 0.1); transform: scale(1); }
            50% { box-shadow: 0 0 50px rgba(34, 197, 94, 0.3); transform: scale(1.02); }
        }

        .mic-btn {
            position: absolute;
            inset: 10px;
            border-radius: 50%;
            background: #151520;
            border: none;
            cursor: pointer;
            display: flex;
            align-items: center;
            justify-content: center;
            transition: background 0.2s;
        }

        .mic-btn:hover { background: #1a1a28; }
        .mic-btn:active { background: #0f0f18; }

        .mic-btn svg {
            width: 36px;
            height: 36px;
            fill: none;
            stroke: #666;
            stroke-width: 1.5;
            stroke-linecap: round;
            stroke-linejoin: round;
            transition: stroke 0.2s;
        }

        .mic-btn.active svg { stroke: #ef4444; }
        .mic-btn.thinking svg { stroke: #f59e0b; }
        .mic-btn.speaking svg { stroke: #22c55e; }

        /* Status */
        .status {
            font-size: 0.9rem;
            color: #555;
            margin-bottom: 2rem;
            min-height: 1.4em;
            transition: color 0.2s;
        }

        .status.active { color: #ef4444; }
        .status.thinking { color: #f59e0b; }
        .status.speaking { color: #22c55e; }

        /* Transcript */
        .transcript-area {
            width: 100%;
            max-height: 200px;
            overflow-y: auto;
            text-align: left;
            padding: 1rem;
            background: #111118;
            border-radius: 12px;
            border: 1px solid #1a1a25;
        }

        .transcript-line {
            font-size: 0.85rem;
            line-height: 1.6;
            margin-bottom: 0.5rem;
        }

        .transcript-line .role {
            font-weight: 600;
            margin-right: 0.4rem;
        }

        .transcript-line .role.user { color: #818cf8; }
        .transcript-line .role.assistant { color: #34d399; }

        /* Config bar */
        .config-bar {
            position: fixed;
            bottom: 1.5rem;
            display: flex;
            gap: 1rem;
            font-size: 0.75rem;
            color: #444;
        }

        .config-badge {
            padding: 0.3rem 0.7rem;
            background: #111118;
            border-radius: 6px;
            border: 1px solid #1a1a25;
        }

        /* Visualizer */
        .visualizer {
            display: flex;
            align-items: center;
            justify-content: center;
            gap: 3px;
            height: 24px;
            margin-bottom: 1rem;
        }

        .visualizer .bar {
            width: 3px;
            background: #333;
            border-radius: 2px;
            transition: height 0.05s ease;
            height: 4px;
        }

        .visualizer.active .bar { background: #ef4444; }
        .visualizer.speaking .bar { background: #22c55e; }
    </style>
</head>
<body>
    <div class="container">
        <h1>OpenClaw Voice</h1>
        <p class="subtitle">Talk to your agent</p>

        <div class="visualizer" id="visualizer">
            <!-- bars generated by JS -->
        </div>

        <div class="mic-container">
            <div class="mic-ring" id="micRing"></div>
            <button class="mic-btn" id="micBtn" onclick="toggleMic()">
                <svg viewBox="0 0 24 24">
                    <path d="M12 1a4 4 0 0 0-4 4v7a4 4 0 0 0 8 0V5a4 4 0 0 0-4-4z"/>
                    <path d="M19 10v2a7 7 0 0 1-14 0v-2"/>
                    <line x1="12" y1="19" x2="12" y2="23"/>
                    <line x1="8" y1="23" x2="16" y2="23"/>
                </svg>
            </button>
        </div>

        <p class="status" id="status">Click the mic to start</p>

        <div class="transcript-area" id="transcript"></div>
    </div>

    <div class="config-bar" id="configBar">
        <span class="config-badge" id="sttBadge">STT: —</span>
        <span class="config-badge" id="ttsBadge">TTS: —</span>
        <span class="config-badge" id="latencyBadge">Latency: —</span>
    </div>

    <script>
        // ---- State ----
        let isActive = false;
        let ws = null;
        let audioContext = null;
        let micStream = null;
        let scriptNode = null;
        let playbackQueue = [];
        let isPlaying = false;
        let currentState = "idle"; // idle, listening, thinking, speaking

        const SAMPLE_RATE = 16000;

        // ---- Visualizer Setup ----
        const vizEl = document.getElementById("visualizer");
        const NUM_BARS = 24;
        for (let i = 0; i < NUM_BARS; i++) {
            const bar = document.createElement("div");
            bar.className = "bar";
            vizEl.appendChild(bar);
        }
        const bars = vizEl.querySelectorAll(".bar");

        // ---- UI Updates ----
        function setState(state) {
            currentState = state;
            const btn = document.getElementById("micBtn");
            const ring = document.getElementById("micRing");
            const statusEl = document.getElementById("status");

            btn.className = "mic-btn" + (state !== "idle" ? " " + state : "");
            ring.className = "mic-ring" + (state !== "idle" ? " " + state : "");
            statusEl.className = "status" + (state !== "idle" ? " " + state : "");
            vizEl.className = "visualizer" + (state === "listening" ? " active" : state === "speaking" ? " speaking" : "");

            const labels = {
                idle: "Click the mic to start",
                listening: "Listening...",
                thinking: "Thinking...",
                speaking: "Speaking..."
            };
            statusEl.textContent = labels[state] || state;
        }

        function addTranscript(role, text) {
            const el = document.getElementById("transcript");
            const line = document.createElement("div");
            line.className = "transcript-line";
            line.innerHTML = `<span class="role ${role}">${role === "user" ? "You" : "Agent"}:</span>${text}`;
            el.appendChild(line);
            el.scrollTop = el.scrollHeight;
        }

        // ---- Audio Playback ----
        function playAudioChunk(pcmBase64) {
            playbackQueue.push(pcmBase64);
            if (!isPlaying) drainQueue();
        }

        async function drainQueue() {
            if (playbackQueue.length === 0) {
                isPlaying = false;
                if (currentState === "speaking") setState("listening");
                return;
            }
            isPlaying = true;

            const b64 = playbackQueue.shift();
            const raw = atob(b64);
            const bytes = new Uint8Array(raw.length);
            for (let i = 0; i < raw.length; i++) bytes[i] = raw.charCodeAt(i);

            // Convert Int16 PCM to Float32
            const int16 = new Int16Array(bytes.buffer);
            const float32 = new Float32Array(int16.length);
            for (let i = 0; i < int16.length; i++) float32[i] = int16[i] / 32768.0;

            const buffer = audioContext.createBuffer(1, float32.length, SAMPLE_RATE);
            buffer.getChannelData(0).set(float32);

            const source = audioContext.createBufferSource();
            source.buffer = buffer;
            source.connect(audioContext.destination);
            source.onended = () => drainQueue();
            source.start();

            // Update visualizer with audio energy
            updateVizFromAudio(float32);
        }

        function updateVizFromAudio(samples) {
            const chunkSize = Math.floor(samples.length / NUM_BARS);
            for (let i = 0; i < NUM_BARS; i++) {
                let sum = 0;
                for (let j = 0; j < chunkSize; j++) {
                    sum += Math.abs(samples[i * chunkSize + j] || 0);
                }
                const avg = sum / chunkSize;
                const height = Math.max(4, Math.min(24, avg * 200));
                bars[i].style.height = height + "px";
            }
        }

        // ---- Mic Input ----
        async function startMic() {
            audioContext = new AudioContext({ sampleRate: SAMPLE_RATE });
            micStream = await navigator.mediaDevices.getUserMedia({
                audio: {
                    sampleRate: SAMPLE_RATE,
                    channelCount: 1,
                    echoCancellation: true,
                    noiseSuppression: true,
                    autoGainControl: true,
                }
            });

            const source = audioContext.createMediaStreamSource(micStream);
            scriptNode = audioContext.createScriptProcessor(4096, 1, 1);

            scriptNode.onaudioprocess = (e) => {
                if (!ws || ws.readyState !== WebSocket.OPEN) return;
                const float32 = e.inputBuffer.getChannelData(0);

                // Convert to Int16 PCM
                const int16 = new Int16Array(float32.length);
                for (let i = 0; i < float32.length; i++) {
                    const s = Math.max(-1, Math.min(1, float32[i]));
                    int16[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
                }

                ws.send(int16.buffer);

                // Update visualizer from mic input
                if (currentState === "listening") {
                    for (let i = 0; i < NUM_BARS; i++) {
                        const idx = Math.floor(i * float32.length / NUM_BARS);
                        const h = Math.max(4, Math.abs(float32[idx]) * 120);
                        bars[i].style.height = h + "px";
                    }
                }
            };

            source.connect(scriptNode);
            scriptNode.connect(audioContext.destination);
        }

        function stopMic() {
            if (scriptNode) { scriptNode.disconnect(); scriptNode = null; }
            if (micStream) { micStream.getTracks().forEach(t => t.stop()); micStream = null; }
            if (audioContext) { audioContext.close(); audioContext = null; }
        }

        // ---- WebSocket ----
        function connectWS() {
            const proto = location.protocol === "https:" ? "wss:" : "ws:";
            ws = new WebSocket(`${proto}//${location.host}/ws/voice`);

            ws.onopen = () => {
                setState("listening");
                fetchConfig();
            };

            ws.onmessage = (evt) => {
                const msg = JSON.parse(evt.data);

                if (msg.type === "audio") {
                    if (currentState !== "speaking") setState("speaking");
                    playAudioChunk(msg.data);
                }
                else if (msg.type === "status") {
                    if (msg.status === "thinking") setState("thinking");
                    else if (msg.status === "listening") {
                        if (currentState !== "speaking") setState("listening");
                    }
                }
                else if (msg.type === "transcript") {
                    addTranscript(msg.role, msg.text);
                }
            };

            ws.onclose = () => {
                setState("idle");
                isActive = false;
            };

            ws.onerror = (err) => {
                console.error("WebSocket error:", err);
                setState("idle");
                isActive = false;
            };
        }

        async function fetchConfig() {
            try {
                const resp = await fetch("/health");
                const data = await resp.json();
                document.getElementById("sttBadge").textContent = `STT: ${data.stt_provider}`;
                document.getElementById("ttsBadge").textContent = `TTS: ${data.tts_provider}`;
            } catch (e) {}
        }

        // ---- Toggle ----
        async function toggleMic() {
            if (isActive) {
                isActive = false;
                if (ws) ws.close();
                stopMic();
                setState("idle");
                playbackQueue = [];
                isPlaying = false;
            } else {
                isActive = true;
                await startMic();
                connectWS();
            }
        }

        // Reset visualizer bars periodically when idle
        setInterval(() => {
            if (currentState === "idle" || currentState === "thinking") {
                bars.forEach(b => b.style.height = "4px");
            }
        }, 200);
    </script>
</body>
</html>


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)