npm package: @omote/r3f
Description: React Three Fiber adapter for Omote AI Characters
Latest version: 0.3.3

# @omote/r3f

React Three Fiber adapter for [Omote](https://github.com/omoteai/muse) AI characters. Drop-in `<OmoteAvatar>` component with lip sync, gaze tracking, emotion blending, and animation support.

## Installation

```bash
npm install @omote/r3f @omote/three @omote/core @react-three/fiber @react-three/drei three three-stdlib
```

**Required peer dependencies**: `@omote/three` (>=0.3.0), `@react-three/drei` (>=9.0.0), `three-stdlib` (>=2.0.0), `three` (>=0.150.0), `react` (>=18.0.0).

## Quick Start

```tsx
import { Suspense } from 'react';
import { Canvas } from '@react-three/fiber';
import { OmoteAvatar } from '@omote/r3f';

function App() {
  return (
    <Canvas>
      <ambientLight />
      <Suspense fallback={null}>
        <OmoteAvatar src="/models/avatar.glb" />
      </Suspense>
    </Canvas>
  );
}
```

> **Important**: `<OmoteAvatar>` uses R3F's `useLoader` which triggers React Suspense. Always wrap it in a `<Suspense>` boundary.

## Lip Sync (TTS Playback)

```tsx
import { Suspense } from 'react';
import { Canvas } from '@react-three/fiber';
import { OmoteAvatar, usePlaybackPipeline } from '@omote/r3f';

function AvatarWithLipSync() {
  const pipeline = usePlaybackPipeline({
    lamModelUrl: '/models/lam.onnx',
    autoLoad: true,
  });
  return <OmoteAvatar src="/models/avatar.glb" frameRef={pipeline.frameRef} />;
}

function App() {
  return (
    <Canvas>
      <Suspense fallback={null}>
        <AvatarWithLipSync />
      </Suspense>
    </Canvas>
  );
}
```

## Conversational Avatar

```tsx
import { useRef, Suspense } from 'react';
import { Canvas } from '@react-three/fiber';
import { OmoteAvatar, type OmoteAvatarRef } from '@omote/r3f';
import { createKokoroTTS } from '@omote/core';

function ConversationalAvatar() {
  const avatarRef = useRef<OmoteAvatarRef>(null);

  async function handleReady() {
    await avatarRef.current?.connectVoice({
      mode: 'local',
      tts: createKokoroTTS(),
      onTranscript: async (text) => {
        const res = await fetch('/api/chat', { method: 'POST', body: text });
        return await res.text();
      },
    });
  }

  return (
    <OmoteAvatar
      ref={avatarRef}
      src="/models/avatar.glb"
      onReady={handleReady}
    />
  );
}

function App() {
  return (
    <Canvas>
      <Suspense fallback={null}>
        <ConversationalAvatar />
      </Suspense>
    </Canvas>
  );
}
```

## Props Reference

### `<OmoteAvatar>`

| Prop | Type | Default | Description |
|------|------|---------|-------------|
| `src` | `string` | *required* | URL to `.glb` avatar model |
| `frameRef` | `RefObject<Float32Array \| null>` | — | 52-channel blendshape frame from pipeline hooks |
| `rawFrameRef` | `RefObject<Float32Array \| null>` | — | Raw (unscaled) blendshape frame |
| `compositorConfig` | `FaceCompositorConfig` | — | FaceCompositor configuration |
| `emotion` | `string \| EmotionWeights \| null` | — | Emotion label (`'happy'`, `'sad'`, etc.) or weights object |
| `stateRef` | `RefObject<ConversationalState>` | — | Conversational state (`'idle'`, `'listening'`, `'speaking'`) |
| `isSpeakingRef` | `RefObject<boolean>` | — | Speaking state for blink suppression |
| `gaze` | `GazeConfig` | `{ enabled: true, target: 'camera' }` | Gaze tracking configuration |
| `animations` | `AnimationSource[]` | — | External FBX animation sources |
| `currentAnimation` | `string` | — | Active animation ID (triggers crossfade) |
| `boneFilter` | `BoneFilterConfig` | `DEFAULT_BONE_FILTER` | Bones excluded from animation playback |
| `avatarScale` | `number` | `1` | Uniform scale multiplier |
| `onReady` | `(event: AvatarReadyEvent) => void` | — | Called when avatar is loaded and configured |
| `onAnimationChange` | `(id: string) => void` | — | Called when animation changes |
| `onAnimationFinished` | `(event) => void` | — | Called when a non-looping animation finishes |

All standard `<group>` props (position, rotation, scale, etc.) are also supported.

### `useOmoteAvatar` (Headless)

For custom rendering pipelines without the `<OmoteAvatar>` component:

```tsx
const gltf = useGLTF('/models/avatar.glb');
const { update, compositor, animationControls, parts, playAnimation } = useOmoteAvatar({
  scene: gltf.scene,
  embeddedAnimations: gltf.animations,
  frameRef: pipeline.frameRef,
  gaze: { enabled: true, target: 'camera' },
});

useFrame((state, delta) => {
  update(delta, state.camera, myGroup.current?.rotation.y);
});
```

### Pipeline Hooks

| Hook | Description |
|------|-------------|
| `usePlaybackPipeline` | TTS audio playback with A2E lip sync |
| `useTTSPlayback` | High-level speak(text) with lip sync (wraps TTSSpeaker) |
| `useMicLipSync` | Microphone input with real-time lip sync |
| `useListener` | Speech listener with VAD + ASR (wraps SpeechListener) |

All pipeline hooks return `frameRef` and `rawFrameRef` — pass directly to `<OmoteAvatar>`.

#### useTTSPlayback

```tsx
import { useTTSPlayback } from '@omote/r3f';

const { frameRef, speak, isSpeaking, stop } = useTTSPlayback({
  tts: myTTSBackend, // required — any TTSBackend (KokoroTTSInference, etc.)
  profile: { mouth: 1.0, jaw: 1.0, brows: 0.6 },
});

// Speak text with lip sync
await speak("Hello world!");

// Pass frameRef to OmoteAvatar
<OmoteAvatar src="/avatar.glb" frameRef={frameRef} />
```

#### useListener

```tsx
import { useListener } from '@omote/r3f';

const { transcript, start, stop, isListening } = useListener({
  config: {
    models: {
      senseVoice: { modelUrl: '/models/sensevoice/model.int8.onnx' },
      vad: { modelUrl: '/models/silero-vad.onnx' },
    },
  },
  autoLoad: true,
});
```

### Utilities

| Export | Description |
|--------|-------------|
| `preloadAvatar(url)` | Preload a GLB model for instant display |
| `preloadAnimation(url)` | Preload an FBX animation file |
| `resolveEmotion(label)` | Convert emotion string to `EmotionWeights` |
| `discoverScene(scene)` | Pure function: traverse scene graph, find bones + morph targets |

## Browser Compatibility

See [@omote/core browser support](https://github.com/omoteai/muse#-browser-support) for WebGPU/WASM details.

## Migration from v0.1.x

See [MIGRATION.md](../../MIGRATION.md) for a detailed guide on breaking changes.

## License

MIT