npm package: @phantasy/live2d-sdk
Description: Reusable SDK for Live2D models with TTS and lip sync integration
Author: Phantasy Team
Latest version: 0.2.14

# Phantasy Live2D SDK

Reusable Live2D SDK for web apps that need:

- Cubism 4 model loading
- pluggable TTS providers
- real-time lip sync
- PIXI filter access
- a React hook on top of the core class

## Package

```bash
npm install @phantasy/live2d-sdk
```

React users can import either from the package root or the explicit subpath:

```ts
import { usePhantasyLive2D } from '@phantasy/live2d-sdk';
// or
import { usePhantasyLive2D } from '@phantasy/live2d-sdk/react';
```

## Runtime Requirements

By default, the SDK can auto-load the pinned Live2D browser runtime for you. The current compatibility stack is:

1. Cubism Core
2. `pixi.js@6.5.2`
3. `pixi-live2d-display` Cubism 4 build

Detailed setup and troubleshooting live in [LIVE2D_SETUP.md](./LIVE2D_SETUP.md).

## Quick Start

### Vanilla / Framework-Agnostic

```ts
import { PhantasyLive2D } from '@phantasy/live2d-sdk';

const phantasy = new PhantasyLive2D({
  container: '#live2d-container',
  model: {
    path: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/Haru.model3.json',
    scale: 0.3,
    position: { x: 0.5, y: 0.7 },
    expressions: {
      F01: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F01.exp3.json',
      F02: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F02.exp3.json',
      F03: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F03.exp3.json',
      F04: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F04.exp3.json',
    },
  },
  runtime: {
    autoLoad: true,
    cubismCoreUrls: [
      '/live2d-sdk/live2dcubismcore.min.js',
      'https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js',
    ],
    pixiUrls: [
      '/vendor/pixi-6.5.2.min.js',
      'https://cdn.jsdelivr.net/npm/pixi.js@6.5.2/dist/browser/pixi.min.js',
    ],
    pluginUrls: [
      '/pixi-live2d-display-cubism4.min.js',
      'https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js',
    ],
  },
  tts: {
    primary: {
      provider: 'agent',
      apiUrl: '/api/tts',
    },
    timeout: 30000,
  },
  lipSync: {
    enabled: true,
    sensitivity: 0.8,
    smoothing: 0.3,
    amplification: 1.3,
    parameterNames: ['ParamMouthOpenY'],
  },
  features: {
    breathing: true,
    mouseTracking: false,
    orbitControls: {
      enabled: true,
    },
  },
});

await phantasy.initialize();
await phantasy.speak('Hello from Phantasy.');
phantasy.setExpression('F01');
```

### React

```tsx
import { usePhantasyLive2D } from '@phantasy/live2d-sdk/react';

export function Character() {
  const {
    phantasy,
    speak,
    setExpression,
    stopSpeech,
    isInitialized,
    isLoading,
    error,
  } = usePhantasyLive2D({
    containerId: 'live2d-container',
    model: {
      path: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/Haru.model3.json',
      scale: 0.3,
      position: { x: 0.5, y: 0.7 },
      expressions: {
        F01: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F01.exp3.json',
        F02: 'https://raw.githubusercontent.com/Live2D/CubismWebSamples/develop/Samples/Resources/Haru/expressions/F02.exp3.json',
      },
    },
    runtime: {
      autoLoad: true,
    },
    tts: {
      primary: {
        provider: 'agent',
        apiUrl: '/api/tts',
      },
      timeout: 30000,
    },
  });

  if (error) return <p>{error}</p>;

  return (
    <>
      <div id="live2d-container" style={{ width: 480, height: 640 }} />
      <button onClick={() => speak('Hello!')} disabled={!isInitialized || isLoading}>
        Speak
      </button>
      <button onClick={() => setExpression('F01')} disabled={!phantasy}>
        F01
      </button>
      <button onClick={stopSpeech} disabled={!phantasy}>
        Stop
      </button>
    </>
  );
}
```

The examples in this repository use Live2D's public `Haru` sample model so they can run without Rally-owned assets. Speech still requires a working TTS endpoint.

## Public API

### `PhantasyLive2D`

Core lifecycle:

- `initialize()`
- `destroy()`
- `getStatus()`
- `getModel()`
- `getPixiApp()`

Speech and lip sync:

- `speak(text)`
- `stopSpeech()`

Expressions:

- `setExpression(expression)`
- `setExpressionStack(expressions)`
- `getAvailableExpressions()`

Model transforms:

- `setPosition(x, y)`
- `setScale(scale)`

PIXI filter hooks:

- `setModelFilters(filters)`
- `clearModelFilters()`
- `setStageFilters(filters)`
- `clearStageFilters()`

Orbit controls:

- `enableOrbitControls()`
- `disableOrbitControls()`
- `setZoom(zoom)`
- `setPan(x, y)`
- `resetOrbitControls()`
- `getOrbitControlsState()`

Events:

- `initialized`
- `destroyed`
- `modelLoaded`
- `expressionChanged`
- `speechStarted`
- `speechEnded`
- `speechError`
- `lipSyncData`
- `lipSyncStarted`
- `lipSyncEnded`
- `error`
- `warning`

Event helpers come from the built-in emitter:

- `on(event, callback)`
- `off(event, callback)`
- `once(event, callback)`

### `usePhantasyLive2D`

The React hook returns:

- `phantasy`
- `isLoading`
- `isInitialized`
- `error`
- `speak(text)`
- `setExpression(expression)`
- `stopSpeech()`
- `initialize()`
- `destroy()`

## Configuration

```ts
type PhantasyLive2DConfig = {
  container: string | HTMLElement;
  model: string | {
    path: string;
    scale?: number;
    position?: { x: number; y: number };
    expressions?: Record<string, string>;
    responsive?: {
      mobile?: Partial<Live2DModelConfig>;
      tablet?: Partial<Live2DModelConfig>;
      desktop?: Partial<Live2DModelConfig>;
    };
  };
  tts?: {
    primary: TTSProviderConfig;
    fallback?: TTSProviderConfig;
    timeout: number;
  };
  lipSync?: {
    enabled: boolean;
    sensitivity: number;
    smoothing: number;
    amplification: number;
    parameterNames: string[];
  };
  features?: {
    mouseTracking?: boolean;
    breathing?: boolean;
    autoExpressions?: boolean;
    responsivePositioning?: boolean;
    blinking?: boolean;
    idleMotion?: boolean;
    speakingMotion?: boolean;
    orbitControls?: {
      enabled?: boolean;
      enableZoom?: boolean;
      enablePan?: boolean;
      zoomSpeed?: number;
      panSpeed?: number;
      minZoom?: number;
      maxZoom?: number;
    };
  };
  runtime?: {
    autoLoad?: boolean;
    verifyTimeoutMs?: number;
    exposeGlobals?: boolean;
    cubismCoreUrls?: string[];
    pixiUrls?: string[];
    pluginUrls?: string[];
  };
  debug?: boolean;
  onReady?: () => void;
  onError?: (error: Error) => void;
};
```

## TTS Providers

Supported providers:

- `agent`
- `alkahest`
- `elevenlabs`
- `gptsovits`
- `animalese`

`speak()` only works if your chosen provider is reachable and configured correctly.

The SDK defaults to the `agent` provider with a relative `/audio/speech` endpoint, so local app integrations do not hardcode a third-party SaaS URL by default.

## Examples

### Vanilla example

Files live in [examples/vanilla](./examples/vanilla).

Run after building the SDK:

```bash
npm install
npm run build

# serve tmp/sdk-live2d from any static server, then open /examples/vanilla/
```

### React example

A runnable Vite example lives in [examples/react](./examples/react).

```bash
cd examples/react
npm install
npm run dev
```

## Development

```bash
npm install
npm run type-check
npm run test:run
npm run build
npm run lint
```

## Repository

- Issues: [https://github.com/phantasy-bot/sdk-live2d/issues](https://github.com/phantasy-bot/sdk-live2d/issues)
- Source: [https://github.com/phantasy-bot/sdk-live2d](https://github.com/phantasy-bot/sdk-live2d)

## License

MIT