npm package: @movementinfra/expo-audiosync
Description: High-performance audio synchronization for React Native/Expo using Rust
Author: Movement Infra
Latest version: 0.1.0

# @movementinfra/expo-audiosync

High-performance audio synchronization for React Native/Expo using Rust-based spectral fingerprinting.

[![npm version](https://img.shields.io/npm/v/@movementinfra/expo-audiosync.svg)](https://www.npmjs.com/package/@movementinfra/expo-audiosync)
[![Platform](https://img.shields.io/badge/platform-iOS-lightgrey.svg)](https://github.com/movementinfra/audiosync)

## Overview

This Expo module wraps the AudioSync Rust library, providing native iOS functionality to find the time offset between two audio files with high accuracy. It uses a Shazam-style algorithm with spectral fingerprinting for robust matching even with background noise.

**Platform Support:** iOS only (Android not yet implemented)

**Simulator Support:** Not included in npm package. See [Local Development](#local-development-with-simulator) to enable.

**Supported Audio Formats:** MP3, M4A/AAC, WAV

## Installation

```bash
npx expo install @movementinfra/expo-audiosync
```

This module requires a [development build](https://docs.expo.dev/develop/development-builds/introduction/) — it won't work with Expo Go.

```bash
npx expo prebuild
npx expo run:ios
```

## Quick Start

### Full Sync Pipeline (Recommended)

Sync a video with a reference song in one call:

```typescript
import { syncVideoWithAudio } from '@movementinfra/expo-audiosync';

const result = await syncVideoWithAudio(
  '/path/to/dance-video.mp4',   // Video with misaligned audio
  '/path/to/song.mp3',          // Reference song
  '/path/to/synced-output.mp4'  // Output path
);

console.log(`Synced with ${result.offsetMs}ms offset`);
console.log(`Confidence: ${result.confidence}`);
```

### Audio-Only Offset Detection

Find the time offset between two audio files:

```typescript
import { findOffset, isAvailable } from '@movementinfra/expo-audiosync';

if (!isAvailable()) {
  console.log('AudioSync is not available on this platform');
  return;
}

const result = await findOffset(
  '/path/to/reference.mp3',  // Full song
  '/path/to/video-audio.mp3' // Audio extracted from video
);

console.log(`Offset: ${result.offsetMs}ms`);
console.log(`Confidence: ${result.confidence}`);
```

## API Reference

### Core Functions

#### `syncVideoWithAudio(videoPath, audioPath, outputPath, config?)`

Full pipeline: extracts audio from video, detects offset, creates synced video.

```typescript
const result = await syncVideoWithAudio(videoPath, audioPath, outputPath);
// Returns: { outputPath, offsetMs, confidence, correlation, matchCount }
```

#### `findOffset(referencePath, queryPath, config?)`

Find the time offset between two audio files.

```typescript
const result = await findOffset(referencePath, queryPath);
// Returns: { offsetMs, offsetSamples, confidence, correlation, matchCount }
```

#### `extractAudio(videoPath, outputPath, options?)`

Extract audio track from a video file.

```typescript
const result = await extractAudio(videoPath, outputPath, { format: 'm4a' });
// Returns: { outputPath, durationMs, sampleRate, channels }
```

#### `createSyncedVideo(videoPath, audioPath, outputPath, offsetMs)`

Create a synced video with a known offset.

```typescript
const result = await createSyncedVideo(videoPath, audioPath, outputPath, offsetMs);
// Returns: { outputPath, durationMs, offsetAppliedMs }
```

#### `isAvailable()`

Check if the module is available (iOS only).

#### `getVersion()`

Get the native library version string.

### Configuration

```typescript
interface SyncConfig {
  sampleRate?: number;           // Default: 16000
  fftSize?: number;              // Default: 1024
  hopSize?: number;              // Default: 256
  numThreads?: number;           // Default: 2
  refinementEnabled?: boolean;   // Default: true
  refinementWindowSecs?: number; // Default: 1.5
  minConfidence?: number;        // Default: 1.5
}
```

### Error Handling

```typescript
import { AudioSyncError, AudioSyncErrorCode } from '@movementinfra/expo-audiosync';

try {
  const result = await findOffset(refPath, queryPath);
} catch (error) {
  if (error instanceof AudioSyncError) {
    switch (error.code) {
      case AudioSyncErrorCode.NoMatch:
        console.log('No matching audio found');
        break;
      case AudioSyncErrorCode.AudioTooShort:
        console.log('Audio must be at least 500ms');
        break;
      // ... handle other cases
    }
  }
}
```

**Error Codes:** `FileReadError`, `DecodeError`, `AudioTooShort`, `NoMatch`, `LowConfidence`, `InvalidConfig`, `InternalError`, `PlatformNotSupported`, `NoAudioTrack`, `NoVideoTrack`, `ExportFailed`

## Local Development with Simulator

The npm package only includes device binaries to minimize size. To enable iOS Simulator support for local development:

```bash
# Clone the repository
git clone https://github.com/movementinfra/audiosync.git
cd audiosync

# Build with simulator support (requires Rust)
./build-ios.sh --release

# Link to your project
cd your-expo-app
npm install ../audiosync/expo-module
npx expo prebuild --clean
npx expo run:ios
```

**Requirements:** Rust 1.70+ with `aarch64-apple-ios-sim` target, Apple Silicon Mac.

## Example App

An example app is included in the repository at `expo-module/example/`. See its [README](./example/README.md) for setup instructions.

## How It Works

The library uses a Shazam-style audio fingerprinting algorithm:

1. **Audio Loading** — Decode audio files and convert to mono
2. **Spectrogram** — Compute Short-Time Fourier Transform (STFT)
3. **Peak Detection** — Find local maxima in time-frequency domain
4. **Fingerprinting** — Generate 32-bit hashes from peak pairs
5. **Matching** — Build offset histogram from hash matches
6. **Refinement** — Cross-correlation for sub-sample precision

## Building from Source

### Prerequisites

- Rust 1.70+ with iOS targets: `rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios`
- Xcode command line tools
- Node.js 18+

### Build Steps

```bash
# 1. Build Rust library for iOS
./build-ios.sh --release

# 2. Build TypeScript module
cd expo-module
npm install
npm run build
```

## Troubleshooting

### "Module not found" error

Ensure you're using a development build, not Expo Go:
```bash
npx expo prebuild --clean
npx expo run:ios
```

### "No match found" errors

- Ensure audio files have at least 500ms overlap
- Check that query audio contains recognizable content from reference
- Try lowering `minConfidence` in config

### Linker errors with libc++

Add to your app's `Podfile`:
```ruby
post_install do |installer|
  installer.pods_project.targets.each do |target|
    target.build_configurations.each do |config|
      config.build_settings['OTHER_LDFLAGS'] ||= ['$(inherited)']
      config.build_settings['OTHER_LDFLAGS'] << '-lc++'
    end
  end
end
```

## License

MIT

## Credits

Built with [Rust](https://www.rust-lang.org/), [Symphonia](https://github.com/pdeljanov/Symphonia), [RustFFT](https://github.com/ejmahler/RustFFT), and [Expo Modules](https://docs.expo.dev/modules/overview/).