npm package: @memoreco/recorder
Description: Memoreco screen and camera recorder SDK for user websites
Latest version: 0.3.1

# @memoreco/recorder

React-based screen/camera/area recorder for Memoreco SDKs. Preview-only (no upload) in this version.

## Recent Updates
- 2026-01-19: **v0.3.0** - Parallel audio service for transcription optimization, improved video recording segmentation, ErrorRecoveryOverlay for incomplete recording recovery, reduced log clutter, and enhanced error handling in waveform functions.
- 2025-11-19: Release 0.2.7 speeds up Voice Mode activation by lazy-loading the streaming service after `getUserMedia`, keeps the mic warm between chats, and exposes the new Voice Mode recorder hooks consumed by the SDK wrapper.
- 2025-11-12: Release 0.2.6 guards streaming status polling against race conditions so cleanup no longer throws “Cannot set properties of null” when the first session ends.
- 2025-11-12: Release 0.2.5 permanently disables shared AudioContext reuse after Firefox sample-rate mismatches, guaranteeing recording restarts can rebuild a fresh graph without user intervention.
- 2025-11-11: Release 0.2.4 adds a Firefox-safe AudioContext fallback so streaming transcription continues after sample-rate mismatches reported in production.
- 2025-11-11: Release 0.2.2 aligns with the regenerated API SDK and exposes the webhook test helpers required by the Web SDK while keeping recorder internals unchanged.
- 2025-11-11: Release 0.2.1 improves Speechmatics streaming diagnostics and guards against provider mismatches surfaced during voice-task sessions.

## Install

Monorepo workspace dependency:

```bash
pnpm add @memoreco/recorder -w
```

## Quick Start (React / Next.js)

```tsx
"use client";
import { MemorecoRecorder } from "@memoreco/recorder";

export default function Page() {
  return <MemorecoRecorder mode="auto" />;
}
```

## Hook Usage

```tsx
"use client";
import { useMemorecoRecorder } from "@memoreco/recorder";

export default function CustomRecorder() {
  const {
    state,
    isRecording,
    selectArea,
    startRecording,
    stopRecording,
    recordings,
    enablePreview,
    disablePreview,
    previewStream,
    previewAudioData,
    pause,
    resume,
    isPaused,
  } = useMemorecoRecorder();

  const recordArea = async () => {
    await selectArea("click");
    await startRecording("area", { maxDuration: 10, countdown: 3 });
  };

  return (
    <div>
      <button onClick={() => enablePreview("video")}>Enable Camera Preview</button>
      <button onClick={() => enablePreview("audio")} className="ml-2">Enable Mic Preview</button>
      <button onClick={disablePreview} className="ml-2">Disable Preview</button>
      <button onClick={recordArea} className="ml-2">Record Selected Area</button>
      {isRecording && (
        <>
          <button onClick={pause} className="ml-2">Pause</button>
          <button onClick={resume} className="ml-2">Resume</button>
          <button onClick={stopRecording} className="ml-2">Stop</button>
        </>
      )}
      {recordings.map((r) => (
        <div key={r.id}>{r.file.name}</div>
      ))}
    </div>
  );
}
```

## Area Selection API

- `selectArea(method?: 'click' | 'drag')` — returns selection bounds and element
- Pass selection implicitly via `startRecording('area')` after `selectArea()`
- Or provide `element` or `coordinates` in `RecordingOptions`

```tsx
await selectArea("drag"); // user drags to pick a rectangle
await startRecording("area", { maxDuration: 15 });
```

## Features Implemented

- Modes
  - `audio`: microphone recording with analyser data
  - `video`: camera recording
  - `screen`: screen/window/tab recording
  - `area`: selected area recording (element image or cropped screen video)
  - `snapshot`: camera still photo
  - `screenshot`: screen still photo (optionally cropped)
- UI & Controls
  - Countdown before start
  - Max-duration auto-stop
  - Pause/resume (MediaRecorder-based modes)
  - Pre-recording preview:
    - Camera preview (video-only)
    - Mic meter (audio-only) via analyser data
  - Live in-recording waveform for audio
- Advanced Features
  - **Gaze tracking**: Eye-gaze heatmap generation using WebGazer.js
  - **Face detection**: MediaPipe-based face validation for photos
  - **NSFW detection**: TensorFlow.js-based content filtering
  - **Streaming transcription**: Real-time audio transcription during recording
- Error surfacing via `onError` and hook `error`

## Gaze Tracking (Eye-Gaze Heatmaps)

The SDK includes built-in gaze tracking powered by WebGazer.js, allowing you to track where users look during video and screen recordings.

### Features

- ✅ **Client-side eye tracking** using WebGazer.js
- ✅ **Face detection validation** - only tracks when exactly 1 face is detected
- ✅ **Auto-pause on camera off** - pauses tracking if camera is turned off
- ✅ **Live visualization options** - face overlay, prediction dots, and live heatmap
- ✅ **Post-processing utilities** - generate static heatmaps, timeline heatmaps, gaze paths, and attention grids
- ✅ **Per-frame gaze data** - stores x/y coordinates, confidence, timestamps, and viewport dimensions

### Basic Usage

#### 1. Enable in SDK Settings Panel

The easiest way is to use the SDK's built-in Settings Panel:

```typescript
// Render settings panel with gaze tracking options
const settingsPanel = memoreco.ui.renderSettings('settings-container', {
  features: {
    gazeTrackingEnabled: {
      enabled: true,
      label: '👁️ Enable Gaze Tracking',
      description: 'Track where eyes look during recording',
      default: false
    },
    gazeShowFaceOverlay: {
      enabled: true,
      label: 'Show Face Tracking Overlay',
      description: 'Display face detection overlay',
      default: false
    },
    gazeShowPredictionDots: {
      enabled: true,
      label: 'Show Gaze Prediction Dots',
      description: 'Display real-time gaze points',
      default: false
    },
    gazeShowLiveHeatmap: {
      enabled: true,
      label: 'Show Live Heatmap',
      description: 'Display heatmap overlay during recording',
      default: false
    },
    gazeHeatmapRadius: {
      enabled: true,
      label: 'Heatmap Point Radius',
      description: 'Size of heatmap points in pixels',
      default: 30,
      min: 10,
      max: 100
    }
  },
  onSettingsChange: (settings) => {
    // Settings are automatically applied
    console.log('Gaze tracking settings:', settings);
  }
});
```

#### 2. Start Recording with Gaze Tracking

```typescript
import { startRecording } from '@memoreco/recorder';

// Start video recording with gaze tracking
const result = await startRecording('video', {
  gazeTracking: {
    enabled: true,
    mode: 'video', // or 'screen'
    calibrationEnabled: false, // false = automatic calibration (default), true = manual calibration flow
    showFaceOverlay: false, // Hidden by default
    showPredictionDots: false,
    showHeatmapDuringRecording: false,
    heatmapPointRadius: 30,
    useKalmanFilter: true
  }
});
```

**Calibration Options:**
- `calibrationEnabled: false` (default) - Uses **automatic calibration** during recording (invisible to user)
- `calibrationEnabled: true` - Shows **manual calibration flow** before recording (30-second guided process for better accuracy)

#### 3. Access Gaze Data After Recording

```typescript
const result = await stopRecording();

if (result.gazeTracking) {
  console.log('Gaze samples:', result.gazeTracking.samples);
  console.log('Sample count:', result.gazeTracking.sampleCount);
  console.log('Mode:', result.gazeTracking.mode);
  
  // Each sample contains:
  // - x, y: coordinates on screen
  // - probability: confidence (0-1)
  // - relativeTimeMs: timestamp from recording start
  // - mode: 'video' or 'screen'
  // - viewport: {width, height}
}
```

### Post-Processing Heatmaps

The SDK provides utilities to generate various visualizations from gaze data:

#### Static Heatmap (Summary)

```typescript
import { generateStaticHeatmap, saveCanvasAsPNG } from '@memoreco/recorder';

// Generate heatmap showing all gaze points
const canvas = generateStaticHeatmap(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  pointRadius: 30,
  opacity: 0.6,
  color: { r: 255, g: 0, b: 0 }, // Red
  blurRadius: 25
});

// Download as PNG
await saveCanvasAsPNG(canvas, 'gaze-heatmap.png');
```

#### Timeline Heatmaps (Video Overlay)

```typescript
import { generateTimelineHeatmaps } from '@memoreco/recorder';

// Generate heatmap for each video frame
const frames = generateTimelineHeatmaps(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  duration: result.duration, // milliseconds
  fps: 30,
  pointRadius: 30,
  opacity: 0.6
});

// frames[0] = heatmap for frame 0
// frames[1] = heatmap for frame 1
// ... etc
```

#### Gaze Path Visualization

```typescript
import { generateGazePath } from '@memoreco/recorder';

// Show eye movement path
const canvas = generateGazePath(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  opacity: 0.8,
  color: { r: 0, g: 150, b: 255 } // Blue
});
```

#### Attention Grid (Heatmap Grid)

```typescript
import { generateAttentionGrid } from '@memoreco/recorder';

// Divide screen into grid and show attention per cell
const { canvas, data } = generateAttentionGrid(result.gazeTracking.samples, {
  width: 1920,
  height: 1080,
  gridSize: 10, // 10x10 grid
  opacity: 0.7
});

// data[row][col] = number of gaze samples in that cell
```

### Configuration Options

```typescript
interface GazeTrackingOptions {
  /** Enable gaze tracking (default: false) */
  enabled?: boolean;
  
  /** Mode: 'video' for camera, 'screen' for screen recording */
  mode?: "video" | "screen";
  
  /** Enable manual calibration flow (default: false - uses automatic) */
  calibrationEnabled?: boolean;
  
  /** Show face tracking overlay (default: false - hidden) */
  showFaceOverlay?: boolean;
  
  /** Show prediction dots/gaze points (default: false) */
  showPredictionDots?: boolean;
  
  /** Show live heatmap overlay (default: false) */
  showHeatmapDuringRecording?: boolean;
  
  /** Heatmap point radius in pixels (default: 30) */
  heatmapPointRadius?: number;
  
  /** Use Kalman filter for smoothing (default: true) */
  useKalmanFilter?: boolean;
  
  /** Pause WebGazer after init (default: true) */
  pauseOnInitialize?: boolean;
  
  /** Parent element for overlays (default: document.body) */
  overlayParent?: HTMLElement;
}
```

### Server-Side Post-Processing

**Automatic Heatmap Video Generation:**

When gaze tracking is enabled, the Memoreco API automatically processes gaze data after recording completion and generates a **new video** with the heatmap overlay. The original video remains untouched.

**How It Works:**

1. **Recording Upload**: SDK sends gaze samples to API with video completion
2. **Background Processing**: Trigger.dev job generates heatmap frames and composites them onto video using FFmpeg
3. **New Asset Creation**: Heatmap video is stored as a separate asset (linked to original via `metadata.parentVideoAssetId`)
4. **Media Proxy Access**: Heatmap video is served via the same media proxy as thumbnails and audio

**Access Heatmap Video:**

```typescript
// Get recording assets
const response = await fetch(`/v1/recordings/${recordingId}/assets`, {
  headers: { 'X-API-Key': apiKey }
});

const assets = await response.json();

// Find gaze heatmap video
const heatmapVideo = assets.data.find(asset => 
  asset.type === 'video' && 
  asset.metadata?.renderType === 'gaze_heatmap'
);

if (heatmapVideo) {
  // Access via media proxy (same as thumbnails/audio)
  const heatmapUrl = `/v1/media/${heatmapVideo.id}`;
  console.log('Heatmap video:', heatmapUrl);
}
```

**Cascade Deletion:**

Heatmap videos are automatically deleted when the original recording is deleted (via `ON DELETE CASCADE` on `recording_id` foreign key).

**Processing Status:**

Check if heatmap processing is complete:

```typescript
const recording = await fetch(`/v1/recordings/${recordingId}`, {
  headers: { 'X-API-Key': apiKey }
}).then(r => r.json());

if (recording.data.metadata?.gazeHeatmapAssetId) {
  console.log('Heatmap processed:', recording.data.metadata.gazeHeatmapAssetId);
} else if (recording.data.metadata?.gazeHeatmapError) {
  console.error('Heatmap processing failed:', recording.data.metadata.gazeHeatmapError);
}
```

### Gaze Sample Structure

```typescript
interface GazeSample {
  /** X coordinate on screen (pixels) */
  x: number;
  
  /** Y coordinate on screen (pixels) */
  y: number;
  
  /** Confidence/probability (0-1 scale) */
  probability: number;
  
  /** Time offset from recording start (ms) */
  relativeTimeMs: number;
  
  /** Recording mode when captured */
  mode: "video" | "screen";
  
  /** Viewport dimensions at capture time */
  viewport: {
    width: number;
    height: number;
  };
}
```

### Important Notes

⚠️ **Face Detection Requirement**: Gaze tracking only records samples when **exactly 1 face** is detected in the camera feed. If no face or multiple faces are detected, tracking automatically pauses.

⚠️ **Camera Status**: If the camera is turned off during a video recording, gaze tracking also pauses automatically.

⚠️ **Screen Mode**: For screen recordings, gaze tracking works without camera validation (WebGazer can function without camera feed).

⚠️ **Performance**: Gaze tracking adds minimal overhead (~5-10ms per frame) and runs in parallel with recording.

⚠️ **Privacy**: All gaze tracking happens client-side. No eye images or facial data are sent to servers - only x/y coordinates are stored.

### Complete Example

```typescript
import { 
  startRecording, 
  stopRecording,
  generateStaticHeatmap,
  generateGazePath,
  saveCanvasAsPNG
} from '@memoreco/recorder';

// 1. Start recording with gaze tracking
await startRecording('video', {
  gazeTracking: {
    enabled: true,
    mode: 'video',
    showHeatmapDuringRecording: true // Show live heatmap
  }
});

// 2. Record...
// User records their video while gaze tracking runs

// 3. Stop and get results
const result = await stopRecording();

if (result.gazeTracking) {
  console.log(`Captured ${result.gazeTracking.sampleCount} gaze samples`);
  
  // 4. Generate static heatmap
  const heatmap = generateStaticHeatmap(result.gazeTracking.samples, {
    width: result.dimensions?.width || 1920,
    height: result.dimensions?.height || 1080
  });
  
  await saveCanvasAsPNG(heatmap, 'gaze-heatmap.png');
  
  // 5. Generate gaze path
  const gazePath = generateGazePath(result.gazeTracking.samples, {
    width: result.dimensions?.width || 1920,
    height: result.dimensions?.height || 1080
  });
  
  await saveCanvasAsPNG(gazePath, 'gaze-path.png');
}
```

### FFmpeg Video Overlay

To overlay heatmaps onto the recorded video using FFmpeg:

```bash
# Generate timeline heatmap frames first (in your code)
# Then use FFmpeg to overlay them:

ffmpeg -i recording.mp4 \
  -framerate 30 -i heatmap-frames/frame_%04d.png \
  -filter_complex "[0:v][1:v]overlay=0:0" \
  -c:a copy output-with-heatmap.mp4
```

Or use the helper function:

```typescript
import { generateFFmpegOverlayCommand } from '@memoreco/recorder';

const command = generateFFmpegOverlayCommand(
  'recording.mp4',
  'heatmap-frames',
  'output-with-heatmap.mp4'
);

console.log('Run this command:', command);
```

---

The recorder SDK supports optional eye-gaze tracking during video and screen recordings using [WebGazer.js](https://webgazer.cs.brown.edu/). This feature captures where users are looking frame-by-frame, enabling post-processing heatmap overlays.

### Basic Usage

```tsx
import { useMemorecoRecorder } from "@memoreco/recorder";

function MyRecorder() {
  const { startRecording, stopRecording } = useMemorecoRecorder();

  const recordWithGaze = async () => {
    await startRecording("video", {
      gazeTracking: {
        enabled: true,
        showFaceOverlay: false,        // Hide from user (default)
        showPredictionDots: false,      // Hide gaze dots (default)
        showHeatmapDuringRecording: false  // Don't show live heatmap
      }
    });
  };

  const stopAndGetGazeData = async () => {
    const result = await stopRecording();
    
    if (result.gazeTracking) {
      console.log('Gaze samples:', result.gazeTracking.samples);
      console.log('Sample count:', result.gazeTracking.sampleCount);
      console.log('Mode:', result.gazeTracking.mode); // 'video' or 'screen'
      
      // Optional: Heatmap data URL (PNG)
      if (result.gazeTracking.heatmapDataUrl) {
        console.log('Heatmap:', result.gazeTracking.heatmapDataUrl);
      }
    }
  };

  return (
    <>
      <button onClick={recordWithGaze}>Start Recording with Gaze</button>
      <button onClick={stopAndGetGazeData}>Stop & Get Data</button>
    </>
  );
}
```

### Configuration Options

```typescript
interface GazeTrackingOptions {
  enabled?: boolean;                    // Enable gaze tracking (default: false)
  mode?: "video" | "screen";            // Recording mode (auto-detected)
  showFaceOverlay?: boolean;            // Show face tracking box (default: false)
  showPredictionDots?: boolean;         // Show gaze prediction dots (default: false)
  showHeatmapDuringRecording?: boolean; // Show live heatmap overlay (default: false)
  heatmapPointRadius?: number;          // Heatmap point radius in pixels (default: 30)
  useKalmanFilter?: boolean;            // Smooth gaze predictions (default: true)
  pauseOnInitialize?: boolean;          // Pause after init, resume on start (default: false)
  overlayParent?: HTMLElement;          // Parent element for overlays (default: document.body)
}
```

### Gaze Data Structure

```typescript
interface GazeSample {
  x: number;              // Screen X coordinate (pixels)
  y: number;              // Screen Y coordinate (pixels)
  probability: number;    // Confidence score (0-1)
  relativeTimeMs: number; // Milliseconds from recording start
  mode: "video" | "screen";  // Capture mode
  viewport: {
    width: number;        // Viewport width at capture time
    height: number;       // Viewport height at capture time
  };
}

// Recording result includes:
result.gazeTracking = {
  samples: GazeSample[];       // All captured gaze points
  sampleCount: number;         // Total samples
  heatmapDataUrl?: string;     // Optional PNG data URL
  mode: "video" | "screen";    // Recording mode
};
```

### Post-Processing Heatmaps

Use the gaze samples to render heatmaps post-recording:

**Option 1: Using the built-in heatmap**
```tsx
// Enable live heatmap generation during recording
await startRecording("screen", {
  gazeTracking: {
    enabled: true,
    showHeatmapDuringRecording: true,  // Generate heatmap
    heatmapPointRadius: 30              // Customize point size
  }
});

const result = await stopRecording();
// Access heatmap as PNG data URL
const heatmapImg = result.gazeTracking?.heatmapDataUrl;
```

**Option 2: Custom post-processing with Remotion**
```tsx
// Remotion example for video overlay
import { Composition } from "remotion";

export const GazeOverlay = ({ gazeS samples, fps }) => {
  const frame = useCurrentFrame();
  const timeMs = (frame / fps) * 1000;
  
  // Find gaze points for this frame
  const currentGaze = samples.filter(s => 
    Math.abs(s.relativeTimeMs - timeMs) < 50
  );
  
  return (
    <AbsoluteFill>
      {currentGaze.map((point, i) => (
        <div
          key={i}
          style={{
            position: "absolute",
            left: point.x,
            top: point.y,
            width: 60,
            height: 60,
            borderRadius: "50%",
            background: `rgba(255, 0, 0, ${point.probability * 0.3})`,
            transform: "translate(-50%, -50%)"
          }}
        />
      ))}
    </AbsoluteFill>
  );
};
```

**Option 3: FFmpeg filter**
```bash
# Export gaze samples as JSON
# Use FFmpeg drawbox filter to add gaze points per frame
ffmpeg -i video.webm -vf "drawbox=..." output.mp4
```

### Screen Mode vs Video Mode

- **Video mode**: Tracks where user looks relative to their camera view
- **Screen mode**: Tracks where user looks on the screen (preferred for usability studies)
- **Precedence**: Screen recordings automatically use screen mode when both video and screen are possible

### Performance Notes

- WebGazer.js loads ~2MB from CDN on first use
- Gaze tracking adds minimal CPU overhead (~5-10%)
- Initialization takes 1-2 seconds for model loading
- Samples are captured at ~30-60 Hz depending on device performance
- Each sample is ~100 bytes; a 5-minute recording generates ~18-36K samples (~1.8-3.6 MB)

### Browser Compatibility

- Chrome/Edge: Full support
- Firefox: Full support
- Safari: Supported (may require camera permission even for screen recordings)
- Mobile: Limited support (WebGazer.js works best on desktop)

## How to Test (Local)

1. Start the web app:
   - Visit `/recorder-demo` (Next.js demo page is included)
2. Try modes:
   - Audio: enable mic preview (optional), start recording, observe waveform, stop and play back
   - Video: enable camera preview (optional), start, pause, resume, stop
   - Screen: start and select a window/tab/screen; stop and play back
   - Area: click “Start Recording” in area mode; you’ll be prompted to select an element (overlay), then recording begins
   - Snapshot/Screenshot: select mode, click “Start Recording” to capture a still automatically
3. Permission flows:
   - If a preview fails (denied), re-enable preview to retry
   - The demo page will show a toast on `onError`

### Where files are saved

- All captures are created in-memory and shown via `URL.createObjectURL(...)`.
- In the demo, files are not uploaded or persisted; use the Download button to save locally.
- In your app, use the `onRecordingComplete(result)` callback from `MemorecoRecorder` or the hook’s `onRecordingComplete` to upload or store `result.file`.

## Examples

- Next.js demo page: `/apps/web/app/recorder-demo/page.tsx`

## Roadmap

- Upload integration via API SDK
- Better permission guidance panel and tailored tips
- Screen/tab preview options
- File processing/transcoding (mp4/mp3 conversion)

## Modes Overview

- Audio: Records your microphone only. Live meter shows levels; output is an audio file (webm/opus). Countdown/max duration/pause/resume apply.

- Video: Records your webcam (with mic). Optional live camera preview before recording; output is a video file (webm). Countdown/max duration/pause/resume apply.

- Screen: Records your screen/window/tab. Checkbox lets you also record your microphone. Some browsers may also include tab/system audio when available. Output is a video file (webm). Countdown/max duration/pause/resume apply.

- Area: Records a specific on‑page area you pick.
  - If an element is selected, it captures that element as an image (png).
  - If a rectangle capture is used, it records that region as a video (webm) by cropping the screen stream.
  - Countdown/max duration apply when it’s a video capture; not for single-image captures.

- Snapshot: Takes a single still photo from the webcam (png). One tap (“Take Photo”) — no countdown/max duration/pause.

- Screenshot: Takes a single still image (png) of the screen/window/tab you choose in the browser picker. One tap (“Take Screenshot”) — no countdown/max duration/pause.

Notes:
- “Show live preview” is for video/screen modes to verify camera before recording.
- “Record microphone audio with screen” merges your mic with the screen capture; you’ll see a separate mic permission prompt.
- If mic is denied or muted, audio wave/meter may stay flat and recordings will be silent.