Skill: etrobot/code2animation (version 1.0.1)

# AgentSaaS Video Editor Skill A comprehensive video editing and rendering skill that enables AI agents to create code-driven animations with text-to-speech narration and smooth transitions. ## Purpose This skill allows agents to: - Create and preview interactive video projects with animations - Generate TTS audio narration using Microsoft Edge TTS - Render complete videos with synchronized audio and visual effects - Support both portrait and landscape video formats - Apply smooth transition effects between media elements ## Core Capabilities ### 1. Interactive Video Preview - Real-time preview of video projects in the browser - Playback controls for testing and debugging - Support for transitions, media clips, and timing adjustments - Frame-by-frame seeking for precise editing - Live transition preview with easing effects ### 2. Transition System - **transitionIn**: Each media defines its own entrance animation - **Supported transitions**: fade, zoom, slide2Left, slideUp, none - **Easing**: Built-in easeOutCubic for smooth slide and zoom animations - **stayInClip**: Media can persist throughout entire clip duration - **Cross-clip transitions**: Automatic handling of clip boundaries ### 3. TTS Audio Generation - Automated text-to-speech using Microsoft Edge TTS (msedge-tts) - Support for multiple voices (English and Chinese) - Word-level timing metadata for lip-sync and animations - Audio file caching for faster previews ### 4. Video Rendering - Automated frame-by-frame rendering using Puppeteer - FFmpeg integration for video encoding and audio mixing - 30 FPS output at 1920x1080 (landscape) or 1080x1920 (portrait) - Deterministic rendering for consistent results - Transition effects preserved in final output ## Project Configuration Format ### Media Item Properties - **src**: HTML filename in the footage directory - **words**: Trigger phrase from speech that activates this media - **transitionIn**: Entrance animation type (optional) - **transitionDuration**: Duration in seconds (optional, default: 0.6s) - **stayInClip**: If true, media remains visible until clip ends (optional) ### Transition Types - **fade**: Opacity transition (0 → 1) - **zoom**: Scale transition (2x → 1x) with opacity - **slide2Left**: Horizontal slide from right (100% → 0%) - **slideUp**: Vertical slide from bottom (100% → 0%) - **none**: No transition effect ### Transition Behavior - **transitionIn**: Defines how this media enters the scene - **transitionDuration**: Duration in seconds (default: 0.6s) - **stayInClip**: If true, media remains visible until clip ends - **Easing**: slide2Left and slideUp use easeOutCubic for smooth deceleration ### Reference Implementation See `public/projects/agentSaasPromoVideo.json` for a complete working example demonstrating all transition types and stayInClip behavior. ## Technical Requirements ### System Dependencies - **Node.js**: 18 or higher - **FFmpeg**: Required for video encoding and audio mixing - **Chromium/Chrome**: Used by Puppeteer for headless rendering ### Node.js Dependencies - **React & Vite**: Frontend framework and build tool - **Puppeteer**: Headless browser for frame capture - **msedge-tts**: Microsoft Edge TTS for audio generation - **Express**: Optional HTTP server (for TTS API endpoint) - **Motion (Framer Motion)**: Animation library - **Tailwind CSS**: Styling framework ## Shell Commands Used This skill executes the following shell commands: ### Audio Generation ```bash npx tsx scripts/generate-audio.ts <projectId> ``` - Reads project JSON configuration - Generates MP3 audio files using Edge TTS - Saves word-level timing metadata ### Video Rendering ```bash node scripts/render.js <projectId> [--portrait] ``` - Starts a local Vite dev server - Launches Puppeteer to capture frames - Uses FFmpeg to encode video and mix audio - Cleans up temporary files ### FFmpeg Operations - Frame encoding: `ffmpeg -framerate 30 -i frames/frame-%05d.jpg -c:v libx264 ...` - Audio mixing: `ffmpeg -i video.mp4 -i audio1.mp3 -i audio2.mp3 -filter_complex ...` ### Browser Detection - Uses `which` command to find Chrome/Chromium on Linux/macOS - Respects `PUPPETEER_EXECUTABLE_PATH` environment variable ## API Endpoints (Optional) The skill may expose an HTTP endpoint for TTS generation: ``` POST /api/tts Content-Type: application/json { "text": "Text to speak", "voice": "en-US-GuyNeural", "rate": "+0%", "pitch": "+0Hz" } ``` This endpoint is optional and only used when pre-generated audio files are not available. ## Security Considerations ### File System Access - Reads from: `public/projects/<projectId>/` - Writes to: `public/projects/<projectId>/audio/`, `public/video/` - Creates temporary directories for frame storage - Cleans up temporary files after rendering ### Network Access - Starts local HTTP server on port 5175+ (configurable) - Connects to Microsoft Edge TTS service (external) - No external API keys required for basic functionality ### Process Execution - Spawns child processes for: Vite dev server, FFmpeg encoding - Uses `execSync` for: browser detection, audio generation trigger - All commands are predefined and not user-controllable ## Environment Variables Optional configuration: - `PUPPETEER_EXECUTABLE_PATH`: Custom browser path for Puppeteer - `FASTMCP_LOG_LEVEL`: Logging level (default: ERROR) ## Project Structure ``` public/ projects/ <projectId>/ <projectId>.json # Project configuration footage/ # HTML/CSS media components audio/ # Generated TTS audio files 0.mp3, 1.mp3, ... 0.json, 1.json, ... # Word timing metadata video/ render-<projectId>-landscape.mp4 render-<projectId>-portrait.mp4 ``` ## Usage Example ```bash # 1. Generate audio for a project pnpm generate-audio agentSaasPromoVideo # 2. Preview in browser pnpm dev # 3. Render final video pnpm render agentSaasPromoVideo # 4. Render portrait version pnpm render agentSaasPromoVideo --portrait ``` ## HTML Animation Guidelines When creating HTML animations for video rendering, use the **CSS variable timeline** model. ### Core Model - Renderer controls time: Puppeteer sets `--t` every frame. - Page only renders state: `DOM = f(t)`. - No lifecycle animation APIs (`play/start/reset`) and no hidden runtime state. - **Transition system handles entrance effects**: Don't implement slide/fade transitions in HTML - use the project's `transitionIn` property instead. ### ✅ Required Patterns - Define timeline root: ```css :root { --t: 0; } ``` - Every animated property must derive from `--t`. - Always clamp normalized progress values: ```css --p: clamp(0, calc((var(--t) - var(--start)) / var(--duration)), 1); ``` - Express initial/ending states directly in CSS (seek-safe at any frame). - Use small deterministic JS only for content mapping (e.g., subtitle/text index from `t`). - **Let transition system handle entrance**: Focus on content animation, not entrance effects. ### 🚫 Forbidden Patterns - `transition` - `animation` / `@keyframes` - `window.registerFrameAnimation(...)` - `requestAnimationFrame` loops for timeline progression - Implicit time from `Date.now()` / `performance.now()` for visual state - **Manual entrance transitions**: Don't implement slide/fade in HTML - use `transitionIn` in project config - **Fade-out effects**: Elements should not disappear after animation completes. Use `opacity: var(--p)` instead of `opacity: calc(var(--p) * (1 - var(--fade)))` to keep elements visible at their final state. ### Recommended Template ```css .element { --start: 0.5; --duration: 1; --p: clamp(0, calc((var(--t) - var(--start)) / var(--duration)), 1); opacity: var(--p); transform: translateY(calc((1 - var(--p)) * 20px)); } ``` ### Easing (without transition) Use math on progress directly: ```css --p: clamp(0, calc((var(--t) - var(--start)) / var(--duration)), 1); --ease-out: calc(1 - (1 - var(--p)) * (1 - var(--p))); opacity: var(--ease-out); ``` ### JS Hook Pattern (text/content only) ```html <script> const labels = ['A', 'B', 'C']; const el = document.getElementById('label'); window.onTimelineUpdate = (t) => { const idx = Math.floor(Math.max(0, t) / 1.2) % labels.length; el.textContent = labels[idx]; }; </script> ``` ### Time Semantics (`t` vs `globalTime`) - `onTimelineUpdate(t, globalTime)` supports two time domains: - `t`: clip-local time (resets to `0` when clip changes). This is the default for most HTML animations. - `globalTime`: continuous timeline across clips. Use only when an element must stay continuous through cross-clip transitions. - Do **not** assume `t` is media-local. If a media appears mid-clip, `t` may already be large when it first becomes visible. - For media-local behavior (e.g., toggle starts animating when this media appears), anchor from first visible `globalTime` and derive: - `local = globalTime - mediaStartGlobalTime` - Keep fallback for compatibility: ```js window.onTimelineUpdate = (t, globalTime) => { const g = Number.isFinite(globalTime) ? globalTime : t; // use `t` for normal clip-local animation, `g` only when continuity is required }; ``` ### Determinism Checklist - Seeking to any `t` yields exactly one deterministic frame. - Animation state must not depend on "previous frame". - Cross-clip transition visuals should be continuous in both clips. - Final frame (`t = totalDuration`) must remain on the last clip (no wrap to first clip). ## Limitations - Requires FFmpeg to be installed on the system - TTS generation requires internet connection (Microsoft Edge TTS) - Rendering is CPU-intensive and may take several minutes - Maximum TTS text length: ~1000 characters per clip - Frame capture requires sufficient disk space ## Transparency Statement This skill executes shell commands and spawns child processes for video rendering. All operations are limited to: 1. Starting a local development server (Vite) 2. Running FFmpeg for video encoding 3. Launching Puppeteer for frame capture 4. Detecting browser executables on the system No arbitrary code execution or user input is passed to shell commands. All file paths and commands are predefined and validated.

path: index.html
content: <!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>My Google AI Studio App</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>



path: metadata.json
content: {
  "name": "AgentSaaS Video Editor",
  "description": "A video clip preview tool to preview and export videos.",
  "requestFramePermissions": []
}


path: README.md
content: # AgentSaaS Video Editor

Code-driven video creation with TTS narration and automated rendering.

## Features

- Interactive video preview with real-time playback controls
- TTS audio generation using Microsoft Edge TTS
- Automated frame-by-frame rendering (30 FPS, 1920x1080/1080x1920)
- HTML/CSS animation support with deterministic timing
- Portrait and landscape video formats

## Quick Start

**Prerequisites:** Node.js 18+, FFmpeg

```bash
# Install dependencies
pnpm install

# Generate TTS audio
pnpm generate-audio agentSaasPromoVideo

# Preview in browser
pnpm dev

# Render final video
pnpm render agentSaasPromoVideo [--portrait]
```

## Commands

```bash
pnpm dev                          # Start development server
pnpm generate-audio <projectId>   # Generate TTS audio files
pnpm render <projectId>           # Render landscape video (1920x1080)
pnpm render <projectId> --portrait # Render portrait video (1080x1920)
```

## Project Structure

```
public/projects/<projectId>/
├── <projectId>.json      # Project configuration
├── footage/              # HTML/CSS media components
└── audio/                # Generated TTS files (MP3 + timing JSON)

public/video/             # Rendered output videos
```

## Animation Guidelines

**Supported patterns:**
- setTimeout-based style changes
- CSS transitions (auto-converted to frame-based)
- Typewriter effects with character timing
- Fade in/out and slide animations

**Best practices:**
```html
<!-- Auto-detected fade in -->
<div style="opacity: 0;">Content</div>
<script>
setTimeout(() => element.style.opacity = '1', 100);
</script>
```

## System Requirements

- **FFmpeg**: Video encoding and audio mixing
- **Chromium/Chrome**: Headless rendering via Puppeteer
- **Internet**: Microsoft Edge TTS service

## Installation

### FFmpeg
```bash
# macOS
brew install ffmpeg

# Linux
sudo apt-get install ffmpeg
```

### Chromium (if needed)
```bash
npx puppeteer browsers install chrome
# or set custom path
export PUPPETEER_EXECUTABLE_PATH=/path/to/chrome
```


path: package.json
content: {
  "name": "react-example",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite --port=3000 --host=0.0.0.0",
    "build": "vite build",
    "preview": "vite preview",
    "clean": "rm -rf dist",
    "lint": "tsc --noEmit",
    "generate-audio": "npx tsx scripts/generate-audio.ts",
    "render": "npx tsx scripts/render.ts"
  },
  "dependencies": {
    "@google/genai": "^1.29.0",
    "@tailwindcss/vite": "^4.1.14",
    "@vitejs/plugin-react": "^5.0.4",
    "better-sqlite3": "^12.4.1",
    "dotenv": "^17.2.3",
    "express": "^4.21.2",
    "lucide-react": "^0.576.0",
    "motion": "^12.23.24",
    "msedge-tts": "^2.0.4",
    "react": "^19.0.0",
    "react-dom": "^19.0.0",
    "vite": "^6.2.0"
  },
  "devDependencies": {
    "@types/express": "^4.17.21",
    "@types/node": "^22.14.0",
    "@types/react": "^18.2.79",
    "@types/react-dom": "^18.2.25",
    "autoprefixer": "^10.4.21",
    "puppeteer": "^23.0.0",
    "tailwindcss": "^4.1.14",
    "tsx": "^4.21.0",
    "typescript": "~5.8.2",
    "vite": "^6.2.0"
  },
  "packageManager": "pnpm@10.14.0+sha512.ad27a79641b49c3e481a16a805baa71817a04bbe06a38d17e60e2eaee83f6a146c6a688125f5792e48dd5ba30e7da52a5cda4c3992b9ccf333f9ce223af84748"
}


path: tsconfig.json
content: {
  "compilerOptions": {
    "target": "ES2022",
    "experimentalDecorators": true,
    "useDefineForClassFields": false,
    "module": "ESNext",
    "lib": [
      "ES2022",
      "DOM",
      "DOM.Iterable"
    ],
    "skipLibCheck": true,
    "moduleResolution": "bundler",
    "isolatedModules": true,
    "moduleDetection": "force",
    "allowJs": true,
    "jsx": "react-jsx",
    "paths": {
      "@/*": [
        "./src/*"
      ]
    },
    "allowImportingTsExtensions": true,
    "noEmit": true
  }
}


path: vite.config.ts
content: import tailwindcss from '@tailwindcss/vite';
import react from '@vitejs/plugin-react';
import path from 'path';
import {defineConfig, loadEnv} from 'vite';
import { exec } from 'child_process';
import { promisify } from 'util';
import fs from 'fs';

const execAsync = promisify(exec);

export default defineConfig(({mode}) => {
  const env = loadEnv(mode, '.', '');
  return {
    plugins: [
      react(), 
      tailwindcss(),
      // Custom plugin to handle API routes
      {
        name: 'audio-api',
        configureServer(server) {
          // API: List all available projects
          server.middlewares.use('/api/projects', async (req, res) => {
            if (req.method !== 'GET') {
              res.statusCode = 405;
              res.end('Method Not Allowed');
              return;
            }

            try {
              const projectsDir = path.resolve(process.cwd(), 'public/projects');
              
              if (!fs.existsSync(projectsDir)) {
                res.setHeader('Content-Type', 'application/json');
                res.end(JSON.stringify({ projects: [] }));
                return;
              }

              const entries = fs.readdirSync(projectsDir, { withFileTypes: true });
              const projectMap = new Map<string, { id: string; name: string; description: string }>();

              // Support top-level format: public/projects/<projectId>.json
              const topLevelJsonFiles = entries.filter(entry => entry.isFile() && entry.name.endsWith('.json'));
              for (const file of topLevelJsonFiles) {
                const projectId = path.basename(file.name, '.json');
                const jsonPath = path.join(projectsDir, file.name);
                try {
                  const projectData = JSON.parse(fs.readFileSync(jsonPath, 'utf-8'));
                  projectMap.set(projectId, {
                    id: projectId,
                    name: projectData.name || projectId,
                    description: projectData.description || ''
                  });
                } catch (e) {
                  console.error(`Failed to parse ${jsonPath}:`, e);
                }
              }

              // Support nested format: public/projects/<projectId>/<projectId>.json
              const directories = entries.filter(entry => entry.isDirectory() && !entry.name.startsWith('.'));
              for (const entry of directories) {
                const projectId = entry.name;
                const jsonPath = path.join(projectsDir, projectId, `${projectId}.json`);
                if (!fs.existsSync(jsonPath)) continue;

                try {
                  const projectData = JSON.parse(fs.readFileSync(jsonPath, 'utf-8'));
                  projectMap.set(projectId, {
                    id: projectId,
                    name: projectData.name || projectId,
                    description: projectData.description || ''
                  });
                } catch (e) {
                  console.error(`Failed to parse ${jsonPath}:`, e);
                }
              }

              const projects = Array.from(projectMap.values());

              res.setHeader('Content-Type', 'application/json');
              res.end(JSON.stringify({ projects }));
            } catch (error: any) {
              console.error('[API] Failed to list projects:', error);
              res.statusCode = 500;
              res.setHeader('Content-Type', 'application/json');
              res.end(JSON.stringify({ error: 'Failed to list projects', details: error.message }));
            }
          });

          // API: Generate audio for a project
          server.middlewares.use('/api/generate-audio', async (req, res) => {
            console.log('[API] Received request:', req.method, req.url);
            
            if (req.method !== 'POST') {
              console.log('[API] Method not allowed:', req.method);
              res.statusCode = 405;
              res.end('Method Not Allowed');
              return;
            }

            let body = '';
            req.on('data', chunk => {
              body += chunk.toString();
            });

            req.on('end', async () => {
              try {
                console.log('[API] Request body:', body);
                const { projectId } = JSON.parse(body);
                
                if (!projectId) {
                  console.log('[API] Missing projectId');
                  res.statusCode = 400;
                  res.setHeader('Content-Type', 'application/json');
                  res.end(JSON.stringify({ error: 'projectId is required' }));
                  return;
                }

                console.log(`[API] Starting audio generation for project: ${projectId}`);
                
                // Call the generate-audio script
                const scriptPath = path.resolve(process.cwd(), 'scripts/generate-audio.ts');
                const command = `npx tsx "${scriptPath}" ${projectId}`;
                
                try {
                  const { stdout, stderr } = await execAsync(command);
                  
                  if (stderr) {
                    console.error('[API] Script stderr:', stderr);
                  }
                  
                  console.log('[API] Script stdout:', stdout);
                  console.log('[API] Audio generation completed');
                  
                  res.setHeader('Content-Type', 'application/json');
                  res.end(JSON.stringify({ 
                    success: true, 
                    message: `Audio generation completed for ${projectId}`,
                    output: stdout
                  }));
                } catch (execError: any) {
                  console.error('[API] Script execution failed:', execError);
                  res.statusCode = 500;
                  res.setHeader('Content-Type', 'application/json');
                  res.end(JSON.stringify({ 
                    error: 'Audio generation failed', 
                    details: execError.message,
                    stderr: execError.stderr
                  }));
                }
              } catch (error: any) {
                console.error('[API] Request processing failed:', error);
                res.statusCode = 500;
                res.setHeader('Content-Type', 'application/json');
                res.end(JSON.stringify({ 
                  error: 'Request processing failed', 
                  details: error.message 
                }));
              }
            });
          });
        }
      }
    ],
    define: {
      'process.env.GEMINI_API_KEY': JSON.stringify(env.GEMINI_API_KEY),
    },
    resolve: {
      alias: {
        '@': path.resolve(__dirname, './src'),
      },
    },
    server: {
      // HMR is disabled in AI Studio via DISABLE_HMR env var.
      // Do not modifyâfile watching is disabled to prevent flickering during agent edits.
      hmr: process.env.DISABLE_HMR !== 'true',
    },
  };
});


path: src/App.tsx
content: import { useState, useEffect, useRef, useMemo } from 'react';
import PlaybackControls from './components/PlaybackControls';
import { Player } from './components/Player';
import { useTTS } from './hooks/useTTS';
import { useProject } from './hooks/useProject';
import { usePlayback } from './hooks/usePlayback';
import { VideoClip } from '@/types';

import { generateAudio, loadAudioFiles, checkAudioExists, getSpeechClips } from './utils/audioManager';
import { 
  calculateCompleteTimeline, 
  calculateEstimatedTimeline, 
  getCurrentMediaFromTimeline, 
  getClipPositionFromGlobalTime,
  getGlobalTimeFromClipPosition,
  seekToTime,
  PlaybackTimeline 
} from './utils/playbackEngine';
import { getActiveMediaStates } from '@/utils/transitionEngine';

export default function App() {
  const [audioCache, setAudioCache] = useState<Map<string, HTMLAudioElement>>(new Map());
  const [isLoadingAudio, setIsLoadingAudio] = useState(false);
  const [currentWord, setCurrentWord] = useState<string>('');
  const [mediaItems, setMediaItems] = useState<any[]>([]);
  const [timeline, setTimeline] = useState<PlaybackTimeline | null>(null);
  const [audioInitialized, setAudioInitialized] = useState(false);
  const [iframesLoaded, setIframesLoaded] = useState(0);

  // Check for render mode from URL params
  const urlParams = new URLSearchParams(window.location.search);
  const isRecordMode = urlParams.get('record') === 'true';
  const urlOrientation = urlParams.get('orientation');
  const urlProject = urlParams.get('project');

  // Override defaults if in record mode
  const initialProject = urlProject || 'agentSaasPromoVideo';
  const initialPortrait = urlOrientation === 'portrait';

  // Use custom hooks
  const { projects, availableProjects, activeProject, setActiveProject, isLoadingProject, currentProject } = useProject(initialProject);

  const [configError, setConfigError] = useState<string | null>(null);

  // Calculate complete timeline once
  useEffect(() => {
    if (!currentProject) {
      setTimeline(null);
      setMediaItems([]);
      return;
    }

    try {
      setConfigError(null);
      
      // First, create estimated timeline for immediate use
      const estimatedTimeline = calculateEstimatedTimeline(currentProject);
      setTimeline(estimatedTimeline);
      setMediaItems(estimatedTimeline.mediaItems);
      console.log('[Timeline] Estimated timeline ready:', estimatedTimeline);
      
      // Then, calculate accurate timeline with word boundaries
      calculateCompleteTimeline(currentProject).then(accurateTimeline => {
        setTimeline(accurateTimeline);
        setMediaItems(accurateTimeline.mediaItems);
        console.log('[Timeline] Accurate timeline ready:', accurateTimeline);
      }).catch(error => {
        console.warn('[Timeline] Failed to load accurate timeline, using estimates:', error);
      });
    } catch (error: any) {
      setConfigError(error.message || 'Failed to process project');
      console.error('[Config Error]', error);
      setTimeline(null);
      setMediaItems([]);
    }
  }, [currentProject]);

  const processedClips = useMemo(() => {
    return currentProject?.clips || [];
  }, [currentProject]);

  const [isPortrait, setIsPortrait] = useState(initialPortrait);

  const {
    currentClipIndex,
    setCurrentClipIndex,
    currentTime,
    setCurrentTime,
    isPlaying,
    setIsPlaying,
    resetCounter,
    nextClip,
    prevClip,
    reset
  } = usePlayback(processedClips);

  useEffect(() => {
    if (urlProject && urlProject !== activeProject) {
      setActiveProject(urlProject);
    }
  }, [urlProject, activeProject, setActiveProject]);

  // In record mode, auto-initialize audio state so renderState computes properly
  useEffect(() => {
    if (isRecordMode && !audioInitialized) {
      setAudioInitialized(true);
    }
  }, [isRecordMode, audioInitialized]);

  // TTS Integration - use preloaded audio from cache
  const currentClip = processedClips[currentClipIndex] as VideoClip;
  const shouldUseTTS = currentClip?.type !== 'transition' && currentClip?.speech;

  // Calculate the audio file index by counting non-transition clips up to and including current
  const audioClipIndex = shouldUseTTS
    ? processedClips.slice(0, currentClipIndex + 1).filter(c => c.type !== 'transition').length - 1
    : undefined;

  // Get preloaded audio from cache
  const cachedAudio = audioClipIndex !== undefined
    ? audioCache.get(`${activeProject}-${audioClipIndex}`)
    : undefined;

  const {
    isLoading: isTTSLoading,
    speak,
    pause: pauseTTS,
    resume: resumeTTS,
    stop: stopTTS,
    duration: ttsDuration
  } = useTTS({
    clip: shouldUseTTS ? currentClip : undefined,
    projectId: shouldUseTTS ? activeProject : undefined,
    clipIndex: audioClipIndex,
    preloadedAudio: cachedAudio,
    onWordBoundary: (word) => {
      console.log('[TTS] Word boundary:', word);
      setCurrentWord(word);
    },
    onEnd: () => {
      console.log('[TTS] Audio ended');
      // Don't auto-advance here, let the normal clip timing handle it
    }
  });

  const containerRef = useRef<HTMLDivElement>(null);
  const [scale, setScale] = useState(1);

  const CANVAS_WIDTH = isPortrait ? 1080 : 1920;
  const CANVAS_HEIGHT = isPortrait ? 1920 : 1080;

  const clipDuration = useMemo(() => {
    if (!timeline || currentClipIndex >= timeline.clipDurations.length) return 4;
    return timeline.clipDurations[currentClipIndex];
  }, [timeline, currentClipIndex]);

  const globalPlaybackTime = useMemo(() => {
    if (!timeline) return 0;
    return getGlobalTimeFromClipPosition(timeline, currentClipIndex, currentTime);
  }, [timeline, currentClipIndex, currentTime]);

  useEffect(() => {
    const updateScale = () => {
      if (containerRef.current) {
        const { clientWidth, clientHeight } = containerRef.current;
        const availableWidth = clientWidth - 64; // 32px padding on each side
        const availableHeight = clientHeight - 64;

        const scaleX = availableWidth / CANVAS_WIDTH;
        const scaleY = availableHeight / CANVAS_HEIGHT;
        setScale(Math.min(scaleX, scaleY));
      }
    };

    updateScale();
    window.addEventListener('resize', updateScale);
    return () => window.removeEventListener('resize', updateScale);
  }, [isPortrait, CANVAS_WIDTH, CANVAS_HEIGHT]);

  // Handle clip advancement
  useEffect(() => {
    if (currentTime >= clipDuration && isPlaying) {
      if (currentClipIndex < processedClips.length - 1) {
        const nextIndex = currentClipIndex + 1;
        setCurrentClipIndex(nextIndex);
        setCurrentTime(0);
        setCurrentWord(''); // Reset word on clip change
      } else {
        setIsPlaying(false);
        setCurrentTime(clipDuration);
      }
    }
  }, [currentTime, clipDuration, isPlaying, currentClipIndex, processedClips, setCurrentClipIndex, setCurrentTime, setIsPlaying]);

  // Auto-start TTS when clip changes and is playing (only if audio is initialized)
  useEffect(() => {
    if (isPlaying && shouldUseTTS && currentTime === 0 && audioInitialized) {
      speak();
    }
  }, [currentClipIndex, shouldUseTTS, isPlaying, speak, currentTime, audioInitialized]);

  // Sync time with all iframes
  useEffect(() => {
    const syncWithIframes = () => {
      const iframes = document.querySelectorAll('iframe');
      iframes.forEach(iframe => {
        if (iframe.contentWindow) {
          try {
            iframe.contentWindow.postMessage({
              type: 'seek',
              time: currentTime,
              globalTime: globalPlaybackTime
            }, '*');
          } catch (e) {
            // Ignore cross-origin errors
          }
        }
      });
    };

    // Sync when time changes and we're playing, or when iframes are loaded
    if (isPlaying || currentTime > 0 || iframesLoaded > 0) {
      syncWithIframes();
    }
  }, [currentTime, globalPlaybackTime, isPlaying, iframesLoaded]);

  const handleIframeLoad = () => {
    setIframesLoaded(prev => prev + 1);
    // Sync current time with newly loaded iframe
    setTimeout(() => {
      const iframes = document.querySelectorAll('iframe');
      iframes.forEach(iframe => {
        if (iframe.contentWindow) {
          try {
            iframe.contentWindow.postMessage({
              type: 'seek',
              time: currentTime,
              globalTime: globalPlaybackTime
            }, '*');
          } catch (e) {
            // Ignore cross-origin errors
          }
          }
        });
      }, 100); // Small delay to ensure iframe script is loaded
  };

  const togglePlay = async () => {
    console.log('[togglePlay] Function called, isPlaying:', isPlaying);

    // If trying to play and audio not initialized, handle audio loading/generation
    if (!isPlaying && !audioInitialized) {
      console.log('[togglePlay] Starting audio loading process...');
      setIsLoadingAudio(true);

      try {
        const speechClips = getSpeechClips(currentProject);
        console.log('[togglePlay] Found', speechClips.length, 'speech clips');
        console.log('[togglePlay] Speech clips:', speechClips.map((c: VideoClip) => c.speech?.substring(0, 30)));

        if (speechClips.length > 0) {
          // Check if audio files exist
          console.log('[togglePlay] Checking if audio exists for project:', activeProject);
          const audioExists = await checkAudioExists(activeProject, speechClips);
          console.log('[togglePlay] Audio exists:', audioExists);

          if (!audioExists) {
            console.log(`[togglePlay] Audio missing, calling generateAudio API...`);

            const result = await generateAudio(activeProject);
            console.log('[togglePlay] Generate audio result:', result);

            if (!result.success) {
              throw new Error(result.error || 'Audio generation failed');
            }

            console.log('[togglePlay] Audio generation successful:', result.message);
          } else {
            console.log('[togglePlay] Audio files already exist, skipping generation');
          }

          // Now load all audio files
          console.log('[togglePlay] Loading audio files...');
          const newCache = await loadAudioFiles(activeProject, speechClips);
          setAudioCache(newCache);
          console.log(`[togglePlay] Successfully loaded ${newCache.size} audio files`);
        } else {
          console.log('[togglePlay] No speech clips found, skipping audio processing');
        }

        setAudioInitialized(true);
        setIsLoadingAudio(false);
      } catch (error: any) {
        console.error('[togglePlay] Audio loading failed:', error);
        alert(`Failed to load audio: ${error.message}`);
        setIsLoadingAudio(false);
        return;
      }
    }

    if (currentTime >= clipDuration && currentClipIndex >= processedClips.length - 1) {
      setCurrentClipIndex(0);
      setCurrentTime(0);
    }

    const newIsPlaying = !isPlaying;
    console.log('[togglePlay] Setting isPlaying to:', newIsPlaying);
    setIsPlaying(newIsPlaying);

    // Sync TTS audio
    if (shouldUseTTS && audioInitialized) {
      if (newIsPlaying) {
        if (currentTime === 0) {
          speak(); // Start from beginning
        } else {
          resumeTTS(); // Resume from current position
        }
      } else {
        pauseTTS();
      }
    }
  };

  const handleNextClip = () => {
    stopTTS(); // Stop current TTS
    setCurrentWord(''); // Reset word
    nextClip();
  };

  const handlePrevClip = () => {
    stopTTS(); // Stop current TTS
    setCurrentWord(''); // Reset word
    prevClip();
  };

  const handleReset = () => {
    stopTTS(); // Stop current TTS
    setCurrentWord(''); // Reset word
    setAudioInitialized(false); // Reset to uninitialized state
    setAudioCache(new Map()); // Clear audio cache
    setIframesLoaded(0); // Reset iframe counter
    reset();
  };

  const toggleOrientation = () => {
    setIsPortrait(!isPortrait);
  };

  // 使用预计算的时间轴渲染状态
  const renderState = useMemo(() => {
    if (!timeline) {
      return { activeMedias: [], currentClip: null };
    }

    const activeMedias = getActiveMediaStates(timeline, globalPlaybackTime);
    let mediasToRender = activeMedias;

    if (mediasToRender.length === 0) {
      const fallbackMedia = getCurrentMediaFromTimeline(timeline, globalPlaybackTime);
      if (fallbackMedia) {
        mediasToRender = [{ media: fallbackMedia, style: { opacity: 1, transform: 'translate3d(0, 0, 0)', zIndex: 1 } }];
      }
    }

    if (mediasToRender[0]?.media) {
      const firstMedia = mediasToRender[0].media;
      console.log(`[Render] Global time ${globalPlaybackTime.toFixed(2)}s, Media: ${firstMedia.src.split('/').pop()}, Words: "${firstMedia.words}"`);
    }

    return {
      activeMedias: mediasToRender,
      currentClip: processedClips[currentClipIndex] || null,
      timeline // Pass timeline to Player
    };
  }, [timeline, globalPlaybackTime, processedClips, currentClipIndex]);

  // Expose functions for rendering mode
  useEffect(() => {
    if (isRecordMode && timeline) {
      // Get total duration using timeline
      const getTotalDurationFunc = () => {
        return timeline.totalDuration;
      };

      // Seek to specific time using timeline
      const seekTo = (time: number) => {
        const position = seekToTime(timeline, time);
        setCurrentClipIndex(position.clipIndex);
        setCurrentTime(position.localTime);
        setIsPlaying(false);
      };

      // Get audio log with timings from timeline
      const getAudioLog = () => {
        let accumulated = 0;
        return timeline.clipDurations.map((duration, index) => {
          const result = {
            file: `${activeProject}/audio/${index}.mp3`,
            startTime: accumulated
          };
          accumulated += duration;
          return result;
        });
      };

      // Get current clip index
      const getCurrentClipIndex = () => {
        return currentClipIndex;
      };

      // Expose to window for puppeteer
      (window as any).seekTo = seekTo;
      (window as any).getTotalDuration = getTotalDurationFunc;
      (window as any).getAudioLog = getAudioLog;
      (window as any).getCurrentClipIndex = getCurrentClipIndex;
      (window as any).suppressTTS = true;

      console.log('[Render Mode] Functions exposed to window');
    }
  }, [isRecordMode, timeline, activeProject, currentClipIndex]);

  if (isLoadingProject || !currentProject) {
    return (
      <div className="min-h-screen bg-neutral-950 text-white flex items-center justify-center">
        <div className="animate-pulse">Loading project...</div>
      </div>
    );
  }

  if (configError) {
    return (
      <div className="min-h-screen bg-neutral-950 text-white flex items-center justify-center p-8">
        <div className="max-w-2xl">
          <div className="bg-red-900/20 border border-red-500 rounded-lg p-6">
            <h2 className="text-2xl font-bold text-red-400 mb-4">Configuration Error</h2>
            <p className="text-red-200 mb-4 whitespace-pre-wrap">{configError}</p>
            <button
              onClick={() => {
                setConfigError(null);
                window.location.reload();
              }}
              className="px-4 py-2 bg-red-600 hover:bg-red-700 rounded transition-colors"
            >
              Reload
            </button>
          </div>
        </div>
      </div>
    );
  }

  return (
    <div className={`h-screen w-screen bg-neutral-950 text-white flex flex-col font-sans overflow-hidden ${isRecordMode && !isLoadingAudio ? 'ready-to-record' : ''}`}>
      {/* Main Canvas Area */}
      <main ref={containerRef} className="flex-1 relative flex items-center justify-center bg-[#0a0a0a] overflow-hidden">
        <div
          className="relative bg-black shadow-2xl ring-1 ring-neutral-800 overflow-hidden shrink-0"
          style={{
            width: CANVAS_WIDTH,
            height: CANVAS_HEIGHT,
            transform: `scale(${scale})`,
            transformOrigin: 'center center'
          }}
        >
          <Player renderState={renderState} background={currentProject.background} resetCounter={resetCounter} isPlaying={isPlaying} onIframeLoad={handleIframeLoad} />
        </div>
      </main>

      {/* Bottom Controls Bar */}
      {!isRecordMode && (
        <PlaybackControls
          projects={projects}
          availableProjects={availableProjects}
          activeProject={activeProject}
          isGenerating={isTTSLoading}
          isLoadingAudio={isLoadingAudio}
          isPlaying={isPlaying}
          currentClipIndex={currentClipIndex}
          currentTime={currentTime}
          clipDuration={clipDuration}
          totalClips={currentProject.clips.length}
          isPortrait={isPortrait}
          onProjectChange={(projectId: string) => {
            stopTTS(); // Stop current TTS when switching projects
            setCurrentWord(''); // Reset word
            setActiveProject(projectId);
            setCurrentClipIndex(0);
            setCurrentTime(0);
            setIsPlaying(false);
            setAudioInitialized(false); // Reset audio initialization
            setAudioCache(new Map()); // Clear audio cache
            setIframesLoaded(0); // Reset iframe counter
          }}
          onTogglePlay={togglePlay}
          onNextClip={handleNextClip}
          onPrevClip={handlePrevClip}
          onReset={handleReset}
          onToggleOrientation={toggleOrientation}
        />
      )}
    </div>
  );
}


path: src/main.tsx
content: import {StrictMode} from 'react';
import {createRoot} from 'react-dom/client';
import App from './App.tsx';
import './index.css';

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <App />
  </StrictMode>,
);


path: src/index.css
content: @import "tailwindcss";


path: src/types.ts
content: export interface AudioAlignment {
  characters: string[];
  character_start_times_seconds: number[];
  character_end_times_seconds: number[];
}

export type TransitionKind = 'fade' | 'zoom' | 'slide2Left' | 'slideUp' | 'none';

export interface MediaItem {
  src?: string;
  words?: string;
  type?: string;
  duration?: number;
  transitionIn?: TransitionKind;
  transitionDuration?: number;
  stayInClip?: boolean; // If true, media remains until clip end
  stay?: number; // Legacy support: number of switches to stay visible
}

export interface VideoClip {
  type: string;
  speech?: string;
  media?: MediaItem[];
  voice?: string;
  duration?: number;
}

export interface VideoProject {
  name: string;
  background?: string;
  clips: VideoClip[];
}

export interface WordBoundaryEvent {
  word: string;
  startTime: number;
  endTime: number;
}


path: src/hooks/useProject.ts
content: import { useState, useEffect } from 'react';

interface ProjectInfo {
  id: string;
  name: string;
  description?: string;
}

export function useProject(initialProject: string) {
  const [projects, setProjects] = useState<any>({});
  const [availableProjects, setAvailableProjects] = useState<ProjectInfo[]>([]);
  const [activeProject, setActiveProject] = useState<string>(initialProject);
  const [isLoadingProject, setIsLoadingProject] = useState(true);

  // Load available projects list on mount
  useEffect(() => {
    const loadAvailableProjects = async () => {
      try {
        const response = await fetch('/api/projects');
        if (response.ok) {
          const data = await response.json();
          setAvailableProjects(data.projects || []);
        } else {
          console.error('Failed to load available projects');
        }
      } catch (error) {
        console.error('Error loading available projects:', error);
      }
    };
    
    loadAvailableProjects();
  }, []);

  // Load project JSON dynamically
  useEffect(() => {
    const loadProject = async () => {
      try {
        setIsLoadingProject(true);
        const response = await fetch(`/projects/${activeProject}.json`);
        if (response.ok) {
          const projectData = await response.json();
          
          // Process background path: convert file to full path
          const processedProject = {
            ...projectData,
            background: projectData.background 
              ? `/projects/${activeProject}/footage/${projectData.background}`
              : projectData.background
          };
          
          setProjects((prev: any) => ({ ...prev, [activeProject]: processedProject }));
        } else {
          console.error(`Failed to load project ${activeProject}`);
        }
      } catch (error) {
        console.error('Error loading project:', error);
      } finally {
        setIsLoadingProject(false);
      }
    };
    
    if (activeProject && !projects[activeProject]) {
      loadProject();
    } else {
      setIsLoadingProject(false);
    }
  }, [activeProject, projects]);

  return {
    projects,
    availableProjects,
    activeProject,
    setActiveProject,
    isLoadingProject,
    currentProject: activeProject ? projects[activeProject] : null
  };
}

path: src/hooks/useTTS.ts
content: import { useState, useEffect, useRef, useCallback } from 'react';
import { AudioAlignment, VideoClip } from '@/types';

interface UseTTSProps {
  clip?: VideoClip;
  projectId?: string;
  clipIndex?: number;
  preloadedAudio?: HTMLAudioElement;
  onWordBoundary?: (word: string) => void;
  onEnd?: () => void;
}

export function useTTS({ clip, projectId, clipIndex, preloadedAudio, onWordBoundary, onEnd }: UseTTSProps = {}) {
  const [alignment, setAlignment] = useState<AudioAlignment | null>(null);
  const [duration, setDuration] = useState(0);
  const [audio, setAudio] = useState<HTMLAudioElement | null>(null);
  const [isSpeaking, setIsSpeaking] = useState(false);
  const [isLoading, setIsLoading] = useState(false);
  
  const audioRef = useRef<HTMLAudioElement | null>(null);
  const onWordBoundaryRef = useRef(onWordBoundary);
  const onEndRef = useRef(onEnd);
  const currentUrlRef = useRef<string | null>(null);

  useEffect(() => {
    onWordBoundaryRef.current = onWordBoundary;
    onEndRef.current = onEnd;
  }, [onWordBoundary, onEnd]);

  const stop = useCallback(() => {
    if (audioRef.current) {
      audioRef.current.pause();
      audioRef.current.currentTime = 0;
    }
    setIsSpeaking(false);
  }, []);

  const cleanup = useCallback(() => {
    stop();
    // Don't revoke preloaded audio URLs
    if (currentUrlRef.current && !preloadedAudio) {
      URL.revokeObjectURL(currentUrlRef.current);
      currentUrlRef.current = null;
    }
    if (audioRef.current && !preloadedAudio) {
      audioRef.current.src = '';
      audioRef.current = null;
    }
    if (!preloadedAudio) {
      setAudio(null);
      setAlignment(null);
      setDuration(0);
    }
  }, [stop, preloadedAudio]);

  const loadResource = useCallback(async (pId: string, cIdx: number, speech?: string, options?: { voice?: string }) => {
    let isMounted = true;
    let objectUrl: string | null = null;

    try {
      if (!speech || !speech.trim()) return null;

      setIsLoading(true);
      cleanup(); // Clean up previous resources

      const audioPath = `/projects/${pId}/audio/${cIdx}.mp3`;
      const jsonPath = `/projects/${pId}/audio/${cIdx}.json`;

      console.log(`[useTTS] Loading audio from ${audioPath}`);

      // Try to load static audio file
      const audioRes = await fetch(audioPath);
      if (audioRes.ok && !isMounted) return null;

      if (audioRes.ok) {
        const blob = await audioRes.blob();
        objectUrl = URL.createObjectURL(blob);
        
        const newAudio = new Audio(objectUrl);
        
        await new Promise<void>((resolve, reject) => {
          newAudio.onloadedmetadata = () => {
            if (!isMounted) {
              reject(new Error('Component unmounted'));
              return;
            }
            resolve();
          };
          newAudio.onerror = () => reject(new Error('Audio load failed'));
        });

        if (!isMounted) return null;

        setDuration(newAudio.duration);
        setAudio(newAudio);
        audioRef.current = newAudio;
        currentUrlRef.current = objectUrl;

        // Load metadata for word boundaries
        try {
          const metaRes = await fetch(jsonPath);
          if (metaRes.ok) {
            const data = await metaRes.json();
            const parsedAlignment: AudioAlignment = {
              characters: [],
              character_start_times_seconds: [],
              character_end_times_seconds: []
            };

            if (Array.isArray(data)) {
              data.forEach((item: any) => {
                // Handle the nested structure: item.Metadata[0].Data
                if (item.Metadata && Array.isArray(item.Metadata) && item.Metadata.length > 0) {
                  const metadata = item.Metadata[0];
                  if (metadata.Type === 'WordBoundary' && metadata.Data) {
                    const start = metadata.Data.Offset / 10000000; // Convert from 100ns to seconds
                    const dur = metadata.Data.Duration / 10000000;
                    const text = metadata.Data.text?.Text || '';
                    
                    if (text) {
                      parsedAlignment.characters.push(text);
                      parsedAlignment.character_start_times_seconds.push(start);
                      parsedAlignment.character_end_times_seconds.push(start + dur);
                    }
                  }
                }
              });

              if (isMounted && parsedAlignment.characters.length > 0) {
                setAlignment(parsedAlignment);
                console.log(`[useTTS] Loaded ${parsedAlignment.characters.length} word boundaries`);
                
                // Set up word boundary tracking
                newAudio.ontimeupdate = () => {
                  if (onWordBoundaryRef.current && parsedAlignment.characters.length > 0) {
                    const currentTime = newAudio.currentTime;
                    const wordIndex = parsedAlignment.character_start_times_seconds.findIndex((startTime, i) => {
                      const endTime = parsedAlignment.character_end_times_seconds[i];
                      return currentTime >= startTime && currentTime < endTime;
                    });
                    
                    if (wordIndex >= 0) {
                      const word = parsedAlignment.characters[wordIndex];
                      if (word) onWordBoundaryRef.current(word);
                    }
                  }
                };
              }
            }
          }
        } catch (metaErr) {
          console.warn('[useTTS] Failed to load metadata:', metaErr);
        }

        // Set up end handler
        newAudio.onended = () => {
          setIsSpeaking(false);
          onEndRef.current?.();
        };

        console.log(`[useTTS] Audio loaded successfully, duration: ${newAudio.duration}s`);
        return newAudio;
      } else {
        // If static file doesn't exist, try generating via TTS API
        console.log(`[useTTS] Static audio not found, trying TTS generation...`);
        
        const ttsResponse = await fetch('/api/tts', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({
            text: speech,
            voice: options?.voice,
            rate: '+0%',
            pitch: '+0Hz'
          })
        });

        if (ttsResponse.ok && isMounted) {
          const audioBlob = await ttsResponse.blob();
          objectUrl = URL.createObjectURL(audioBlob);
          
          const newAudio = new Audio(objectUrl);
          
          await new Promise<void>((resolve, reject) => {
            newAudio.onloadedmetadata = () => {
              if (!isMounted) {
                reject(new Error('Component unmounted'));
                return;
              }
              resolve();
            };
            newAudio.onerror = () => reject(new Error('TTS Audio load failed'));
          });

          if (!isMounted) return null;

          setDuration(newAudio.duration);
          setAudio(newAudio);
          audioRef.current = newAudio;
          currentUrlRef.current = objectUrl;

          newAudio.onended = () => {
            setIsSpeaking(false);
            onEndRef.current?.();
          };

          console.log(`[useTTS] TTS audio generated successfully, duration: ${newAudio.duration}s`);
          return newAudio;
        }
      }

      throw new Error('Failed to load or generate audio');

    } catch (error) {
      console.error('[useTTS] Error loading audio:', error);
      if (objectUrl) {
        URL.revokeObjectURL(objectUrl);
      }
      return null;
    } finally {
      if (isMounted) {
        setIsLoading(false);
      }
      return () => {
        isMounted = false;
      };
    }
  }, [cleanup]);

  const speak = useCallback(async () => {
    if (!audioRef.current) return false;
    
    try {
      setIsSpeaking(true);
      audioRef.current.currentTime = 0;
      await audioRef.current.play();
      return true;
    } catch (error) {
      console.error('[useTTS] Error playing audio:', error);
      setIsSpeaking(false);
      return false;
    }
  }, []);

  const pause = useCallback(() => {
    if (audioRef.current) {
      audioRef.current.pause();
      setIsSpeaking(false);
    }
  }, []);

  const resume = useCallback(async () => {
    if (!audioRef.current) return false;
    
    try {
      setIsSpeaking(true);
      await audioRef.current.play();
      return true;
    } catch (error) {
      console.error('[useTTS] Error resuming audio:', error);
      setIsSpeaking(false);
      return false;
    }
  }, []);

  // Auto-load when clip changes
  useEffect(() => {
    // If we have preloaded audio, use it directly
    if (preloadedAudio && clipIndex !== undefined) {
      console.log(`[useTTS] Using preloaded audio for clip ${clipIndex}`);
      setAudio(preloadedAudio);
      audioRef.current = preloadedAudio;
      setDuration(preloadedAudio.duration);
      
      // Set up end handler
      preloadedAudio.onended = () => {
        setIsSpeaking(false);
        onEndRef.current?.();
      };
      
      // Try to load metadata
      if (projectId !== undefined && clipIndex !== undefined) {
        const jsonPath = `/projects/${projectId}/audio/${clipIndex}.json`;
        fetch(jsonPath)
          .then(res => res.ok ? res.json() : null)
          .then(data => {
            if (data && Array.isArray(data)) {
              const parsedAlignment: AudioAlignment = {
                characters: [],
                character_start_times_seconds: [],
                character_end_times_seconds: []
              };

              data.forEach((item: any) => {
                // Handle the nested structure: item.Metadata[0].Data
                if (item.Metadata && Array.isArray(item.Metadata) && item.Metadata.length > 0) {
                  const metadata = item.Metadata[0];
                  if (metadata.Type === 'WordBoundary' && metadata.Data) {
                    const start = metadata.Data.Offset / 10000000;
                    const dur = metadata.Data.Duration / 10000000;
                    const text = metadata.Data.text?.Text || '';
                    
                    if (text) {
                      parsedAlignment.characters.push(text);
                      parsedAlignment.character_start_times_seconds.push(start);
                      parsedAlignment.character_end_times_seconds.push(start + dur);
                    }
                  }
                }
              });

              if (parsedAlignment.characters.length > 0) {
                setAlignment(parsedAlignment);
                console.log(`[useTTS] Loaded ${parsedAlignment.characters.length} word boundaries for preloaded audio`);
                
                // Set up word boundary tracking
                preloadedAudio.ontimeupdate = () => {
                  if (onWordBoundaryRef.current && parsedAlignment.characters.length > 0) {
                    const currentTime = preloadedAudio.currentTime;
                    const wordIndex = parsedAlignment.character_start_times_seconds.findIndex((startTime, i) => {
                      const endTime = parsedAlignment.character_end_times_seconds[i];
                      return currentTime >= startTime && currentTime < endTime;
                    });
                    
                    if (wordIndex >= 0) {
                      const word = parsedAlignment.characters[wordIndex];
                      if (word) onWordBoundaryRef.current(word);
                    }
                  }
                };
              }
            }
          })
          .catch(err => console.warn('[useTTS] Failed to load metadata:', err));
      }
      
      return;
    }
    
    // Fallback to loading audio if not preloaded
    if (projectId !== undefined && clipIndex !== undefined && clip?.speech) {
      loadResource(projectId, clipIndex, clip.speech, { voice: clip.voice });
    }
  }, [projectId, clipIndex, clip?.speech, clip?.voice, preloadedAudio, loadResource]);

  // Cleanup on unmount
  useEffect(() => {
    return cleanup;
  }, [cleanup]);

  return {
    alignment,
    duration,
    audio,
    isSpeaking,
    isLoading,
    speak,
    pause,
    resume,
    stop,
    loadResource,
    cleanup
  };
}

path: src/hooks/usePlayback.ts
content: import { useState, useEffect, useRef } from 'react';

export function usePlayback(processedClips: any[]) {
  const [currentClipIndex, setCurrentClipIndex] = useState(0);
  const [currentTime, setCurrentTime] = useState(0);
  const [isPlaying, setIsPlaying] = useState(false);
  const [resetCounter, setResetCounter] = useState(0);

  const requestRef = useRef<number>(0);
  const previousTimeRef = useRef<number | undefined>(undefined);
  const isPlayingRef = useRef(isPlaying);

  useEffect(() => {
    isPlayingRef.current = isPlaying;
  }, [isPlaying]);

  const animate = (time: number) => {
    if (previousTimeRef.current != undefined && isPlayingRef.current) {
      const deltaTime = (time - previousTimeRef.current) / 1000;
      setCurrentTime((prevTime: number) => prevTime + deltaTime);
    }
    previousTimeRef.current = time;
    if (isPlayingRef.current) {
      requestRef.current = requestAnimationFrame(animate);
    }
  };

  useEffect(() => {
    if (isPlaying) {
      previousTimeRef.current = performance.now();
      requestRef.current = requestAnimationFrame(animate);
    } else {
      cancelAnimationFrame(requestRef.current);
      previousTimeRef.current = undefined;
    }
    return () => cancelAnimationFrame(requestRef.current);
  }, [isPlaying]);

  const nextClip = () => {
    if (currentClipIndex < processedClips.length - 1) {
      const nextIndex = currentClipIndex + 1;
      setCurrentClipIndex(nextIndex);
      setCurrentTime(0);
    }
  };

  const prevClip = () => {
    if (currentClipIndex > 0) {
      const prevIndex = currentClipIndex - 1;
      setCurrentClipIndex(prevIndex);
      setCurrentTime(0);
    } else {
      setCurrentTime(0);
      setResetCounter((c) => c + 1);
    }
  };

  const reset = () => {
    setCurrentClipIndex(0);
    setCurrentTime(0);
    setIsPlaying(false);
    setResetCounter((c: number) => c + 1);
  };

  return {
    currentClipIndex,
    setCurrentClipIndex,
    currentTime,
    setCurrentTime,
    isPlaying,
    setIsPlaying,
    resetCounter,
    nextClip,
    prevClip,
    reset
  };
}

path: scripts/animation/timer-overrides.js
content: // Timer function overrides - COMPLETELY DISABLED for now
export class TimerOverrides {
  constructor(core, typewriterHandler) {
    this.core = core;
    this.typewriterHandler = typewriterHandler;
    console.log('[Animation] Timer overrides completely disabled - all animations work normally');
  }
}

Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)