npm package: @gdnaio/react-polly-text-to-speech
Description: React hook for text-to-speech using Amazon Polly with secure Cognito-based authentication
Author: g/d/n/a
Latest version: 1.0.2

# @gdnaio/react-polly-text-to-speech

React hook for text-to-speech using **Amazon Polly** with secure Cognito-based authentication.

No API keys in the browser. No third-party TTS services. Just AWS Polly called securely through Cognito Identity Pool credentials — the same pattern used by [`@gdnaio/react-transcribe-streaming`](https://www.npmjs.com/package/@gdnaio/react-transcribe-streaming).

## Features

- **Single-hook API** — `usePollyTextToSpeech` returns speak/stop controls, loading/playing state, and audio data
- **Secure by default** — Uses Cognito Identity Pool for temporary AWS credentials (no keys in frontend)
- **All Polly engines** — Standard, Neural, Long-form, and Generative
- **SSML builder** — Built-in `ssml` utility for pauses, emphasis, prosody, whispering, and more
- **Voice catalogue** — Curated voice map with `getVoicesByLanguage()` and `getVoiceInfo()` helpers
- **Configurable output** — MP3, OGG, PCM with custom sample rates
- **No hidden audio** — The hook does NOT create its own `Audio` object; you control playback via your own `<audio>` element and the `ref` callback
- **Client caching** — The `PollyClient` and AWS credentials are cached at module level across all hook instances; only the very first call triggers Cognito round-trips
- **Audio Blob exposed** — Use `audioUrl` with your own `<audio>` player or download the blob
- **TypeScript** — Full type definitions included
- **Lightweight** — Only depends on `@aws-sdk/client-polly` and `@aws-sdk/credential-providers`

## Installation

```bash
npm install @gdnaio/react-polly-text-to-speech
```

Both ESM and CommonJS builds are included. TypeScript declarations ship with the package.

## AWS Prerequisites

You need three things (identical setup to `@gdnaio/react-transcribe-streaming`):

### 1. Cognito User Pool

Your existing User Pool for authenticating users.

### 2. Cognito Identity Pool

Link it to your User Pool. This provides temporary AWS credentials to the browser.

### 3. IAM Role for Authenticated Users

Attach this inline policy to the Identity Pool's authenticated role:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["polly:SynthesizeSpeech"],
      "Resource": "*"
    }
  ]
}
```

> **Tip:** If you already have a Transcribe Identity Pool stack, you can add the `polly:SynthesizeSpeech` permission to the same IAM role and reuse the Identity Pool.

## Quick Start

```tsx
import { usePollyTextToSpeech } from '@gdnaio/react-polly-text-to-speech'

function TextToSpeechButton({ idToken }: { idToken: string }) {
  const { speak, stop, ref, loading, playing, error, audioUrl } = usePollyTextToSpeech({
    config: {
      region: 'us-east-1',
      identityPoolId: 'us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
      userPoolId: 'us-east-1_XXXXXXXXX',
      idToken, // from your auth provider
    },
  })

  return (
    <div>
      <button onClick={() => speak('Hello! Welcome to our application.')} disabled={loading}>
        {loading ? 'Generating...' : 'Speak'}
      </button>
      {playing && <button onClick={stop}>Stop</button>}
      {error && <p style={{ color: 'red' }}>{error}</p>}

      {/* IMPORTANT: attach ref so the hook can track play/pause/duration */}
      {audioUrl && <audio ref={ref} autoPlay controls src={audioUrl} />}
    </div>
  )
}
```

## API Reference

### `usePollyTextToSpeech(options)`

#### Options

```ts
{
  // Required — AWS Cognito credentials
  config: {
    region: string            // AWS region (e.g. "us-east-1")
    identityPoolId: string    // Cognito Identity Pool ID
    userPoolId: string        // Cognito User Pool ID
    idToken: string           // JWT ID token from your auth provider
  },

  // Optional — voice settings
  voice: {
    voiceId?: string          // Polly voice (default: "Joanna")
    engine?: PollyEngine      // "standard" | "neural" | "long-form" | "generative" (default: "neural")
    languageCode?: string     // Only needed for bilingual voices (e.g. "hi-IN" for Aditi)
  },

  // Optional — audio output settings
  audio: {
    format?: PollyOutputFormat    // "mp3" | "ogg_vorbis" | "pcm" (default: "mp3")
    sampleRate?: PollySampleRate  // "8000" | "16000" | "22050" | "24000" | "44100" | "48000"
    lexiconNames?: string[]       // Custom pronunciation lexicons (max 5)
    speechMarkTypes?: PollySpeechMarkType[]  // "sentence" | "ssml" | "viseme" | "word"
  }
}
```

#### Return Value

```ts
{
  speak: (text: string, textType?: 'text' | 'ssml') => Promise<void>
  stop: () => void
  ref: (el: HTMLAudioElement | null) => void  // callback ref for your <audio> element
  loading: boolean          // true while API call is in-flight
  playing: boolean          // true while audio is playing
  error: string | null      // last error message, or null
  audioBlob: Blob | null    // raw audio Blob from last synthesis
  audioUrl: string | null   // Object URL for <audio src> usage
  duration: number | null   // audio duration in seconds (after metadata loads)
}
```

## Configuring Voices

```tsx
// Neural voice (default) — natural sounding
const tts = usePollyTextToSpeech({
  config,
  voice: { voiceId: 'Matthew', engine: 'neural' },
})

// Generative voice — most expressive
const tts = usePollyTextToSpeech({
  config,
  voice: { voiceId: 'Ruth', engine: 'generative' },
})

// Long-form voice — optimised for articles/stories
const tts = usePollyTextToSpeech({
  config,
  voice: { voiceId: 'Danielle', engine: 'long-form' },
})

// Spanish voice
const tts = usePollyTextToSpeech({
  config,
  voice: { voiceId: 'Lupe', engine: 'neural', languageCode: 'es-US' },
})
```

## Using SSML

For fine-grained control over speech output, use the built-in `ssml` builder:

```tsx
import { usePollyTextToSpeech, ssml } from '@gdnaio/react-polly-text-to-speech'

function SsmlExample() {
  const { speak } = usePollyTextToSpeech({ config })

  const handleSpeak = () => {
    const text = ssml.speak(
      ssml.sentence('Hello there!') +
      ssml.pause('500ms') +
      ssml.emphasis('This is really important.', 'strong') +
      ssml.pause('300ms') +
      ssml.prosody('And this part is spoken slowly.', { rate: 'slow' }) +
      ssml.pause('200ms') +
      ssml.whisper('This is a secret.')
    )

    speak(text, 'ssml')
  }

  return <button onClick={handleSpeak}>Speak with SSML</button>
}
```

### SSML Builder Methods

| Method | Description | Example |
|--------|-------------|---------|
| `ssml.speak(content)` | Wrap in `<speak>` root | `ssml.speak('Hello')` |
| `ssml.pause(time)` | Insert a break | `ssml.pause('500ms')` |
| `ssml.emphasis(text, level)` | Emphasise text | `ssml.emphasis('wow', 'strong')` |
| `ssml.prosody(text, opts)` | Control rate/pitch/volume | `ssml.prosody('slow', { rate: 'slow' })` |
| `ssml.paragraph(text)` | Paragraph with natural pause | `ssml.paragraph('First para.')` |
| `ssml.sentence(text)` | Sentence boundary | `ssml.sentence('A sentence.')` |
| `ssml.sayAs(text, type)` | Interpret as date/number/etc | `ssml.sayAs('2025', 'cardinal')` |
| `ssml.phoneme(text, ph)` | Phonemic pronunciation | `ssml.phoneme('pecan', 'pɪˈkɑːn')` |
| `ssml.sub(text, alias)` | Substitution | `ssml.sub('AWS', 'Amazon Web Services')` |
| `ssml.lang(text, lang)` | Switch language mid-speech | `ssml.lang('Bonjour', 'fr-FR')` |
| `ssml.whisper(text)` | Whispering voice | `ssml.whisper('secret')` |
| `ssml.amazonEffect(text, name)` | Polly-specific effects | `ssml.amazonEffect('news', 'drc')` |

## Voice Discovery

Browse available voices with the included catalogue:

```tsx
import { getVoicesByLanguage, getVoiceInfo, POLLY_VOICES } from '@gdnaio/react-polly-text-to-speech'

// Get all English (US) voices
const usVoices = getVoicesByLanguage('en-US')
// → [{ voiceId: 'Joanna', name: 'Joanna', gender: 'Female', engines: [...] }, ...]

// Look up a specific voice
const info = getVoiceInfo('Matthew')
// → { voiceId: 'Matthew', name: 'Matthew', gender: 'Male', engines: ['neural', 'standard'] }

// Access the full catalogue
console.log(Object.keys(POLLY_VOICES))
// → ['en-US', 'en-GB', 'en-AU', 'en-IN', 'es-US', 'es-ES', 'fr-FR', ...]
```

## Using `ref` with Your Own Audio Player

The hook does **not** play audio internally. It synthesises audio and returns `audioUrl` — you render the `<audio>` element and pass the `ref` callback so the hook can track `playing`, `duration`, and `stop()` state.

```tsx
function CustomPlayer() {
  const { speak, stop, ref, audioUrl, playing, duration, loading } = usePollyTextToSpeech({ config })

  return (
    <div>
      <button onClick={() => speak('Hello world')} disabled={loading}>
        Generate Audio
      </button>
      {playing && <button onClick={stop}>Stop</button>}

      {/* ref is required — without it, playing/duration/stop() won't work */}
      {audioUrl && <audio ref={ref} autoPlay controls src={audioUrl} />}

      {duration && <p>Duration: {duration.toFixed(1)}s</p>}
    </div>
  )
}
```

### Why `ref` matters

The `ref` callback is how the hook connects to the actual `<audio>` DOM element. Without it:

- `playing` will always be `false`
- `duration` will always be `null`
- `stop()` will have no effect (there's no element to pause)

```tsx
// WRONG — hook can't track the audio element
{audioUrl && <audio autoPlay controls src={audioUrl} />}

// CORRECT — hook is connected to the element
{audioUrl && <audio ref={ref} autoPlay controls src={audioUrl} />}
```

## Using with Vite

```tsx
const { speak } = usePollyTextToSpeech({
  config: {
    region: import.meta.env.VITE_AWS_REGION,
    identityPoolId: import.meta.env.VITE_AWS_IDENTITY_POOL_ID,
    userPoolId: import.meta.env.VITE_AWS_USER_POOL_ID,
    idToken: token, // from your auth hook
  },
  voice: { voiceId: 'Joanna', engine: 'neural' },
  audio: { format: 'mp3' },
})
```

## Using with Next.js

Since this hook uses browser APIs (`Audio`, `URL.createObjectURL`), use dynamic import with SSR disabled:

```tsx
// components/TtsButton.tsx
'use client'

import { usePollyTextToSpeech } from '@gdnaio/react-polly-text-to-speech'

export function TtsButton({ idToken }: { idToken: string }) {
  const { speak, ref, audioUrl, loading } = usePollyTextToSpeech({
    config: {
      region: process.env.NEXT_PUBLIC_AWS_REGION!,
      identityPoolId: process.env.NEXT_PUBLIC_AWS_IDENTITY_POOL_ID!,
      userPoolId: process.env.NEXT_PUBLIC_AWS_USER_POOL_ID!,
      idToken,
    },
  })

  return (
    <div>
      <button onClick={() => speak('Hello from Next.js')} disabled={loading}>
        Speak
      </button>
      {audioUrl && <audio ref={ref} autoPlay controls src={audioUrl} />}
    </div>
  )
}
```

## Token Retrieval

The hook needs a Cognito **ID token** (not access token). Common patterns:

```ts
// @gdnaio/cognito-auth
const { getIdToken } = useAuth()
const token = await getIdToken()

// AWS Amplify v6
import { fetchAuthSession } from 'aws-amplify/auth'
const { tokens } = await fetchAuthSession()
const token = tokens?.idToken?.toString()

// amazon-cognito-identity-js
cognitoUser.getSession((err, session) => {
  const token = session.getIdToken().getJwtToken()
})
```

## Audio Output Formats

| Format | MIME Type | Use Case |
|--------|-----------|----------|
| `mp3` (default) | `audio/mpeg` | Best browser compatibility, small file size |
| `ogg_vorbis` | `audio/ogg` | Open format, good quality-to-size ratio |
| `pcm` | `audio/pcm` | Raw audio for processing pipelines |

## Supported Voices

The built-in `POLLY_VOICES` catalogue includes the following voices. You can pass **any valid Polly VoiceId** directly to the hook — the catalogue is a convenience helper, not a restriction.

For the full and most up-to-date list of all voices, engines, and languages, see the [Amazon Polly Voice List](https://docs.aws.amazon.com/polly/latest/dg/voicelist.html).

### English (US) — `en-US`

| Voice | Gender | Engines |
| --- | --- | --- |
| Joanna | Female | neural, standard, long-form |
| Matthew | Male | neural, standard |
| Ruth | Female | neural, long-form, generative |
| Stephen | Male | neural, long-form, generative |
| Danielle | Female | neural, long-form, generative |
| Gregory | Male | neural, long-form, generative |
| Ivy | Female | neural, standard |
| Kendra | Female | neural, standard |
| Kimberly | Female | neural, standard |
| Salli | Female | neural, standard |
| Joey | Male | neural, standard |
| Justin | Male | neural, standard |
| Kevin | Male | neural, standard |

### English (GB) — `en-GB`

| Voice | Gender | Engines |
| --- | --- | --- |
| Amy | Female | neural, standard |
| Emma | Female | neural, standard |
| Brian | Male | neural, standard |
| Arthur | Male | neural |

### English (AU) — `en-AU`

| Voice | Gender | Engines |
| --- | --- | --- |
| Olivia | Female | neural |
| Nicole | Female | standard |
| Russell | Male | standard |

### English (IN) — `en-IN`

| Voice | Gender | Engines |
| --- | --- | --- |
| Kajal | Female | neural |
| Aditi | Female | standard |
| Raveena | Female | standard |

### Spanish (US) — `es-US`

| Voice | Gender | Engines |
| --- | --- | --- |
| Lupe | Female | neural, standard |
| Pedro | Male | neural |
| Penelope | Female | standard |
| Miguel | Male | standard |

### Spanish (ES) — `es-ES`

| Voice | Gender | Engines |
| --- | --- | --- |
| Lucia | Female | neural, standard |
| Sergio | Male | neural |
| Enrique | Male | standard |
| Conchita | Female | standard |

### Spanish (MX) — `es-MX`

| Voice | Gender | Engines |
| --- | --- | --- |
| Mia | Female | standard |
| Andres | Male | neural |

### French (FR) — `fr-FR`

| Voice | Gender | Engines |
| --- | --- | --- |
| Léa | Female | neural, standard |
| Rémi | Male | neural |
| Mathieu | Male | standard |
| Céline | Female | standard |

### French (CA) — `fr-CA`

| Voice | Gender | Engines |
| --- | --- | --- |
| Gabrielle | Female | neural |
| Chantal | Female | standard |

### German — `de-DE`

| Voice | Gender | Engines |
| --- | --- | --- |
| Vicki | Female | neural, standard |
| Daniel | Male | neural |
| Hans | Male | standard |
| Marlene | Female | standard |

### Italian — `it-IT`

| Voice | Gender | Engines |
| --- | --- | --- |
| Bianca | Female | neural, standard |
| Adriano | Male | neural |
| Carla | Female | standard |
| Giorgio | Male | standard |

### Portuguese (BR) — `pt-BR`

| Voice | Gender | Engines |
| --- | --- | --- |
| Camila | Female | neural, standard |
| Vitória | Female | neural, standard |
| Thiago | Male | neural |
| Ricardo | Male | standard |

### Japanese — `ja-JP`

| Voice | Gender | Engines |
| --- | --- | --- |
| Kazuha | Female | neural, long-form |
| Tomoko | Female | neural, long-form |
| Takumi | Male | neural, standard |
| Mizuki | Female | standard |

### Korean — `ko-KR`

| Voice | Gender | Engines |
| --- | --- | --- |
| Seoyeon | Female | neural, standard |

### Chinese (Mandarin) — `cmn-CN`

| Voice | Gender | Engines |
| --- | --- | --- |
| Zhiyu | Female | neural, standard |

### Hindi — `hi-IN`

| Voice | Gender | Engines |
| --- | --- | --- |
| Kajal | Female | neural |
| Aditi | Female | standard |

### Arabic (UAE) — `ar-AE`

| Voice | Gender | Engines |
| --- | --- | --- |
| Hala | Female | neural |
| Zayd | Male | neural |

### Arabic (Standard) — `arb`

| Voice | Gender | Engines |
| --- | --- | --- |
| Zeina | Female | standard |

## Browser Support

| Browser | Minimum Version |
|---------|----------------|
| Chrome | 66+ |
| Firefox | 76+ |
| Safari | 14.1+ |
| Edge | 79+ |

## Gotchas and Common Mistakes

### 1. Forgetting `ref` on the `<audio>` element

This is the most common mistake. Without `ref={ref}`, the hook has no connection to the DOM audio element. `playing`, `duration`, and `stop()` will all be broken. Always pass the `ref`.

### 2. Multiple components = one shared client

The `PollyClient` and AWS credentials are cached at **module level**. If you render multiple `usePollyTextToSpeech` instances (e.g. one per chat message), they all share the same client. The Cognito `GetId` + `GetCredentialsForIdentity` API calls only happen once — every subsequent `speak()` call goes directly to Polly.

### 3. Pass `idToken` as state, not a stale string

The hook needs a valid Cognito ID token. If you pass an empty string or a stale token, the credential provider will fail. Fetch the token asynchronously and pass it via React state:

```tsx
// WRONG — token is empty on first render, hook will error
const token = '' // or some stale value
const { speak } = usePollyTextToSpeech({ config: { ...rest, idToken: token } })

// CORRECT — fetch token, set state, then call speak()
const [idToken, setIdToken] = useState('')

useEffect(() => {
  getIdToken().then(setIdToken)
}, [])

const { speak } = usePollyTextToSpeech({ config: { ...rest, idToken } })

// Only call speak() after idToken is set
useEffect(() => {
  if (idToken) speak('Hello')
}, [idToken])
```

### 4. Conditionally rendered `<audio>` loses the ref

If your `<audio>` element is inside a conditional (`{audioUrl && <audio ref={ref} ... />}`), the ref detaches when `audioUrl` becomes `null` (e.g. on a new `speak()` call). This is expected — the hook handles ref attach/detach cleanly.

### 5. SSML text must use `textType: 'ssml'`

If your text contains SSML tags like `<break>` or `<prosody>`, you must pass `'ssml'` as the second argument to `speak()`. Otherwise Polly will read the tags as literal text.

```tsx
// WRONG — tags read out loud as text
speak('<speak>Hello <break time="500ms"/> world</speak>')

// CORRECT
speak('<speak>Hello <break time="500ms"/> world</speak>', 'ssml')
```

## Error Handling

The hook catches errors and exposes them via the `error` state. It never throws. Common scenarios:

- **Invalid credentials** — expired token or misconfigured Identity Pool
- **Invalid voice/engine combo** — e.g. using `long-form` engine with a voice that doesn't support it
- **Text too long** — Polly has a 3,000 character limit for `SynthesizeSpeech` (6,000 for SSML including tags)
- **Autoplay blocked** — some browsers block `audio.play()` without user interaction

## License

MIT