tiny-swara

tiny-swara is a tiny, absurdly fast Bengali text-to-speech model with two voices, BF (female) and BM (male).

It packs the entire pipeline — phoneme encoder, duration predictor, prosody stacks and a source-filter waveform generator — into 4.5M parameters and an 8.2 MiB INT8 file. On just 2 CPU threads it renders 4.75 seconds of speech in 103 ms: 45.9× faster than real time, with about 4.6 seconds of headroom before playback even catches up. No GPU. No cloud. No waiting.

Built on the tiny-voca architecture, with a source-filter waveform generator instead of a neural vocoder.

Parameters 4,537,483
Voices BF, BM (Bengali)
Input Bengali text, phonemized with espeak-ng (bn)
Output 24 kHz mono
Format ONNX (opset 17)
Recommended model onnx/tiny-swara-int8.onnx
License Apache-2.0

architecture

Files

File Size Contents
onnx/tiny-swara-int8.onnx 8.24 MiB INT8-quantized model, recommended
onnx/tiny-swara.onnx 20.4 MiB FP32 model, reference
config.json Model settings, phoneme vocabulary, voices
inventory.json Phoneme inventory and fallback mappings
lexicon.json English-to-Bengali word mappings for code-mixed text
tiny_voca/text.py Bengali-patched frontend: text to phoneme tokens
tiny_voca/__init__.py Package marker
inference.py ONNX inference, CLI and Python API
SETUP.md Beginner setup and troubleshooting
requirements.txt Python dependencies
samples/ MP3 samples used on this page (64 kbps mono)

SHA256:

  • onnx/tiny-swara.onnx: cef829cff4b6cb38554e32d8f96b9b51f96a4c0e9cb2e406b53ac79d5923d630
  • onnx/tiny-swara-int8.onnx: f3080233715cf22ebf519cfc1f81a8ebe4c937c0cdce0285b2cf51998c64eabb

Samples

The first table uses recordings exported from the notebook. The second table was generated with the packaged inference.py and the INT8 model, so it also serves as an INT8 check. All files here are MP3 at 64 kbps mono to keep the repository small; the model itself outputs 24 kHz mono WAV.

Female (BF) and male (BM)

Text Female (BF) Male (BM)
আমার order কোথায়?
গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি।
Hello বন্ধুরা, আপনারা কেমন আছেন?
বাহ, এই ছোট মডেলটি তো দারুণ! আমি মুগ্ধ।
আমি তোমাকে পছন্দ করি।

INT8 model

Text Voice Audio
আমার সোনার বাংলা, আমি তোমায় ভালোবাসি। BF
আজকের তারিখ ২০ নভেম্বর ২০২৬, তাপমাত্রা ৩২ ডিগ্রি সেলসিয়াস। BM
Hello, আমি আপনার order টি confirm করেছি। BF

Usage

First time here? SETUP.md has copy-paste steps and a troubleshooting table.

Python script

pip install -r requirements.txt
sudo apt install espeak-ng          # or: brew install espeak-ng

python inference.py \
  --text "আমি ভালো আছি, আপনার অর্ডার কনফার্ম হয়েছে।" \
  --voice BF \
  --output hello.wav
Option Default Meaning
--text Text to speak
--text-file Read the text from a file instead of --text
--voice BF BF (female) or BM (male)
--output tiny-swara.wav Output WAV path
--model onnx/tiny-swara-int8.onnx Path to an ONNX file
--speed 1.0 Speaking rate; larger is faster
--pitch-shift 0.0 Added to log F0; 0.058 is about one semitone
--pitch-range 1.0 Scales pitch movement around the utterance mean

Text is read one sentence at a time, so text of any length works. Output is peak-normalized to 0.9 before it is written.

Python API

from inference import TinySwara

tts = TinySwara(voice="BF")        # INT8 model by default
tts.save("আমি ভালো আছি।", "hello.wav")

# Sentence-level streaming: each chunk is complete before it is yielded.
for audio in tts.stream("প্রথম বাক্য। দ্বিতীয় বাক্য।"):
    play(audio)                    # float32 mono at 24 kHz

inference.py sets TINY_VOCA_INVENTORY, TINY_VOCA_LEXICON and TINY_VOCA_FRONTEND before importing tiny_voca.text, so the patched frontend is wired up automatically. The frontend must be the Bengali-patched version shipped here; the upstream tiny-voca text.py has no Bengali lexicon support.

ONNX directly

The graph takes phoneme ids and stress values; text is converted to ids outside the graph by tiny_voca/text.py. The ids are the positions in vocab in config.json.

import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("onnx/tiny-swara-int8.onnx")
one = lambda value: np.array([value], dtype=np.float32)

audio, durations = session.run(None, {
    "tokens": np.array([[1, 62, 23, 64, 6, 2]], dtype=np.int64),  # <bos> ami . <eos>
    "stress": np.array([[0, 1, 0, 0, 0, 0]], dtype=np.int64),
    "voice": np.array([0], dtype=np.int64),      # 0 = BF, 1 = BM
    "speed": one(1.0),
    "pitch_shift": one(0.0),
    "pitch_range": one(1.0),
})
Input Type Shape Meaning
tokens int64 [1, N] Phoneme ids, including <bos> and <eos>
stress int64 [1, N] 0 none, 1 primary, 2 secondary
voice int64 [1] 0 = BF, 1 = BM
speed float32 [1] 1.0 as predicted; larger is faster
pitch_shift float32 [1] Added to log F0
pitch_range float32 [1] Scales pitch movement
Output Type Shape Meaning
audio float32 [1, samples] Waveform at 24 kHz
durations int64 [1, N] Length of each token in frames of 256 samples

The model reads one utterance per call, up to 512 tokens and 32 seconds. Split longer text into sentences. The raw graph output can be quiet, so scale it to a usable level before saving 16-bit samples:

audio = audio[0]
audio *= 0.9 / np.abs(audio).max()

Model

Text is converted to phonemes with espeak-ng (-v bn --ipa) by tiny_voca/text.py and mapped onto a Bengali phoneme inventory. Phonemes that are not in the inventory are rewritten with the fallback rules in inventory.json. Known English words in code-mixed text are replaced with their Bengali spelling using lexicon.json; other Latin words are phonemized with the English (en-us) voice and mapped back through the same fallbacks. The token stream matches training: <bos>, <sp> between words, explicit punctuation tokens, and <eos>.

A transformer encoder (4 blocks) reads the phonemes and a duration predictor gives each one a length in frames. Two frame-level transformer stacks (2 and 5 blocks) predict pitch, voicing and energy and then the parameters of the waveform generator. All blocks use 192 channels, 4 heads and rotary position embeddings. The waveform is produced by a source-filter synthesizer instead of a neural vocoder: a band-limited pulse train at the predicted pitch and a noise signal are each shaped by a time-varying filter predicted once per frame (256 samples, 10.7 ms).

Vocabulary 77 tokens (Bengali IPA phonemes, punctuation, special tokens)
Encoder 4 blocks, 192 channels, 4 heads
Frame stacks 2 + 5 blocks, 192 channels, 4 heads
Waveform Source-filter, hop 256
Training steps 60,000

Training

Dataset ai4bharat/Rasa, Bengali subset, styles WIKI, CONV, BOOK, NEWS, SANGRAH, INDIC
Hours / clips 11.5 h, 6,275 train / 112 validation / 61 test clips after filtering
Steps 60,000
Hardware 1 × Tesla T4 on Kaggle, in two sessions (about 10.5 h and 7 h)
Alignment facebook/wav2vec2-lv-60-espeak-cv-ft, relaxed gates (weak-phone fraction 0.5, token probability 0.3); 97% of training clips kept
  • Architecture and recipe from tiny-voca: multi-resolution spectral losses for the first 8,000 steps, then adversarial (GAN) training with multi-period and multi-resolution discriminators.
  • All 60,000 steps ran on one Kaggle Tesla T4 across the two sessions above.

Evaluation

tiny-swara was evaluated using AI4Bharat Indic-Transcribe Core as the Bengali automatic speech recognition (ASR) evaluator.

Quantitative results

Dataset Clips Synthesized WER Synthesized CER Reference WER Reference CER
Validation 100 13.85% 5.66% 12.10% 2.22%
Test 64 13.28% 4.39% 13.16% 2.38%

Speaker-wise results

Dataset Speaker Clips WER CER
Validation BF 46 18.04% 8.69%
Validation BM 54 10.24% 2.99%
Test BF 29 12.39% 4.91%
Test BM 35 14.00% 3.95%

Test WER is very close to the reference (13.28% vs 13.16%), but CER is noticeably higher for generated speech (4.39% vs 2.38%). The female voice has the largest validation gap, while the test split shows the opposite WER ordering — too early to call one voice consistently worse. The splits also have different difficulty distributions, so their scores should not be read as a direct improvement or regression.

Evaluation methodology

  • ASR evaluator: AI4Bharat Indic-Transcribe Core.
  • Reference audio: Original Bengali Rasa dataset recordings.
  • Synthesized audio: Generated by tiny-swara.
  • Metrics: Word Error Rate (WER) and Character Error Rate (CER).
  • Text normalization: Unicode NFC normalization and punctuation removal.

Internal acoustic metrics

These are internal validation numbers on held-out Rasa clips.

Metric Value
Mel loss, teacher-forced (lower is better) 0.70
Mel loss, predicted durations 0.72
Pitch error (semitones) 2.07
Voicing accuracy 0.93
Pause accuracy 0.90

Limitations: Results depend on the ASR evaluator and text normalization. These metrics measure transcription accuracy rather than subjective speech naturalness, prosody or speaker similarity. The evaluation covers 100 validation and 64 test clips and should be treated as a preliminary benchmark.

Performance

This is the fun part. Measured with the INT8 model on CPU using 2 ONNX Runtime threads:

Inference (2 threads) 103 ms
Generated audio 4.75 s
Real-time factor 45.9×
Headroom over playback ~4.6 s

The test sentence is the 66-token line "গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি।". The 103 ms covers the ONNX graph only; phonemization with espeak-ng adds about 70 ms on a desktop CPU, and writing the WAV is negligible. Speed depends on the CPU and thread count.

Intended use

Research and personal projects. Not for impersonating any real person.

Limitations

  • Bengali with limited code-mixed English; two fixed voices, no voice cloning.
  • Code-mixed English is handled word by word through a small lexicon (213 entries) plus fallback rules. Foreign names, unusual spellings and words outside the lexicon are often mispronounced.
  • Numbers and abbreviations depend on espeak-ng's Bengali frontend.
  • The same text is always read the same way. There is no sampling of prosody.
  • No frame-level streaming; inference.py streams sentence by sentence.
  • The INT8 model is smaller and faster, but its output differs slightly from the FP32 reference.

License

The weights and code are released under Apache-2.0. tiny-swara is built on the tiny-voca architecture by Muhammed Nazeem; see n4ze3m/tiny-voca. The phoneme inventory and lexicon were produced for this model. The training recordings come from AI4Bharat Rasa; follow its terms when redistributing derived audio.

Credits

  • tiny-voca architecture and training code by n4ze3m (Apache-2.0).
  • Bengali recordings from AI4Bharat Rasa.
  • Aligner: facebook/wav2vec2-lv-60-espeak-cv-ft.
  • Phonemizer: espeak-ng.
Downloads last month
38
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train Xeven7/tiny-swara