tiny-swara
tiny-swara is a tiny, absurdly fast Bengali text-to-speech model with two voices, BF (female) and BM (male).
It packs the entire pipeline — phoneme encoder, duration predictor, prosody stacks and a source-filter waveform generator — into 4.5M parameters and an 8.2 MiB INT8 file. On just 2 CPU threads it renders 4.75 seconds of speech in 103 ms: 45.9× faster than real time, with about 4.6 seconds of headroom before playback even catches up. No GPU. No cloud. No waiting.
Built on the tiny-voca architecture, with a source-filter waveform generator instead of a neural vocoder.
| Parameters | 4,537,483 |
| Voices | BF, BM (Bengali) |
| Input | Bengali text, phonemized with espeak-ng (bn) |
| Output | 24 kHz mono |
| Format | ONNX (opset 17) |
| Recommended model | onnx/tiny-swara-int8.onnx |
| License | Apache-2.0 |
Files
| File | Size | Contents |
|---|---|---|
onnx/tiny-swara-int8.onnx |
8.24 MiB | INT8-quantized model, recommended |
onnx/tiny-swara.onnx |
20.4 MiB | FP32 model, reference |
config.json |
Model settings, phoneme vocabulary, voices | |
inventory.json |
Phoneme inventory and fallback mappings | |
lexicon.json |
English-to-Bengali word mappings for code-mixed text | |
tiny_voca/text.py |
Bengali-patched frontend: text to phoneme tokens | |
tiny_voca/__init__.py |
Package marker | |
inference.py |
ONNX inference, CLI and Python API | |
SETUP.md |
Beginner setup and troubleshooting | |
requirements.txt |
Python dependencies | |
samples/ |
MP3 samples used on this page (64 kbps mono) |
SHA256:
onnx/tiny-swara.onnx:cef829cff4b6cb38554e32d8f96b9b51f96a4c0e9cb2e406b53ac79d5923d630onnx/tiny-swara-int8.onnx:f3080233715cf22ebf519cfc1f81a8ebe4c937c0cdce0285b2cf51998c64eabb
Samples
The first table uses recordings exported from the notebook. The second table was
generated with the packaged inference.py and the INT8 model, so it also serves
as an INT8 check. All files here are MP3 at 64 kbps mono to keep the repository
small; the model itself outputs 24 kHz mono WAV.
Female (BF) and male (BM)
| Text | Female (BF) | Male (BM) |
|---|---|---|
| আমার order কোথায়? | ||
| গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি। | ||
| Hello বন্ধুরা, আপনারা কেমন আছেন? | ||
| বাহ, এই ছোট মডেলটি তো দারুণ! আমি মুগ্ধ। | ||
| আমি তোমাকে পছন্দ করি। |
INT8 model
| Text | Voice | Audio |
|---|---|---|
| আমার সোনার বাংলা, আমি তোমায় ভালোবাসি। | BF | |
| আজকের তারিখ ২০ নভেম্বর ২০২৬, তাপমাত্রা ৩২ ডিগ্রি সেলসিয়াস। | BM | |
| Hello, আমি আপনার order টি confirm করেছি। | BF |
Usage
First time here? SETUP.md has copy-paste steps and a
troubleshooting table.
Python script
pip install -r requirements.txt
sudo apt install espeak-ng # or: brew install espeak-ng
python inference.py \
--text "আমি ভালো আছি, আপনার অর্ডার কনফার্ম হয়েছে।" \
--voice BF \
--output hello.wav
| Option | Default | Meaning |
|---|---|---|
--text |
Text to speak | |
--text-file |
Read the text from a file instead of --text |
|
--voice |
BF |
BF (female) or BM (male) |
--output |
tiny-swara.wav |
Output WAV path |
--model |
onnx/tiny-swara-int8.onnx |
Path to an ONNX file |
--speed |
1.0 |
Speaking rate; larger is faster |
--pitch-shift |
0.0 |
Added to log F0; 0.058 is about one semitone |
--pitch-range |
1.0 |
Scales pitch movement around the utterance mean |
Text is read one sentence at a time, so text of any length works. Output is peak-normalized to 0.9 before it is written.
Python API
from inference import TinySwara
tts = TinySwara(voice="BF") # INT8 model by default
tts.save("আমি ভালো আছি।", "hello.wav")
# Sentence-level streaming: each chunk is complete before it is yielded.
for audio in tts.stream("প্রথম বাক্য। দ্বিতীয় বাক্য।"):
play(audio) # float32 mono at 24 kHz
inference.py sets TINY_VOCA_INVENTORY, TINY_VOCA_LEXICON and
TINY_VOCA_FRONTEND before importing tiny_voca.text, so the patched frontend
is wired up automatically. The frontend must be the Bengali-patched version
shipped here; the upstream tiny-voca text.py has no Bengali lexicon support.
ONNX directly
The graph takes phoneme ids and stress values; text is converted to ids outside
the graph by tiny_voca/text.py. The ids are the positions in vocab in
config.json.
import numpy as np
import onnxruntime as ort
session = ort.InferenceSession("onnx/tiny-swara-int8.onnx")
one = lambda value: np.array([value], dtype=np.float32)
audio, durations = session.run(None, {
"tokens": np.array([[1, 62, 23, 64, 6, 2]], dtype=np.int64), # <bos> ami . <eos>
"stress": np.array([[0, 1, 0, 0, 0, 0]], dtype=np.int64),
"voice": np.array([0], dtype=np.int64), # 0 = BF, 1 = BM
"speed": one(1.0),
"pitch_shift": one(0.0),
"pitch_range": one(1.0),
})
| Input | Type | Shape | Meaning |
|---|---|---|---|
tokens |
int64 | [1, N] | Phoneme ids, including <bos> and <eos> |
stress |
int64 | [1, N] | 0 none, 1 primary, 2 secondary |
voice |
int64 | [1] | 0 = BF, 1 = BM |
speed |
float32 | [1] | 1.0 as predicted; larger is faster |
pitch_shift |
float32 | [1] | Added to log F0 |
pitch_range |
float32 | [1] | Scales pitch movement |
| Output | Type | Shape | Meaning |
|---|---|---|---|
audio |
float32 | [1, samples] | Waveform at 24 kHz |
durations |
int64 | [1, N] | Length of each token in frames of 256 samples |
The model reads one utterance per call, up to 512 tokens and 32 seconds. Split longer text into sentences. The raw graph output can be quiet, so scale it to a usable level before saving 16-bit samples:
audio = audio[0]
audio *= 0.9 / np.abs(audio).max()
Model
Text is converted to phonemes with espeak-ng
(-v bn --ipa) by tiny_voca/text.py and mapped onto a Bengali phoneme
inventory. Phonemes that are not in the inventory are rewritten with the
fallback rules in inventory.json. Known English words in code-mixed text are
replaced with their Bengali spelling using lexicon.json; other Latin words are
phonemized with the English (en-us) voice and mapped back through the same
fallbacks. The token stream matches training: <bos>, <sp> between words,
explicit punctuation tokens, and <eos>.
A transformer encoder (4 blocks) reads the phonemes and a duration predictor gives each one a length in frames. Two frame-level transformer stacks (2 and 5 blocks) predict pitch, voicing and energy and then the parameters of the waveform generator. All blocks use 192 channels, 4 heads and rotary position embeddings. The waveform is produced by a source-filter synthesizer instead of a neural vocoder: a band-limited pulse train at the predicted pitch and a noise signal are each shaped by a time-varying filter predicted once per frame (256 samples, 10.7 ms).
| Vocabulary | 77 tokens (Bengali IPA phonemes, punctuation, special tokens) |
| Encoder | 4 blocks, 192 channels, 4 heads |
| Frame stacks | 2 + 5 blocks, 192 channels, 4 heads |
| Waveform | Source-filter, hop 256 |
| Training steps | 60,000 |
Training
| Dataset | ai4bharat/Rasa, Bengali subset, styles WIKI, CONV, BOOK, NEWS, SANGRAH, INDIC |
| Hours / clips | 11.5 h, 6,275 train / 112 validation / 61 test clips after filtering |
| Steps | 60,000 |
| Hardware | 1 × Tesla T4 on Kaggle, in two sessions (about 10.5 h and 7 h) |
| Alignment | facebook/wav2vec2-lv-60-espeak-cv-ft, relaxed gates (weak-phone fraction 0.5, token probability 0.3); 97% of training clips kept |
- Architecture and recipe from tiny-voca: multi-resolution spectral losses for the first 8,000 steps, then adversarial (GAN) training with multi-period and multi-resolution discriminators.
- All 60,000 steps ran on one Kaggle Tesla T4 across the two sessions above.
Evaluation
tiny-swara was evaluated using AI4Bharat Indic-Transcribe Core as the Bengali automatic speech recognition (ASR) evaluator.
Quantitative results
| Dataset | Clips | Synthesized WER | Synthesized CER | Reference WER | Reference CER |
|---|---|---|---|---|---|
| Validation | 100 | 13.85% | 5.66% | 12.10% | 2.22% |
| Test | 64 | 13.28% | 4.39% | 13.16% | 2.38% |
Speaker-wise results
| Dataset | Speaker | Clips | WER | CER |
|---|---|---|---|---|
| Validation | BF | 46 | 18.04% | 8.69% |
| Validation | BM | 54 | 10.24% | 2.99% |
| Test | BF | 29 | 12.39% | 4.91% |
| Test | BM | 35 | 14.00% | 3.95% |
Test WER is very close to the reference (13.28% vs 13.16%), but CER is noticeably higher for generated speech (4.39% vs 2.38%). The female voice has the largest validation gap, while the test split shows the opposite WER ordering — too early to call one voice consistently worse. The splits also have different difficulty distributions, so their scores should not be read as a direct improvement or regression.
Evaluation methodology
- ASR evaluator: AI4Bharat Indic-Transcribe Core.
- Reference audio: Original Bengali Rasa dataset recordings.
- Synthesized audio: Generated by tiny-swara.
- Metrics: Word Error Rate (WER) and Character Error Rate (CER).
- Text normalization: Unicode NFC normalization and punctuation removal.
Internal acoustic metrics
These are internal validation numbers on held-out Rasa clips.
| Metric | Value |
|---|---|
| Mel loss, teacher-forced (lower is better) | 0.70 |
| Mel loss, predicted durations | 0.72 |
| Pitch error (semitones) | 2.07 |
| Voicing accuracy | 0.93 |
| Pause accuracy | 0.90 |
Limitations: Results depend on the ASR evaluator and text normalization. These metrics measure transcription accuracy rather than subjective speech naturalness, prosody or speaker similarity. The evaluation covers 100 validation and 64 test clips and should be treated as a preliminary benchmark.
Performance
This is the fun part. Measured with the INT8 model on CPU using 2 ONNX Runtime threads:
| Inference (2 threads) | 103 ms |
| Generated audio | 4.75 s |
| Real-time factor | 45.9× |
| Headroom over playback | ~4.6 s |
The test sentence is the 66-token line "গতকাল আমি খুব ব্যস্ত ছিলাম, তাই সন্ধ্যায় তোমাকে ফোন করতে পারিনি।". The 103 ms covers the ONNX graph only; phonemization with espeak-ng adds about 70 ms on a desktop CPU, and writing the WAV is negligible. Speed depends on the CPU and thread count.
Intended use
Research and personal projects. Not for impersonating any real person.
Limitations
- Bengali with limited code-mixed English; two fixed voices, no voice cloning.
- Code-mixed English is handled word by word through a small lexicon (213 entries) plus fallback rules. Foreign names, unusual spellings and words outside the lexicon are often mispronounced.
- Numbers and abbreviations depend on espeak-ng's Bengali frontend.
- The same text is always read the same way. There is no sampling of prosody.
- No frame-level streaming;
inference.pystreams sentence by sentence. - The INT8 model is smaller and faster, but its output differs slightly from the FP32 reference.
License
The weights and code are released under Apache-2.0. tiny-swara is built on the tiny-voca architecture by Muhammed Nazeem; see n4ze3m/tiny-voca. The phoneme inventory and lexicon were produced for this model. The training recordings come from AI4Bharat Rasa; follow its terms when redistributing derived audio.
Credits
- tiny-voca architecture and training code by n4ze3m (Apache-2.0).
- Bengali recordings from AI4Bharat Rasa.
- Aligner:
facebook/wav2vec2-lv-60-espeak-cv-ft. - Phonemizer: espeak-ng.
- Downloads last month
- 38
