SparkTTS-Thai-Emotion-Control

Fine-tuned SparkTTS models for experiments on emotion-controlled Thai text-to-speech. This project explores whether adding explicit emotion tokens can help a text-to-speech model produce Thai speech with a requested emotion. It is based on Unsloth/Spark-TTS-0.5B, uses the THAI-SER dataset, and follows the original SparkTTS implementation.

The focus here is emotion control and Thai text representation, rather than voice cloning as the main research goal. The checkpoints are experimental and should not be treated as production-ready models.

What is included

This project uses PyThaiNLP to preprocess Thai text and prepare different text representations for the SparkTTS training experiments and using for inference to compare four ways of representing Thai text:

• Raw Thai text: Original Thai text without additional text segmentation or phonetic conversion.
• Tokenized Thai text: Thai text segmented into tokens using Thai word-tokenization tools. (word_tokenize(newmm))
• Romanized Thai text: Tokenized-Thai text converted into a Romanized representation.(transliterate->romanize)
• Phoneme representation: Tokenized-Thai text converted into a phonetic representation for speech synthesis. (transliterate->tltk_ipa)

Emotion labels

The training experiments use five emotion categories represented in THAI-SER tokens: <|emotion_neutral|>, <|emotion_happy|>, <|emotion_sad|>, <|emotion_angry|>, <|emotion_frustrated|>

Checkpoints in this repo

This repo contains 8 fine-tuned LLM checkpoints (4 text representations x with/without emotion tokens). Each subfolder is a complete transformers model (weights + tokenizer + config).

Subfolder	Text representation	Emotion tokens
raw	Raw Thai text	yes
raw_baseline	Raw Thai text	no
tokenized	Thai word tokens (newmm)	yes
tokenized_baseline	Thai word tokens (newmm)	no
romanized	Romanized Thai	yes
romanized_baseline	Romanized Thai	no
phoneme	IPA phonemes (tltk_ipa)	yes
phoneme_baseline	IPA phonemes (tltk_ipa)	no

Overall Performance

Model	WER ↓	CER ↓	MOS ↑	Emotion Match ↑
Baseline	0.0936	0.0552	3.332	-
Emotion	0.1086	0.0903	4.142	0.779

Text representation	Baseline	Angry	Frustrated	Happy	Neutral	Sad
Phoneme	0.5708	0.8212	0.8396	0.6561	0.9313	0.6737
Raw	0.5716	0.7762	0.8292	0.7086	0.9255	0.6523
Romanized	0.5438	0.7723	0.8041	0.6661	0.9305	0.6493
Tokenized	0.5831	0.8098	0.8296	0.6759	0.9302	0.6967

Load the fine-tuned LLM

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "Mametoyas/SparkTTS-Thai-Emotion-Control"  # change after upload
SUBFOLDER = "phoneme"  # one of: raw, tokenized, romanized, phoneme (+ _baseline variants)

tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUBFOLDER)
model = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder=SUBFOLDER, torch_dtype=torch.bfloat16, device_map="auto"
)

Or download only that checkpoint with huggingface_hub:

from huggingface_hub import snapshot_download

snapshot_download(
    "Mametoyas/SparkTTS-Thai-Emotion-Control",
    allow_patterns=["phoneme/*"],
    local_dir="pretrained_models/ThaiEmotion/phoneme",
)

Text preprocessing module

thai_text_preprocess.py contains the Thai text preprocessing used for training and inference (raw / tokenized / romanized / phoneme), plus helpers for emotion and prompt tokens.

from thai_text_preprocess import TTSDataProcessor, build_prompt, get_text_for_model

processor = TTSDataProcessor()
processed = processor.process("สวัสดีครับ ยินดีต้อนรับ")
# {'raw': ..., 'tokenized_text': ..., 'romanized_text': ..., 'phoneme': ...}

text = get_text_for_model(processed, "phoneme")   # matches the checkpoint subfolder
prompt = build_prompt(text, emotion="happy", gender="male")
# <|task_tts|><|emotion_happy|><|gender_male|><|start_content|>...<|end_content|><|start_global_token|>

Install dependencies:

pip install pythainlp "pythainlp[thai2rom]" tltk

# thai2rom model weights for romanize(); without these, romanized_text
# silently falls back to the tokenized text
python -c "from pythainlp.corpus import download; download('thai2rom-pytorch-attn')"

Colab inference notebook

inference_colab.ipynb runs the full pipeline on Google Colab (GPU): install → download BiCodec from the base model → load a fine-tuned LLM checkpoint from this repo → upload a reference voice → generate speech for each emotion.

Full inference setup (BiCodec / vocoder)

The fine-tuned checkpoints replace only the LLM component of Spark-TTS. BiCodec, the vocoder, and the rest of the runtime still come from the base model:

from huggingface_hub import snapshot_download

snapshot_download("SparkAudio/Spark-TTS-0.5B", local_dir="pretrained_models/Spark-TTS-0.5B")

Install

Clone and Install

• Clone the repo

git clone https://github.com/SparkAudio/Spark-TTS.git
cd Spark-TTS

• Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
• Create Conda env:

conda create -n sparktts -y python=3.12
conda activate sparktts
pip install -r requirements.txt
# If you are in mainland China, you can set the mirror as follows:
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

Model Download

Download via python:

from huggingface_hub import snapshot_download


snapshot_download("SparkAudio/Spark-TTS-0.5B", local_dir="pretrained_models/Spark-TTS-0.5B")

Download via git clone:

mkdir -p pretrained_models


# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install


git clone https://huggingface.co/SparkAudio/Spark-TTS-0.5B pretrained_models/Spark-TTS-0.5B