EmoSphere-PP-16kHz

EmoSphere++ is an emotion controllable zero shot text to speech model by Cho et al. (IEEE Transactions on Affective Computing, 2025). It synthesizes a 16 kHz, 80 band log mel spectrogram from English text, a reference recording of the target speaker and an emotion given as a point in a sphere: the radius sets the intensity, the elevation and azimuth set the style.

This repository holds the authors' released checkpoint (model_ckpt_steps_1100000.ckpt, 1.1M steps) converted to the transformers format. The weights are the authors' float32 values; only the flow matching estimator is stored in float16 (see Precision). No weight was retrained or calibrated.

The model outputs a mel spectrogram. generate renders waveforms with the 16 kHz vocoder RetentionLabs/BigVGAN-16kHz-80band, since the authors did not release theirs.

• Paper: doi:10.1109/TAFFC.2025.3561267
• Original code: github.com/Choddeok/EmoSpherepp, commit 3907152
• Demo of the authors: choddeok.github.io/EmoSphere-Demo

Model summary

Input	Phonemes (CMUdict and g2p_en), WavLM x-vector, emotion2vec+ base embedding, emotion-adaptive spherical vector $(r, \vartheta, \phi)$
Output	Natural log mel spectrogram, 80 bands, 16 kHz, hop 256
Parameters	28.2M (text encoder 15.3M, flow matching estimator 11.4M, style encoder 1.4M)
Precision	Mixed float32 and float16, see below
Training data	ESD English (10 speakers, 5 emotions), by the authors

Architecture

• Joint attribute style encoder. The WavLM x-vector (microsoft/wavlm-base-sv) and the emotion2vec+ base utterance embedding each pass a two layer perceptron. The elevation and azimuth select one of 2 and one of 4 learned embeddings, and the radius is projected. The result is a 192 channel style vector.
• Text encoder. The Glow-TTS encoder of Matcha-TTS: phoneme embeddings, a convolutional pre-net, the style vector concatenated to every token, six relative position transformer layers, the prior mean and a duration predictor.
• Decoder. The conditional flow matching U-Net of Matcha-TTS with snakebeta feed forwards, conditioned on the aligned prior and on the style vector. Generation uses 100 Euler steps, temperature 1 and guidance 0, as the authors' inference does.

The processor builds every model input from a reference recording: phoneme ids from the NATSpeech front end with a port of g2p_en, the x-vector, the emotion2vec+ embedding, and the spherical vector from the audEERING arousal, dominance and valence reading of the reference, measured against the emotion center and quartiles of the authors' training data. Neutral uses $(0, 0, 0)$.

Precision

Part	dtype	Reason
Text encoder and duration predictor	float32	Durations are rounded up ($\lceil e^{\log d} \rceil$), so a small error changes the frame count of a phoneme and the length of the whole utterance.
Style encoder	float32	Its output conditions the duration predictor.
Processor encoders (WavLM x-vector, emotion2vec+, audEERING VAD)	float32	Their outputs feed the style encoder and so the durations.
ODE state (Euler integration of the flow)	float32	The 100 step sum accumulates rounding error.
Flow matching estimator	float16	Its values stay in the float16 range, and float16 has 3 more mantissa bits than bfloat16.
Vocoder (RetentionLabs/BigVGAN-16kHz-80band)	float16	Same reason as the estimator.

_keep_in_fp32_modules_strict keeps the text and style encoders in float32 for any load dtype, config.dtype (float16) applies to the estimator, and config.vocoder_dtype sets the vocoder. The estimator inputs are cast to its dtype and its output back to float32 at every Euler step. The processor loads its encoders in float32 (processor_config.json, dtype).

The choice comes from a precision ablation: 40 held-out ESD reference and target pairs (20 seen, 20 unseen speakers), seed 0, 100 Euler steps, each variant compared with the all float32 run of the same pair. "Same length" counts generations whose frame count equals the float32 one. Mel L1 is on the generated log mel, log spectrum L1 on the vocoded waveform. WER uses Whisper large-v3, UTMOS is UTMOSv2.

Variant	Same length	Mel L1	Log spectrum L1	WER (%)	UTMOS
All float32	40/40	0	0	5.90	2.71
This release (loaded from this repository)	40/40	0.0013	0.089	5.90	2.71
Estimator float16 only	40/40	0.0013	0.063	5.90	2.71
Estimator bfloat16 only	40/40	0.017	0.140	5.56	-
Vocoder float16 only	40/40	0	0.082	5.90	2.71
Vocoder bfloat16 only	40/40	0	0.279	5.90	-
Text encoder bfloat16 only	23/40	0.043	0.194	6.25	-
Speaker encoder bfloat16 only	18/40	0.064	0.234	7.99	-
Estimator float16, encoders and style bfloat16	23/40	0.017	0.304	5.56	2.67
Everything bfloat16	12/40	0.204	0.602	7.64	-

Requirements

pip install voicestudio==5.18.0 soundfile torchaudio

The remote code imports the flow matching decoder from voicestudio and everything else from transformers. The processor downloads three encoders from their own repositories at pinned commits on first use: microsoft/wavlm-base-sv (speaker x-vector), emotion2vec/emotion2vec_plus_base (emotion embedding, converted on load) and audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim (arousal, dominance and valence, CC BY-NC-SA 4.0). The g2p_en and NLTK resources are downloaded from their pinned GitHub commits and checked by SHA-256.

Usage

import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor

processor = AutoProcessor.from_pretrained("RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True, device="cuda")
model = AutoModel.from_pretrained("RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True).to("cuda").eval()

inputs = processor(
    text="The weather is lovely today.",
    reference_audio="reference.wav",  # any sampling rate, read mono
    emotion="happy",                   # neutral, angry, happy, sad or surprise
).to("cuda")
output = model.generate(**inputs)
audio = output.audio_values[0, : output.audio_lengths[0]]
sf.write("output.wav", audio.cpu().numpy(), output.sampling_rate)

Load the model without a dtype argument to keep the mixed precision above. generate returns the mel spectrogram (spectrogram, spectrogram_lengths) and the waveform, loudness normalized to -20 LUFS as the authors' vocoder wrapper does. Pass output_audio=False to get the mel spectrogram only. emotion_intensity and emotion_style can be passed to the processor to override the spherical vector read from the reference.

Evaluation

Paper protocol

ESD English, the 1,500 reference and target pairs of the paper's protocol (1,200 seen speaker pairs, 300 unseen speaker pairs), one generation per pair with seed 0, float32 model, vocoded by RetentionLabs/BigVGAN-16kHz-80band. WER avg is the mean of the Whisper large-v3 and wav2vec 2.0 (facebook/wav2vec2-large-960h-lv60-self) WER, SECS avg the mean of the Resemblyzer and WavLM speaker cosines. ECA and EECS use emotion2vec+ base, SVAS the audEERING arousal and valence. Paper rows are Tables II and III of the paper, which uses the authors' unreleased vocoder. The mixed precision of this release keeps every duration of the float32 run on the 40 pairs above; it was not run on the 1,500 pairs.

Speakers	Row	WER avg (%)	SECS avg	ECA (%)	SVAS	EECS
Seen (1,200)	Paper, EmoSphere++	17.19	0.8181	93.53	0.8717	0.9270
Seen (1,200)	Ours, float32	9.87	0.8377	93.42	0.8799	0.9515
Seen (1,200)	Paper, ground truth	13.21	0.8311	95.53	-	0.9487
Seen (1,200)	Ours, ground truth	2.76	0.8627	95.58	-	0.9410
Unseen (300)	Paper, EmoSphere++	16.42	0.7592	94.61	0.8458	0.9358
Unseen (300)	Ours, float32	8.78	0.7681	91.33	0.8411	0.9086
Unseen (300)	Paper, ground truth	14.11	0.8503	100.00	-	0.9944
Unseen (300)	Ours, ground truth	4.27	0.8581	100.00	-	0.9938

Our WER setup reads far lower than the paper's even on the ground truth recordings (2.76 % against 13.21 % for seen speakers), so WER is comparable only between rows of our setup. The paper does not give its normalizer or its exact wav2vec 2.0 checkpoint; we normalize with the Whisper English normalizer.

Authors' demo samples

The 4 non-parallel style transfer items of the authors' demo page (2 seen, 2 unseen speakers; Sad and Happy), regenerated from the demo's reference recordings and texts with seed 0. "Demo" rows are the audio files of the demo page. SIM is the WavLM speaker cosine to the reference recording and to the ground truth recording. Our rows ran the vocoder in bfloat16. WER is 0 % for every row.

Row	UTMOS	SIM to reference	SIM to ground truth
Demo, ground truth	3.17	0.680	1.000
Demo, ground truth vocoded by the authors	3.14	0.684	0.967
Ours, ground truth vocoded by BigVGAN-16kHz-80band	3.09	0.686	0.945
Demo, EmoSphere++	2.86	0.362	0.372
Ours, float32	2.97	0.371	0.378

Limitations

• English only, trained on 10 ESD speakers. Zero shot quality on other voices and recording conditions is lower than on ESD (see the unseen speaker rows).
• The spherical vector is computed from the emotion label and the reference recording; an emotion other than the five ESD emotions is not supported.
• The text front end follows NATSpeech: numbers are not expanded and are dropped.
• The vocoder is ours, not the authors'; results differ from the paper by that and by evaluation details.

License

Research and non-commercial use only. See LICENSE and NOTICE.

• Code: MIT, Copyright (c) 2026 Deok-Hyeon Cho (EmoSphere++), with the MIT notices of Matcha-TTS, Glow-TTS and Grad-TTS whose modules it ports.
• Weights: released by the authors without a license and trained on ESD, whose terms allow research use only. Ask the authors before any other use.
• Encoders the processor downloads (not redistributed here): microsoft/wavlm-base-sv (Microsoft, see its model card), emotion2vec/emotion2vec_plus_base (FunASR model license) and audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim (CC BY-NC-SA 4.0).
• g2p resources: CMUdict (BSD-2-Clause), g2p_en (Apache 2.0) and the NLTK averaged perceptron tagger (Apache 2.0).

Citation

@article{cho2025emosphere++,
  title={EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector},
  author={Cho, Deok-Hyeon and Oh, Hyung-Seok and Kim, Seung-Bin and Lee, Seong-Whan},
  journal={IEEE Transactions on Affective Computing},
  volume={16},
  number={3},
  pages={2365--2380},
  year={2025},
  doi={10.1109/TAFFC.2025.3561267}
}

Conversion and evaluation by TICTAP-11 and b-re-w (Retention Labs).