GreenBitAI/Drama-3-8bit-paged

Drama 3 8-bit, paged for gbx-lm

Built with Fish Audio. Fish Audio Drama 3, the expressive multi-speaker text-to-speech model, packaged for gbx-lm on Apple silicon: an OpenAI-compatible /v1/audio/speech server that streams audio, and runs on Macs that cannot hold the whole model in memory.

This is a community build by GreenBitAI. It is not made, sponsored or endorsed by Fish Audio.

• Where the weights come from. Fish Audio's release, as converted to MLX by mlx-community/fish-audio-drama-3-8bit, whose routed experts are 8-bit. Here the layers it kept in bfloat16 are 8-bit too, in the same format: a quarter faster, with speech quality unchanged within what we can measure (below). The routed experts live in experts.bin, one record per expert, so that a machine can read just the experts it needs. NOTICE lists every change.
• Real time on 64 GB and up. 56-58 frames per second on an M3 Ultra, 2.6x real time, first audio in under 0.3 s when streaming.
• Runs on 32-48 GB, for offline use. With less memory the experts stream from the SSD: three quarters of real time on a 48 GB Mac, half on 32 GB. Generate the file, then play it.

What you need

Machine	Apple silicon Mac, 64 GB or more for real time, 32 GB at least; or a DGX Spark
Disk	40 GB for the model (37 GiB)
gbx-lm	a release with speech support
Python	3.10 or newer, for the audio codec (below)

gbx-lm generates the model's audio codes; turning codes into sound is the job of a small companion package, gbx-codecs, which gbx-lm starts in its own process. It needs only mlx, numpy and miniaudio -- no PyTorch, no ffmpeg -- and about 3 GB of memory while it decodes. The codec weights are in this repository.

Setup

1. gbx-lm. The prebuilt binary, as in the gbx-lm README:

curl -fL -o gbx_lm-darwin-arm64.tar.gz 'https://github.com/GreenBitAI/gbx-lm/releases/latest/download/gbx_lm-darwin-arm64.tar.gz' \
  && tar -xzf gbx_lm-darwin-arm64.tar.gz gbx_lm

2. The codec, in a virtual environment of its own (this location is the one gbx-lm looks in by default):

python3 -m venv ~/.gbx-codecs
~/.gbx-codecs/bin/pip install gbx-codecs

3. Voices (optional). To speak in a particular voice, put a short clean recording (5-30 s, WAV; flac and mp3 need pip install "gbx-codecs[formats]") and its exact transcript side by side in a folder:

~/voices/
    narrator.wav
    narrator.txt      # the words spoken in narrator.wav

The request's voice is the file name, narrator. Use only voices you have the right to use -- see License. Without a voice ("voice": "default") the model chooses one; fix it with a seed.

On a DGX Spark (or another Linux machine with an NVIDIA GPU), use the Linux build of gbx-lm and install the codec with the CUDA build of MLX: ~/.gbx-codecs/bin/pip install "mlx[cuda13]" gbx-codecs. The model runs in memory there, at about 1.3x real time.

4. Start the server. The model downloads on first start:

./gbx_lm --model GreenBitAI/Drama-3-8bit-paged --voices_dir ~/voices

If the codec is not in ~/.gbx-codecs, add --codec_python /path/to/that/python. To check the codec on its own once the model is downloaded:

~/.gbx-codecs/bin/gbx-codecs check ~/.libra/cache/models/hub/models--GreenBitAI--Drama-3-8bit-paged/snapshots/*/

Use

curl http://127.0.0.1:11688/v1/audio/speech -H "Content-Type: application/json" -d '{
  "model": "GreenBitAI/Drama-3-8bit-paged",
  "input": "[sighs] Fine. If we miss the train, that is on you. [laughs] Okay, okay, I am going.",
  "voice": "narrator",
  "response_format": "wav"
}' -o out.wav

With the OpenAI SDK, streaming as it is generated:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11688/v1", api_key="local")
with client.audio.speech.with_streaming_response.create(
    model="GreenBitAI/Drama-3-8bit-paged",
    voice="narrator",
    input="<|speaker:0|> Are you coming tonight? <|speaker:1|> I wouldn't miss it.",
    response_format="pcm",                # 24 kHz, 16-bit, mono
    extra_body={"stream_format": "audio", "seed": 7},
) as response:
    for chunk in response.iter_bytes():
        ...                               # play or write it

field
input	the text. Inline cues such as [laughs], [sighs], [whispering]; speakers `<
voice	a name from --voices_dir, or default
response_format	wav (44.1 kHz, the default) or pcm (24 kHz, as OpenAI's). Other formats are refused
stream_format	audio to stream; absent, the whole file is returned
seed, temperature, top_p, sample_rate	extensions; from the OpenAI SDK, pass them in extra_body

speed other than 1.0 and instructions are not supported by this model: emotion and delivery come from the inline cues.

Pauses longer than a second are shortened in the output: the model occasionally goes silent mid-sentence for many seconds before finishing it.

Quality

32 texts in English and Chinese (plain, cued, two-speaker, and long enough to be split), transcribed by Whisper large-v3-turbo:

	English word error	Chinese character error
mlx-audio reference implementation, mlx-community weights (one take each)	0.0%	1.1-1.5%
this repository on gbx-lm (three takes each)	0.0-2.9%, mean 1.2%	1.1-1.9%, mean 1.5%
mlx-community weights on gbx-lm (three takes each)	0.0-0.4%, mean 0.2%	1.3-7.8%, mean 3.7%

Single takes occasionally drop the end of a line -- in both builds, 1-2 takes in 96 -- so retry with another seed when one does. With experts streamed from disk the output is the same as in memory up to rounding.

Speed

M3 Ultra, experts read from a cold SSD where they stream; the model makes 21.5 frames for each second of audio:

Mac	how	frames/s	x real time
64 GB and up	in memory	56-58	2.6-2.7
48 GB	streaming experts	15.5-17	0.7-0.8
32 GB	streaming experts	10-11.5	0.5

The smaller tiers are measured with the expert cache gbx-lm would give those Macs, on a Mac Studio's SSD; a laptop's is slower, so expect less. 24 GB Macs can load the model and run at about a third of real time. gbx-lm decides how to load the model from the memory it finds; GBX_PAGING=off forces it into memory.

License

The Fish Audio Community License, Version 1.0 applies to this model, including the outputs you make with it. In short, and the license itself governs:

• Free for research and non-commercial use, and for commercial use by organizations with annual revenue under USD 500,000. Above that, or to offer text-to-speech to third parties as a hosted service (an API or app whose purpose is speech generation from their text or voices), you need a separate license from Fish Audio (business@fish.audio).
• Do not generate speech imitating an identifiable person without their consent, impersonate anyone to deceive, or use the outputs to train other speech or foundation models.
• NOTICE records the attribution and every change made to Fish Audio's weights, including this repository's.

gbx-lm is Apache 2.0 and gbx-codecs is MIT. The codec implementation in gbx-codecs and the model implementation in gbx-lm are adapted from mlx-audio (MIT).

Credits

Fish Audio for Drama 3. cguldogan for the MLX 8-bit conversion published as mlx-community/fish-audio-drama-3-8bit. Prince Canuma and the mlx-audio contributors for the MLX implementation this build and its codec are based on.