EvoAwaken-Workshop/dots-tts-edit-gguf

dots.tts Edit - GGUF

This repository contains a GGUF conversion of the dots-studio/dots.tts.edit checkpoint. It supports instruction-controlled speech editing with a source audio file, including text replacement, insertion and deletion, emotion and prosody control, pauses, enhancement, and background-audio operations.

The files are intended for a runtime that implements the custom dotstts audio-generation pipeline, such as rust-model-inference. A generic text-only GGUF runner cannot execute the complete editing pipeline.

References

• Original checkpoint: dots-studio/dots.tts.edit
• Official implementation: studio-dots-ai/dots.tts
• GGUF exporter and Rust runtime: Liyulingyue/rust-model-inference
• Exporter source used for this conversion: tools/dots/convert_dots_tts.py
• Exporter contract tests: tools/dots/test_convert_dots_tts.py
• Rust dots.tts runtime: src/models/dots
• Technical report: dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model

Files

The checkpoint is split into an LLM file and an audio-pipeline file:

File	Precision	Contents
dots-tts-edit-BF16.gguf	Mostly BF16; normalization tensors are F32.	Qwen2-compatible language and latent-token model.
dots-tts-edit-mmproj-BF16.gguf	Mixed BF16/F32; counters are I64.	Speaker encoder, patch encoder, flow-matching DiT, latent statistics, and AudioVAE vocoder.

These files are not Q4 or Q8 quantizations. The BF16 suffix identifies the primary model-weight precision. The mmproj generation core uses BF16, while the speaker encoder and AudioVAE vocoder retain F32 weights.

The LLM file uses standard Qwen2 GGUF tensor names. The mmproj file uses general.architecture = clip metadata and the custom dotstts.* metadata required by the Rust runtime. Both files are required.

Export Command

Clone the exporter and download the original checkpoint:

git clone https://github.com/Liyulingyue/rust-model-inference.git
cd rust-model-inference

hf download dots-studio/dots.tts.edit \
  --local-dir models/dots.tts.edit

Install the exporter's only Python dependency and run the conversion:

python3 -m pip install numpy

python3 tools/dots/convert_dots_tts.py \
  models/dots.tts.edit \
  --variant edit \
  --out-dir models

The command creates:

models/dots-tts-edit-BF16.gguf
models/dots-tts-edit-mmproj-BF16.gguf

The exporter infers the edit variant when --variant is omitted and the source directory name contains edit. Existing outputs are not overwritten unless --overwrite is supplied:

python3 tools/dots/convert_dots_tts.py \
  models/dots.tts.edit \
  --variant edit \
  --out-dir models \
  --overwrite

Conversion Details

Source files

The exporter reads the following files from the original checkpoint:

dots.tts.edit/
|-- model.safetensors
|-- speaker_encoder.safetensors
|-- vocoder.safetensors
|-- llm_config.json
|-- config.json
|-- tokenizer_config.json
|-- vocab.json
|-- added_tokens.json
|-- merges.txt
`-- latent_stats.pt

PyTorch and the safetensors Python package are not required. The exporter reads Safetensors headers and payloads directly, uses Python standard-library parsers for configuration and tokenizer files, and uses NumPy only to decode the array values stored in latent_stats.pt.

Before conversion, required files, source tensor dtypes, and tensor shapes are validated against llm_config.json and config.json.

LLM GGUF

The LLM output is GGUF v3 with Qwen2 architecture metadata:

general.architecture         = qwen2
general.name                 = dots.tts-edit
general.file_type            = 32
general.quantization_version = 2

The tokenizer vocabulary, added control tokens, merge rules, BOS/EOS IDs, context length, embedding size, feed-forward size, attention heads, RMS norm epsilon, and RoPE settings are written into GGUF metadata.

The main tensor mapping is:

Source tensor	GGUF tensor
llm.model.embed_tokens.weight	token_embd.weight and output.weight
llm.model.norm.weight	output_norm.weight
llm.model.layers.{i}.input_layernorm.weight	blk.{i}.attn_norm.weight
llm.model.layers.{i}.post_attention_layernorm.weight	blk.{i}.ffn_norm.weight
llm.model.layers.{i}.self_attn.q_proj.*	blk.{i}.attn_q.*
llm.model.layers.{i}.self_attn.k_proj.*	blk.{i}.attn_k.*
llm.model.layers.{i}.self_attn.v_proj.*	blk.{i}.attn_v.*
llm.model.layers.{i}.self_attn.o_proj.weight	blk.{i}.attn_output.weight
llm.model.layers.{i}.mlp.gate_proj.weight	blk.{i}.ffn_gate.weight
llm.model.layers.{i}.mlp.up_proj.weight	blk.{i}.ffn_up.weight
llm.model.layers.{i}.mlp.down_proj.weight	blk.{i}.ffn_down.weight

Most LLM tensors remain BF16. Normalization weights are converted from BF16 to F32 for the runtime contract. This is a storage conversion, not a quantization pass.

Audio mmproj GGUF

The mmproj output is GGUF v3 with an audio-capable CLIP-style container:

general.architecture           = clip
clip.has_vision_encoder        = false
clip.has_audio_encoder         = true
clip.has_gen_audio_encoder     = true
clip.audio.projector_type      = dotstts_spkenc
clip.gen.audio.projector_type  = dotstts_gen

It contains:

1. Latent mean and variance tensors from latent_stats.pt.
2. Projection heads connecting LLM, latent, speaker, and EOS states.
3. The patch encoder and its transformer blocks.
4. The flow-matching DiT and its time, attention, feed-forward, AdaLN, and output layers.
5. The CAM++-style speaker encoder and resampling kernel.
6. The AudioVAE encoder and vocoder used to encode source audio and decode the generated latent sequence.

Core model tensors remain BF16. Speaker and vocoder tensors remain F32. Required integer tensors, including batch-normalization counters, remain I64.

Tensor layout and derived weights

Source dimensions are reversed when written because the checkpoint follows PyTorch dimension order while GGUF uses the runtime tensor order.

Legacy vocoder weight normalization is folded before writing:

weight = (weight_g / norm(weight_v_row)) * weight_v_row

Fixed resampling and filter tensors are included in the mmproj payload, so the runtime does not need the original Safetensors files.

Output validation

Each output is first written to a temporary file. Before publishing it, the exporter reads the complete file back and validates:

• metadata values;
• tensor names, GGML types, dimensions, and byte lengths;
• the raw byte payload of every tensor.

Duplicate metadata keys, duplicate tensor names, invalid dimensions, unexpected dtypes, and configuration shape mismatches are rejected.

Run the focused exporter tests with:

cd tools/dots
python3 -m unittest -v test_convert_dots_tts.py

Speech Editing With rust-model-inference

Build the runtime:

cargo build --release

Run a text substitution edit:

cargo run --release --bin rust-model-inference -- \
  --model dots-tts-edit-BF16.gguf \
  --mmproj dots-tts-edit-mmproj-BF16.gguf \
  --tts \
  --edit \
  --source-audio source.wav \
  --instruction 'Hello <sub targ="small">brave</sub> world.' \
  --use-xvector auto \
  --out edited.wav

--source-audio, a non-empty --instruction, and --out are required. --source-text and --target-text can override transcripts derived from the instruction:

cargo run --release --bin rust-model-inference -- \
  --model dots-tts-edit-BF16.gguf \
  --mmproj dots-tts-edit-mmproj-BF16.gguf \
  --tts \
  --edit \
  --source-audio source.wav \
  --instruction '<sub targ="A local bank">Washington Mutual</sub>' \
  --source-text 'Washington Mutual' \
  --target-text 'A local bank' \
  --use-xvector auto \
  --seed 42 \
  --out edited.wav

The current runtime accepts the following structural tags:

Tag	Purpose
<del>...</del>	Delete speech.
<ins>...</ins>	Insert speech.
<sub targ="replacement">...</sub>	Replace speech. The targ value must be quoted.
<emo>...</emo>	Local emotion control.
<pitch>...</pitch>	Pitch control.
<rate>...</rate>	Speaking-rate control.
<enhance>...</enhance>	Speech enhancement.
<bg>...</bg>	Background-audio operation.
<pause/>	Pause operation.
<spk_transfer/>	Speaker-transfer operation.

--use-xvector accepts auto, on, or off. In auto mode, speaker guidance is disabled only when every operation is emo, bg, or enhance; it remains enabled for text, pitch, rate, pause, speaker-transfer, and mixed edits.

Useful generation controls include:

Option	Meaning
--max-tokens	Maximum number of generated target latent patches.
--temp	Sampling temperature.
--steps	Flow-matching Euler steps.
--seed	Reproducible random seed.
--threads	CPU worker thread count.

The generated file is mono PCM16 audio at 48 kHz.

Limitations and Responsible Use

• The two GGUF files must be used together.
• These files require the custom dots.tts edit pipeline and are not compatible with generic text-only GGUF inference.
• No Q4 or Q8 version is included in this repository.
• Generated or edited speech may contain pronunciation, timing, speaker-similarity, or instruction-following errors.
• Do not use speech editing to impersonate people, mislead listeners, violate consent, or create unlawful content. Clearly disclose synthetic or edited audio where appropriate.
• Quality, speed, and memory use depend on the runtime, CPU architecture, source audio, instruction, and generation settings.

Provenance and License

The source checkpoint is dots-studio/dots.tts.edit. This repository preserves the original checkpoint's license: other metadata. Review the original model card and the studio-dots-ai/dots.tts repository for the terms that apply to the model weights and code.

This model card documents the GGUF conversion and does not replace the terms of the original checkpoint. Users are responsible for complying with the applicable license, consent requirements, and local laws.