CSM-1B-ONNX-WebGPU

sesame/csm-1b split into ONNX graphs for running in a web browser with ONNX Runtime Web's WebGPU backend, as used by Quick Inference.

Changes from the original

• The backbone (16-layer Llama) and the depth decoder (4-layer Llama) were converted with ONNX Runtime GenAI's model builder (-p int4 -e webgpu, k-quant, block size 32, inputs_embeds in, float16 activations and KV cache). The backbone's head is the codebook-0 head; the depth decoder's head is its 31 per-codebook heads stacked (codebook j: columns (j-1)2051 .. j2051-1). The backbone also outputs its final hidden state.
• The text and audio embedding tables, and the depth decoder's input projection, are separate small graphs (onnx/text_embed_q4.onnx, onnx/audio_embed_q4.onnx, onnx/depth_project_q4.onnx; 4-bit GatherBlockQuantized / MatMulNBits). audio_embed also returns the rows through the input projection, for the depth decoder.
• The Mimi decoder (onnx/mimi_decoder_q4.onnx, codes [1, 32, frames] -> 24 kHz audio) was exported from the checkpoint's codec with a dynamic number of frames and quantized without calibration (4-bit MatMuls, int4 convolution weights).
• tokenizer.json, tokenizer_config.json, special_tokens_map.json, chat_template.jinja, config.json and generation_config.json are the original files.

These are modified files: their outputs are close to, but not identical to, the original model's. On eight test sentences (one seed, sampling as in generation_config.json), Whisper small.en heard the quantized model with the same word error rate as the original (11.4%).

Inference loop

Prompt <|begin_of_text|>[0]{text}<|end_of_text|> -> backbone prefill on its text embeddings. Per frame: sample codebook 0 from the backbone (temperature 0.9, top-k 50); run the depth decoder from a fresh cache on [depth_project(hidden), depth_embeds(code0)], then one step per codebook (depth_embeds(j*2051 + code_j)), sampling codebooks 1..31 from heads 1..31; feed the sum of the frame's 32 embeds(k*2051 + code_k) back to the backbone. Stop at a frame whose codebooks 0..30 are all zero; decode the frames with the Mimi decoder.

The default voice is the model's own for speaker 0, with no reference audio (no recording is included), so it can differ from one generation to the next.

Licence

CSM-1B is licensed under the Apache License 2.0 (LICENSE). The Mimi codec weights inside it are Kyutai's, licensed under CC BY 4.0 (LICENSE-Mimi); credit: Kyutai, Mimi (https://huggingface.co/kyutai/mimi), modified (exported and quantized). Sesame's model card prohibits impersonation without consent, misleading content and other harmful use.

Files

• onnx/backbone/ and onnx/depth_decoder/: model.onnx, model.onnx.data, genai_config.json
• onnx/text_embed_q4.onnx, onnx/audio_embed_q4.onnx, onnx/depth_project_q4.onnx, onnx/mimi_decoder_q4.onnx
• tokenizer.json, tokenizer_config.json, special_tokens_map.json, chat_template.jinja, config.json, generation_config.json
• LICENSE, LICENSE-Mimi