NightingaleCen/moonshine-tiny-zh-onnx

Moonshine Tiny ZH ONNX and tokenizer artifacts

This repository contains onnx models derived from Moonshine Tiny ZH.

The ONNX models provide KV-cache decoding with dynamic INT8 quantization, and have been optimized using onnxsim and ONNX Runtime.

moonshine_tiny_zh_decode.bin contains the decode-only tokenizer data in a compact binary format.

This is a derivative work of the original Moonshine Tiny model. It is distributed under the Moonshine AI Community License. See LICENSE.txt and NOTICE.

tokenizer_decode.bin format

All integer fields are fixed-width little-endian values. The file layout is:

Header
special_bitset[(vocab_size + 7) / 8]
byte_token_ids[256]       uint32
piece_offsets[vocab_size] uint32
piece_lengths[vocab_size] uint32
piece_blob                 UTF-8 bytes

piece_offsets[id] and piece_lengths[id] point into piece_blob.

The header is 40 bytes and contains, in order:

char[4] magic       "TKD1"
uint32 version      1
uint32 header_size  40
uint32 vocab_size   32768
uint32 bos_id       1
uint32 eos_id       2
uint32 pad_id       2
uint32 unk_id       0
uint32 special_bytes
uint32 blob_size

special_bitset uses one bit per token id to indicate whether the token is a special token. byte_token_ids[b] maps byte b (0x00–0xFF) to the corresponding token ID used by the tokenizer's ByteFallback mechanism. A decoder should build the inverse mapping once during initialization. 0xffffffff means that no mapping exists. 0xffffffff means that no mapping exists.

Decode conventions

• Replace U+2581 with one ASCII space.
• Convert <0xXX> tokens to their raw byte values.
• Remove one leading ASCII space after concatenation pieces.