JimHue/TASTE-S-Tokenizer

TASTE-S tokenizer

A speech tokenizer: it turns audio into a short sequence of discrete tokens, and turns those tokens back into audio. This repository contains everything needed to run it.

Install

# git-xet handles the large weight files: https://hf.co/docs/hub/git-xet
git xet install
git clone https://huggingface.co/JimHue/TASTE-S-Tokenizer
cd TASTE-S-Tokenizer

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

A GPU with ~16 GB of memory is recommended. CPU works but is slow.

Check that it works

Three sample utterances ship with reference outputs:

python inference.py --wav "samples/audio/*.flac" --out_dir samples/my_output
python samples/check.py samples/my_output

This should print OK -- all 3 samples match the reference. See samples/README.md for what "match" means (text tokens are exact; taste tokens are compared as an agreement rate, because a small number of codes shift between hardware).

Run

python inference.py --wav your_audio.wav --out_dir out/

Multiple files and globs are accepted:

python inference.py --wav "my_audio/*.wav" --out_dir out/

For each input NAME.wav you get:

File	Contents
out/NAME_recon.wav	the audio resynthesized from the tokens (24 kHz)
out/NAME_tokens.json	taste_tokens, text_token_ids, speech_unit_ids

taste_tokens is the tokenizer's output: one token per text token, each token being 16 integers. text_token_ids is the transcription the model produced on its own — you do not need to supply a transcript. speech_unit_ids are the intermediate acoustic units the vocoder consumes.

Console output looks like:

[load] 2750M params on cuda in 17s
[your_audio] 4.91s -> 18 taste tokens x 16 codebooks, 120 units, 4.80s audio  (encode 1.4s, decode 4.9s)

Options

Flag	Default	Meaning
--wav	—	input audio file(s) or glob; any format soundfile/librosa reads
--out_dir	./taste_s_out	where to write results
--device	cuda if available	e.g. cuda:0, cpu
--seed	42	sampling seed; the same seed reproduces the same output
--top_k	25	sampling cutoff
--top_p	0.8	sampling cutoff
--temperature	1.0	sampling temperature
--tokenizer_dir	none	optional text tokenizer, only to print the transcript as readable text
--model_dir	this directory	where config.json and the weight shards live
--campplus	./campplus.onnx	speaker-embedding model, used to match the input voice

Audio longer than 30 s is truncated to the first 30 s.

Loading the model in your own code

from transformers import AutoModel

model = AutoModel.from_pretrained("JimHue/TASTE-S-Tokenizer", trust_remote_code=True)

inference.py shows how the encoder and decoder are driven end to end; its Encoder and UnitDecoder classes are small and meant to be copied into your own scripts.

Files

inference.py	the entry point — the only file you need to read
configuration_taste_s.py, modeling_taste_s.py	the model
third_party_cosyvoice_*.py	vendored upstream code (see Attribution); not specific to this model
samples/	test audio and reference outputs

Notes

• On the same hardware, a given --seed reproduces outputs exactly. Across different hardware, text tokens and taste tokens are stable but generated units are not — see samples/README.md.
• The vocoder is conditioned on a speaker embedding extracted from the input audio, so the resynthesized audio keeps the original voice.

Attribution

Parts of this repository are derived from other open-source projects and remain under their original licenses. third_party_cosyvoice_conformer.py and third_party_cosyvoice_vocoder.py hold upstream code from CosyVoice2 (Apache-2.0) and Matcha-TTS (MIT), concatenated into one file each. The speech encoder architecture derives from Whisper (MIT). campplus.onnx is the speaker-embedding model distributed with CosyVoice2. The sample audio is from LibriSpeech (CC BY 4.0) — see samples/README.md.