ManyOtherFunctions/Yue2-WannabeTokenizer

What This Is:

Just a humble wannabe semantic tokenizer to make the semantic tokens to train the Yue2 music model with

Q/A:

Q: But.. wasn't there already another one?

A: Yes. But the dataset used to train it wasn't very robust. Many song segments devolved into noise.

Q: How good is this one then? Can it tokenize any song then?

A: More or less

Q: .. what do you mean more or less?

A: Yknow how if you don't have the real Nine Yin Manual, you have to practice the fake Nine Yin Manual? This is the fake Nine Yin Manual. It gets alot more right. It gets the overall song right. There are some slight hiccups with consonants occasionally. But it is our hope that that is something the extra text conditioning on the NAR diffusion stage will change.

Q: What about the joint nar lora or all that from mothersuperior, do I use th-

A: Thou shalt have no other tokenizers before me (hahahaa) ... But no, don't do that. This tokenizer will sound mostly right with the base model NAR stage. WILD, amirite? But that's how you make sure the AR model won't shit the bed trying to learn your token sequences that may or may not describe production grade music. You don't HAVE to train on production music. Sometimes you just want an amateur grade school orchestra to troll with. But yeah no we don't need no stinkin NAR lora. It's counterproductive. How would you know if Yue2 can even learn it or not if you can't even hear what the NAR stage normally sounds like? NAR loras are for the final polish stage.

How to actually train Yue2:

The first step is obviously build a good foundation. The official Yu2 tokenizer would be that good foundation. But if you don't have the real Nine Yin, practice the fake Nine Yin.

Next is the AR or autoregressive LLM part of Yue2. This is why having accurate semantic tokens matters so much. The tokens carry ALL the meaning. What sounds to say to make up lyrics. What those vocals sound like. That part takes your lyrics, your text prompt, and most importantly, sheet music melody, and spits out these semantic tokens here These semantic tokens therefore have to be correct

SO! It means you have to use this tokenizer and listen to the reconstructed audio And if it fails to remake your song, then you can't train it. Simply that. Try to train and you'll destroy the model and give it brain damage. Most of those people who look at us like we're crazy for doin all this extra shit, well guess what? They like music the model was already good at and the tokenizer built on limited data already knew

But what if you wanted to make a chinese version of "I Want It That Way" What if you wanted it to be Dutch? (yes this model knows Dutch. Crazy, I know.) What if you wanted backing vocals? What if you didnt wanna sound like AI slop, but wanted to actually fool people into thinking a cracked band made an awesome banger? With the old tokenizer you'd be shit out of luck. With this tokenizer... you have a better chance. No guarantees.

Lastly we come to the NAR, the diffusion stage! That stage turns these tokens here, which describe exactly what instruments, what they sound like, what the vocals sound like exactly, that stage turns these tokens into audio basically. (I mean not really, but as far as alot of you are concerned, that's good enough of an understanding) The NAR / diffusion stage takes these IDs and it also takes the text prompt.

HOWEVER!!! We tried throwing in all kindsa trash and garbage instructions into the captions for the NAR stage IT DID NOT CHANGE THE SONG AT ALL. the NAR ONLY LISTENS TO THESE SEMANTIC TOKENS. The only thing that might have changed a little bit was MAYBE, JUST MAYBE, supplying the right lyrics improved pronunciation. Maybe. Or I could be trippin balls and coping hard.

So which part of Yue2 Do I train?!!:

If you want a little extra polish on your sound, train the NAR stage.

Most of you want more than that. You want your own singers. You want a sick arrangement without having to prompt for one. You want it to make banger melodies so you don't have to compose songs by hand.

That's all the AR, but in two stages. ONE of these stages is unaffected by a dogshit wannabe semantic tokenizer.

If you only need it to make good melodies, and nothing else, YOU ARE IN LUCK! Freeze the losses on everything BUT the sheetABC part of the AR stage and just train based on melodies. It will at least make banger melodies.

If you wanna train everything else, that's where this tokenizer comes into play. You wanna train the melodies first. Then on the main AR stage, freeze the losses on the ABC section and only compute losses on the tokens. (Ask your friendly AI assistant for help with that) Now when it comes to the generating semantic tokens stage You caption your song and everything, maybe even supply a corrected sheetsage ABC melody file. And then you supply these semantic tokens.

The AR has to output the tokens that make the audio. This tokenizer is a sanity check for that. It will generate and reconstruct. If you don't hear something that sounds enough like what you want, you're not gonna get what you want with training (And in fact I can't guarantee this won't blow up the AR after alotta steps either, but it should help somewhat)

WannabeTokenizer inference

A self-contained, offline inference package for:

1. converting an audio file into YuE2 semantic token IDs at 25 Hz; and
2. reconstructing audio from those IDs with YuE2's NAR and VAE decoder.

The unified tokenizer checkpoint includes both its MERT2 encoder and cosine quantizer. The YuE2 checkpoint is supplied separately. No Hub downloads, remote model code, training code, dataset tooling, or pickle checkpoints are used.

Requirements

• Python 3.10+
• PyTorch 2.5+ and a matching torchaudio build
• A CUDA GPU is strongly recommended for YuE2 reconstruction
• The bf16 YuE2 all-in-one checkpoint (yue2_3b_bf16.safetensors)

Install the Python dependencies in your environment:

python -m pip install -r requirements.txt

Run

From this folder:

python run_tokenizer.py /path/to/input.wav --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output

Outputs:

• output/tokens.npy — one-dimensional int32 NumPy array, IDs in 0..32767
• output/reconstructed.wav — 48 kHz stereo PCM-24 WAV

Existing outputs are protected unless --overwrite is supplied. Common input formats supported by libsndfile or torchaudio include WAV, FLAC, OGG, MP3, and M4A.

Optional settings:

python run_tokenizer.py input.flac --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output --device cuda:0 --seed 1 --steps 32

Use python run_tokenizer.py --help for all options. If installed as a package, the equivalent entry point is wannabe-tokenize.

Behavior and resource use

Reconstruction is generative and seed-dependent, not a lossless inversion of the recording. It uses YuE2's base NAR with an empty tags/lyrics prompt. Different seeds produce slightly different audio from the same tokens.

The tokenizer is released before YuE2 loads so the encoder and renderer do not occupy accelerator memory at the same time. The loader skips YuE2's unused language-model head and VAE encoder. VAE decode tiles automatically shrink after an out-of-memory error. Longer inputs require more memory and time.

Python API

from wannabe_tokenizer import WannabeTokenizer, YuE2Renderer
from wannabe_tokenizer.audio import load_audio, save_wav_atomic

waveform = load_audio("input.wav")
tokenizer = WannabeTokenizer("checkpoints/WannabeTokenizer.safetensors", device="cuda")
tokens = tokenizer.tokenize(waveform)

del tokenizer  # release encoder memory before loading YuE2 in a real application
renderer = YuE2Renderer("/path/to/yue2_3b_bf16.safetensors", device="cuda")
reconstructed = renderer.reconstruct(tokens, seed=0, steps=32)
save_wav_atomic("reconstructed.wav", reconstructed)