rec7 — an audio encoder for MiniMax-Music3
MiniMax-Music3 ships a language model that emits music codes and hidden states, a depth decoder, a condition encoder and a flow-matching renderer — but not the audio→codes quantizer. Nothing in the release lets you put a real recording into the model.
rec7 is a substitute for that missing piece. It reads a recording and produces, per 40 ms frame, the 4096-d hidden state the language model would have produced for that frame. From those states everything else follows deterministically using the released components: the 8 code streams, and the full condition the renderer expects.
What that enables, today:
- Covers. Real recording in → M3 renders it back. Timbre, guitar articulation and performance come through; the language model is skipped entirely. On acoustic singer-songwriter material the result is close to indistinguishable from the pipeline's own output quality.
- Supervision signal. Hidden states read from real audio are the one thing the released M3 never had. They are usable as training targets for anything that consumes those states.
- Reading codes off real audio (with an important caveat below).
Quick start
pip install -r requirements.txt
# weights/trunk.pt and weights/head.pt come with this repo
# you also need the MiniMax-Music3 checkpoint directory (not included)
# encode: states + codes for a recording
python rec7_encode.py song.mp3 --m3 /path/to/minimax_music3
# cover: render the recording back through M3
python rec7_render.py song.mp3 --m3 /path/to/minimax_music3 \
--lyrics lyrics.txt --seconds 60 --out cover.flac
dav_loader.py (a thin loader for the Flow-VAE in dav.pth) ships
alongside. The 8B language model is never loaded — the two 16k-row tables
it needs are sliced straight out of the safetensors.
Memory: encoding needs 3 GB. Covers need the full renderer (10 GB in
bf16).
Using the outputs
rec7_encode.py writes a .pt with h [T, 4096] (float16), codes
[T, 8] (int16: c0 is 16384-way, c1–c7 are 1024-way), and optionally
cond [T, 32768] — the renderer condition — and z, the Flow-VAE
latents. 25 frames per second.
To drive the renderer yourself, rec7_model.states_to_streams() turns
states into the condition, and rec7_render.render_with_condition()
injects it into the pipeline (it patches the semantic generation step to
hand over the states instead of running the LM).
If you build on the states, the frame→latent geometry matters:
rec7_model.frame_latent_starts() is the exact piecewise mapping the
renderer uses (200-frame windows on a 100-frame hop, stitched at a
345-latent hop). A constant ratio is wrong by up to a frame and it shows.
How it was made
- Architecture: conv stem + dilated residual blocks on the Flow-VAE latents (86 Hz), exact-span mean-pool to 25 Hz, 8-layer transformer (d=1088), a 2-layer head to 4096. ~170M parameters.
- Bootstrapped by self-distillation: ~56k tracks generated by M3 itself, storing codes, latents and audio, with the LM's true hidden states dumped by teacher-forcing.
- Then trained on non-synthetic audio (~22k files) with a reconstruction objective: states → released depth chain → renderer denoising step, judged against the recording's own latents. Anchored throughout by supervised code prediction on the synthetic pool so the code head doesn't drift.
Measured performance
| value | |
|---|---|
| held-out code accuracy (c0, synthetic) | 55.7% top-1 |
| state agreement vs. true LM states (synthetic) | 0.88 cosine |
| cover of a dense rock master: content tracking | 0.95 |
| cover: spectral clarity vs. original | 90% |
| cover: transient sharpness vs. original | 127% (over-sharp) |
| cover: high band (5–12 kHz) energy vs. original | 81–84% |
The Flow-VAE round-trip itself preserves ~64% of 5–12 kHz energy on the same material, so the encoder slightly overshoots the high band and adds some grain — renders read a touch bright and hyped rather than dull. On sparse acoustic recordings this is much less audible than the numbers suggest.
Limitations — read these
Its codes are not plausible language-model sequences. Teacher-forced through the base LM, rec7's code sequences score ~6.2 nats per frame, against ~2.1 for the model's own samples and ~3.2–3.5 for an encoder trained to regress the LM's states directly. The states render beautifully — the renderer doesn't care about sequence plausibility — but do not use rec7's codes as training targets for the language model. Two fine-tunes trained that way collapsed into droning above modest adapter strength. Use the states; get codes elsewhere.
Timbre does not transfer through code-level training. Fine-tuning the LM (LoRA or full) on codes teaches composition, phrasing and articulation and leaves the voice generic. The voice lives in the hidden states, which code-level losses never touch.
Trained on 44.1 kHz stereo, 40 ms frames. Mono is duplicated. Very short inputs (< ~6 s) are rejected.
Training
See rec7_train.py for training script.
What we found the states are good for
The LM's own hidden states and rec7's states for the same song are related by an almost linear transform — a fitted 4096×4096 map plus offset lifts held-out agreement from 0.28 to 0.98, and a constant offset alone gets 0.86. Rendering the LM's teacher-forced states through that map produces the recording's singer, where the raw states produce a generic one. That is a route to voice identity without touching the language model; details and tooling to follow.
Files
weights/trunk.pt the encoder (EMA weights)
weights/head.pt the state head
rec7_model.py architecture, geometry, loading, inference helpers
rec7_encode.py audio -> states/codes
rec7_render.py audio -> M3 cover
dav_loader.py Flow-VAE loader