Confirming the DAV encoder accepts real audio — and a local app built on Music 3
Posting this partly as a data point and partly because #16 ("Humming as Audio as INPUT") and #19 (the arbitrary-audio toolkit) are asking the same question I spent a while on.
The short version: the DAV encoder in this repo will take real audio and produce a latent the render stage accepts. I'd assumed for weeks it wouldn't, because the thing I kept reading was that Music 3 takes text and returns audio. That's true of the composition stage. It is not true of the render stage, and the two never seem to get separated out loud — so one accurate statement about the first gets applied to the second.
To be clear about credit: I'm not claiming to be first here. #19 is clearly further along on arbitrary-audio encoding than I am, and this is corroboration from a different direction rather than a discovery.
What I measured, on a 4070 Ti SUPER, in case the numbers are useful:
- Encoding a real recording and rendering from that latent works, and carries the performance rather than a text approximation of it.
- The useful control is the denoise level. At 0.60 the output is still recognisably a copy of the input. At 0.85 waveform correlation against the source has collapsed to about 0.004 — a genuinely new performance that keeps the feel. Past ~0.95 the reference stops steering it at all.
- ComfyUI won't do this step, so I run the encoder outside it. That was the part that took longest to work out.
The app, since it's the reason I went looking: a local desktop studio built on Music 3 — you write a style description and lyrics, it renders on your own machine. It also does cover art, stems, word-level timed lyrics, short video clips and a small editor. Apache-2.0, and it ships no weights: every model shows its size and licence before anything downloads, and the download goes to the [publisher.
https://senzube4n.github.io/AIPLAY-Studio/
The pipelines ship as plain API-format JSON in workflows/ and the measurement scripts in scripts/, so the numbers above are re-runnable rather than just asserted. Happy to be told I've got any of it wrong.
Thanks for putting the weights out.
