cdismus4/voivent-utakoe

Voivent Utakoe — melody-conditioned accompaniment model

Browser demo (no install)

• Official Web Studio (PC & Mobile): https://daw.voivent.com/lab/utakoe

Utakoe （うたごえ） is a small symbolic music model for finding what a sung melody wants to become. You sing (or play) a melody first, and the model rolls many takes of an accompaniment for an 8-bar, 4/4 phrase — a chord progression plus a counter-melody — as raw material to search through, keep, and recombine. The counter-melody is the point: a partner line that sings back against your melody. The chords are not the point; they are a protective floor underneath it — in key, never clashing with long melody notes, and returning home at phrase ends — so that wild rhythms and lines have something steady to stand on.

• Input: one monophonic melody (8 bars, 4/4, 16th-note grid), normalized to C major / A minor.
• Output: one chord token per bar (12 roots × 10 qualities) and a counter-melody line, as symbolic tokens.
• Size: 5.11M parameters. The int8 ONNX build is 5.4 MB and runs entirely in the browser (see demo/) or anywhere ONNX Runtime runs.

Status

This release is a beta. The model in this release is the second generation of Utakoe; the first generation was an internal prototype and was never publicly released. It is a search tool, not a finished accompaniment: it proposes many takes, and the user picks, locks and recombines the good bars. Choosing, locking and recombining happen outside the model, in the demo workflow — the weights only propose.

What it is good for today: rolling takes from a sung melody and finding a counter-melody that fits it. What it is not: an accurate harmonizer. Chord accuracy against the reference scores is modest by design of the measurement (see Evaluation), and functional cadences are not guaranteed — dominant-function chords are used as color and do not reliably resolve (0 of 10 cases in a theory audit of 2 real sung takes). The honest numbers, including the gates we did not meet, are in Evaluation and Known limitations below, not hidden. A future release targeting functional harmony and robustness to low-register, ornamented singing is in design; this card will be updated when the full release lands.

Quick start

Open demo/index.html from a static host. It loads the int8 model lazily on first use, extracts pitch from a sung take with SwiftF0, estimates the bar grid, and generates an accompaniment in 8-bar chunks. Everything runs locally in the browser; no audio leaves the device. (The bundled demo/assets/sample.wav is a synthetic voice-like test signal rendered for development, not a real recording.)

ONNX inference

Files:

File	Contents
model/utakoe_v2_int8.onnx	int8 dynamic-quantized model with KV cache, 5,408,994 bytes — the shipping build
model/utakoe_v2.onnx	fp32 model with KV cache, 20,673,942 bytes
model/vocab_v3.json	token vocabulary (208 tokens)
model/v2_final.pt	fp32 PyTorch checkpoint (20,457,039 bytes) — the trained original from which both ONNX builds above were exported

Graph interface (both builds):

• inputs: input_ids int64 [1, S] (the full prefix on the first call, then one token per step), position_offset int64 [1] (0 for the prefill; afterwards the 0-based index of the token being fed), past float32 [12, 1, 8, S_past, 32] (zeros with S_past = 0 for the first call)
• outputs: logits [1, S, 208], present (same shape as past with the extended length) — feed present back as past.

smoke_test.py in this repo is a minimal, self-contained reference: it loads only model/utakoe_v2_int8.onnx from this package, encodes a sample melody, greedily generates with the product guards, and prints the resulting chord progression.

Token format (vocabulary v3, 208 tokens)

Range	Meaning
0	PAD
1	BOS
2	EOS
3	BAR
4–19	POS: position inside the bar (0–15 sixteenths)
20–140	CHORD: 20 + root_pc * 10 + quality (121 values incl. "none"; qualities: maj, min, dim, aug, dom7, maj7, min7, hdim7, dim7, other)
141–189	PITCH: MIDI 36–84
190–205	DUR: duration 1–16 sixteenths
206	COND_START
207	TGT_START

A sequence is BOS COND_START <melody bars> TGT_START <target bars> EOS. Each bar is BAR, then in the condition a melody (POS/PITCH/DUR events), and in the target a CHORD token followed by counter-melody events. Melodies are transposed so the key is C major or A minor before encoding; renderers transpose back.

Reference generation uses sampling (temperature 1.0, top-k 40) with musical guards: EOS is banned until at least 4 bars are complete, a generated event identical to a condition event in a conflicting register is re-rolled, over-long runs of one token are cut, and pitch tokens must be followed by a duration token.

Model

Decoder-only Transformer: d_model 256, 6 layers, 8 heads, context 1024, dropout 0.1; 5,107,712 parameters. This model differs from the first-generation model only in the chord vocabulary: the first generation collapsed every 7th-type chord into a catch-all other (49 chord tokens); this model predicts 121 chord tokens directly.

Training data

9,097 songs → 42,322 training sequences / 10,201,648 tokens (8-bar windows, stride 4, at most 3 windows per song; transposition augmentation ±1–6 semitones; "sung-style" timing/pitch wobble with p = 0.4). Only public-domain / CC material was used:

Source	License	Songs used
PDMX (Zenodo DOI 10.5281/zenodo.15571083)	CC BY 4.0 (dataset)	8,358 — only rows flagged no_license_conflict; license-conflicting rows were strictly excluded
OpenScore Lieder	CC0 1.0	431
JSB Chorales	public-domain compositions	308

Data-side attributions live in NOTICE and are separate from the model license: the model and code in this repository are Apache-2.0.

Training

AdamW (lr 3e-4, warmup 500, cosine decay), batch 32, 26,000 steps, seed 777; loss on the target section only. Final train loss 0.584; full teacher-forced validation loss 0.926 (first generation: 0.973). Environment: Python 3.12, PyTorch 2.15.0 (training), ONNX Runtime 1.31.1 (export parity checks and browser runtime).

Evaluation

What these numbers do not measure. Every metric in the table below is about chords. Chord-token accuracy is agreement with the first chord written in each bar of the source score — it says nothing about taste, and nothing about the counter-melody. The fit between counter-melody and main melody — how well the two lines sit together, bar by bar — is the quality this model is actually judged by in use, and this release has no number for it yet: the first measure of that fit (a per-bar clash / density / seam check used to pick and lock good bars) was built after this release, as a separate, still uncalibrated tool. Read the chord numbers as a floor check — in key, no clashes, home at phrase ends — not as the score of the music.

Measured on melody-matched pairs between the first-generation and second-generation validation sets (identical melodies for both models), sampling temperature 1.0 / top-k 40 unless noted. The first-generation model has the same architecture with the collapsed 49-chord vocabulary.

Metric	Generation 1	This model	Note
7th-family chord rate	collapsed into "other"	48.43%	training distribution: 47.97%
"other" chord rate	75.64%	16.34%	the first-generation bias this release set out to fix
Early EOS (< 8 bars), raw	4.69%	1.56%	0.00% for both with the min-bars guard
In-key chord-root rate (n = 256 pairs)	94.05%	94.60%	absolute floor 80% also met
Generation diversity (8 conditions × 8 seeds)	1.000	1.000	fraction of distinct progressions
Validation loss (teacher-forced, full sets)	0.973	0.926

Reported honestly — a gate we did not meet. The project required this model's teacher-forced chord-token accuracy to be ≥ the first-generation model's on the strict ruler (exact root+quality match). This model scores 0.3743 vs the first generation's 0.3889, so that gate is recorded as not met. The comparison is structurally uneven — this model chooses among 121 chords, the first generation among 49 — and on the first generation's collapsed 4-quality mapping this model scores 0.4080 vs 0.3889; root-only accuracy is 0.4983. Per-quality exact accuracy is uneven: maj 0.64, min 0.47, maj7 0.32, dom7 0.29, min7 0.24, "other" 0.22, and ≤ 0.06 for the rare dim/aug/hdim7/dim7 classes. We publish the model anyway because the bias fix (above) and the loss improvement are real, and the strict-accuracy gap is documented here rather than hidden.

A variant we rejected. We also trained a feature-bridge variant conditioned on per-bar F0 statistics of the sung input. In an A/B contrast test the conditioning turned out to be effectively ignored (25 of 32 generations were token-identical when the features were swapped for a very different melody's), because in training the features were a deterministic function of the melody itself. That variant is not included in this release.

ONNX parity: greedy decoding of the exported graphs matches PyTorch for 60/60 tokens, for both the fp32 and int8 builds.

Known limitations

• Do not generate much beyond 8 bars in one run. The model was trained on 8-bar windows; in free-running continuation the counter-melody's note density grows explosively from around bar 10. Products should generate in 8-bar chunks (the bundled demo does, and additionally caps each chunk's target at the condition's bar count).
• 4/4 only. Other meters were filtered out of the training data.
• Keys are handled by normalizing to C major / A minor; modulation inside a phrase is not modelled.
• One chord per bar (the bar's first chord in the reference data); no inversions or voice-leading control. Chord rendering uses fixed interval templates; the catch-all other quality renders with a dominant-7th template. (The other entry in model/vocab_v3.json describes a different layer — the data side: when the training data was built, source chords whose pitch-class sets matched sus4 / sus2 / maj6 / min6 / power templates were folded into the other token. Rendering does not reconstruct those original voicings.)
• Monophonic input only; pitch range MIDI 36–84; durations are quantized to the 16th-note grid (max 16 steps).
• Style bias: the corpus is classical/folk public-domain material. Expect that idiom; pop/rock/jazz harmony is out of distribution.
• Tempo in the demo comes from an externally supplied grid when available (e.g. a click/DAW tempo); the built-in estimate from note onsets is a fallback and can be off on rubato singing.
• Harmony is locally fit, not functional. A music-theory audit of generated output found that chord choice tracks the melody of the moment rather than functional harmony: dominant-function chords are used as color and are not guaranteed to resolve, and phrases have no designed cadences — a take can end on a non-tonic chord even when the melody itself ends on the tonic. Expect moment-to-moment vertical fit, not textbook progressions.
• Quantization caps harmonization on difficult inputs. With low-register or heavily ornamented singing, pitch extraction and quantization produce many chromatic notes (in audited takes, about a third of melody duration fell outside the detected scale), which limits how diatonic any harmonization of that input can be. Cleaner, mid-register singing harmonizes best.
• Reference-pipeline theory filter. The bundled demo and the take-processing chain apply a conservative post-generation filter with two rules: (1) a clash guard — if a long melody note (half a bar or longer) semitone-clashes with the bar's chord, the chord is substituted when a vocabulary chord removes the clash, adds no new counter-melody clash, and does not lower the bar's melody coverage; (2) a final-bar tonic pull — when the melody ends on the tonic but the final chord is not tonic-family, a fitting tonic chord is substituted under the same conditions. Bars with no qualifying substitute are left unchanged. The filter does not add cadences or dominant resolutions elsewhere.

Rights in generated output

The model and code are Apache-2.0. Musical output generated with this model is yours to use, including commercially, as far as the model itself is concerned; as with any generative tool, you are responsible for checking that a specific output does not reproduce a specific copyrighted work. Training used only public-domain / CC0 / CC BY material (see NOTICE).

Samples

samples/ contains, for 6 validation melodies and 4 real sung takes: the generated MIDI plus WAV renders in two timbres — piano (FluidSynth + MuseScore_General, MIT) and an additive synth. The sung takes are the project's own recordings; only the generated accompaniment is included — no raw voice audio is in this package. See samples/README.md for the generation conditions of every sample and a chord-progression table for all of them.

Citation / provenance

Voivent Utakoe, a second-generation model developed for the Voivent DAW project (Voivent Satellite). Finalization program and evaluation reports were produced alongside the model; the headline numbers above are the ones recorded in those reports, including the ones that did not meet their gates.