JacobLinCool/urfm-vega-preview

URFM Vega Preview

URFM Vega Preview writes playable rhythm game charts from a song's audio, for Taiko and the osu! modes. Tempo and bar lines come from the audio, a song planner lays out every section, and a game-conditioned decoder writes the notes. An experimental tool also writes music for an existing chart. It is a research preview of the Universal Rhythm Foundation Model (URFM), which learns from the charts of several games at once.

Try it in the Space

What it does

• Audio → chart. Taiko charts as .tja (Easy to Edit, with an official level) and osu! charts as .osu for osu!taiko, osu!catch, osu!mania (1 to 18 keys) and osu!standard (Easy to Expert). You choose the courses, the note density and the sampling settings, and you can edit each course's section plan before the notes are written.
• Chart → music (experimental). YuE2-3B composes music for a chart's tempo, meter and section cues and renders several takes, and the compatibility model keeps the take that fits the chart best. The music covers the whole chart at one tempo (up to six minutes) and is not note-synchronous.

Part	Status
Taiko	release
osu!taiko	release
osu!catch	preview
osu!standard	experimental
osu!mania	experimental
Chart → music	experimental

Components

Component	Role	Parameters	Files
Beat model	Beats, downbeats and tempo from the log-mel, decoded into one timing map	5.0M	beat.safetensors
Notation-level classifier	Chooses half, the same or double the tracked tempo, the way charters of each game notate the song	logistic regression	octave.json
Song planner	Plans every section of each course (density, peaks, fine notes, rests, colour, long notes, repetition, variety) from the audio, with a song-level style latent	1.43M	planner.safetensors
Chart decoder	Encoder–decoder Transformer conditioned on the game, course and plan; a grammar from each game's mechanics spec keeps every output a valid chart	18.5M	decoder.safetensors
Compatibility model	Scores how well a chart's timing fits audio, without seeing note positions on screen; picks the music take	3.8M	compat.safetensors

The chart model (planner and decoder) has 19.98M parameters. Each .safetensors file comes with a .json file of the same name that holds its configuration.

Training

The decoder and planner learn from about 111k charts of about 22k songs with audio: Taiko (4,023 training charts) and the osu! modes (osu!taiko 32,639, osu!mania 25,878, osu!catch 11,245, osu!standard 37,696). One song-level split is shared by every game, so a test song is unseen in all of them.

• Supervised. The decoder learns all games together by next-token prediction. Tempo and notation augmentation (tempo changes, half and double tempo notation, the beat model's own grid) lets it write on whatever timing the audio yields.
• Song planner. Trained on the same charts to predict each section's plan.
• Reinforcement learning. Song-level GRPO on long spans of the decoder's own output (Taiko and osu!taiko), with rewards for onset accuracy, placement precision, rhythm distribution, plan adherence and structure, and penalties for stagnation and monotony.
• Supporting models. The beat model learns from beats marked in charts, the notation-level classifier from how charters notate tempo, and the compatibility model contrastively from audio–chart pairs of every game.

Evaluation

All results compare generated charts with human charts of held-out test songs, with timing predicted from the audio.

Fit to the music. A separately trained compatibility model scores how well each chart's timing fits the audio. It is not the released one and is used neither in training nor for selection. Scores are scaled per game so that the human chart shifted by 120 ms scores 0 and the human chart scores 1. Charts were generated through the Space's request path with default settings, on 30 test songs per game.

Game	Status	Test charts	Fit (human = 1)	Charts beating the shifted human chart
Taiko	release	129	0.75	95%
osu!taiko	release	120	0.81	99%
osu!catch	preview	120	0.77	99%
osu!standard	experimental	115	0.69	100%
osu!mania	experimental	87	0.66	100%

Human charts beat the shifted chart on 95–100% of the same charts. osu!mania was measured before the release's chord correction (a logit bias of −0.2 against stacking another note at the same time). With it, the chord ratio matches human charts (0.41 vs 0.41) and the fit is unchanged (+0.01 logit, 95% CI [−0.38, 0.20]).

Musical shape. Section contrast is the correlation of note density over 8-beat sections with the human chart; the human reference is a second human chart of the same song. Excess monotone time is the seconds per minute in which one rhythm repeats longer than Taiko charters keep one rhythm at the same density; the same allowance applies to every game. Taiko: 599 test charts × 3 seeds. osu!: a 120-song test panel × 2 seeds, of which each game covers a part (osu!taiko 139, osu!catch 43, osu!standard 233, osu!mania 114 charts).

Game	Section contrast	Human reference	Excess monotone time (s/min)	Human charts
Taiko	0.38	0.72	0.32	0.26
osu!taiko	0.58	0.86	0.11	0.37
osu!catch	0.49	0.71	0.03	0.04
osu!standard	0.40	0.64	0.06	0.05
osu!mania	0.46	0.78	4.1	4.6

These were measured with the released planner and decoder before the notation-level classifier and the osu!mania chord correction were added. On Taiko the classifier leaves onset F1 and placement unchanged.

Known limitations.

• Section contrast reaches about half to two thirds of the agreement between two human charts.
• Movement in osu!catch and osu!standard is more conservative than in human charts (slower, with more pauses on osu!standard). The fit score does not see positions on screen.
• The mix of note types in osu!mania still differs from human charts.
• Chart → music fits a chart's tempo, meter, pulse and density, not its particular rhythm.

Usage

The Space runs the whole system on a shared GPU and works from any browser. The Space repository also holds the code, so the same system runs locally. With Python 3.12 and ffmpeg installed:

git clone https://huggingface.co/spaces/JacobLinCool/urfm-vega-preview
cd urfm-vega-preview
pip install -r requirements.txt gradio==5.50.0 spaces
python -m urfm.app.pipeline song.mp3 --courses oni:9 hard:7 --out out/
python -m urfm.app.pipeline song.mp3 --game osu_mania --keys 4 --courses hard insane --out out/
python app.py    # the Space's interface at http://127.0.0.1:7860

The weights download from this repository on first use, and a CUDA GPU is used when one is available. Taiko courses are written course:level (optionally :notes per second), osu! courses difficulty (optionally :notes per second). --game is one of taiko, osu_taiko, osu_catch, osu_mania and osu_std. Chart → music needs a CUDA GPU.

License

Research only, non-commercial use: see the URFM Vega Preview Research-Only Non-Commercial License. The chart → music tool uses YuE2-3B, which is licensed CC BY-NC 4.0.