Steering Autoregressive Music Generation with Recursive Feature Machines
Authors: Daniel Zhao
Daniel Beaglehole
Julian McAuley
Taylor Berg-Kirkpatrick
Zachary Novack
Venue: ICLR 2026 Poster (2026)
Keywords: music generation
probing
interpretability
music ai
multimodal LLMs
steering
inference
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We adapt Recursive Feature Machines to steer pre-trained music models in real time, enabling fine-grained, interpretable control of musical attributes without retraining.

Abstract: Controllable music generation remains a significant challenge, with existing methods often requiring model retraining or introducing audible artifacts. We introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained, interpretable control over frozen, pre-trained music models by directly steering their internal activations. RFMs analyze a model's internal gradients to produce interpretable "concept directions", or specific axes in the activation space that correspond to musical attributes like notes or chords. We first train lightweight RFM probes to discover these directions within MusicGen's hidden states; then, during inference, we inject them back into the model to guide the generation process in real-time without per-step optimization. We present advanced mechanisms for this control, including dynamic, time-varying schedules and methods for the simultaneous enforcement of multiple musical properties. Our method successfully navigates the trade-off between control and generation quality: we can increase the accuracy of generating a target musical note from 0.23 to 0.82, while text prompt adherence remains within approximately 0.02 of the unsteered baseline, demonstrating effective control with minimal impact on prompt fidelity.

Published as a conference paper at ICLR 2026
STEERING AUTOREGRESSIVE MUSIC GENERATION WITH RECURSIVE FEATURE MACHINES
Daniel Zhao, Daniel Beaglehole, Taylor Berg-Kirkpatrick, Julian McAuley, Zachary Novack
University of California, San Diego (dizhao, dbeaglehole, tberg, jmcauley, =novack]@ucsde
ABSTRACT
Controllable music generation remains significant challenge, with existing meth- ods often requiring model retraining or introducing audible artifacts. We introduce MusicRFM, framework that adapts Recursive Feature Machines (RFMs) (Rad-) hakrishnan et al., 2023) to enable fine-grained, interpretable control over frozen, pre-trained music models by directly steering their internal activations. RFMs an- alyze model's 's internal gradients to produce interpretable "concept directions", or specific axes in the activation space that correspond to musical attributes like notes chords. We first train lightweight RFM probes discover these direc- tions within MUSICGEN'S hidden states; then, during inference, we inject them back into the model to guide the generation process in real-time without per-step optimization. We present advanced mechanisms for this control, including dy- namic, time-varying schedules and methods for the simultancous enforcement of multiple musical properties. Our method successfully navigates the trade-off be- tween control and generation quality: can increase the accuracy of generating target musical note from 0.23 to 0.82, while text prompt adherence remains within approximately 0.02 of the unsteered baseline, demonstrating effective control with minimal impact on prompt fidelity. We release code to encourage further explo- ration on RFMs in the music domain.
INTRODUCTION
Large autoregressive (AR) models, powered by neural audio codecs, have made remarkable strides in mext-to-music (TTM) generation, producing audio with impressive fidelity and coherence (Copet ct al., 2024] Yuan et al. 2025). Despite a growing body of work in conditioning TTM AR models on varying controls (Novack et al. 2024b at Wu al., 2024 Lin al., 2023 Koo al. 2025 achieving precise control over fine-grained music-theoretic (e.g. pitch classes and chord qualities)
content across time in generations remains challenging. Current approaches often focus on broad temporal controls like dynamics or polyphonic melody, and may either require intense finetuning runs or costly per-step optimization during inference avoid large-scale training.
We argue that a more direct and principled path to controllability lies in activation space interven- tion. If we can identify directions within a model's hidden states that reliably correspond human- interpretable music-theoretie concepts, such as specific pitches, chord qualities, or tempo. we can then steer the generation along these axes, guiding the creative process without retraining the base model or altering its decoding procedure. The critical question then becomes how to discover these semantic directions in a robust and interpretable manner.
Recursive Feature Machines (RFMs) provide powerful answer (Radhakrishnan et al., 2023 Bea- glehole et al. 2025a b). By forming an Average Gradient Outer Product (AGOP) from lightweight task probes, RFMs yield set of orthogonal, cigenvalue-ranked directions that capture the most salient axes of variation for a given concept within a model's 's representation space. These directions directly represent the model's principal axes of sensitivity to specific features.
In this work, we introduce MusicKFM, the first framework that adapts RFMs for TTM generation by steering frozen MusIcGEN-Large model directly in its activation space. Our approach is

---

Published as a conference paper at ICLR 2026
twofold: first, we train extremely lightweight, layer-wise RFM probes the SYNTHEORY dataset (Wei et al. 2024) to extract concept-aligned directions. Then, at inference time, we inject them into the model's residual stream via forward hooks, enabling real-time, fin-grrained control over the generated output. We deploy this framework on a suite of novel music-theoretic controls, controlling for diverse concepts such as the presence of specifie intervallic relationships, chord qualities, and scale modes. To ensure that audio quality and fidelity is not sacrificed for steering controllability, we introduce layer-based methods that apply steering selectively across the model's 48 decoder blocks, using top-K selection or an exponential weighting scheme based on each layer's 's probe performance. We also show that RFMs can be used to control the presence global attributes as a function of time, using time-based schedules that modulate steering strength throughout the generation with functions like linear fades, sinusoidal patterns, and sparse, stochastic application. Furthermore, MusicRFM supports multi-direction steering, allowing for simultaneous or staggered enforcement of multiple attributes, such as jointly controlling notes and tempos. This comprehensive approach to control proves highly effective: our primary analysis shows that steering can increase the classification accuracy of a target note from 0.23 to over 0.82, while CLAP score for text alignment remains within =0.02 of the unsteered baseline- a highly favorable trade-off between control and fidelity.
In brief, MusicRFM establishes an efficient framework for fine-grained, interpretable control in text-to-music generation, requiring only lightweight training of RFM probes, with finetuning or costly optimization at inference time. Its layer- and time-aware mechanisms along with support for multi-direction steering, enable controllable modulation of audio while preserving high fidelity.
2 RELATED WORK
Research on controllable generation spans several communities, from activation-level steering in large language models to decoding-time time control methods and controllable music generation. Our work, MusicRFM, builds on and unifies these threads by adapting RFMs to the domain of music while adding new temporal and architectural control mechanisms.
2.1
CONTROLLABLE MUSIC AND AUDIO GENERATION
We focus particularly on mext-to-music (TTM) generation that relies on neural audio codecs and au- toregressive sequence models in architectures like MUSICGEN. MUSICLM, and JUKEBOX (Copet
ct al. 2024 Défossez et al.) 2022 Agostinelli et al. 2023 Dhariwal et al. 2020: Yuan et al. 2025
Team et al. 2025). Additionally, number of controllable TTM systems exist the parallel diffu- sion domain (Novack et al., 2024b a; Wu et al. 2024 Nistal et al. 2024a b). Most existing control- lable methods for AR focus on muultimodda controls (e.g. video) (Kim et al. 2025) or common mu- sical controls like piano rolls (Lin et al. 2024). These approaches, while generally performant, still require reasonably compute-heavy finetuning runs and thus necessitate changing the base model; even parameter-efficient fine-tuning methods (Wu al. 2024 Lin al. 2024 Baker & Nistal, 2023) require 10s to 100s of GPU hours for each control added. Additionally, prior methods risk breaking the core generative capabilities of the models, especially if the finetuning data is ill-chosen.
2.2 ACTIVATION-LEVEL STEERING IN GENERATIVE MODELS
Beyond music, a growing body of work investigates activation-level steering language models. Activation Addition (ACTADD) constructs steering vectors from paired prompts and injects them into hidden states for sentiment or style shifts, without retraining or optimization (Turner et al.) 2024 Contrastive Activation Addition (CAA) extends this idea by contrasting positive/negative contexts to obtain more targeted steering directions Llama-style models (Panickssery et al.) 2024). These methods illustrate broader trend: interpretable steering can often be achieved by modifying internal activations, rather than logits or decoding heuristics. Within music, existing approaches either focus solely on binary controls (Facchiano al. 2025) broad concepts like instrument presense (Koo et al. (2025), and thus it remains to be seen whether such approaches can extended to time-varying, theoeeti control.
2.3 RECURSIVE FEATURE MACHINES (RFMS)
Recursive Feature Machines (RFMs) Radhakrishhan et al. 2023) were introduced as probing meth- ods that iteratively recondition features via AGOP matrices to uncover task fask-sensitive subspaces.
2

---

Published as a conference paper at ICLR 2026
More recently. RFM-derived directions have been re-injected into activations for steering in LLMs (Beaglehole et al. 2025b). We extend this paradigm to autoregressive music generation with three innovations: (i) layer-based control through top-K K and exponential weighting across 48 layers, (ii) time-based control using dynamic schedules, and (iii) mulii-divection control via simultaneous or staggered application of concept directions.
METHODS
Our overall goal is enable fine-grained, interpretable control in AR music generation, steering wards concepts like specific notes, chord types, or slow/fast tempo. To do this, we train lightweight RFM probes extract concept-aligned directions and re-inject them into MUSICGEN activations inference time. This framework allows us to generate music samples that still follow text condition- ing with high accuracy, while also reflecting controlled variations targeted musical attributes.
3.1 BACKGROUND ON RECURSIVE FEATURE MACHINES
We first provide some more background on Recursive Feature Machines before describing our appli- cation to music generation. RFMs (Radhakrishnan al. 2023) were originally proposed as a prob- ing method, iteratively reconditioning features with Average Gradient Outer Product (AGOP) matri- ces to identify sensiiive subspaces. Given training data {(T,, y4))== and predictor f IR + R, define per-sample gradients gi
vz((x;) E æd and the AGOP
M A
9:9;
T Rdddd
(I)
i=1
M is PSD, with eigendecomposition M QAQT. Directions {hp} are orthonormal, with eigenval-
ues X5 2 0 measuring sensitivity:
d = q, Mg, = 1 Σ 9.)2.
(2)
RFM implements feature learning by iterating: (i) train base learner (kernel ridge regression as described in App. A) features (!)) to obtain f(1), (ii) compute (t) via equation | and (iii) update features with
((++)) - T (Q0)))
where a > O amplifies high-sensitivity directions. Importantly, this process is backpropagaaioon-fee.
Recent work has extended RFMs to steering: injecting a concept direction q; back into hidden activations biases a frozen model toward that attribute during inference (Beaglehole al. (2025b).
In practice, steering is implemented by registering hooks on subset of layers 5 and adding broadcast control vector to each residual stream:
hig h4, ne()) 48.""
where qr ERd is reshaped to (1, 1, d,y). Steering only uses the top component per direction.
3.2 MUSICRFM: RFM STEERING FOR MUSIC GENERATION
We adapt RFMs to steer MUSICGEN large (L=48 decoder blocks). a Transformer over EnCodec tokens conditioned on text (Copet et al. 2024, Défossez et al. (022). Our pipeline has three stages: (i) audio - ENCODEC codes, (ii) layerwise RFM probes that yield AGOP eigendirections, and (iii) steering applied at inference as described above.
Synthetic Dataset for Probe Training. SYNTHEORY (Wei et al. 2024) is recently designed syn- thetic dataset made to study interpretable representations of music theory concepts in large models, divided into categories: tempo, notes, chord progressions, chord types, scales, intervals, and time signatures. Compared prior music datasets, SYNTHEORY offers clean, fine-grained supervision
3

---

Published as a conference paper at ICLR 2026
of musical properties, enabling controlled experiments model interpretability and controllability. This dataset is particularly well-suited for probing approaches, as its labeled attributes align directly with theoretical concepts that can be mapped onto latent representations. In our setting, SYNTHE- ORY allows us to train lightweight RFM probes on layerwise activations of MusicCien, yielding gradient-based directions that correspond to human-interpretable musical attributes.
Feature extraction. Audio clips are resampled 32 kHz, encoded with ENCODEC, and passed through MUSICGEN. For clip and layer f, we mean-pool over tokens, zi,t = + EL hig E Rd, yielding clip-level vectors. Unlike last-token pooling used in text-based RFMs (Beaglehole et al. 20256 a), mean pooling better captures temporal structure and improves probe performance.
Probe training and steering. For each concept and layer f, we train RFM probes for 15 itera- tions (fit predictor, compute AGOP, apply PSD map), keeping the probe with best validation metric (AUC for classification, MSE for regression). Binary concepts use {0, 1} labels and regression tar- gets are z-normalized. The resulting eigendirections Qr.j form interpretable axes used for steering inference. Steering is performed by the same process described Eq. For classification tasks, we additionally train multiclass RFMs that simply replace binary labels with one-hot- ane-hot-encoded target vectors, predicting through softmaxing final outputs.
3.3 IMPROVING ROBUSTNESS IN AUDIO- DOMAIN STEERING
As extend the existing text-stecring framework of RFMs provided by Beaglehole et al. (2025b) to audio domain music, we introduce additional modifications to help reduce out-of-distribution behavior and improve control. In particular, given the difference between the discrete, variable- sampling rate nature of text and the continuous, fixed-sampling rate nature of audio-domain music, found that many of the algorithmic choices made by [Baglehhle al. (2025b) were ill-suited for TTM generation. All modifications are only applied during inference time.
3.3.1 LAYER PRUNING
Naïve steering, where we inject RFM directions uniformly across all L=48 layers at every step as is done in the original RFM paper (Beaglehole et al. 2025a), leads to noticeable degradation in audio quality and weaker alignment to text prompts. To address this, we introduce layer pruning strategies inference time that prioritize informative layers and downweight noisy ones, thereby improving both perceptual fidelity and controllability (see App. C for full results).
Top-K selection. We rank each layer e E {I,..., L} by its validation probe performance AUC,, then restrict steering to the top-K K layers.
Exponential weighting. Instead of hard pruning, also apply continuous weighting across layers. For each layer f, we normalize its probe score 8t into & E [0, 1], and define wy = wy 8 2//x with K E (0, 1). This concentrates steering strength higf-perg layers, reducing unwanted artifacts and incorrect directions produced by the lower-scoring ones.
3.3.2 TIME-CONTROL SCHEDULES
We modulate steering strength over time as (t) = 70 we d(t) (t). where To is a global coeffi- cient, we layer weight, o(t) deterministic schedule, and (t) optional stochastic gate.
Deterministic schedules d(t). Linear/logistic rise, linear/exponential decay, and sinusoidal modu- lation let us increase or decrease concept's influence over time (e.g., fade out note class, ramp in a chord progression, or periodically modulate tempo). Closed-form expressions are given App.
Stochastic application (t). At each step, apply control with probability p (Bernoulli gating). Similarly to layer pruning, this method reduces over-steering and cumulative artifacts while pre- serving the expected bias toward the target. Ablations are in App.
3.3.3 DIRETI-IRECTION AND STAGGERED CONTROL
We further extend MusicRFM to support multi-direction steering, combining multiple concept vec-
tors {gt,im. J m== in parallel. At each step we inject hi. -h, -,, in We Φm (1) (t) Peii.
M
4

---

Published as a conference paper at ICLR 2026
where each direction m has its own coefficient fu and schedule (t). This enables both (i) si- multaneous enforcement of multiple attributes and (ii) staggered control where different concepts activated at different times. For example, one schedule may enforce tempo strongly during the opening segment, while another gradually ramps in harmonic structure later.
4 CLASSIFICATION RESULTS
With MusicRFM, we additionally train separate multiclass probes (different from the binary probes used to steer) to compare RFM clasification against the original probing methods used in SYN- THEORY. We see that RFMs have better or comparable performance to the 2-layer FFN probes used in the original SYNTHEORY paper across all categories. We highlight that RFMs beat the baseline probes in accuracy on scales, progressions, and intervals and R2 score on the tempo dataset, resulting in higher overall average score. We also find that mean-pooled RFMs outper- form last-token activations. Using only the final token assumes the model compresses all relevant musical information into single position, which is unlikely for temporally structured attributes. Mean-pooling instead aggregates information across the entire sequence, better capturing temporal patterns, especially for tempo, chord progressions, and scales.
Furthermore, we argue that FFNs do not naturally yield orthogonal, eigenvalue- directions suitable for steering. contrast, RFMs produce PSD AGOP matrix whose eigenvectors corre- spond to stable, interpretable axes of sensitivity. These axes can be directly injected into the model inference, making RFMs uniquely suited for controlled generation.
Model
Notes
Intervals
Scales
Chords
Prog.
Time Sig.
Tempos
Avg.
MusicREM
0.850
0.975
0.956
0.984
0.943
0.900
0.985
0.942
mean pooled (ours)
RFM (last token)
0.734
0.743
0.546
0,866
0,811
0.771
0.959
0.776
Linear Probe
0.761
0.618
0.158
0.834
0.725
0.729
0.972
0.685
Syntheory FFN
0.866
0.972
0.905
0.989
0.901
0.905
0.965
0,929
Table 1: Classification results for base SYNTHFORY FFN (in Wei et al. (0024) ) simple linear probes, RFMs trained last-token activations, and MusicREM (ours). We report R2 racy on the others. We don't it record performance on logistic probes as some fail to converge. Bold indicates best performing model per category.
the tempos dataset and accu-
5
MUSICRFM STEERING RESULTS
We report results on how well binary directions trained using MusicRFM are able to steer genera- tions towards interpretable concepts, exploring both quantitative and subjective metries.
5.1 QUANTITATIVE METRICS
We first quantify distributional shift, prompt adherence, and control accuracy of generations steered along single concept direction using four metrics as a function of the control coefficient rJo (i) Fréchet Distance (FD) (Gui et al. 2024) (lower is better), and (ii) Maximum Mean Discrepancy (MMD) @Jaysuumnna et al.] 2024) (lower better), (iii) CLAP alignment (Wu et al. 2023) using bn-test.......e.ooo...ee.ee.e. pt checkpoint (higher is better), and (iv) classification ac- curacy of generated samples using the multiclass RFM probes described Sec. (higher is better).
In our setup, we compare MusicRFM steering to simple baseline: prompt-based conditioning. For each concept category (e-g., notes, tempo), we append a textual hint that explicitly specifies the target attribute (c.g., "Note; C#" or "Slow Tempo") ") and generate audio using MUSICGEN-LARGN without any steering. We thus compare this settings: (i) prompt-only setting, (ii) a MusicRFM- only setting, and (iii) combined Prompt+MusicRFM setting where the prompt conditioning and RFM directions are applied simultaneously. This allows us to disentangle what can be achieved through prompt engineering from what is uniquely enabled by RFM intervention.
For all experiments, we evaluate on a fixed evaluation set of 250 prompts sampled from the SONG- DESCRIBER dataset (Manco et al. 2023). using all 3 settings described above. For each setting.
5

---

Published as a conference paper at ICLR 2026
FD
MMD
CLAPT 1
Probe Aec. 1
Category
Control coefficient No
0.15
0.30
0.45
0.60
| 0.15
0.30
0.45
0.60 0.15
0.30
0.45
0.60
0.15
0.30
0.45
0.60
MusicRFM-only steering
Chords (0.250)
0.063
0.086
0.040
0.095
0.324
0.330
Intervals (0.083)
0.110
0.128
0.169
0.232
0.078
0.119
0.400
0.817
0.315
0.324
0.311
0.307
0.121
0.156
0.187
0.223
Notes (0.083)
0.113
0.130
0.138
0.180
0.052
0.127
0.217
0.476
0.315
0.31
0.318
0.303
0.231
0.461
0.684
0.824
Scales (0.143)
0.075
0.061
0.081
0.318
0.328
0.322
0.324
0.161
0.176
Progs (0.053)
0.131
0.142
0.173
0.207
0.157
0.233
0.443
0.650
0.315
0.309
0.296
0.297
0.070
0.079
0.096
0.114
Tempos
0.122
0.150
0.206
0.377
0.112
0.324
0.717
1 1,880
0
0.325
0.307
0.280
Time sigs (0.125)
0.162
0.264
0.492
0.356
1.046
1,980
2.647
0.320
0.317
0.278
0.264
0.172
0.204
0.238
0.245
Prompt RFM steering
Choris (0.250)
0.077
0.071
0.080
0.095
0.120
0.114
0.154
0.243
0.330
0.326
0.328
0.333
0.273
0.276
0.309
0.344
Intervals (0.083)
0.078
0.077
0.091
0.119
0.184
0.169
0.232
0.417
0.351
0.353
0.345
0.328
0.125
0.163
0.209
0.245
Notes (0.083)
0.108
0.119
0.133
0.159
0.438
0.479
0.563
0.713
0.343
0.325
0.321
0.329
0.657
0.826
0.921
0.952
Scales (0.143)
0.141
0.127
0.131
0.138
0.566
0.473
0.472
0.500
0.348
0.346
0.346
0,340
0.179
0.212
0.209
0.230
Progs (0.053)
0.186
0.715
0.758
0.085
0.106
0.129
Tampos
0.163
0.199
0.270
0.442
0.370
0.630
1.5
2.342
0.318
0.314
0.293
0.270
Time sigs (0.125)
0.090
0.099
0.150
0.261
0.251
0.212
0.338
0.790
0.342
0.329
0.328
0,300
0.198
0.235
0.253 0.267
Prompt-only baseline
Chords (0.25)
0,069
0.078
0,331
0.267
Intervals (0.083)
0.082
0.216
0.356
0.104
Notes (0.083)
0.107
0.414
0.342
0.436
Scales (0.143)
0,146
0.630
0,344
0.190
Progs (0.053)
0.184
0,739
0.323
0.065
Tempos
0.087
0.111
0.325
Time sigs (0.125)
0.101
0,352
0.352
0.139
Table Single direction steering metrics. top black reports only steering stochastic application (p((), p
exponential layer weighting (mio I, K 0.05). The middle block (Prompr + RFM) shows combined promptin and RFM steering. The bottom block (Prompt-only) reports baseline where only prompt modified (independent of no). Parentheses denote random chance for each CLAP and Probe Accuracy (mean class,, Ground-truth MUsiCGEN-LAAGGI has CLAP 0.332. Probe acc is undelined for tempor (regression). We conduct experiments on 250 samples per class in each category.
category, Lower better for FD/MMD and higher better for
Steering Type
Chords
Intervals
Notes
Tempo
No Steering
59.71 + 6.01
54.75 + 5.52
57.08 + 6.37
55.75 + 7.08
Naive RFM (ours)
69.21 + 5.25
62.58 + 5.84
68.
5.97
73.33 + 4.35
MusicRFM (ours, optimal)
73.46 4.18
70.33 1 4.02
72.88 5.67
73.38 1 4.75
Table 3: Listening test results (mean + standard deviation) across musical attributes.
we generate 250 samples for each class in each category, and for each control coefficient 7o E {0.15, 0.30, 0.45, 0.60) (for cases (ii) and (iii). All results are reported on generations steered with RFM probes using stochastic application d (t) with p = 0.3 and exponential layer weighting with u/p = and x 0.95; these are settings we found to most optimal when creating high-quality, conceptually accurate generations. For the tempos category, results from each Yo are averaged among the absolute value of the coefficient (e.g. the results from -0.15 and 0.15 are averaged into the 0.15 column). FD and MMD distributions are compared against MUSICGEN-LARGE generations produced from the original prompts without any control. We show these results Table
5.2 BINARY-PROBE QUANTITATIVE STEERING RESULTS
Across all categories, our quantitative metries follow consistent trends. Distributional metrics (FD and MMD) are consistently lower at smaller control coefficients, since weak steering leaves gener- ations closer to the reference distribution. As TJu increases, stronger injections deviate more from ground truth and raise FD/MMD. By contrast, CLAP alignment remains essentially flat across con- trol strengths, indicating that textual conditioning is preserved regardless of steering intensity, only with slight degradation in some categories as control coefficient increases. Probe-based classifica- tion exhibits the same monotonic behavior. Accuracy highest for notes, rising sharply from 0.23 at 07.1.1 15 to 0.82 at =0.660 and increases monontonically for all other categories. Thus, mod- crate values of r/o can balance concept control with distributional fidelity while maintaining prompt adherence. We provide additional visual graphs for the reader in App.
We observe that our baseline, prompt-only conditioning, except for on the notes categories, yields almost cendom-chance level accuracy, showing that simple textual descriptions do not provide good control. By contrast, RFM-only steering produces clear, 7/0 -dedependent improvements. Additionally, combining prompt conditioning with RFM typically yields the strongest results, especially in cases

---

Published as a conference paper at ICLR 2026
where prompting alone already gets accuracy to a higher level (e.g. in notes where accuracy exceeds 95% at higher n7)) These comparisons highlight that RFM activation-level steering enables forms of musical control that do not emerge from prompt engineering alone.
We note that probe accuracies should be interpreted as relative indicators rather than absolute ground truth. The RFM probes were trained on SYNTHEORY, synthetic dataset with simplified musical attributes, and therefore may not generalize perfectly to natural MUSICGEN outputs. Nonetheless, their trends across To provide a reliable signal that MusicRFM is correctly steering music.
Note Dominance (%)
Chord Dominance (%)
Mean Event Rate (eventsk)
Method no
0.15
0.30
0.45
0.60
0.15
0.30
0.45
0.60
0.600
00.55
00.30
00.15
0.15
0.30
0.45
0.60
MusicRFM
18:50
34.47
$2.50
66.47
24.40
28.40
30,50
35.00
18.66
20.97
25.07
26.24
30.48
30.01
30.88
31.65
Prompe-RRM
53.57
67.83
78.23
85.13
26.60
27.80
27.30
33.60
15.19
19.02
21.13
22.43
31,66
34.10
33.55
32,51
Prompt only
35.97
26.40
25.03 (slow), 30.63 (fast)
Table 4: External evaluations
chonds, and tempo. Note/chord columns show target dominance (higher is better); tempo columns
report mean event rate across control coefficients to-
5.3 EXTERNAL EVALUATION METRICS FOR MUSICAL. CONTROI
On some categories, we introduce external evaluators that operate directly on the waveform and do not rely our multiclass RFM probes in order evaluate the accuracy of RFM steering. For notes, we compute chromagrams and label a sample as correct if the target pitch class has the highest mean energy across all classes. For chords, we apply Essent ia-based chord estimator (Bogdanov) iet al. 2013) and mark a sample as correct when its most frequently predicted class matches the target. Table shows that RFMs work better than prompt-only injections, and accuracy increases as 1oo increases. For notes, we see an even higher accuracy increase if we combine both methods. However, for chords, we see a performance degradation. likely due to the fact that prompting alone has a low accuracy (so combining the two may push the generation in the wrong direction).
As we qualitatively found that traditional BPM detectors did a poor job of picking up on stylistic differences between generations with "fast" vs. "slow", for tempo we instead use a peak-weighted onset event rate (events/s) as a measure of rhythmic density (i.e. the average number of onsets per second, weighted by onset strength) using 1 Librosa onset detection (McFee & et al. 2023). Table provides a horizontal comparison across all steering strengths. Prompt-only conditioning yields a consistent fast-slow separation, whereas RFM steering exhibits a clear monotonically increasing lationship with 7/. with a Spearman coefficient of 0.283. Combining RFM steering with prompting achieves even better results, with Spearman coefficient of 0.433.
5.4 LISTENING TEST AND AUDIO SAMPLES
We provide results of a listening test, where asked 12 participants to score 3 different audio samples for 4 control types (24 total samples, 3 control setups for control examples each across 4 different control types), where they judge based audio quality and adherence of the audio to the specified control. The clips were randomly chosen base model generations (without control), naive RFM generations, and optimal RFM generations (steering with p = 0.3 and exponential layer weighting with We 1 and = 0.95). Participants were randomly chosen from a departmental computer science forum at an R1 research institution, with mean age of 23.6 and mean musical experience of 9.6 years. We show mean and STD of each type of steering in Table 3 Overall, the results indicate that both naive and MusicRFM steering substantially improve perceived control compared to the base model, with MusicRFM consistently achieving the highest ratings across all attributes. In particular, chord and interval control benefit most from RFM steering, while tempo control shows the largest relative gain over the no-steering baseline.
To the reader, we also provide representative audio samples from the listening test, illustrating single-direction control (notes), multi-direction control (notes+chords), and time-based schedules (rise/decay and crossfades). Each clip is paired with its text prompt and steering metadata (7o- schedule), where all clips are steered with the "optimal" parameters listed above. An interactive demo of some of the clips used our listening test is available at the project page
htttps:/mmsiirrfnn wtw.h/ub..oc///ooroooo aaaal-lmmssa musi mnraic-rfm/
7

---

Published as a conference paper at ICLR 2026
5.5 EVALUATION ON MUSICBENCH (REAL MUSIC)
To test transfer beyond synthetic data, we evalu-
ate RFM probes MUSICBENCH (Melechovsky et al. (024)) a real-music corpus with ground-truth tempo. notes, and keys. Using the same pipeline as in Sec. 3.2 we mean-pool MUsIcGEN-large hid-
FD
MMD 4
CLAP 1
Acc. t
0.15
0.424
0.478
0.315
0.148
0.30
0.495
0.908
0.308
0.264
0.45
0.576
1.563
0.276
0.479
0.60
0.717
2.615
0.247
0.619
den states and fit layerwise RFMs (train/val/test split 70/15/15). For tempo we report normalized MSE, for classification overall accuracy. RFM probes
Table RFM steering on MUSICBENCH (key).
reach 75.3% accuracy notes and 67.5% on keys, while tempo regression proves difficult (MSE 0.862). Steering experiments (Table 5) mirror SYNTHEORY: higher To increases FD/MMD and reduces CLAP, showing that moderate control preserves text adherence but aggressive coefficients destabilize generations. Overall, MusicBench confirms that though sensitivity varies by concept difficulty.
attributes can be steered,
MULTI-DIRECTION AND TIME-BASED STEERING RESULTS
We also evaluate MusicRFM when (i) multiple concept directions are injected simultaneously and (ii) when steering strength varies over time. We report the same quantitative metrics used in the single-direction setting and for (ii) introduce temporal analyses based on RFM probe softmax scores.
6.1 MUL.TI-DIRECTION STEERING: PAIRWISE
CONTROL
To test whether MusicRFM can jointly enforce multiple musical attributes, we examine all pairwise combinations among notes, chords, intervals). For each pair ba, we sample random target class from category a (e.g., note C`) and a random class from category b (e.g.. major chord), then generate music conditioned on both controls simultancously.
At inference, we inject two steering directions per selected layer, one for each concept, following Sec 3.3.3 Each direction scaled by independent global coefficient 7o {0.3, 0.6]. We evaluate all four cross- combinatisss [[0.3,0.3]. [0.3,0.6], [0.6,0.3], [0.6,0.6]], where the first value corresponds category a and the second samples per configuration, yielding 3 x 4 x N = 1200 total generations. To report results concisely, we reorganize outputs by attribute rather than by pair. For instance, all samples where notes were steered with coefficient 0.3 D.grregardsess of whether they were paired with chords or intervals- are averaged together. This gives us per-category summaries across all pairings, shown Table
category
For each pair. we generate N 100
Concept
%o
MMD
CLAP
Probe Acc.
Chords Chords
0.3
0.604
2.564
0.207
0.385
0.6
0.747
3.539
0.167
0.390
Intervals
0.3
0.572
2.351
0.209
0.298
Intervals
0.6
0.8861
4.470
0.134
0.300
Notes
0.3
0.566
2.394
0.205
0.770
Notes
0.6
0.927
4.725
0.133
0.920
Table 6: Multi-direction (pairwise) steering. Each cell reports the average over 200 generations.
Findings. We observe several trends: (i) Probe accuracy still rises with stronger coefficients. For notes in particular, accuracy increases from 0.770 at 7o = 0.3 to 0.920 at 0.6. indicating that control strength directly improves enforcement even in multi-direction cases. (ii) Distributional metrics and CLAP scores degrade at higher strengths. Both FD and MMD grow substantially as 7o increases, consistent with the single-direction case, where aggressive steering pushes samples away from the reference distribution. CLAP alignment also degrades significantly. (iii) Accuracy in multi-direction steering exceeds single-direction. We actually observe higher probe accuracy in the multi-direction setting, which hypothesize arises because stronger aggregate constraints reduce adherence to the text prompt (lower CLAP) and, in turn. compress the generative manifold. This yields less stylistic variance in the music, making classes easier for probes detect.
8

---

Published as a conference paper at ICLR 2026
Interprefation. These results highlight that multi-direction steering can indeed enforee multiple concepts, but doing so amplifies distributional drift and weakens prompt adherence. Notably, notes remain most controllable (large probe gains with modest 7o), while more abstract concepts like intervals yield smaller improvements. This suggests that balancing coefficients across attributes or staggering them temporally (Sec. 3.3.3. may be necessary for high-quality joint control.
6.2 TIME-DEPENDENT CONTROL: SMOOTH SCHEDULES
To study temporal schedules in isolation, we analyze the notes dataset with per-step steering strength (1))) = "No Pr (tt) and track the softmax score of the ground-truth note class under the RFM probe as a function of time (generation steps). For the experiments in this section, only analyze on notes because are they are the highest quality in terms of following control, and also can give us a measurable accuracy when evaluating with RFM probes.
We use per-direction coefficients 70.m and sched- ules om (t) [0, 1], so (t) f7 (t). The schedules we ablate are exponential decay, lin-
Schedule
FD
MMD
4 CLAP t
Linear increase
0.358
1.917
0.227
decay & increase, logistic increase, and sine
Linear decay
0.321
1.636
0.257
wave. We put formulas used in Appendix E and record FD, MMD, and CLAP scores in Table 7
Exponential decay
0.229
1.052
0.312
Logistic increase
0.360
1.999
0.208
Sine modulation
0.413
2.347
0.225
Correct note probability over time. For each schedule plot the probe softmax of the correct
Table Merrics time dependepnddett contmlled generations
note over time in Figure Tal We see that the distribution over time follows exactly what we would expect from each of the smooth scheduling functions exponential linear decay look like decay functions, sine is very similar to sine wave, and logistic & linear increase show an increase in predicted probability.
Crossfading Between Concepts. We additionally study a controlled cross-fade between two notes 11 - 122 using complementary schedules over a fixed window of 0-1500 steps: for n, we decay from V/o = 0.45 to O, and for n2 we rise from 0 to 0.45. Formally,
(t) TJo ddccay (t), nid
(nz) Ti
1oo (t)seet,,, t E [0, 1500].
We then log and display the RFM-probe softmax scores for both n and n2 at each timestep in Figure [b As expected, the first note falls in probability while the second note rises. On average over 500 randomly sampled note pairs, crossfaded generations achieve FD of 0.350, MMD of 1.922, and CLAP alignment of 0.250, indicating modest distributional drift but stable prompt adherence.
Time Control Effects on Note Classification
Crossfade Analysis: First Note Second Note
Avarage Sohmer Soore (Conei (Coreca Clocs)
0.14
0.14
0.13
0.13
0.12
Average Soltmax
0.11
0.30
0.68
1000
Decay
Timestap
Docay
Logisic Increase
First Note (Feding Out)
Second Note (Feding
(a) Temporal softmax traces (notes). Curves show the probe probu- bility the ground-truth note across timesteps for different schedules (limeurlepp riseldeaay, log. increase, sine). We choose the probe the hest performing layer (37) as our representative probe.
(b) Twu note crossfade (solumax probahiities). The score for de- cays (red) while ni rises (blue). We again choose layer 37 our representative probe and average 500 samples.
Figure 1:
Time-based steering analyses. (a) Probe softmax follows prescribed schedules faithfully. (b) Cross- fade experiments show expected decay-rise patterns between two target notes.

---

Published as a conference paper at ICLR 2026
7 LIMITATIONS & FUTURE WORK
While MusicRFM demonstrates that RFM-derived directions can steer music generation in inter- pretable ways, several limitations remain.
First, our probes rely on mean-pooled features, which discard temporal ordering. This limits per- formance on concepts with strong sequential dependencies, such as scales, chord progressions, and time signatures, where the temporal dynamics are essential for accurate classification and control. As a result, RFM probes underperform on these attributes compared temporally local concepts like notes or chords. Future work should explore temporally aware pooling strategies (e.g., attention pooling, recurrent aggregation, convolutional pooling) or sequence-level RFMs that directly model time-evolving representations. Similarly, extending beyond the top eigenvector to incorporate multi- ple components could capture richer subspaces of variation, but we have not yet performed variance analyses to quantify how much information higher-order components retain.
Another promising direction is to modify the steering process itself: by selectively masking portions of the autoregressive context during inference, one could reduce model over-dependence previ- ously generated tokens, thus increasing the model's sensitivity to injected steering signals. Such approaches could make activation-space interventions both more controllable and also reduce the number of unwanted artifacts, particularly for longer generations.
Additionally, experiments so far are limited to SYNTHEORY- symbolic music- thhor-che concepts such as notes, chords, and tempo. Future work could extend MusicRFM to attributes more directly tied perceptual production-level qualities, including instrument identity, timbre, or articulation style. While we perform preliminary analysis on MUSICBENCH, extended RFM training and steering on real-music-based datasets remains an open direction. These studies would connect RFM steering more directly to interpretability in real-world generation tasks.
Finally, our experiments target MusicGEN-large, but other large audio models open complemen- tary directions for RFM steering. OpenAl's JUKEBOX (Dhariwal et al.) 2020) uses multi-scale VQ-VAE codes and hierarchical AR decoders, while Google's 's recent MAGENTA-RT (Team et al.] 2025 framework supports real-time audio generation. Applying RFMs in these contexts would re- quire adapting probe extraction to multi-level codebooks (for Jukebox) and to low-latency streaming architectures (for Magenta-RT) In particular, real-time models highlight the possibility of real-time steering: dynamically injecting directions during live playback, enabling interactive control (i.c. live DJ-ing). Extending MusicRFM into these setups could bridge interpretability with performance- critical generative applications such as interactive music tools and live performance systems.
8 CONCLUSION
We presented MusicRFM, a framework that leverages RFM-derived, eigenvalue-ranked directions to steer frozen MUSICGEN-laye model directly in activation space. By combining concept- aligned directions with layer-aware weighting and time-dependent schedules, MusicRFM enables fine-grained, interpretable control over attributes such as notes, chords, and tempo without modify- ing the base model or relying on per-step optimization.
Across synthetic and real-music settings, we observed consistent trade-offs governed by the control coefficient no: moderate steering improves alignment to targeted concepts with limited distribu- tional drift (FD/MMD) and minimal degradation in prompt adherence (CLAP), while aggressive steering yields stronger control at the cost of artifacts. Notes are the most reliably controllable, multi-direction steering is feasible but amplifies drift, and simple schedules (e.g.. decay/rise) sup- port intuitive manipulations like crossfades. Time-based control is accurate and true-to- scchedll terms of evaluating on softmax probability of classes. Layer pruning and stochastic (Bernoulli) application help stabilize generations by limiting cumulative bias.
By enhancing fine-grained controllability, this line of research can significantly broaden the practical applications of generative models. In the long term, improving the steerability and interpretability of generative models will expand their usefulness in domains like music production and game audio.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. Musiclm: Generating music from text, 2023. URL
org/abs/ 2301 11325
Tom Baker and Javier Nistal. Lilac: A lightweight latent controlnet for musical audio generation. arXiv preprint arXiv:2506. 114.7, 2025.
Daniel Beaglchole, David Holzmüller, Adityanarayanan Radhakrishnan, and Mikhail Belkin. xrfm::
Accurate, scalable, and interpretable feature learning models for tabular data. arXiv preprint arXiv:2508. /0053, 2025a.
Daniel Beaglchole, Adityanarayanan Radhakrishnan, Enric Boix- and Mikhail Belkin. To- ward universal steering and monitoring of ai models, 2025b. URL https://axxiv.. org/
abs/2502. 03708
Dmitry Bogdanov, Nicolas Wack, Emilia Gomez, Sankalp Gulati, Oscar Boyer, Oscar Mayor, Ger- ard Roma, Justin Salamon, Jose Ramon Zapata, and Xavier Serra. Essentia: An audio analysis library for music information retrieval. In Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR), pp. 493-498, 2013.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation, 2024. URL httpt::/axxiii.......m org/ 0os/2306.05282
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv:2005.00344, 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression, 2022. /Rt.atttss///arrr...r////////// 223..1338111
Simone Facchiano, Giorgio Strano, Donato Crisostomi, Irene Tallini, Tommaso Mencattini, Fabio Galasso, and Emanuele Rodolà. Activation patching for interpretable steering in music genera- tion, 2025, URL https: /ttxpps://axxi..oo
org/ abs/ 205/2504.000 04479
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1331-1335. IEEE, 2024.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards better evaluation metric for image generation. In CVPR, 2024.
Haven Kim, Zachary Novack, Weihan Xu, Julian McAuley, and Hao- Dong. Video-guided text- to-music generation using public domain movie collections. arXiv preprint arXiv:2506. 1225: 2025.
Junghyun Koo, Gordon Wichern, François Germain, Sameer Khurana, and Jonathan Le Roux. Smitin: Self-monitored inference-tin catrennce-timeeeee intervention for generative music transformers. JEEE Open Journal of Signal Processing, 2025.
Liwei Lin, Gus Xia, Junyan Jiang, and Yixiao Zhang. Content-based controls for music large lan- guage modeling. 11X11:2310.17162, 2023.
Liwei Lin, Gus Xia, Yixiao Zhang, and Junyan Jiang. Arrange, inpaint, and refin:: Steerable long- term music audio generation and editing via content-based controls. arXiv:2402.09508 2024.
Iariia Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bodganov, Yusong
Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv:2311.1003. 2023.
Brian McFee and et al. librosa/librosa: 0.10.., August 2023.

---

Published as a conference paper at ICLR 2026
Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Sou- janya Poria. Mustango: Toward controllable text-to-music generation, 2024. URL https:
7/ /arxiv.org/aaa htt.o/rggaaaaa/.la1..aaa 08355
Javier Nistal, Marco Pasini, Cyran Aouameur, Maarten Grachten, and Stefan Lattner. Diff-a-riff: Musical accompaniment co-creation via latent diffusion models. arXiv:2406.08338 2024a.
Javier Nistal, Marco Pasini, and Stefan Lattner. Improving musical accompaniment co-creation via diffusion transformers. arXiv:2410.23005, 2024b.
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrickk and Nicholas J. Bryan. DITTO-2: Dis-
tilled diffusion inference-time t-optimization for music generation. In ISMIR, 2024a.
Zachary Novack, Julian McAuley, Taylor KinkKiikpatrric,, and Nicholas J. Bryan. DITTO: Diffu- sion inference-time T-optimization for music generation. In ICML, 2024b.
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024. URL https://arxiv.
org/abs 2312. 06681
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mecha- nism of feature learning in deep fully connected networks and kernel machines that recursively learn features, 2023. URL httphtts:/// /tt.ttpps:///...v..oooo http//axxxv..../a/.//// a22p:///bb////b...e/.. 112112.18888 13881
Lyria Team, Antoine Caillon, Brian MeWilliams, Cassie Tarakajian, Ian Simon, Ilaria Manco, Jesse Engel, Noah Constant, Yunpeng Li, Timo I. Denk, Alberto Lalama, Andrea Agostinelli, Cheng- Zhi Anna Huang. Ethan Manilow, George Brower, Hakan Erdogan, Heidi Lei, Itai Rolnick, Ivan Grishchenko, Manu Orsini, Matej Kastelic. Mauricio Zuluaga, Mauro Verzetti, Michael Dooley, Ondrej Skopek, Rafael Ferrer, Zalán Borsos, Aaron van den Oord, Douglas Eck, Eli Collins, Jason Baldridge, Tom Hume, Chris Donahue, Kehang Han, and Adam Roberts. Live music models, 2025. URL htt.::tts:////t
04651
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL
/tttss://erxis htgg/bb//250 11.00..11200 1248
Megan Wei, Michael Freeman, Chris Donahue, and Chen Sun. Do music generation models encode music theory?, 2024. URL httpltttt///axx..crrr a22p///.///242....oo 00872
Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan. Music ControlNet: Multiple time-varying controls for music generation. IEEE/ACM Transactions Audio, Speech, and Language Processing (TASLP), 2024.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kong-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023.
Ruibin Yuan, Hanfeng Lin, Shayue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation, arXiv preprint arXiv:2503s 08638, 2025.
12

---

Published as a conference paper at ICLR 2026
A OVERVIEW OF KERNEL RIDGE REGRESSION
Kerne ridge regression (KRR) is the base model with which we apply the RFM procedure for iterative feature learning via the AGOP. We briefly explain the KRR model. Let X E Raxd denote
training samples with 20() denoting the sample in the 7th row of X for i E [n] and y gaxe denote training labels, where с is the number of output channels (e.g. one-hot encoded classes for c > 2 classes). Let K IR x IRd + R denote kernel function (a positive semi-definite, symmetric function), such as the Gaussian/RBF kernel (K (z, 2) exp(-Ile
= = or the Laplace kernel = =I12)/ L) used in this work. Given a ridge regularization parameter 2 0, KRR solved on the data (X, y) gives a predictor, f IR + R of the form:
(K(x, 2) exp(-
f(x) =
(4)
where a is the solution to the following linear system:
(K(X, + x1)
(5)
Here the notation (xx, X) E Rix= is the n-dimensional row vector with K K(x, = xz,x11))
and K(X, X) E R2xx denotes the kernel matrix of pair-wise kernel evaluations ((X,)) X); )ij K k(x!(0, ,.))). The advantage of kernel functions in the context of this work is that the predictor admits closed form solution, which can be robustly computed and generally has fast training times for datasets under 70k samples.
B TUNING PROCEDURE FOR RFM PROBING
We use 70/15/15 train/valid/test split on RFM training, 15 RFM iterations, and mean pooling over all timesteps. For multiclass training of simple progressions, we use 700 examples per class (there are 1100 per class in dataset, but cannot fit them given our A6000 GPU memory size. However, we note that even without all training data, we still get significantly better accuracy than baseline in this category). For all other classes, 100 random choices of hyperparameters listed below for layer-wiss wise probes and 300 for aggregation. We maximize on AUC for layer-wise probes and accuracy for aggregation.
the entire dataset for our training & validation. We use
When tuning the number of components calculated with our RFM probes, we tried a lower number of components (2-10) for categories with less data points and less perceived complexity (e.g. notes, time signatures). For categories with larger dataset size and higher perceived complexity (c.g. simple progressions, scales), we choose number of components ranging from 8 to 24.
Hyperparameter
Layer-wise
Aggregation model
Bandwidth (I.) Center gradients Exponent g Kernel Type
(o, 100)
10g2((1, 100) (False, True} ((0.7,1.4)
[False, True)
u(007, 1.4)
K2g
Kp.g
P (when kernel type is Kp.a)
u(g 2)
Regularization
logi((10055 10)
log 1og2((10-5,100 10)
Table 8: Search spaces for MusicRFM on individual layers and for the aggregation model.
Note we tune over a more general class of kernels Kp, x²) = exp(-IIx- x' ='Ilg/T.)) (indicated by the kernel type hyperparameter) for the aggregation model, which has been shown to improve the performance of RFM on tabular datasets (Beaglehole et al. 2025a). We also tune over whether to center the gradients in each iteration of RFM, which can help de-noise the gradients in high- dimensional settings (Beaglehole et al. 2025b). Gradient centering modifies the AGOP computation to give the following centered M matrix in the RFM iteration, where g Ei= 96:
M =
Σ(
(a. - 9) - 9)
(6)
13

---

Published as a conference paper at ICLR 2026
C STEERING ABLATIONS
For generation, we ablate two steering knobs that most strongly impact generation quality and con- cept alignment: (i) the effective number of layers contributing control via both flat top-k value and an exponential, score-weit weighte layer scheme ("layer pruning"), and (ii) per-timestep injection probability p that sparsifies when control is applied.
C.I
SETUP AND METRICS
We follow Sec. 3.2 and inject layerwise RFM directions into the residual stream with strength
ne(t)
17 Pe <(t) (t))))
where for ablations we (6()= and vary layer weighting and p.
C.2 ABLATING LAYER PRUNING
We study three strategies that control how many (and how strongly) layers contribute to steering: (i) exponential score-weighted steering, (ii) a simple linear score-weighted scheme, and (iti) hard top-K selection. We show results Table I0 and Table
Continuous weighting (Linear Exponential). layer weight Pt in (t) = 17u Pe ((t) using either:
Given base scale Wp. instantiate the per-
Linear:
We $e. we
Exponential:
Wi
exp wi $g
1/k
where & E (0, 1) is decay rate (smaller x increases contrast, concentrating weight on high-scoring layers). Linear is the minimal "from 1 to 0" mapping; exponential recovers linear 1 and becomes more selective as x..
Discrete selection (Top-K). layers by &g:
(K) We also ablate hard selection mask mi
f0, 1} over the top-K
m
(K)
1[ E TopK(s, K)),
w top-K wy m (K)
We sweep K {4,8, 12, 16, 24, 48) with K=48 meaning all layers.
Scheme
Hyperparams
FD 1
MMD
CLAP
Classification Acc. 1
Linear
wp8:
0.482
2.701
0.166
0.959
Exponential
K. 0.98
0.487
2.710
0.186
0.954
Exponential (ours)
0.95
0.465
2.575
0.194
0.961
Exponential
A 0.92
0.483
2.687
0.175
0.954
Uniform (naive)
0.599
3.44
0.155
0.964
Table Layer weighting ablation (continuous schemes), Exponential decay interpolates between flat (N < 1) and highly concentrated (k 0). Linear maps the best layer to wo and the worst to 0.
Top-K
FD 1
MMD 4
CLAP
Classification Acc. t
= 4
0.109
0.192
0,309
0.398
K 8
0.157
0.448
0.291
0,678
= 12
0,225
0.919
0.263
0.882
K 16
0.347
1.781
0.225
0.941
24
0,555
3.218
0.158
0.967
K 32
0.586
3.395
0.158
0,958
K = 48 (naive)
0.599
3.44
0.155
0.964
Table 10: Layer selection ablation (top-K hard pruning). K controls the effective number of controlled layers.
14

---

Published as a conference paper at ICLR 2026
C.3 ABLATING INJECTION PROBABILITY p
At each generation step t, we sample gate by ~ Bernoulli(p) and apply:
M4e dee + by ne(t) qe,
control fires stochastically with probability p. We show results in Table I1
p
FD 4
MMD 4
CLAP f
Classification Acc. f
0.15
0.108
0.163
0.348
0.348
0.30 (ours)
0.118
0.272
0.306
0.697
0,45
0.197
0.769
0.287
0.884
0.6
0,281
1.343
0.265
0.931
0.75
0.399
2.145
0.207
0.961
0.9
0.510
2.853
0.172
0.962
1.0 (naive)
0.599
3.44
0.155
0.964
Table 11:
Injection probability ablation. Lower p reduces artifacts may weaken alignment; higher p creases control strength but risks over-steering.
D SINGLE DIRECTION METRIC GRAPHS
Single Direction Metrics va Control Coeflicient by Category
FO by Control
MMD by Control Coof
0.50
Clacde
chands
Srades
Brne signinitnnn signnaums
2.5
0.30
0.20
0.5
8.10
2.15
0.0
045
1.0
0.5
0.38
6.45
0.00
CLAP Alggmmentet by Comtoll Cocl
Clossilcttoon Accuracy by Control Coof
0.50
28
$re paane
2.5
0.200
0.20
0.7
Corno Confeioon
Figure 2: dingle-direction steering metrics function of control coefficient fje- Top Left: Fréchet Distance (FD; J) increases with stronger control. Top Right: Maximum Mean Discrepancy (MMD; 1) shows a similar trend. Bottom Left: CLAAP alignment (f) to the text prompt remains relatively stable most categories. Bottom Right: Probe accuracy shows that, despite poor performance on generated data, there is an upwards trend in accuracy we increase the control coef yo..
15

---

Published as a conference paper at ICLR 2026
E CONTROL SCHEDULES USED FOR TIME CONTROL ABLATIONS ON NOTE CLASSIRICATION
(1a)(1) min max( 1500 L 0).
(1)) max 1500 0), 1).
(1) X², (A 0.998),
Glugt
i-exp! 7501/ 200)
@sin (1) = 2 (1 sin(2m//1500))
F RFM STEERING PSEUDOCODE
Algorithm MusicRFM steering
1: Input: Directions {qr,e}: control scale roo layer weights wy; schedule <(t); gate probability p.
total timesteps T',
2: Output: Generated sequence (y1....,,f") yr))
3: y {BOS}
4: for = to T do
5: 14,0 TOKENEMBED (y)
6:
for L to do
7:
hr. TRANSSORMERBLOCXX(LA..-
8:
if We > O then if Bernoull(pp) (t)
9:
then
10:
me(t) t- hi, = h4,e + ne (1) gr.c
11:
12:
end if end if
13:
14:
end for
15: y + y ① SAMPLE(hh,.)
16: end for
17: return
16


Reviews:
[{"review_id": "n7zxB1vW3J", "reviewer": "Reviewer_UgTQ", "summary": "This paper presents a training-free method to steer MusicGen with RFMs. The controllable features include tempo, chords, notes, time signatures etc.", "questions": "1. Line 257: I cannot quite understand the tempo issue. what happened to the tempo category and what specific methods did you used for tempo?\n\n2. I do not quite understand the correspondence between table 2 and the demo page. From the demo page it seems that the controllability is relatively good but the audio quality/musicality is harmed. However, table 2 shows that the quality is relatively ok but the controllability is low. Why?\n\n3. For time varying controls (changing $\\phi(t)$), do you have any demos that can produce i.e., tempo changes or time signature changes within a song? As far as I know these are very difficult for current controllable generation models. Currently, I see no time varying control in the demo page.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The idea of fine tuning-free steering of multiple music concepts in a universal way is intriguing, and the time-variant control is a useful direction. \n\n2. In the demo page it seems that some controls (e.g., augmented chords; interval 6) are effective. But it is strange why the probing results are low (see questions 2).", "weaknesses": "1. The model focuses on global controls that are described by labels. This is not a novel task and can be done by many fine-tuning methods, using either text-based or other controls. The methodology itself is not novel either.\n\n2. The performance of the model seems to be low. Previous works seem to provide better controllability and musicality as in Wu et al (2024) & Lin et al (2023). There is no comparative experiments against previous fine-tuning methods either.\n\n3. Sec 5.1: For some tasks like notes, chord etc., a pretrained model (e.g., audio chord estimator) provides a better evaluation metrics compared to subjective and objective (by reusing the prober) test.\n\n4. Line 243: incomplete sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9pFDy44H7N", "reviewer": "Reviewer_h1Zj", "summary": "This paper introduces MusicRFM, a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained control over frozen autoregressive music generation models like MUSICGEN. By training lightweight RFM probes on the SYNTHEORY dataset, the authors extract interpretable \"concept directions\" corresponding to musical attributes (e.g., notes, chords, tempos). These directions are injected into the model's activations during inference to steer generation without retraining or per-step optimization. Key innovations include layer-based pruning (top-K and exponential weighting), time-varying schedules (e.g., linear fades, sinusoidal modulation), and multi-direction steering for simultaneous control of multiple attributes. Experiments demonstrate improved control accuracy (e.g., note classification from 0.23 to 0.82) with minimal impact on text prompt fidelity (CLAP score within ~0.02 of baseline), supported by quantitative metrics (FD, MMD, CLAP) and a small listening test.", "questions": "1. Could the authors address the underestimation of related work? For instance, how does MusicRFM compare directly to PEFT-based methods in terms of computational cost and control granularity? Also, why were papers like Zhu et al. (2025) and Zhang et al. (2024) not discussed, given their focus on inference-time control in music generation?\n\n2. To improve readability, could the authors suggest additions to the methods section, such as pseudocode for the steering injection process or a simple worked example of a time schedule (e.g., linear rise) applied to a generation?\n\n3. Regarding the trade-off between control strength and metrics like FD/MMD, do the authors have plans for larger-scale subjective evaluations of audio fidelity and musicality? How might this affect the method's usability in real-world applications, and are there mitigation strategies?\n\n4. For the listening test, could the authors provide more details on the protocol? Specifically, what were the participant demographics, expertise levels (e.g., musicians vs. general listeners), and how were samples randomized?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a solid paper that explores the application of Recursive Feature Machines (RFMs) to music generation, specifically for steering codec-based autoregressive models like MusicGen. \n\nThe work addresses an important challenge in controllable music generation by enabling fine-grained, interpretable control over musical attributes such as notes, chords, intervals, scales, progressions, tempos, and time signatures, without requiring model retraining or heavy optimization. \n\nThe use of the SYNTHEORY dataset for probe training is well-motivated, as it provides clean, music-theoretic labels. \n\nThe proposed extensions—layer pruning strategies (top-K and exponential weighting), dynamic time schedules, and multi-direction steering—are novel and practical, allowing for more robust and flexible control. \n\nExperiments are comprehensive, including classification results showing RFMs outperforming baselines (e.g., average score 0.942 vs. 0.929 for SYNTHEORY FFNs), quantitative metrics on steering trade-offs (FD, MMD, CLAP, probe accuracy), and a listening test demonstrating perceptual improvements. \n\nOverall, the paper makes a meaningful contribution to activation-level steering in generative audio models, with potential for broader applicability.", "weaknesses": "First, the related work section underestimates prior methods by claiming they require \"intense finetuning runs,\" when many actually use parameter-efficient fine-tuning (PEFT) techniques, which are computationally lighter than full finetuning (though still more than RFMs). This portrayal lacks objectivity and overlooks the fact that some methods already achieve control over notes and chords without architectural gaps. \n\nAdditionally, relevant papers are missing, such as Zhu et al. (2025) on efficient fine-grained guidance for diffusion-based symbolic music generation via inference-time control, and Zhang et al. (2024) on zero-shot text-to-music editing with diffusion models, which also focuses on inference-time interventions. Second, although the writing is clear overall, the methods section can be dense and challenging to follow, particularly the details on RFM adaptations, layer pruning, and schedule formulations; more explanatory text or examples would help. \n\nThird, objective experiments show that increasing steering strength degrades FD and MMD scores, which seem to imply a degradation in musicality as well—objective metrics like FD and MMD are designed to capture distributional shifts that often correlate with perceptual quality, suggesting that stronger steering may introduce artifacts or incoherence that harm the overall musical experience. This is my biggest concern, as it raises questions about the method's ability to maintain high-fidelity generations under aggressive control, yet the paper lacks subjective evaluations of fidelity and musicality beyond the small listening test—relying on proxies like CLAP may not fully capture these aspects, potentially limiting the method's practical value. \n\nFinally, the listening test protocol uses only 12 participants, which is insufficient for robust conclusions, and omits details on participant selection criteria, demographics, or distribution.\n\n[1] Zhu, T., Liu, H., Wang, Z., Jiang, Z., & Zheng, Z. Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation. In Forty-second International Conference on Machine Learning.\n\n[2] Zhang, Y., Ikemiya, Y., Xia, G., Murata, N., Martínez-Ramírez, M. A., Liao, W. H., ... & Dixon, S. (2024). Musicmagus: Zero-shot text-to-music editing via diffusion models. arXiv preprint arXiv:2402.06178.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEbVe6qkmi", "reviewer": "Reviewer_6iX2", "summary": "This paper introduces MusicRFM, a framework for activation-level steering of pre-trained autoregressive music models. Building on RFMs, the authors identify interpretable “concept directions” in MusicGen’s hidden states that correlate with musical attributes such as notes, chords, or tempo. These directions are injected into the model’s residual stream during inference, allowing fine-grained control without retraining or step-wise optimization. Experiments on synthetic datasets (SYNTHEORY) and real-music benchmarks (MUSICBENCH) evaluate classification accuracy, FAD/MMD/CLAP metrics, and small-scale listening tests.", "questions": "- Table 1: what is the implication of performance difference between proposed and RFM (last token)?\n- Table 2: mae for Tempos?\n- Table 2: Classification results were very high from Table 1. Doesn’t this mean it’s a bit reliable to trust the metric?\n    1. If the authors claim it’s not reliable for real music (according to Table 2 caption), then correlation between human evaluation should be included to observe how different it is and find the best way for evaluation (e.g., models specific for each downstream task)\n    2. If this Probe Acc is reliable, then the successful controllability is below chance level for all tasks except for Notes. Is this acceptable? Especially since there’re no other baselines to compare.\n- details of participants of listening test? and 3 questionnaire seems too small\n- what’s the benefit compared to diffusion-based controls\n- so what is the best $\\eta_0$? and any way to automate this for each steering direction?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed approach enables interpretable, fine-grained control over musical attributes such as notes, chords, and tempo without retraining or per-step optimization. The introduction of layer-aware steering, time-varying schedules, and multi-directional control makes the framework flexible and musically relevant. Empirical results demonstrate effective controllability with minimal loss of fidelity, showing a clear advancement toward interpretable and lightweight control in music generation systems.", "weaknesses": "The paper lacks essential experimental details, including the datasets, prompts, and number of samples used for evaluation. While the experiments appear to partially rely on the SynTheory dataset, this is never explicitly stated or described in detail, making it difficult to assess reproducibility. In addition, the absence of a direct baseline would be the critical issue of the paper. Most naively, authors can compare the proposed steering approach to a simple text-conditioning (e.g., simply adding “fast music” to the input prompt) limits the clarity of the paper’s contribution. Such a comparison is crucial to demonstrate the unique benefits of activation-space control over standard prompt-based methods.\n\n**missing references**\n\n- citation for metrics? (FD, MMD, CLAP) it’s not mentioned of which CLAP model was used for evaluation\n\n**minor**\n\n- Table 1: correct the caption (”We train using 7 We report…”) and place the proposed method at the most bottom row\n- reporting FD and MMD together is a bit redundant\n- Table 2 caption: “higher better for” → “higher is better for”\n- line 315: “were randomly chosen base model …” → “were randomly chosen from base model ...”\n- couldn’t find listening examples for time-based schedules from the demo page upon time of reviewing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)