Discovering and Steering Interpretable Concepts in Large Generative Music Models
Authors: Nikhil Singh
Manuel Cherep
Patricia Maes
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretability
Generative Models
Music
Mechanistic
Sparse Autoencoders
Primary area: interpretability and explainable AI

Abstract: The fidelity with which neural networks can now generate content such as music presents a scientific opportunity: these systems appear to have learned implicit theories of such content's structure through statistical learning alone. This offers a potentially new lens on theories of human-generated media. When internal representations align with traditional constructs (e.g. chord progressions in music), they show how such categories can emerge from statistical regularities; when they diverge, they expose limits of existing frameworks and patterns we may have overlooked but that nonetheless carry explanatory power. In this paper, focusing on autoregressive music generators, we introduce a method for discovering interpretable concepts using sparse autoencoders (SAEs), extracting interpretable features from the residual stream of a transformer model. We make this approach scalable and evaluable using automated labeling and validation pipelines. Our results reveal both familiar musical concepts and coherent but uncodified patterns lacking clear counterparts in theory or language. As an extension, we show such concepts can be used to steer model generations. Beyond improving model transparency, our work provides an empirical tool for uncovering organizing principles that have eluded traditional methods of analysis and synthesis.

Published as a conference paper at ICLR 2026
DISCOVERING AND STEERING INTERPRETABLE CON- CEPTS IN LARGE GENERATIVE MUSIC MODELS
Nikhil Singh"), Manuel Cherep"? Pattie Maes Dartmouth College, MMI.." Equal contribution.
musicdiscover.meddiaii.ccc
Activation Extraction
Unsupervised Feature Discovery
Automatic Labeling
D(Topki k())
Max. Activating Examples
Generanive Labelig
Sparse Fonture Motrir
H+
Music Corpus
Enc.
Dee
Chasuifee Baxed Labeling
Exampler
Filoring
Drim Roll
Resithall Soreom
Acticationn
Misic Generation Model (eg. MusicGen)
Seeving
Silence
Figure I: Multi-stage pipeline for discovering and steering interpretable concepts in autoregressive music models. (1) Music from a large corpus passed through a pre-trained generator to extract activations from multiple layers. (2) Sparse autoencoders reconstruct activations (also usable for steering), and features are filtered to retain the most viable candidates. (3) Retained features are characterized via musical examples and labeled using generative labeling with multimodal LM and classifier-based labeling with pre-trained models.
ABSTRACT
The fidelity with which neural networks can now generate content such as music presents scientific opportunity: these systems appear have learned implicit theories of such content's structure through statistical learning alone. This of- fers potentially new lens theories of human-generated media. When internal representations align with traditional constructs (c.g. chord progressions in mu- sic), they show how such categories can emerge from statistical regularities; when they diverge, they expose limits of existing frameworks and patterns we may have overlooked but that nonetheless carry explanatory power. In this paper, focus- ing autoregressive music generators, introduce method for discovering interpretable concepts using sparse autoencoders (SAEs), extracting interpretable features from the residual stream of transformer model. We make this approach scalable and evaluable using automated labeling and validation pipelines. Our re- sults reveal both familiar musical concepts and coherent but uncodified patterns lacking clear counterparts theory or language. As an extension, we show such concepts be used to steer model generations. Beyond improving model trans- parency, our work provides an empirical tool for uncovering organizing principles that have eluded traditional methods of analysis and synthesis.
INTRODUCTION
When humans create, draw upon rich vocabulary of concepts. Some of these we can name and explicitly reason about; terms like "symmetry" or "gradient" in the visual arts, for instance. Others shape media long before we have language for them: consider how the "Hero's Journey" narrative structure infused myths, legends, and epics across cultures for millennia before Joseph Camphell famously theorized it in the 20th century (Camphell) 1949). This gap between practice

---

Published as a conference paper at ICLR 2026
and theory reflects more fundamental pattern: our ability to recognize and use structure often precedes our ability to describe it.
This phenomenon poses interesting puzzle for machine learning. Deep generative models produce remarkably convincing content through statistical learning, suggesting they learn coherent internal representations of the building blocks of such content. How do we bridge the gap between these models' raw statistical power and the structured conceptual vocabularies humans use? The answer to this could impact whether generative models will remain opaque mimics or become genuine creative collaborators that can engage with and perhaps even extend human creative frameworks.
Consider what it means for model to "understand" musical concept like chord progressions. Some neural pattern must respond to and enable their generation, yet this pattern might not cor- respond neatly to our theoretical description. Connecting our hypotheses to generative models' opaque mechanisms could enable useful new interactions through shared analytical and synthetic vocabularies. Here, music offers a compelling test domain: it combines centuries-long theoretical vocabulary with perceptual patterns resisting verbal description, and exhibits both statistical regu- larity and cultural variety. It also lacks large-scale data (e.g. paired music-text) that might guide concept discovery. Many efforts have sought to align music generation with human intent, making this a challenging yet useful case for concept discovery in generative models.
This paper introduces method using sparse autoencoders to extract features from transformer-basee music model activations. We demonstrate how these features reveal both familiar patterns aligning with traditional concepts and implicitly learned regularities that elude clear verbal description. In summary, this work makes the following contributions:
1. A general pipeline for unsupervised concept discovery in autoregressive music models. We show how sparse autoencoders applied to MusicGen residual streams reliably surface interpretable features, extending recent interpretability techniques beyond text and vision into audio/music. This is the first application of SAEs in audio, to our knowledge.
2. Automated large-scale evaluation of discovered features. We combine multimodal LLM labeling, pretrained audio classifiers, cross-modal alignment, and other techniques to auto- matically name and score thousands of latent musical concepts at scale.
3. Evidence of well-known and subtle musical concepts. Our method recovers familiar categories (e.g. genres, timbres, instruments) while also surfacing coherent patterns not yet well-described by theoretical constructs, demonstrating that these models may encode musically meaningful structures beyond human- moonand-roviedd labels.
4. Empirical insight into layer- and model-size effects. We show how concept interpretabil- ity and distinctiveness vary with layer depth, sparsity/expansion settings, and model scale, contributing evidence to the study of feature localization in generative models.
5. Demonstration of feature steering in generation. We provide proof-of-concee steering experiments showing that discovered features can be directly manipulated to alter genera- tion outputs, establishing practical utility of the approach for controllable generation.
2 RELATED WORK
2.1 STATISTICAL LEARNING OF MUSICAL STRUCTURE
Computational approaches modeling music span rule-based systems, probabilistic models, and modern neural networks. Early Markov models captured local dependencies like chord transitions but failed at longer-range structure (Conklin| 2003). while coormation-theorette analyses have re- vealed statistical bases for harmonic categories (Jacoby et al.) 2015). For several years, neural architectures have achieved impressive results in both symbolic and audio modeling (Huang et al., 2018 Van Den Oord et al., 2016)] Mehri et al.) 2016). More recent systems often model audio di- rectly with transformers or diffusion approaches (Agostinelli et al.) 2023 Copet et al., 2024 Liu et al. 2023 Huang et al. 2023). Yet as these models generate increasingly sophisticated music, understanding of what musical knowledge they acquire lags behind. The progress in generation has coincided with greater opacity of internal structure, motivating central question: what do these models actually learn about music? Our work seeks to make progress on answering this question (see Appendix G for additional music-theoretic context).
2

---

Published as a conference paper at ICLR 2026
2.2 INTERPRETABILITY THROUGH FEATURE EXTRACTION
Interpreting neural network computations has long been central goal. Recent work shows that LLM representations often encode concepts in surprisingly localized ways (Templeton et al. 2024), echoing classic sparse coding results where mammalian visual cortex was modeled learning effi- cient codes for natural images (Olshausen & Field 1997]. This broader lesson, namely that sparse latent structure can be computationally and biologically advantageous, has motivated the develop- ment of interpretability methods. A recent line trains sparse autoencoders (SAEs) (Mallat & Zhang, 1993 'Templeton al.) 2024) on transformer activations, enforcing sparsity to extract interpretable components. Formally, with encoder E and decoder D, one optimizes:
min E - D [[) E
This constraint is thought to encourage discovery of reusable "atomic" concepts rather than mem- orized reconstructions. Learned features can then be labeled using automated methods (e.g. LLMs applied to maximally activating sequences). Extending such approaches to music generators is non- trivial, given their hierarchical temporal structure and mixed discrete-continuuus features. Our work develops adaptations to apply these principles to the musical domain.
2.3 INTERPRETING MUSIC GENERATION MODELS
Given a vocabulary of known concepts and example data to match, one strategy toward interpreting music generation models is probing. While probing has rich history in NLP, work has recently begun to apply it to music models. Prior work (Wei et al.; 2024a) introduced the SynTheory dataset to systematically probe music foundation models on core music theory concepts like tempo, in- tervals, and chords, revealing that certain layers capture these concepts more strongly than others. Similarly, other work investigates chord and pitch representations in MusicGen and find that musical concepts become more evident in deeper layers. yet may not be linearly separable without additional interventions in the hidden space (Ma & Xial 2024). Beyond these direct probing and manipulation techniques, recent work (Vásquez et al. 2024 uses the DecoderLens (Langedijk et al. 2023) method to interpret intermediate activations, offering an auditory perspective on what each layer "hears" and how it evolves through the network. Such approaches exemplify emerging trend: there is increas- ing interest in understanding how these models do what they do. However, these approaches have primarily focused on probing for known concepts, important but limited strategy. This raises key question: what structures might we be missing by focusing only on concepts we already know to look for? To answer this, we propose an unsupervised concept discovery pipeline. In summary, probing asks "do models encode concept we already know?", while our pipeline asks "what concepts do models encode?" without supervision.
2.4 AUTOMATED CONCEPT DISCOVERY AND EVALUATION
A central challenge in interpretability is to discover concepts model has learned organically, rather than imposing them a priori. Recent frameworks address this across domains. Both [Simon & Zou (224)) and Gujral et al. (2025) train sparse autoencoders on a protein language model (ESM-2), uncovering thousands of latent features, many aligning with known biological concepts and others appearing novel. Concept Bottleneck Models (CBMs) (Koh et al. 2020) instead constrain interme- diate layers to discrete, named concepts, though typically with a hand-specified set. Recent variants invert this paradigm, using SAEs to first discover latent concepts and then name them automati- cally or automaaltmat (Rao et al. 2024). While developed outside of music, these principles broader: generative music models likely encode diverse concepts without explicit labels. Auto- mated discovery and evaluation of such features could reveal how these models "understand" music, and enable finer control over generated outputs.
3 METHODS
Our discovery and steering pipeline is illustrated in Figure]
3

---

Published as a conference paper at ICLR 2026
3.1 DATASET AND ACTIVATION EXTRACTION
The first step in our pipeline is to obtain a dataset of music clips for which we can obtain activations from the generative model. We use the MusicSel (Wei al. 2024b) dataset for this, which is collection of around 160,000 samples, most of which are R TO seconds long. MusicSet is a combina- tion of data from MTG-Jamendo (Bogdanov al., 2019). MusicCaps (Agostinelli et al. 2023), and MusicBench (Melechovský et 2023). All the source datasets are Creative Commons--lcensse (though only MTG-Jamendo explicitly states that this applies to the audio). We chose this dataset because it offers a comparatively diverse range of genres, instrumentation, and musical characteris- ties at a larger scale than many alternatives. We also found the data to be more stylistically diverse and high-quality than FMA (Defferrard et al. 2016), another popular option. We produce activations by feeding the music into two pre-trained MusicGen (Copet et al., 2024) models: MusicGen MurgeGen-Larre (henceforth, MGL) and MusicGen-Smie Small (henceforth, MGS). For each, we extract activation vectors from five residual stream layers: the early layer (layer 2), late layer (second-to-le to-ssss, middle layer, and layers at 25% 50%, and 75% depth. This corresponds to Layer
{2, 6, 12, 18, 22), and MGL E (2, 12, 24, 36, 46). Activation dimensionality is 1024 for MGS and 2048 for MGL. Note: we use unconditional audio; see Appendix B for more details.
3.2 TRAINING THE SPARSE AUTOENCODERS
We train series of SAEs to extract interpretable features from the residual stream activations of MusieGen. Each is trained to reconstruct the original activations x E Rdd from a learned sparse latent representation h E Red where c is an expansion factor (note: this means the latent dimension is typically larger than the input dimension for such SAEs).
The SAE architecture generally consists of an encoder (h- ReLU( e + b.)) and decoder (x = Wah + ba), both implemented as single linear layers where W. Re-dxa Wa E Rdxed b E R'd and ba E Rd are the learned weights and biases. To enforce explicit sparsity, a s-sparse projection operator Ps Rnd + Rdd to h. yielding the sparse latent code 2 = (h).
apply
These are typically called k-sparse autoencoders (Makhzani & Frey 2013 Gao et al. 2024) Where Se is the set of indices of the top-k values E h:
[h; h4 if h4 E (h)
z = Pb(h)= =
o otherwise
We minimize the mean squared reconstruction error, subject to the sparsity constraint. We experi- ment with expansion factors E E {4, 32} and sparsity levels & E {32, 100). all of which are values encountered in prior SAF literature (Gao et al. 2024 Simon & Zou 2024
3.3 FEATURE MAPPING AND FILTERING
After the SAE has trained, its latents presumably correspond to meaningful musical features. Of course, the model itself does not reveal what these are. We need some way work backwards from these latents to objects whose content we can perceptually interpret, in relation to particular latent dimension learned by the SAF, conventional approach is to use max activating examples. In the case of language models, these are token sequences (prompts, perhaps) that most significantly activate a given feature. In the case of music, gets a little trickier. Unlike text, tokens here represent an almost imperceptible chunk of music: about 20 milliseconds. Even with handful of these, interpreting a feature becomes quite challenging.
To characterize the relationship between learned features and input tracks, we compute per-track summary statistic for each feature based on its activation profile over time. Let fa denote the i-th leamned feature, and let at, = (os.j.1...., At,,,T, E RT; xd represent its activation time series over
the T3 time steps of track j. The mean activation of feature f. in track j is: P4,j = Tj E' anjja- This gives us scalar estimate of the feature's 's average prominence over the duration of the track.
To support downstream interpretation and analysis, we then identify a subset of features whose acti- vation patterns suggest both selectivity and relevance. The goal is to exclude features that are either inactive, overly common, or too rare to support generalizable interpretation. Let 6,,5 E {0. 1} denote whether feature fa exhibits nonzero average activation, which corresponds to nonzero activation at
4

---

Published as a conference paper at ICLR 2026
any time step in track j: ōi,j ( (maxigest, Oi,i 7) where 2 0 is small activation thresh- old (we set 7 = 0). We define the corpus-level activation rate of feature fi as r; N Σ}= 55, where is the number of validation tracks.
We discard feature fi if any of the following conditions hold: (a) Inactive: T3 0: the feature never activates the validation set. (b) Excessively Ubiquitous: 7; > m with m = 0.25; the feature activates in more than 25% of tracks, suggesting diffuse or polysemous behavior that limits interpretability. (c) Excessively Obscure: < ri < Omin. with mmin = 0.01; the feature activates in fewer than 1% of tracks, indicating insufficient coverage support reliable interpretation. This filtering step seeks to ensure that only features with non-trivial, non-saturated, and sufficiently fre- quent activation patterns are retained for subsequent analysis. Thresholds are heuristic and informed by prior work in neural feature sparsity and interpretability (Simon & Zou 2024h.
3.4 SELECTING EXAMPLES FOR LABELING
One challenge of labeling concepts relates to rypicality. Extreme values (e-g., the maximum activat- ing example) might not represent the typical set of the distribution of examples that activate a given feature. Let E, = {-r [xxx > 7} denote the set all inputs x that activate feature j with suffi- cient strength [ (ar) (as before, we set r 0 as a conservative threshold). Instead of relying single max-activating example, we select the top 10 highest-activatiin examples, Sy C E.y. and infer the feature's ' label from these. The choice of [S]] 10 is motivated by balance between statisti- cal robustness and noise from weakly activating examples. A larger activating subset improves the likelihood of capturing representative sample from Ej, but diminishing returns arise activation strength decays, especially under the sparsity constraint wherein the distribution might be truncated. Our heuristic aims approximate the mode of the feature's natural stimulus distribution.
3.5 AUTOMATED INTERPRETABILITY
Now that each feature can be represented by its top-activating examples, it must be labeled. Human labeling feasible but does scale. With expansion factor 32x d-2048 layer, an SAE yields 32d=65,536 features; if only ~ 1% activate, that is R 655 features. At 10 examples x 10s per feature, this is ~65,500 (=18.2 h) of listening for one SAE, excluding deliberation, and the cost multiplies across layers and settings. A scalable alternative is required.
Though prior automated interpretability methods use language models to explain linguistic concepts, musical concepts present potentially unevenly across much longer sequences and lack mature music- language models to use in such setup. As such, to generate descriptive, meaningful labels for the discovered features, we use multi-step pipeline:
1. A generative label proposal strategy. We query a large multimodal model (i.e., Gemini Flash 1.5, but explored others in Appendix B with concatenated audio of the top-10 max activating examples for each feature (see prompt in Appendix H). We instruct the model produce set of concept tags, confidence scores for each, and an overarching label and description for the feature.
2. A classifier-bused strategy where extract set of tags (see Appendix D for each fea- ture using pre-trained Essentia (Bogdanov et al. 2013 Alonso-Jiménez et al. 2020) audio classifier models. Multiple top predicted tags from each model are candidate labels.
3. We use CLAP (Wu et al.. (023) to compute semantic alignment between suggested labels and activating examples. This provides a quantitative measure of how well each concept label aligns with the audio content of the feature.
3.6 HUMAN VALIDATION
Ultimately, given the complexity of musical signals and the possible ambiguity of features, we conducted human validation study (IRB approved) to assess labeling quality. For each feature, participants were presented (see Appendix F with three audio examples (sampled from SAES with high valid feature counts) and the candidate labels. They selected the label that best captured the commonality across the examples, rated their confidence in that choice, and provided an optional free-text label.
5

---

Published as a conference paper at ICLR 2026
3.7 GENERATION STEERING
We then seek to test whether we can use discovered features to sleer the model's generation. Steering modifies the forward pass by adding scaled feature vectors to the residual stream the SAE's hook point. For a given SAE feature j. we perform steering by adding the corresponding decoder weight vector W d, the residual stream activations during generation: x' = B W d, where are the original residual stream activations, E (0, 1) is the steering strength, and / is the maximum activation strength for feature j computed over the feature j's max. activating examples.
Experimental Setup. We evaluate steering using neutral prompt "Simple melody" (as in Arad al.] (2025)) for steering strengths for each feature to test the "steerability" of concept. We used MusicGen-Large SAEs with exp. factor 32, {32, 100), and middle to late layers (24, 36, and 46), based on feature statistics.
0.0 (unsteered baseline) and 1.0 (max. steered version)
Feature Selection and Evaluation. We identify the most steerable features by computing the cosine similarity of CLAP embeddings (Wu et al., 2023) for each feature using cosine similarity between the feature's top 10 activating examples and (I) the steered example, and (2) the baseline.
4 RESULTS
4.1 STATISTICS OF DISCOVERED FEATURES
We first quantitatively characterize the distribution feature activation patterns across the valida- tion set, and the impact of SAE hyperparameters the number and nature of discovered features. More details are provided Appendix
Before Filtering Initially, feature activations show a heavy-tailed distribution: some fire in many tracks, while most activate only rarely (see Figureßßfor details). SAE hyperparameters appear to me- diate this balance, i.e. larger & expansion factor ()F) increase overall recovery but also changes the activation frequency distributions. This imbalance makes raw feature counts misleading and motivates our explicit filtering approach, to isolate features that both selective and interpretable.
After Filtering Table reports the number of features that survive filtering across models, layers, and SAE settings. For MGL, certain configurations yield thousands of retained features, while MGS rarely produces more than 100. Expansion factor also has predictable effect: higher EF values increase feature counts, reflecting the greater representational pressure imposed by the wider latent space. Interestingly, we find that this combined well with larger k. possibly suggesting that musical clips tend to have many co-occurring activating features (e.g. in different layers and at different time- steps). Early layers (c.g. L2) largely produce more features than later ones, though interpretabilityy does not necessarily track count. The consistent difference between MGL and MGS indicates that scale does more than add parameters: it appears alter the internal organization of representations ways that make interpretable structure more extractable. Together, these patterns show that feature discovery depends jointly the SAE and the underlying model's representational regime.
Table 1: Feature counts across models, layers, and SAE configurations (total 4697). For each model layer, the configuration providing the max. number features bolded.
MusieGen Large
MusicGen Small
Exp. F
k
L.2
L12
L24
L36
L46
L2
L6
L12
L18
L22
32
12
0
4
0
0
0
32
32
30
117
149
135
71
3
7
4
7
4
100
407
109
147
28
25
6
3
0
0
32
100
2344
222
412
131
177
59
47
4
17
4.2
EXAMPLES OF FINDING CANONICAL MUSICAL CONCEPTS
Retained features often correspond directly to established musical constructs, showing us how these encoded within the model. Figure (left) shows manually annotated examples from MGL::

---

Published as a conference paper at ICLR 2026
Elec. Beeps & Boops
Talko
Hardstyle/
Instr.
Techno
Note
Baroque
Oscillating Bell-Like
Harps.
Guilar Solo
MIDI
Piano
Examples of features discovered using the sparse autoencoders we train. Note: these ex- amples are labeled manually. Spectrograms highlight similarities across examples within a concept.
Figure 2:
1. Taiko Drums (k=32, EF=4): activates resonant Taiko and other low-pitched drums, isolating a timbral family recognizable in theory and practice.
2. Hardstyle Techno (k=32, FFF=32): consistently aligns with the hardstyle subgenre, cap- turing distinct rhythmic and production signature.
3. Baroque Harpsichord (k=32, EF=32): isolates the plucked timbre and contrapuntal tex- ture of harpsichord repertoire, as well as similar stylistic cues in Baroque string writing.
4. Rock Guitar Solos (k=32, FF=4): activates on electric guitar solos, integrating timbre, ornamentation, and melodic phrasing characteristic of the style.
These cases demonstrate that SAEs can discover categories that existing vocabulary already recog- nizes as meaningful. The match between latent features and canonical concepts indicates that the activations are not arbitrary artifacts but can and do internally encode distinctions salient to musi- cians and listeners in ways we can localize and recover from their internal activations.
4.3 EXAMPLES OF EMERGENT MUSICAL REGULARITIES
Not all features map neatly established verbal categories. Some capture patterns that are percep- tually coherent but poorly described in existing theory (Figure right):
1. Electronic Beeps and Boops (k=32, EF==4): activates on diverse synthetic tones and glitches, class of sounds central to electronic genres but not theoretically well-defined.
2. Single Instrument, Single Note (k=32, EF=32): isolates sustained single-note events across instruments, suggesting that the model may have detectors for basic atomic units of musical texture.
3. Oscillating Bell-like Timbres (k=32, EF=32): responds to bell-like sounds with beating or oscillations, pointing to sensitivity to fine-grained spectral phenomena.
4. Romantic Poppy MIDI Piano (k=32, FF=4)): activates on MIDI piano in pop-ballad contexts, apparently tuned to performance artifacts like rigid quantization and compressed dynamics that co-occur with the poppy, romantic stylistic frame.
These examples surface coherent musical regularities that not fully described by standard mu- sical terminology likely to be found in prompts or specified in probing experiments. They show that models reflect production practices, timbral subtleties, or emergent stylistic groupings. Identifying such fea- tures provides concrete evidence that generative models encode structure beyond what human-led interpretability, such as probing, is likely to anticipate. In turn, analyzing situations in which these features arise may prove useful for building them into coherent theoretical constructs when studied in detail over time by music theorists.
limited reproducing canonical categories but also carve out distinctions that
4.4 CONCEPT REPRESENTATION
We next examine how the discovered features are distributed across layers and model scales, and how these patterns connect prior findings.
7

---

Published as a conference paper at ICLR 2026
Escntia Tage
Scoe:: Gamini Concepts
Name
Scote: Gemini Stammary
CLAD Score
MCL
MGS
Relative Depth
Figure 3: Avg. CLAP (Wu et al. 2023) score across layers, comparing feature audio automatic concept labels. For MGL, later layers appear to produce more interpretable features on average.
Do later layers encode more interpretable features? Figure shows average CLAP scores be- tween feature audio and their automatically generated labels, stratified by layer. For MGL, deeper layers yield higher scores, indicating that their features are more readily aligned with human- interpretable concepts. This mirrors probing results from (Ma et al. 2024), which showed that musical properties such chords and roots more strongly encoded in deeper transformer layers. Our results generalize that observation: later layers not only encode specific theory-driven constructs more clearly, but also produce features that can be labeled more consistently across diverse concepts.
Does model scale change how features are organized across layers? To test whether larger models produce more layer-specific features, we trained a simple MLP probe to predict the layer of origin from a feature's 's activation profile. Accuracy substantially higher for MGL (50.29 29 + 3.41) than for MGS (40.51 3.60). This means that features in MGL are more distinctive by layer, whereas those in MGS reflect less differentiated representational structure. These results extend probing-based findings on synthetic datasets (Wei et al. 2024a): scale not only increases the number of recoverable features, but also sharpens the division representational roles across layers.
How do features co-activate? In Appendix analyze feature co-activation patterns, showing examples of cross-layer concept connections which emerge from SAE training.
4.5
AUTOMATED INTERPRETABILITY
Fears, (Score' x)
1006
Agg: Min
Agp
Agg:
Score Source
796
Esentia Tags
Ganini Conoati:
296
Gemini Name
Ganini Semmary
CLAP Score (X)
Figure 4: Distribution of max. CLAP scores across all SAEs. Pooling both Gemini- and Essentia- produced labels, we score them using CLAP, showing the tradc-off between confidence and coverage different potential filter levels.
We assessed label quality using CLAP (Wu al. 2023) alignment between feature audio and assigned labels. Figure 4 shows the distribution of maximum scores across all SAEs. Essentia tags achieve stronger alignment than Gemini with considerable overlap between the two. Overall, single labeling strategy dominates. CLAP scores also provide practical mechanism for filtering: thresholds can be tuned to trade off label quality against feature coverage without exhaustive manual review, though the optimal cutoff may depend corpus and task.
CLAP Scores
Human Evaluation To compare methods directly, evaluated 400 features per pipeline with 80 participants (40 per method, 10 features each). Participants chose the best label, rated confidence, and could supply alternative labels. Confidence Gemini (3.19/5; 47% >4), suggesting that while multimodal LLMs can generate open-ended labels beyond fixed taxonomies, classifier-based tags may be reliable in practice.
higher for Essentia (3.96/5; 71% 24 than for
8

---

Published as a conference paper at ICLR 2026
Baseline
Feucure Excampees
Sterred
Spndance
4150
Autolabel
Apurssice
oria
Figure 5: Examples of steered features. Note: these examples were labeled automatically. (Base- line) Generation without steering for "Simple melody." (Feature Examples) Top max. activating examples for the steering feature. (Steered) Generation steering with the same prompt and seed as the baseline, and maximum strength empirically calculated from the maximum activations. The steering shows close alignment with the feature examples, as seen in the spectrograms.
4.6 STEERING WITH DISCOVERED CONCEPTS
We next tested whether the features uncovered by our SAEs be used for controllable generation via steering. This is a strong intervention: there is no guarantee that every discovered feature should be steerable,
that steering will always preserve perceptual coherence. Limits steering via SAES have heen observed in prior work (Wu et al. 2025). Thus, the goal of these experiments is to establish the existence of steerable concepts.
Table 4.6 summarizes results. Depending on SAE configuration, 15-35% of tested features had improved CLAP alignment with their top-activating examples under steering (with prompt "Simple Melody"). While this may appear modest, it demonstrates that a non-trivee non-trivial fraction of discovered features correspond causal, manipulable directions in activation space even under automatic label- ing: actionable for controlling outputs. Figure illlustrates such cases, where steering shifts outputs toward the intended feature class while holding prompt and seed fixed. This establishes the potential for SAE-driven stecring in controllable generation settings, though primary goal in this work remains feature discovery (Peng et al. 2025).
Table 2: Proportion features per SAE with positive steering improvement.
Model
Exp. F
k
Layer
Steering Improvement
MGL.
32
100
24
96/408 (23.5%)
MGL
32
100
36
46/131 (35.1%)
MGL
32
100
46
27/177 (15.3%)
MGL
32
32
24
44/149 (29.5%)
MGL
39/135 (28.9%)
MGL
32
32
46
16/71 (22.5%)
conducted a listening study with 10 participants to each listen to sets clips; each containing baseline, steered version, random-direction matched-norm steered version, and representative of the steering target, asking them to match the candidates (baseline, randomly steered, and SAE steered). Each participant's 10 sets were equally sampled from the total of top 50 steerable features. Participants largely selected the SAE-steered audio (66/100, compared to 17 each for baseline and random; x2 48.02, p < .0001). This suggests that the steering effects clearly perceptible.
CONCLUSION
Our results demonstrate the feasibility and potential of using sparse autoencoders to discover inter- pretable concepts in large music generative models, i.e. using models for musical discovery (Singh et al. 2024). We have shown that these models learn a rich set of musical patterns, ranging from well-established concepts emergent regularities that warrant further investigation. Through future work in this area, we believe is possible to build a comprehensive understanding how neural networks represent and generate music. Our pipeline offers scalable path towards this goal.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We gratefully acknowledge AWS for providing computational resources through the AWS Research and Engineering Studio (RES) pilot program. We thank Brian McCarthy and Jared Novotny for their valuable support, and Peter Fisher (MIT ORCD) for making this collaboration happen. The authors acknowledge the MIT Office of Research Computing and Data for providing high perfor- mance computing resources that have contributed to the research results reported within this pa- per. MC is supported by fellowship from "la Caixa" Foundation (ID 100010434) with code LCE/BQ/EUU23/2200077 We also thank Michael Casey, Anna Huang, and the HAI-Res lab for supportive comments.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301 1/355, 2023.
Pablo Alonso-Jiménee Dmitry Bogdanov, Jordi Pons, and Xavier Serra. Tensorflow audio models in essentia. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 266-270. IEEE, 2020.
Dana Arad, Aaron Mueller, and Yonatan Belinkov. Saes are good for steering-if you select the right features. arXiv preprint arXiv:2505.20063, 2025.
Dmitry Bogdanov, Nicolas Wack, Emilia Gómez, Sankalp Gulati, Perfecto Herrera, Oscar Mayor, Gerard Roma, Justin Salamon, José Zapata, and Xavier Serra. Essentia: an open-source library for sound and music analysis. In Proceedings of the 21st ACM international conference Mul- timedia, pp. 855-858, 2013.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg- jamendo dataset for automatic music tagging. In International Conference Machine Learning. 2019. URL httLhtts::// /ttp///pp... www.peissemanccccoooloooo
wwt. CrggCorrpsID: 961996188 1966184995
Joseph Campbell. The hero with a thousand faces. New World Library, 1949.
Darrell Conklin. Music generation from statistical models. In Proceedings of the AISB 2003 Sym- posium Artificial Intelligence and Creativity in the Arts and Sciences, pp. 30-35. Citeseer, 2003.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Michaël Defferrard. Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. Fma: dataset for music analysis. arXiv preprint arXiv:1612.01880) 2016.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and leffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0400 04093, 2024.
Robert Gjerdingen. Music in the galant style. OUP USA, 2007.
Robert O Gjerdingen. Categorization of musical patterns by self-organizing neuronlike networks. Music Perception, 7(4):339-366 1990.
Onkar Gujral, Mihir Bafna, Eric Alm, and Bonnie Berger. Sparse autoencoders uncover biologically interpretable features in protein language model representations. Proceedings of the National Academy of Sciences, 122(34)::25003311122 2025.
Cheng- ZZi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv: (00::/800.0000 04281, 2018.
10

---

Published as a conference paper at ICLR 2026
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an aakke-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference Machine Learning, pp. 13916-13932. PMLR, 2023.
Nori Jacoby, Naftali Tishby, and Dmitri Tymoczko. An information theoretic approach chord categorization and functional harmony. Journal of New Music Research, 44(3):219-244 2015.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-5348. PMLR, 2020.
Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema. and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. ArXiv, abs/2310.03666, 2023. URL
https://app www.etettiiieee ias.mnnncccchharrr wwt./ooro/ororoooee ott.aaaa...cccc......o.oooooooaa 266677158
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023.
Wenye Ma and Gus Xia. Exploring the internal mechanisms of music Ilms: A study of root and quality via probing and intervention techniques. ICML 2024 Workshop on Mechanistic Inter- pretability, 2024.
Wenye Ma, Xinyue Li, and Gus Xia. Do music Ilms learn symbolic concepts? a pilot study using probing and intervention. In Audio Imagination: NeurIPS 2024 Workshop Al-Driven Speech, Music, and Sound Generation, 2024.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint ar.iv: arXiv::312..66.3 2013.
Stéphane Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process., 41:3397-3415, 1993. URL hatpos//appl..smantiicccoooo.o.
org/ CrrpCosuus ID: 44277335
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemnn: An unconditional end-to-end neural audio generation model. arXiv preprint arXi::1612.07877. 2016.
Jan Melechovský, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. Mustango: Toward controllable text-to-music generation. In North Ameri- can Chapter of the Association for Computational Linguistics, 2023. URL https://api.
semant cscholar........... www..rrrooooorottao.... 2651177750
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):331133322 1997.
Kenny Peng, Rajiv Movva, Jon Kleinberg, Emma Pierson, and Nikhil Garg. Use sparse autoencoders to discover unknown concepts, not to act on known concepts. arXiv e-prints, pp. arXiv-2506, 2025.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-naat name: Task-agnostic concept bottlenecks via automated concept discovery. In European Conference Computer Vision, pp. 444 461. Springer, 2024.
Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language mod- els via sparse autoencoders. bioRxiv, pp. 2024-11, 11, 2024.
Nikhil Singh, Manaswi Mishra, and Tod Machover. Ai for musical discovery. 2024.
David Temperley. The cognition of basic musical structures. 2004.
David Temperley. Music and probability. 2007.

---

Published as a conference paper at ICLR 2026
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. transformer circuits thread, 2024.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint (eXX.v:60900349999 12, 2016.
Marcel Vélez Vásquez, Charlotte Pouw, John Ashley Burgoyne, and Willem Zuidema. Exploring the inner mechanisms of large generative music models. ISMIR, 2024.
Megan Wei, Michael Freeman, Chris Donahue, and Chen Sun. Do music generation models encode music theory? arXiv preprint arXiv:2410.00872, 2024a.
Shaopeng Wei, Manzhen Wei, Haoyu Wang. Yu Zhao, and Gang Kou. Melody-guided music gener- ation. 2024b. URL BtRpttps:: aap........ semant www.aanttcccooooo icscholar wtg.aaa..rg//oooaeaaeee :ttp:/a.....///ooouus::::::22:::
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- KirkpKirkpat and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christo- pher Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outper- form sparse autoencoders. arXiv preprint 1rXiv:2501.77148, 2025.
A ADDITIONAL RESULTS: STATISTICS OF DISCOVERED FEATURES
Figure presents feature activation statisties. For each SAE, we plot the relationship between the average activation strength of each feature (y-axis) and its prevalence, defined by the fraction of the validation set for which the feature's mean activation is positive (x-axis). Figure visualizes the results from Table for an alternate view, to facilitate overall comparisons.
MG-S 1b-32/ 2E-4
MG-S k-32// BF-32
MG-S/ 10000/ EF-6
MGS 100 18-32
1.0
Log Ave. Activation Scrength
0.8
0.6
MG-1./k330 EF
MG-./k-5 37-32
MG-L/ /--t:/ EF
MG-./ k-1n0/ EF-32
0.4
0.0296
0.35
196
136
100%
0.03%
0.38
296
2086
10085
0.0136
0.185
136
1090
100%
0.02%
0.2%
19
10%6
100%
Feature Prevalence Sampies Acctivated
Figure 6: Distribution of learned feature activation characteristics across the MusicSet validation cohort, prior to filtering, for various SAE configurations (log-log plot). The x-axis shows feature prevalence (fraction of validation tracks on which a feature exhibits non-zero mean activation). The y-axis indicates mean activation strength. The observed heavy-tailed distributions, with many fea- tures exhibiting either very broad or very sparse activation patterns, suggest the need for a more principled filtering method.
B CHOICE OF CAPTIONING MODEL.
We compared several versions of Gemini models supplementary analysis, specifically Flash 1.5, Flash 2.0, Flash 2.5, and Pro 2.5, as automated labeling pipelines. We conducted a preliminary evaluation of label quality using the CLAP score (Wu et al. 2023).
12

---

Published as a conference paper at ICLR 2026
MGL
MGS
Retained Features
107
Params
K-32, EF-4
107
k=32, EF-32
K-100, EF-4
R=100, EFF=32
50%
100%
056
73%
Relative Depch
Figure 7: Impact of SAE byperparameters and target model on yield of viable musical features after filtering. Each bar represents the number of features retained for a specific configuration, varying by MusicGen model size (small or MGS/large or MGL.), target layer, k-sparsity, and expansion factor (EF). The substantial variation in feature counts shows how these parameters affect discovery of statistically robust features.
Figure shows average CLAP scores per MusicGen layer across models. Figure presents the ECDF of maximum CLAP scores over all labeled features. Surprisingly, Gemini Flash 1.5 consis- tently produces the highest CLAP alignment on the whole, despite being the oldest model tested.
Given these results, although acknowledge there are significant limitations arising from using CLAP as the evaluation metric, we use Flash 1.5 as the default captioning model for all automatic labels in the paper's primary analyses.
CLAP: Essentia Tips/ Flch 15 CLAP: Gemini Concepts/ Flash 1,5 CLAP: Gemini Name Mlam/Flahh CLAP: Gemini Summary Fluhh
03
CLAP: Essentia Taps/ Flsh
CLAP: Geminl Concepts/ Flush 2.0
CLAP: Geminl Name Flush
CLAP: Gemini Summary / Fluhh 2.0
01-
02
Modd
CLAP: Essentaa Taps Flah
CLAP: Geminl Concepts Flsh 25
CLAP: Gemial Name / Elakk 25 CLAP: Gemina Summary / Flak
MGL
MGS
03-
03
CLAP: Escntia Tags P55
CLAP: Cemins Coaepu
/Pa225 Pro25
CLAP: Gemini Name/ P1o25
CLAP: Gamini Summary/ / Pio
01-
03
796
Relative Depth
Figure 8: Average CLAP scores by MusicGen layer for each Gemini model used as the captioning pipeline. Gemini Flash 1.5 achieves the highest alignment average.
13

---

Published as a conference paper at ICLR 2026
Agg: Min
Agg: Max
Agg Mean
Model, Flash 1.5
79%
50%
2096
0.30
0.45
Model Flash 2.0
79%
50%
96 Feats, X)
0.30
0.30
0.55
0.45
Score Source
Model Flash 2.5
Eoentis Tage
79%
Concepos
509
Gemini
Ganini Sanmay
0.00
0.30
0.5
0.45
Model: 25
79%
50%
2056
0.30
0.30
0.5
0.45
CLAP Score (X)
Figure 9: ECDF of maximum CLAP scores across all features for each Gemini variant. Flash 1.5 yields higher proportion of strong alignments than newer models.
C FEATURE CO-ACTIVATTONS
We conducted large-scale measurements of feature co-activation across models, layers, and SAE configurations order to understand when distinct learned features might activate on the same musical input. For pair of features (Js, {z), we treat the number of tracks common among their respective top-10 10 activating examples as an indicator functional relatedness. Although the most common non-zero outcome is that two features overlap exactly one track, the long tail of rarer, higher-overlay cases provides (observational) evidence of meaningful structure. Here we summarize the most clear and recurring qualitative patterns observed in this tail.
Same-site convergence across distinct SAEs. First, we find cases where separate SAEs trained on the same model recover nearly identical concepts at the same layer position. Representative examples include:
Distorted Rock (MGL L36) by three separate SAE configurations: 32, EF 32; k 100, EF 32; k 100, EF =
String Orchestra (MGL 1.2) by l 100, FF 4; 100, F.F 32
Reverberant Vocals and Percussion (MGS L22) by k: 100, EF 32; =2, 32, EF
32
These patterns indicate that certain musical concepts may be encoded robustly enough in the model's residual stream that different SAEs can reliably recover them.
Cross-layer hierarchical relationships. We also observe cases where early-layer features and deeper-layer features appear to express related musical ideas at different (e.g. increasing) levels of abstraction. For example:
14

---

Published as a conference paper at ICLR 2026
Wind-Dominated Folk Drone (L/2) Eastern European Folk Wind Music (L36) in MGL 32, EF = 32 with 50% overlap between features (5/10 examples)
Jazzy Brass (L24) + Brass and Syncopation (L45) in MGL k 100, EF= = with 50% overlap
Arpeggiated Electronic Music (1.6) -> Synth Pop (L/8) MGS 32, EF 32 with 40% overlap
Such patterns point potentially semantically hierarchical pathways in the learned features, where deeper layers might develop earlier features.
Overall structure. Across the full set of experiments, all configuration: of MGL and MGS exhibit the same broad pattern: most feature pairs co-activate sparsely at all (see right side of Figure TO). Figure 1T shows statisties of features co-activating across layers. These results suggest that certain SAE configurations are more successful at recovering these than others. For the most successful one, we show layer co-activation statisties in the heatmap the right (within-layer co-activations are zeroed out here). Co-activations are distributed rather than e.g. locally biased. Finally, in Figure [2] we examine what degree different SAE configurations can recover similar features in similar locations. Here, there is a split between cooccurence rates and strengths: MGL L2 has by far the most cooccurring features across different SAE configurations, but the lowest average strength. In contrast, MGS 1.22 has among the fewest cooccurring features, but the highest average strength. This could be due to, for instance, the degree of compression within the model and layer: 1.22 the latest layer test in the small-scale MGS and may have more well-formed features that are consistently recovered, while L2 is the earliest layer we test in the large-scale MGL, and could have more generie and lowelevel features. We note however that these are observations rather than formal, mechanistic findings.
Distribution of Cooccurennce Pair Types
Distribution of Coocuurence Strengths by Type
60000
Steongth (Nam. Examples)
$1,951
Nanber Pa
40,100
30000
20000
10000
10
p-1.30
Layer
Beawom
Between
Beewean Layen
Layer
Layers
Figure 10: Co-activation/cooccereee statistics by unit of analysis: between SAE configurations the same model/layer, between layers on the same model and SAE configuration, within the same layer/model/SA AE configuration, and between MGS and MGL models.
In aggregate, these observations suggest that while the bulk of discovered features behave indepen- dently, there is a nontrivial amount of structure learned across different parts the models that co-activation analysis change under different co-activation strength thresholds, however we acknowledge this requires a much larger scope of experimentation.
help to reveal. A deeper analysis might look at how these observations
15

---

Published as a conference paper at ICLR 2026
Kollecar Gorggesden
MOX Thte Suatd
1761
153
01-002
500 0000 2000
Jome
1500
2000
thangh
Figure 11: Statisties of cross-layer ao-accivatioos.
CrosSAAE Modelilaper
CrowsAAE Smungh Morkll Layer
MGL. Layer
MGS Layer
MGy/Layer12
Layer
MGL//ayer2e
Modd/ Layer
MGS/ Layeró
Modd/ Layer
MGS /Layer
MGL./ Layer 46
MGL. Layer
MGS/Iayer
LoGL//Lyerr 12
d6
MGL./
MGL/ Layer36
MGL/Layer2
MGS 2ayer22
5000
10000
19000
22000
Nomber Cf
Mian Senragth
Moz Cummon SAE Cenfiurrttio Pairs
Seotngthh Dirtibuttion by Medel (Crme-SA))
EP-32 1e-3 ov EF-i 12
100000 co EF-é 30
Concurtenc Stenigh
Configimation
E3-4 12-22 eo 12-6/ 1-100
BF-32 1--2 o WF-4// 100
100 E7-3//--3 (0
E3-32/ 100 1--0/1-100
EF-30/ 30 EF-32/k-100
E3-47 1000000000000000000 ou BF-32/k-1000
$200
MGG
Number of Coseurmenne
Figure 12: Statistics of co-activations across SAE configurations, within the same model and layer.
D ESSENTIA TAGS
ESSENTIA MODEL TAGS
We use the following Essentia feature extractors and tagging models. By default, select the top predicted tags from each model, since we observe too much noise in the results to rely solely on the top predicted tag and these are often used in multi-label settings (exceptions are noted and justified below):
Genre-Diseogs4000 Genre prediction (top 3)
Multi-label mood/theme (top 3)
16

---

Published as a conference paper at ICLR 2026
Instrument-Jament Instrument presence (top 4, to account for common band composi- tions)
Tags-Jamendo: General music tags (top
Tags-MSD: General music tags, different set (top 3)
Moods-Audioset: Mood categories, different set (top 2; these tend to be less distinct)
These in depend VGGish, EffNet, or embeddings, all of which also extracted.
E PROMPI-CONDITONES ACTIVATIONS
Our primary focus in this work was to analyze the musical representations that emerge without text conditioning. These representations organizes musical structure in its own representational space, rather than simply responding to verbal instructions. Whereas prompts are already verbal and thus relatively easy align with human categories, internal musical features may be tacit, difficult to describe, and poorly characterized in existing theory. By targeting this layer of representation, we aim to surface structures that extend beyond what can be easily named, and therefore hold greater potential for discovery.
especially important because they capture how the model
We see text-conditioned analysis as a valuable future direction, but our findings suggest that is not tractable under current conditions. First, open datasets with paired text-music examples remain limited. Captions are often sparse, noisy, or stylistically inconsistent, and prompt- audio alignment insufficient to support reliable, large-scale analysis. This means that conclusions about text- conditioned representations would be confounded by dataset artifacts rather than model behavior. Second, from methodological standpoint, isolating the contribution of text requires ablation and control techniques that go beyond existing ones. Mechanistic interpretability has not yet converged reliable tools for partitioning disentangling multi-modal conditioning, making this substantial open problem.
Addressing these challenges would require both new datasets with carefully designed prompt- audio correspondence and new methodological advances to interpret conditioning effects in multimodal models. We believe this is a promising research agenda, but it demands significant additional work beyond the scope of this paper. Our contribution here is therefore to establish a foundation by studying the non-textual musical representations directly, which offers both theoretical value, in revealing the categories that emerge independently of language, and practical value by providing a baseline for future multimodal interpretability.
F INSTRUCTIONS GIVEN TO PARTICIPANTS
Study Details
We are assessing the quality of a novel method for discovering interpretable concepts in generative music models. You will be presented with a series of short sounds with common pattern or feature. Given set of options, you will be asked to select the one that best describes all the sounds, as well as your confidence in your prediction.
[FOR EACH SAMPLEJ
Select the closest category representing what all these sounds have in common:
[Audio 1] [Audio 2] [Audio 3]
[Option 1]
[Option
[Option
[Option [Option 5]
How confident are you that this label describes the common feature(s) between these well?
17

---

Published as a conference paper at ICLR 2026
(Completely confident]
[Fairly confident]
[Somewhat confident]
[Slightly confident] [Not confident at all]
Write down your own label for what these sounds have common musically
[FREE TEXT]
G CONNECTIONS TO MUSIC THEORY
This work is conceptually related to several core threads we are aware of in music theory, cognitive musicology, and computational modeling of music structure. Here, we clarify how our approach relates to existing work by seeking to uncover "operational" structures (what we refer to as features or concepts) in generative models.
Statistical foundations of music theory Temperley 2004 2007) argues that listeners develop an understanding of many core elements of Western music (such as meter, harmony, and voice leading) by internalizing probabilistic regularities encountered in musical corpora. From this perspective, foundational concepts within music theory, rather than being seen solely elements of a formal axiomatic system, can be understood as principles that often reflect this inductively learned, tacit knowledge of musical likelihoods. Temperley's computational cognitive models are themselves compact and idealized representations which help explain why these music-theoretic constructs are perceptually and musically relevant.
Our approach parallels this an Al setting. Rather than analyzing human musical corpora directly to infer cognitive principles, seek to "reverse "reverse-engineer generative models trained on such data to recover the internal distinctions and abstractions they have formed. The concepts discovered in our pipeline reflect these model abstractions, many of which correspond statistical regularities observed in music (e.g. genre, texture. instrument timbre, harmonic style).
Operational concepts vs. formal theory Gjerdingen's 's schema theory (Geerdingen 1990 2007) discusses the role of learned patterns in shaping musical fluency. His analysis of Galant phrase struc- ture proposes that composers relied on a rich set of prototypical configurations (or slock musical phrases in particular), acquired through exposure and practice rather than via formal rule systems. These schemata were typically only formalized retroactively through pedagogical and analytical dis- course, as we imagine concepts of the type our pipeline discovers might be the future. As such, our work focuses on proto-theoretical regularities: we extract features that appear to be function- ally meaningful to the model regardless of whether they map cleanly to any established theoretical category. Some features do align with known "schemata" but others do not, yet still display internal coherence, this sense, our pipeline seeks to unearth raw material from which a theory (either of model's 's music understanding, or of some aspect of music itself) could potentially be formed.
Meta-theoretical implications This empirical approach may also serve as a testing ground for teta-thhoretical questions about music theory. If large-scale generative models learn internal rep- resentations that resemble certain theoretical concepts but diverge from others, this may point to systematic gaps or biases in how theory abstracts from musical practice. Conversely, the emergence of coherent but non-canonical features suggests that models may encode distinctions that are musi- cally significant but neglected in prior analyses. this way, our work does not propose a theory of music, but can be seen contributing method for investigating the gap between musical practice and theoretical formalism. We propose that such discovered features features be treated as hypothe- about musical structure that may be refined, validated, or rejected through future research.
18

---

Published as a conference paper at ICLR 2026
H
GEMINI DETAILS
Our prompt to automatically label concepts using Gemini's API, developed through iterative testing:
Prompt for Gemini's Labeling
Listen very carefully to this set of audio clips, which consists of song snippets concatenated in random order.
You need to discover common musical patterns across the whole set, to identify what musical feature is shared across all clips. You will need to listen carefully. For each potential concept you identify, output a name, confidence score between and (where is highest confidence), and concise description of the concept.
At a higher level, describe the overall concept shared across the set, give suitable name, and provide an overall confidence score (0 to 1).
Describe the *underlying concepts* * not the specific audio snippets (e.g. your description could say "the concept" but not "the audio snippets" ") However, try to avoid such verbiage altogether and concisely describe the musical concept's main attributes.
Include NO FILLER text.
Focus on being specific. Concepts could relate genre (e.g. hip-hop, salsa, reggueton, balkan), instruments (e.g.. piano, cello. guitar, flute), recording/produccion techniques (e.g.. reverberation, drones, noise, DJ scratching, beatboxing, drum machine, hi-hat patterns. fingerpicking, live recording artifacts, low-pass filtering), or more nuanced musical ideas (e.g., drum solo, chill dance rhythm, serene woodwinds arrangement). These are illustrative examples, NOT fixed list choose from.
We use the following model definition to constrain Gemini's response via structured output genera- tion:
Gemini's Response Schema
pydantic import BaseModel, Field
class Concept (BaseModel):
"""Represents single identified musical concept .**
name: str Field(.... "csnees name for the musical 4 concept.")
confidence: float = Field(.... ge=0.0, 1e=1.0, description=" 4 Confidence score (0.0 to 1.0)."
description: str Field(.... description-"Arie "Brie description of 4 the concept.")
class ConceptLabels (BaseModel):
a """Rereeeents the overall analysis result for set of audio clips
concepts: list [Concopt] Field(.... deseription="Liet specific 4 concepts identificd.""
overall_ surrall_sumary: str Field(..., description="Overee description of the shared musical concept [no full sentences, concise 4 summary of underlying concept, ignore snippets]
overall_nanes str Field(.... description="Coneisss name for the 4 overall shared concept.")
cerrall_confideee float = Field(.... ge=0.0, 1e=1.0, description "Overal1 confidence score (0.0 to 1.0).")
19

---

Published as a conference paper at ICLR 2026
1 COMPUTE RESOURCES
All model training runs in this paper run on node with 4x NVIDIA L40s GPUs, as well as some of the experiments (e.g. CLAP score computation). Other experiments use a CPU compute node with 128 Intel Xeon Platinum 8375C CPUs @ 2.90GHz each for parallelization.
J LLM USE DISCLOSURE
We used large language models for minor copy editing, including improving grammar and phrasing. The authors reviewed all changes.
20


Reviews:
[{"review_id": "gbXEDPwFqP", "reviewer": "Reviewer_jYej", "summary": "This paper uses SAE to discover interpretable concepts in a pretrained music generation model MusicGen. The process is unsupervised, using pretrained audio taggers/LLM for automatic concept labeling.", "questions": "1. Line 210: what is a \"track\" in this context? Does it refer to a single piece of generated music, or a separated stem of a music?\n2. Line 397: If later layers encode more interpretable features, why is the numbers in early layers (e.g., L2) larger compared to later layers in Table 1?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This is the first paper to perform concept discovery with SAE in pretrained music generators, and the results are much better compared to the previous probing works.\n\n2. The design choices in Sec. 3.3 provide very useful insights for future researchers in concept discovery for pretrained audio/music models.\n\n3. The automatic labeling pipeline is promising (with some limitation as stated in weakness) and could be applied to other models (audio generation/understanding models).\n\nIn general, this work has the potential to bring significant impact to explanibility in large audio/music models.", "weaknesses": "1. In sec. 3.5 automated interpretability, the automatic pipeline might harm the interpretation of some concepts (i.e., chord & keys) since neither gemini nor essentia has such ability.\n\n2. Currently the number of examples are very limited. More examples/case study would be useful in the appendix, including possible failure cases where the automatic labeling pipeline fail to conclude. I.e., more examples where:\n\n(1) A concept could be successfully extracted and named;\n(2) A concept could be extracted, but could not be correctly named;\n(3) A concept that could not be extracted.\n\nDifferent types of concepts could also be tested, including genre, instruments, playing techniques, rhythm pattern, groove, sound field, audio quality, mixing techniques, key modes, tempo, velocity, moods etc. Even testing on a portion of them would be very useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3siErYVGmD", "reviewer": "Reviewer_1vEC", "summary": "This paper extends sparse autoencoder (SAE) methodology from language models to generative music models, specifically MusicGen-Small and Large. The authors train k-sparse autoencoders on residual-stream activations, automatically label discovered features using multimodal LLMs and audio analysis tools (filtered by CLAP alignment), and demonstrate concept steering by intervening on SAE decoder directions. The work reports thousands of interpretable features per configuration, shows that deeper layers in larger models yield more interpretable concepts, and achieves measurable steering success on 15–35% of tested features.", "questions": "(1) Can you provide human listening tests comparing steered vs. unsteered outputs on perceptual dimensions (genre, timbre, rhythm)? Please compare this with random-direction baselines.\n\n(2) How do interpretability quality metrics (human agreement, not just counts) vary with θ_min, θ_max, k, and EF? \n\n(3) Add negative controls (steering on low-activation features, cross-layer swaps, random directions with matched norm) and report effect sizes with statistical tests.\n\n(4) Can you demonstrate the pipeline on at least one small-scale diffusion-based model to support cross-architecture claims?\n\n(5) How does SAE steering compare to supervised linear probes, activation patching, or MusicGen's own conditioning mechanisms?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "To the best of my knowledge, this is the first attempt to apply SAE-based interpretability to audio generation LM; this is a meaningful extension beyond NLP/vision that addresses an underexplored modality. The modular pipeline is well-documented and reproducible. The authors conducted systematic exploration across model scales, layers, sparsity levels, and expansion factors provides useful data on where interpretable structure emerges in music LMs. Table 1 and Fig. 3 offer valuable design guidance. The paper is generally well-written and I find the figures to be informative.", "weaknesses": "(1) I notice CLAP serves as both the filter for accepting labels and the metric for evaluating interpretability and steering success. This creates a validation loop where the authors are essentially measuring \"does this feature change CLAP scores\" rather than \"does this feature control meaningful musical concepts\"; The human study is too limited to break this circular argument issue.\n\n(2) 15–35% success rate with a single prompt and CLAP-only evaluation is insufficient to claim \"robust\" controllability. I suggest to add the following: (a) listening studies, (b) diverse prompts, (c) baseline comparisons (e.g., random directions, PCA, supervised probes), (d) negative controls (inactive features, orthogonalized vectors). Per my first point, current results may reflect CLAP bias rather than perceptual control.\n\n(3) Key design choices (θ_min/θ_max thresholds, k values, rarity cutoffs) lack sensitivity analysis. How much do interpretability rates depend on these arbitrary choices? Some statistical report is needed.\n\n(4) Restricting to MusicGen limits this paper's impact. Currently, diffusion/flow-matching based models dominate audio generation; Appendix C acknowledges difficulties but doesn't attempt even a small-scale demonstration (as an example, Stable Audio Open small is small-scale; only 341M parameters). Claims of generality are thus in my opinion unsupported.\n\n(5) Mean-pooling activations across time may obscure temporal dynamics, and there's no analysis of within-track feature stability or polysemantic behavior at different time scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYSAQCeL2d", "reviewer": "Reviewer_eAhf", "summary": "I am a little confused by the premise of this paper, and would appreciate it being explained in the rebuttal (in which case I will be most willing to revise my score).  It seems that the paper is arguing that there are concepts humans don’t have words for and that “Electronic Beeps and Boops” is such an example.  Yet the authors are validating using both a multimodal audio -> text model (gemini) and CLAP, which seems to presuppose (and, as I have seen it used, correctly) that CLAP itself can map “Electronic Beeps and Boops” and audio to a valid score.  If one defines theory as “the things that show up in 20th century textbooks”, “beeps and boops” wouldn’t be there (for more sociological/historical than cognitive reasons), but I wouldn’t label them as “concepts we don’t have names for”, particularly given the success with CLAP.  \n\nBut to summarize the paper's technical contributions - they have used sparse autoencoders (SAE's) to infer common patterns of activation.  They use this in the context of interpretability by using Gemini to provide labels for these common patterns, do some basic analysis of these data patterns, and try to steer the MusicGen model using these patterns (although it's unclear if that was successful or not).", "questions": "1. Do you have any way of measuring “degree to which this is a previously untheorized label”?\n\n2. On line 288 – could you elaborate on what the “baseline” is?  Uncontrolled musicgen?\n\n3. On line 310 – could you possibly provide examples of co-occurring activating features, and mention if there were any there that were unexpected?  I’d imagine that determining cooccurrences might be more enlightening than the concept names themselves.\n\n4. Can you clarify the diagram on line 385?\n\n5. Can you speak as to how your features are different than the features CLAP can recognize, or why your contribution is an improvement over just having access to CLAP?\n\n6.   Could you describe the essentia tags used?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper engages with the interpretability literature to provide a lens into the relationship between a highly complicated model and the sort of language we use as humans to describe music.  This provides much needed research at the intersection of model interpretability and music generation.  They successfully determine that they can use SAE's to uncover important information about how music data is structured, including not only the fact that there are such labels, but that they are in some instances highly correlated with each other.", "weaknesses": "1. I am unconvinced by the contributions of this paper.  Fundamentally, the claim that they are producing concepts we \"don't have a name for\" seems blatantly false given that they use a multimodal model to label them (unless I'm misunderstanding something).  \n\n2. The numbers for the steering example don't seem impressive without more information - having \"any improvement at all\" on 20-25% of features compared to None could be noise.\n\n3. For a paper about music, it is very limiting to take the \"mean activation\" of a feature over 10 seconds.  For instance, unless I'm misunderstanding, the example they gave about chord progressions being something we do have interpretable concepts for, \\textit{unlike} \"beeps and boops\", actually reveals a significant flaw in their approach - I don't think it would be mathematically possible to infer a chord progression feature from mean activations over 10 seconds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyVcvRVca3", "reviewer": "Reviewer_PtMu", "summary": "This paper presents a comprehensive framework for uncovering and manipulating interpretable latent features in pretrained music generation models.\nThe authors apply sparse autoencoders (SAEs) to the residual streams of MusicGen, discovering sparse “concept vectors” that correspond to meaningful musical dimensions (e.g., timbre, rhythm, genre).\nThe framework further proposes automatic labeling of each discovered feature via LLM-based captioning and MIR taggers, and demonstrates preliminary steering by injecting these features back into the model.\n\nOverall, the work is technically sound, conceptually elegant, and well-motivated, providing a significant step toward mechanistic interpretability in multimodal (music–audio–text) systems.", "questions": "1 Could you include an explicit steering demo—for instance, modifying one SAE latent and showing corresponding audio differences or spectrogram shifts?\n2 could you show \"strong emergence\" in the sense that the learned concept cannot be directly used as text prompt to steer MusicGen (show that doing this will lead to poor results); also CLAP cannot output such a concept.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Novel application of mechanistic interpretability to music.\nThe use of SAEs to analyze music model activations is both original and conceptually clean. It connects recent interpretability work (e.g., Anthropic’s neuron-level SAE studies) to a new and highly complex modality.\n\nBridging analysis and control.\nThe proposal to “steer” MusicGen by manipulating discovered features introduces an exciting bridge between interpretability and controllability—potentially enabling a new paradigm of interpretable generation.\n\nStrong conceptual framing.\nThe paper explicitly situates itself within the context of “unsupervised concept discovery,” distinguishing itself from traditional probing methods that rely on predefined human labels. This distinction is theoretically well-grounded and clearly communicated.", "weaknesses": "Steering demonstration missing.\nThe paper’s most exciting claim—that discovered features can steer MusicGen—remains theoretical.\nA simple demo (e.g., modulating one feature and showing changes in audio or spectrogram) would transform this from a conceptual claim to a verifiable capability.\n\nEmergent ability needs stronger evidence.\nTo argue that new control dimensions have emerged, the authors should show that these SAE-derived features are not trivially mappable to existing models such as CLAP or other audio–text embeddings.\nIn other words, the learned features should correspond to new, controllable factors that cannot already be directly used to condition MusicGen or described by current models.\nThis would validate that the SAE uncovers genuinely new interpretable concepts rather than re-labeling known ones. \n\n(my understanding is that because the automatic labeling step relies on existing music/audio to text models, the performance is somehow bounded by their capability. E.g., if CLAP cannot do a good job on recognizing the emotion of music, the proposed model won't have good emotion interpretability. That said, novel combinations of labels are very possible, and such combinations cannot be directly used as the text prompt of musicgen. )\n\nhappy to adjust the rating if these issues are resolved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)