Music Foundation Model as Generic Booster for Music Downstream Tasks
Authors: Wei-Hsiang Liao
Yuhta Takida
Yukara Ikemiya
Zhi Zhong
Chieh-Hsin Lai
Giorgio Fabbro
Kazuki Shimada
Keisuke Toyama
Kin Wai Cheuk
Marco A. Martínez-Ramírez
Shusuke Takahashi
Stefan Uhlich
Taketo Akama
Woosung Choi
Yuichiro Koyama
Yuki Mitsufuji
Venue: Accepted by TMLR ()

Abstract: We demonstrate the efficacy of using intermediate representations from a single foundation model to enhance various music downstream tasks. We introduce SoniDo, a music foundation model (MFM) designed to extract hierarchical features from target music samples. By leveraging hierarchical intermediate features, SoniDo constrains the information granularity, leading to improved performance across various downstream tasks including both understanding and generative tasks. We specifically evaluated this approach on representative tasks such as music tagging, music transcription, music source separation, and music mixing. Our results reveal that the features extracted from foundation models provide valuable enhancements in training downstream task models. This highlights the capability of using features extracted from music foundation models as a booster for downstream tasks. Our approach not only benefits existing task-specific models but also supports music downstream tasks constrained by data scarcity. This paves the way for more effective and accessible music processing solutions.

Published in Transactions on Machine Learning Research (05/2025)
Music Foundation Model as Generic Booster for Music
Downstream Tasks
WeiHsiang Liao Chieh-Hsin Lait,
Yuhta Takida" Giorgio Fabbro
Yukara Ikemiya"', Kazuki Shimada',
Zhi Zhong Keisuke Toyama",
Marco A. Raaiieez--aairz'',, Shusuke Takahashi Stefan Uhlich,, Taketo Akama", Woosung Choi',
Kin Wai Cheuk,,
Yuichiro Koyama',
Yuki Mitsufuji"
A,, Tokyo, Japan
Sooy Group Corporation, Tokyo, Japan
"oon Europe B. V., Stuttgart, Germany "Sony CSL, Tokyo, Japan
"Equal contribution
Reviewed on OpenReview: https: // openreview. net/ forum? id= kH14J2yN2
Abstract
We demonstrate the efficacy using intermediate representations from a single foundation model to enhance various music downstream tasks. We introduce SoNIDo, a music foun- dation model (MFM) designed extract hicrarchical fcatures from target music samples. By leveraging hierarchical intermediate features, SoNIDo constrains the information gran- ularity, leading to improved performance across various downstream tasks including both understanding and generative tasks. We specifically evaluated this approach on representa- tive tasks such as music tagging, music transcription, music source separation, and music mixing. Our results reveal that the features extracted from foundation models provide valuable enhancements
training downstream task models. This highlights the capabil- ity of using features extracted from music foundation models a booster for downst ream tasks. Our approach not only benefits existing task-specific models but also supports music downstream tasks constrained by data scarcity. This paves the way for more effective and accessible music processing solutions.
Step 1: Analysis input audio with foundation mode]
Step 2: Boost downstream tasks
Music Foundation Model
Music Tagging
Insermdiate Resture
Music Transcription
Audio Ingie
Understonnn Downstremm Theks,
Source Separation
Audio Ingur
The come Ingui ataloo
- 1f
Music Mixing
Figure 1: SONIDO extracts hierarchical features of target music samples, which are useful for solving music downstream tasks including understanding and generative tasks.

---

Published in Transactions on Machine Learning Research (05/2025)
1 Introduction
A foundation model is a pre-trained model developed on a large-scale dataset that can be adapted for a variety of downstream lasks (Bommasani et al., 2021). Several language processing models (Radford et al. 2021 Brown et al. 2020; Devlin et al. 2018] Team et al. 2024) are considered foundation models due to their ability to unity all language tasks as sequence prodiction tasks, effectively addressing multiple tasks with a single model. These foundation models have gained significant al attraction and are widely used in everyday applications. In contrast, a powerful music foundation model capable of handling various music downstream tasks for music production is lacking (Ma et al. 2024). We categorize the tasks that a music foundation model primarily addresses into two types: understanding tasks, such as tagging and transcription, and generalive lasks, such as mixing and mastering.
Several multi-task models have been proposed as potential music foundation models (Li et al. 2023a Yang let al.] 2023 Copet et al., 2023 Agostinelli et al., (023). However, this approach necessitates the inclusion of the desired tasks during the training phase. A notable strategy to overcome this limitation is to inject features extracted from pre-trained large-scale model into smaller back-end models for downstream tasks that were not seen during training. This ensemble approach, which combines a large-scale model with various smaller models, can effectively function as music foundation model. The codified audio language modeling (CALM) framework proposed by Castellon et al.] (2021) | is the first work in this direction, utilizing the intermediate representations from Jukebox (Dhariwal et al. 2020) to tackle music information retrieval (MIR) tasks, covering most music understanding tasks. Beyond MIR et al. (2022) leveraged representations from Jukebox for melody transcription. Other studies have followed this approach address time-invariant MIR tasks using the latest generative models based on residual quantized variational Autocncoders (RQ- VAEs) (Zeghidour el al., 2022 Délossez et al., 2023), enhancing the stale-of-the-ar (SOTA). However, these applications remain limited to music understanding tasks. Li et al. (2023b) expanded the focus to include music source separation, generative task, but encountered instability issues during training. The performance of this extension does not yet match that of the baselines mentioned by Mitsufuji et al. (0022 An extensive overview of related work can be found in Appendix A
We extended the methodology from MIR tasks to generic music downstream tasks. To address both un- derstanding and generative tasks, we focus is directly analogous to foundation models in the natural language processing field. Furthermore, We hy- pothesize that the representation structure of foundation models is crucial in this context. Specifically, we propose that hierarchical representations which divide information of varying granularity into different levels of embedding, are expected to provide efficient information hierarchy for all downstream tasks including both understanding and generative tasks. We empirically verify this hypothosis Scction 4 In contrast, music foundation models that have been applied to boost music rownstream lasks do not have such a hierarchical structure. For example, Jukebox (Castellon et al. 2021 Donahue et al. 2022 is trained to have multi-level representation inspired from hierarchical latent representation (Razavi et al. 2019); however, each level is independently trained. RQ-VAEs (Yang et al. 2023 Li et al. (023b) learn factorized representation that has a self-organized coarse-to- fine structure, however, they are not hierarchical.
tokenization-
generative modeling approach, which
In accordance with the aforementioned hypothesis, we outline this study as follows. We propose and train our music foundation model, SONIDO (meaning sound in Spanish), on a high-fidelity internal dataset ! to establish task-agnostic feature extraction pipeline. SONIDO is a generative model consisting of multi-level transformer with multi-level hicrarchical encoder. With proper pre-processing, we infuse its intermediato representation as features to (ask-specific models on various music downstream lasks with data augmentation. Moreover, for understanding tasks, we proposed an on-the-fly data augmentation called token-out to avoid overfitting. Performance evaluation was done by benchmarking with representative tasks from understanding to generative tasks: music tagging, music transcription, music source separation, and music mixing.
'The rights of this internal datasct are trained on liceused content only. Except for as specifically authorized by the rights owner, the rights owner expressly prohibits and has opted out of any text data mining, web scraping similar, reproductions, extractions or uses, of its content for any purposes, including in relation to training, developing, or commercializing any Al System.
2

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Performance overview applying extracted features to various downstream tasks. Bold: best, underline: second best. The result marked with * is obtained with a different evaluation protocol. The results marked with are numbers reported in Castellon et al. (2021)
Downstream Tack
Dataset
Metricc
SoxiDo
MESICUSE
MUSICEEN LARGR
Julaniso: SD
MEITT
Tusk Specific SOTA
SMALLL
Multi-tokk Deggingg
ArsieTagATioee
AUC
9L.1
Husng et
mAP
41.5
38.85
39.0
41.41
401.2
38.4
Pitch Estimation
Neyath
Are.
03.8
93.3
92.8
91.6"
94.4
90.2
McCallum
a002
Cauefication
Acc.
78.0
Emotion Regression
EmoMusie
Averaged gt
64.7
45.0
46.2
06.91
68.0
63.0
Ietellon e m
204
Key Detection
GannSStes
Weighted Acc.
63.5
65.2
62.4
06.71
65.6
79.6
Sel Re et m 674
Genze Chasillaatonn
GTZAN
Ace.
MILI
Siuger Idtetifieation
VocalSet
Are.
87.0
82.3
82.6
8T.1
90.3
mong 3045
Techniqu Identification
Act.
70.4
66.1
63.9
76.71
76.9
65.6
Tamantooo ot
4722
F1
86.45
65.97
85.14
Music Transciptton
MAPS
Note w/ Offect F1
68:27
68.27
66.25
68:25
66.31
Toyrama al. 2025
Note
Valacity
51.34
SDR (baus)
9.50
8.86
8.17
7.12
5.6
11.31
MUSDRI8
SDR (druzs)
(otlber)
6.11
8.65
8.03
5.50
7.50
6.65
0.49
2004
Source Sepucaation
SDR (vorals)
8.07
7.57
7.65
6.84
5.3
10.0A
SDR (bass)
8.14
7.44
7,00
6.58
1.56
MDXDR21
(drums)
8.16
hrbden
SDR (other)
6.21
5:28
4.93
4.59
5.00
SDR (vorals)
8.04
7,81
7,73
7.12
7.70
Storen Envminnt
79,86
MDXDB21-day
Spectroliogg
0.221
0.229
0.228
0.231
0.183
Music Mixing
hidden
Dynonicaage
Panningmape
0.064
0.175
0.072
0.244
0.219
0.073
0.075
0.249
0.070
0,172
Martines Ramizes et al. 2022
Londacte.wge
0.171
The encoder design of SONIDO is inspired by Jukebox but makes the representation hicrarchical by enforcing the fine level to be conditioned by the coarse levels using hierarchical autoencoder framework called hierarchically quantized VAE (HQ-VAB) (Takida et al. 2024 We then use transformer-basee multi- level auto-regressive model to characterize the probability mass of learnt HQ-VAE embeddings. We extract features from the intermediate representation of SoNIDo by first converting input audio with the encoder into tokens, feeding them into the transformers, and extracting the intermediate output from the midst layer. We refer to these extracted features as SoNIDO features.
As shown in Table ! we test SONIDO's feature injection for selected downstream tasks along with several baselines. To the best of our knowledge, this is the first study on enhancing both understanding and generative tasks with the intermediate representation from a single model. We briefly list our major findings:
1. We empirically show that, with an auto-regressive generative model that is established on hierar- chical representation, its intermediate reprosentation can serve as generic booster various music reamstream lasks.
2. We verify that the extracted intermediate representation is beneficial for music understanding tasks even with only an extra shallow back-end network. The extension of the shallow network with attention layers leads to further improvement.
We show that the extracted intermediate representation beneficial for enhancing task-specific models, through the applications to both understanding and generative tasks.
4. Several of the above improvements in each task category result in new SOTA scores. The summary of our results is shown in Table [
2 Proposed Two-stage Hierarchical Model: SONIDO
To explore the elfectiveness of hierarchical modeling in boosting downstream lasks, we adopt a typical two-stage generative modeling (Dhariwal et al.) 2020 Copet et al. 2023 Li et al. (023a). In stage-1, we use an HQ-VAE for hicrarchical representation learning, dividing information into different levels based on granularity. In stago-2, we use auto-regressive modeling learn the multi-level token streams extracted from the stage-1 model. Finally, features from stage-2 model extracted described in Section 3.1
3

---

Published in Transactions on Machine Learning Research (05/2025)
2.1 Stage-1 Model: HQ-VAE
We construct the archilecture of SONIDO to learn hierarchical representation of the larget datasel. Consider music sample æ with length T' where æ E X' C R²" A set of codebooks {BI, B2, B3} is used for learning a throo-layor hicrarchical representation on æ. For l E f1, 3), the ith codebook is denoted as B = (br, / which consists of K2 d--dimensional trainable vectors b E Rd, The architecture is designed to extract a hierarchical latent representation of music samples, which is denoted as Z == Z1 6 Z2 69 Z3 with Zi E B (1 = 1,2, 3), where is the latent sequence length at the (th layer. The discrete tensors Z1, Z2, and Z3 are expected to convey the coarse, medium, and fineggraincd information. The reconstruction can be done with well-optimized neural function f B 69 B X B X, i.e. x Ri f(Z1,2,3)
The architecture is composed of bottom-up and top-down paths, as illustrated in Figure 2 aa, the inference process of which is as follows. A series of encoders in the bottom-up path extracts feature tensors for three different information resolutions, which are denoted as Hx(x) (1 = 1,2,3), from sample æ. The feature (x)x) is used for the top-down path to process the data in a hicrarchical manner. The top-down path has three (top, middle, and bottom) lop-down blocks to model hierarchical discrete latent representations. The top block first quantizes Z1 := H1 (zx), which has the most global (coarse) information amongst the encoded features, into discrete tensor Z by the nearest neighbor search in codebook B At the next step, the middle latent tensor is conditioned on the top Z1 to focus more on local details, with the injection of Hz)x). Therefore, the block takes both tensors processed in the top block and bottom-up paths, i.e., Z1 and H2( generating a raw continuous feature Ze == (2)H2 Z1). The raw feature is then quantized into Zg in the same manner as with codebook B2. The bottom block repeats a similar process with Zg and Hz(x) to further refine the representation with the additional discrete feature Z3- Finally, the set of Z1, Zz, and Z3 is decoded to the data space to reconstruct æ.
We train the architecture including the codebooks within the variational Bayes framework, of HQ-VAE, stochastically quantized VAE-2 (SQ-VAE-2) (Takida et al. 2024p. To establish a gener- ative process in this VAE, we first define the prior probability distribution on Z1,2,3 as P(Z1,23) = (2 Z2 (z) Z1.2). Given a chunk of latent variables Zi, Z2, and Za, a data sample can generated under a conditional probability distribution p( Z1,23). Concretely, we parameterize the con- ditional distribution as normal distribution with function f and a trainable isotropic covariance matrix p(æ Z.2,3)
instance
(Z1,,)))) To summarize, the gencrative process consists of two steps: sam- pling Z1 from the prior distribution and decoding it with the conditional distribution. Note that, in practice, Z1,,, is sampled from an estimated posterior distribution instead of the prior distribu- tion, as presented in Section 2.2 Next, the approximated postcrior distribution for p( 2.2 (x) is set as ((Z121)zz = Q1 (Z, [2) Z2| Z1, x))aa(aa Z1.2; æ). We connect each Q1, Q2, and Q3 with the compo- nents in Figure a). Specifically, the categorical distribution at the Ith layer, Qi (Z) Z<t, x), is defined as a stochastic quantization that is (z, b xn,) exp(-I|ž1,, 1 /22 with trainable positive scalar 8f, where 2,, and žtn indicate the nth vectors in Zi and 2,, respectively. Finally, the resulting training objective consists of terms for reconstruction and latent regularization:
T L1 (xx) = log log o² + EQ(ZI.2a
| - 1(Z
Σ
3
1
Z.12 H(P.(2212))) 2
(1)
2
20
where H()) is the entropy of a probability mass function. Progressive coding to ensure the amount of information is balanced across the three layers.
Takida et al. 2024 is applied
2.2 Stage-2 Model: Sparse Transformers
The stage-2 model addresses the gap between the pre-sct prior distribution (i..,, P(Zz.2.)) and marginalized
posterior distribution (i.e., 2(2a) := Ept [Q( (1.2aa=)))] by directly learning the posterior distribution. We incorporated the contrastive language-audd audio pretraining (CLAP) model proposed by LAIÓN (Wu* all, 2023) into the stago-2 model. To include the CLAP conditioning, we approximate Qø (2.)))) with condi- tioned decomposition
Pri 3.122///addlo) Pr, (Zi Vaudio. Pro
7m,,(2 (Zz Z1, Vandio Pr.. (Zsl Z1,, Wandlo);
(2)
4

---

Published in Transactions on Machine Learning Research (05/2025)
Top tog-doww black
H (x)
Rt
(x(x)
Quantizer B,
Encoderrop
Middle top-dows block
Decoderoop
2, (xH,(x)
Ro
H2(x)
Quantizer B
Encodermaddte
Bottom top down block
Decodermidete
2, G,HH,
(x(x)
Quantizer wi B3
Encoderaatcom
Z3
Decode/bottom
(a) Stage model architecture
Top Prior
2,
embeldinng
Stage-1 Foender
Yondio
Autoregressive Sparse Transformer
SonDDo Festuee
Zz.
Middle Conditional Prior
upsampling
upsampled embedding
Z:
eubelding
Youdio
Autoregressive Sparge Transfrmmr
Midele Somto Feanue
CLAP
Yaudio
Bottom Conditional Prior
Zz
Ze
upsampling
apsampled embedding
2g
embellding
Youdio
Autoregressive Spanse Transformer
2g
Stago-1 Decoder
default dataflow
feature extraction
sample generation
generated audio
(b) Stage-2 model architocture
Figure 2: The two stages of SONIDO.
where II = {T,,,2, feature produced from the CLAP encoder. Thanks to the alignment between the audio and text embeddings of CLAP, even if the audio dataset has no text caption, we can still feed audio in the training phase, whereas allows either audio or text input in the inference stage. The use of a pre-trained encoder is common in modern generative models. For example, MusicGen (Copet et al., (023) uses the pre-trained T5 encoder (Raffel al. 2019) to model the text conditions. The training objective is negative log- logeliielhoood:
T2, T3} is set of neural networks for the stage-2 model, and Yaudin E R512 denotes the
C2(x) = Eq, (e)pcLAr
[e) lel log Prr (1, /
(3)
We follow Jukebox Dhariwal et al. 2020 to construct the networks II with sparse transformers (Vaswani ct al.
2017 Child et al. 2019). As illustrated in Figure (b), we train three auto-rogeessive sparse transform- ers to model P(Zi (andio), P(Zz|2 Zi, yaudio), and P(Za Zi, Yaudio), which we refer to as top prior, middle conditional prior, and bottom conditional prior, respectively. The middle and bottom priors use the token sequenees from the upper levels, with up-sampling achieved through upsampling modules, corresponding the conditioners of Jukebox. We additionally condition each prior on Yaudio- Appendix provides further details of the stage-2 model. Appendix B.3 evaluates the common objective metrics SoNIDo.
5

---

Published in Transactions on Machine Learning Research (05/2025)
2.3 SoNIDO Other Music Foundation Models
This section compares the architecture of SONIDO with those of other well-known music foundation models,
i..,, Jukebox (Dhariwal al. 2020). MusicLM (Agostinelli et al. 2023). and MusicGen (Copet et al. 2023 These models are categorized on the basis of how their stage-1 I models are constructed; (a) SQ-VAE-2 (b)
multi-resolution VQ-VAEs, and (c) residual vector quantization (RVQ), as illustrated in Figure 3
+
Z
2
Z
Z
Zz
Z3
Zq
(a) SQ-VAF-2
(b) multi-resolution VQ-VAEs
(c) RVQ
Figure 3: Stage-1 model comparison.
While SONIDO and Jukebox exhibit some shared characteristics, such as three-level architecture in the stage-1 model, SoNIDo is based on SQ-VAE-2 whereas Jukebox used multi-resolution VQ-VAEs. In Juke- box, token streams Z1, Z2, and Z3 were independently and separately trained for different sampling rates. Consequently, the /th transformer was designed to generate Zi by upsampling the previous token sequence Zi-1 for 1 2. In contrast, SONIDO's token streams from the stage-1 model are jointly trained and collaboratively contribute to the comprehensive modeling of the waveform at the original sampling rate from scratch. Given the tight interrelation between token streams from different levels, SONIDo's Ith transformer is conditioned on all the upper token streams.
Recent approaches such as MusicLM and MusicGen used RVQ a bottleneck feature space instead of applying these hicrarchical quantization methods (c.g., VAE-2).Thhes approaches also use transformers to model the prior of the music-token streams P(Zi.z). In the context of token-sequence length for generating 1 s of audio at target sampling rate sr, the bottom-most token-sequence length in SONIDO and Jukebox is sr/8, while MusicGen requires a token-sequence length of sr 97/640. RVQ-based models excute quantization in highly compressed latent spaces using a series of vector quantization layers, effectively shortening the token sequence to be learned by transformers in the stage-2 model.
3 SONIDO on Music Downstream Tasks
We first obtain SoNIDO features from input audio with task-ngnostic feature extraction process. Depending whether the downstream task is time-invariant or time-varying, then apply different pre-processing steps. Finally, we inject the pre-processed SoNIDO features into a proper location of target task-specific model. The selection of such a location is explained in Section
3.1 Task-agnostic Feature Extraction
We follow the feature extraction pipeline in Castellon et al. (2021): Nizzui et al. 2002 Huang et al. (2022b) based on the pre-trained frozen SONIDO. The music waveform is first converted to multi-level token sequences via the stage-1 encoder of SoNIDo. The tokens are then fod into the top prior, middle conditional, and bottom conditional priors of SONIDO without auto-regressive ileration. The middle and bottom priors are conditioned by the ground-truth tokens produced with the stage-1 encoders. We extract the output of the N-th (N 36 as Castellon et al. (2021h) transformer layer in those prior models as SoNIDO features. If the CLAP audio embedding is not used as the condition of priors, we call the feature extraction unconditional extraction; otherwise, CLAAPconditionn extraction.

---

Published in Transactions on Machine Learning Research (05/2025)
Top feanures
franuees
feanures
apply tokun out with shuffling
Block
vocal.
MLP
vihrato,
single vector for classfication
Figure 4: Attention-based feature aggregation and token-out data augmentation. "T", "M", "B" mean top, middle, and bottom priors, respectively. Token-out augmentation deletes masked tokens from input soquence. Attention block aggregates sequence into single vector and is followed by MLP to prodict tags.
The maximum sequence length of the priors 8192. Since the hown-sampling rates in the stage-1 model are 128x (top); 32x (middle), and 8x (bottom), the same amount of 8192 tokens in different priors cor- respond to 24 s, 6 and 1.5 s in the time domain, forming set of hierarchical multi-rate features. To save compulational resources, the SONIDO features are pre-computed for most deam tasks, except for HTDemucs mentioned in Section 4.2.2, where a clip of music is randomly selected on- on-the-fly during training. To compute features for a long audio input, we treat the input as overlapping segments with the ratio Novip- If Novip sufficiently large such that the overlap is longer than the perception field of the stage-2 model, it is guaranteed that the feature extraction result is not affected by the segmentation.
3.2 Feature Pre- Pre-processing for Time-invariant Downstream Tasks
If the downstream task is time-invariant, we first divide input audio into non-overlapping segments of 24 (top), 6 (middle), and 1.5 (bottom) For cach prior, the SoNIDo features of the segment are reduced to single token via average pooling, forming 3 SONIDO token sequences in the end.
The common practice (Castellon et al. 2021 Li et al. 2023b) suggests using multi-layer perceptron (MLP) with a single hidden layer 512 dimensions to probe the features. However, SONIDO token sequences originate from priors with different time resolutions, which different from prior studies. To effectively use these hierarchical features, a sequence aggregation is required. We thus propose to aggregate the sequences via a standard attention block, which an attention layer followed by a feed-forward layer. This is inspired by the attention- based feature aggregation in instrument classification tasks (Gururani et al. 2019 Zhong et al. 2023a). We first concatenate the hierarchical SONIDo features into a single loken sequence then attach learnable class token at the front. The attention block is trained to aggregate all features into the class token, which is then converted to music tags or emotion scores by the aforementioned MLP. Hyperparameters as well as an ablation study on the sequence aggregation are provided in Appendix C
To prevent overfitting when using the concatenated token sequence train the attention block, we propose an on-the-fly data augmentation method called loken-oul. This inspired by SpecAugment (Park et al. 2019) and masked Transformers (Koutini et al.) 2022 Zhong et al. 2023b Comunità et al. 2024). in which part of the input is masked before feeding into deep neural networks. Unlike prior arts, token-out is applied to the whole token sequence extracted with the multiple layers in SONIDO, as illustrated in Figure The masking ratio is sampled between and 100% uniformly. As shown in Appendix features with the shallow attention layer and token-out augmentation led to performance improvement.
C rggregating the SoNIDo
3.3 Feature Pre- Pre-processing for Time-varying Downstream Tasks
When applying the SONIDO features to a task-specific model on a time-varying downstream task, we face several challenges, such as temporal alignment, proper amount of information compression, and sufficient feature adaptation before injecting features into the task-specific models.
7

---

Published in Transactions on Machine Learning Research (05/2025)
The temporal alignment between the SONIDO features and target model can be achieved by either pooling the SoNIDo features or using linear layers. Examples of these two cases are provided in Appendices F.2 and D respectively. Compared with average pooling or max maxx-pooling, we found that using linear layers can yield better performance, as described in Appendix
E22
Both information compression and feature adaptation can be done with linear layers. The output dimension is simply set to match the feature dimension of the target task-specific model. Empirically, one layer is sufficient for most of the models we have tested, except for music transcription with TFFT-Tranformer TToamm et al.)
2023)) which requires four layers, as described in Appendix D.3
4 Experiments
We conducted experiments to examine the usefulness of features extracted from music foundation models for understanding and generative downstream tasks by addressing two questions: Q1: Do extracted features have useful information for music understanding? Q2: Can extracted features boost current task-specific models for both understanding and generative tasks? To verify the generalizability of the results, we test not only the SoNIDo features, but also features extracted from Jukebox and the two public versions of MUSICGEN (Copet et al. 2023), namely MUSICGEN SMALL and MUSICGEN LARGE. As a major focus of this work is to extend the applicable downstream tasks, we evaluate Jukebox's features specifically for time-varying tasks and report the results from Castellon et al. (2021) for time-invariant tasks in Table !
To address the first question, we selected eight music tagging tasks and music transcription as representatives for understanding tasks. We verified that the extracted features encompass both time-invariant information of overall musical properties and time-varying information of specific musical events. For the second question, we tested the injection of extracted features into several task-spocific models for understanding and generative
tasks. They consist of one transcription model (Toyama el al. 2023 two separation models Mitsuluuji et al. 2022 Fabbro et al., 2023), and two mixing models from Martinez- Ramírez et al. (0022).
The feature extraction described in Section 3.1 is used for all experiments. Before applying the features
into downstream lasks, lask-dependent pre-processing is applied (see Sections 3.2 and 3.3 Details of the experimental sctup, results, and further ablation studies are provided in Appendices respectively. We only use features extracted from the top and middle layers of SONIDO. In the preliminary experiments, we found that including the bottom-layer features does not always improve the performance of understanding tasks. We assume this is due to the bottom layer mostly containing only the fine-grained information irrelevant to the tasks, thus degrading performance. This is discussed further in cach subscction.
and F
4.1 Usefulness of Extracted Features for Music Understanding
The following feature probing experiments demonsstate that features extracted from SoNIDO, MUSICGEN and Jukebox all contain valuable information, which is consistent with Castellon et al. (2021). However, for music transcription, the shallow network remains insufficient to match SOTA models. In Section 4.2 we show that injecting extracted fcatures into task-specific models can boost their performance beyond SOTA.
4.1.1 Music Tagging
We test a wide range of music tagging tasks as well as the emotion regression task: MagnaTaggATune (MTAT) (Law et al.] (2009) for auto tagging, Nsynth (Engel et al., (207]] for pitch and instrument recognition, EmsM (Soleymani et al. 2013) for emotion regression, GTZAN Tzanetakis & Cook 2002) for genre classification, GiantSteps (nnces et al. 2015 Korzeniowski & Widmer 2017) for musical key estimation, and VocalSet (Wilkins et al. 2018) for singer and singing technique identification. A summary of the datasets is shown in Table 9 in Appendix C We followed the pro-processing in previous studies (Li et al. 2023b; Yuan el al.) 2023 and used scikit-learn (Pedregosa et al., 2011) and mir eval (Raffel et al.) 2014) for metric computation. The average R² of arousal and valence axis is reported for EmoMusic. The feature pre-processing for timo-invariant downstream tasks in Section 3.2 including the feature aggregation and token-out augmentation applied for all tasks. Following common practice (Castellon et al. 2021 Li ct al. 2023b), an ML.P is then used to probe the aggregated features.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Music tagging. Benchmark results of SONIDO features in music tagging tasks (bold: top-2 score).
Dalaar
Noynth Pitch
Neyuth Trettronens
EnoMuuic Emosioo regression
GientSlepe Key
GTZAN
VocdiSa Singer
VorilSet Voral techniiques
Tok'
Anto ligging
Geare
Metties
ROC-AUC
mAP
Acc.
Avc.
Average R²
Weighted acc.
Aue.
Aue.
Auc.
Sipproiiadd
MaxCNN Pres Serra 2019
90.6
38.3
64.1
72.8
58.5
12.8
79.0
57.0
70.3
2022
91.7
41.3
70.5
78.1
66.6
28.6
85.5
Julo
Jalebbxx
Dharival
19.7
MubiiCCan Crap stan
Lope oi mh
90.4
88.5
911.
71.9
45.8
65.2
73.2
52.3
(65.1
MasoeGen large
Copee ot
3729
90.5
33.0
92.8
74.2
48.2
62.4
70.3
53.3
63.9
CLMR Spejkeroet Bargoyae
89.4
96.1
67.0
56.8
14.9
63.6
69.9
58.1
Slow Soo-NE Val-p11 Waine pi
$9.5
88.0
78.2
MUULB=contnsiive (M2
91.4
80.1
89.2
74.0
63.9
66.7
73.3
87.6
75.5
Hdall rocossaly
H&BEETT mucic Iim et [021 Li
20255
90.2
37.7
77.4
69.3
54.3
14.7
70.0
75.3
65.9
data2vee mmstr Bhevaki di ni An M
3.2hl
90.0
36.2
93.1
69.4
61.6
50.6
74.1
61.4
71.1
METI
91.3
01.2
72.6
68.0
65.6
79.3
76,9
Hicrustiool onlontigranin (ours] Sozillo
91.7
41.6
93.8
78.0
64.7
68.5
80.7
87.0
74.4
Table 3: Results of feature probing using shallow back-end MAPS (bold: best, underline: second-best).
Input
Note F1(%)
Spectrogram
18.83
Spectrogram + SoNIDo Top
57.20
Spectrogram SoNIDo Middle
64.98
Spectrogram + SoNIDo Top + SONIDO Middle
66. 06.02
Spectrogram MUSICGEN SMALL
53.18
Spectrogram + MUSICGEN LARGE
49.16
Spectrogram Jukebox
57.13
We conduct a preliminery study for SoNIDO with top prior features to compare unconditional extraction and features from the CLAP encoder. We use the tagging task for coarse-grained concepts on MTAT, and the classification task for fine-grained concepts (pitch) on Nsynth. We found that CLAP performs well for coarse concepts, while unconditional extraction results better accuracy for pitch estimation. CLAP-conditional extraction achieves better scores in both tasks. Details can be found in Appendix C and Table [00] Consequently, we report SONIDO's scores with the CLAP-conditional feature extraction for time-invariant understanding tasks.
extraction,
The test results on various datasets and benchmarks with prior studies are listed in Table 2 Probing the SONIDO and MUSIcGEN features both shown competitive scores in most lasks. The SONIDO's fea- tures reached the top-2 performance in auto tagging, pitch estimation, instrument classification, and genre classification. They also performs well for emotion regression and singer identification. While these are prompl-condditoneed generalive models, feature probing using these models reached comparable performance compared with SOTA encoder-only models specialized for understanding tasks.
4.1.2 Music Transcription
Boyond timo-invariant undcrstanding tasks, we continue the test on music transcription, which a time- varying understanding (ask. All of SoNIDo, MUSICGEN, and Jukebox features are obtained with uncon- ditional extraction described in Section 3.1 The dimension and time resolution of extracted features are aligned to those of the spectrogram with lincar layers. Following Castellon et al. (2021). the features are concatenated with the spectrogram. A single-layer shallow back-end network is used to probe these features.
We show the transcription performance the feature probing mentioned above on the MAPS dataset (Emiya ct al. 2010) in Table 3. All the features greatly improved the note-wise F1 score compared with using the spectrogram only. This suggests that all of SONIDO, MUSICGEN, and Jukebox features contain useful information for time-varying understanding tasks.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Music transcription results of Fl scores on MAPS (bold: best score, underline: second best).
"Note" refers to note-wise estimation. First row corresponds Transfooofore
Toyama al. 2023
Training data Input
Frame
Note
Note w/ Offset
Note w/ Offsct& Velocity
Spectrogram Spectrugram + SoNIDo Top
82.89
85.14
66.34
48,20
83.92
86.45
68.27
51.34
Spectrogram + SONIDo Top SoNIDo Middle
84.16
85.96
67.37
50.98
100%%
Spectrogram + MUSICGEN SMALL
82.94
85.97
68.27
50.42
Spectrugram + MUSICGEN LARGE
81.53
85.14
66.28
Spectrogram + Jukebox
84.23
86.54
68.26
50.46
Spectrogram
9.83
0.59
0.17
0.46
Spectrogram + SoNIDO Top
65.91
66.64
39.88
25.87
Spectrogram + SoNIDo Top SoNIDo Middle
71.57
75.00
46.18
30.63
10[%]
Spectrogram + MUSICGEN SMALL
63.73
65.90
39.00
24.94
Spectrogram + MUSICGEN LARGE
61.81
63.27
37.03
24.01
Spectrugram + Jukebox
70.43
73.76
45.80
30.42
4.2 Using Extracted Features to Boost Existing Task-specific Models
In Section 4.1 we showed that the extracted features contain useful knowledge for music understanding, In this section, we test all of SONIDo, MUSICGEN, and Jukebox features on several SOTA task-specific models, the tasks include music transcription, music source separation, and music mixing, which covered both music understanding and generative tasks. The experimental results indicate that the extracted features consistently boost the performances of lask-specifie models. We also observed that injecting the SoNIDO features accelerated the decrement of training loss in early epochs.
4.2.1 Music Transcription: hFT- Transformer
We applied the extracted features to hFT--Tranffrmmr (Toyama et al. 2023), a SOTA music transcription model for piano on MAPS (Emiya et al., 2010). to assess whether it surpasses existing models that rely solely the spectrogram. On the basis of the input spectrogram, AFT-Transformer estimates the frame-based note activation, along with the onsct, offset, and velocity of note (frame, onset, offise,, and velocity). It is transformer-based model consisting of two transformer encoders that work on different axes of the inpul and transformer decoder in the middle of these two encoders. Following the processing pipeline in Section ! we attempted injecting the SoNIDo, MUSICGEN, and Jukebox fcatures before the 1st encoder, 2nd encoder, and decoder. We found that feature injection before the decoder yields the best result, thus we adopt this injection method in the following experiments. All the training hyperparametens were kept. the same as in a previous study (Tyaaaa et al. 2023). Further details are provided in Appendix D.3
Following the common evaluation practice (Gardner et al. 2022 Toyama et al. 2023 report four F1 scores: frame-wise, note-wise, note-wise with offset, and note wise with offset and velocity using the check- point with the best validation Fl score. As shown in Table 4 injecting either SoNIDo, MUSICGEN SMALL, or Jukebox features improves the performance of hFT-Traasfoo Transformer. The performance gap is especially huge when the model is trained with a small subset of MAPS. This demonstrates the usefulness of injecting music foundation model features into downstream task models when training data are scarce. We also observed that the decrement of training loss is faster when either SONIDO, MUSICGEN, or Jukebox features are injected, shown in Figure in Appendix D.3
In the experiment involving the full MAPS, injecting top and middle SoNIDO features yields performance improvement. However, no improvement is observed when all the features from three layers are injected. A similar trend can be observed from the results of MUSICGEN LARGE. We assume that the network capacity required to interpret all the information contained the features could excooded that of hFT- Transformer, negatively impacting the model. This suggests that, disentangling feature information on the basis of information granularity to filter out irrelevant information is erucial for such injection.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Music source separation. Evaluation results on MUSDBI8 and MDXDB21.
MUSDBI8 (BSSEval vd SDR (dB))
MDXDB21 (global
SDR (dB))
Model
Buss
Drums
Other
Vocals
Average
Bass
Druns
Other
Vouals
Average
Open-Unmix (UMX)
4.01
4.35
2.79
5.66
4.20
4.50
4.46
2.66
5.55
4.29
UMX MusicGen Small
4.
4.55
3.18
5.66
4.41
4.61
4.29
2.92
5.42
4.31
UMX MusicGen Large
3.97
4.25
3.13
5.28
4.16
4.53
2.95
5.35
4.22
UMX
SONIDO
1.37
1.16
3.00
5.91
4.36
4.71
5.69
4.37
HTDemues (default)
8,
8.22
7.86
7.13
HTDemues (ublation 1)
7.94
7.24
HTDenues (ablation 2)
7.96
7.17
HTDemues STFT-2048
4.45
5.72
5.84
5.80
HTDemues STFT-4006
6.18
5.91
HTDemues CLAP
8.25
7.37
7.37
6..79
HTDemmes MusieGeu Small
86
7.44
7.21
HTDemmes MusieGen Large
7.50
7.40
7.37
6.86
HTDemues Jukebox
6.65
6.35
6.58
HTDemues SoNDDo
8.14
7.39
HTDemuce (trained on SDXDB23 Bleeding)
3.96
5.52
3.53
5.70
4.65
6.20
5.98
4.53
6.09
5.85
HTDemues + SoniDo (trained on SDXDB23 Bleeding)
5.50
6.06
3.97
5.82
5.43
6.41
6.40
4.64
7.19
6.16
4.2.2 Music Source Separation: UMX, HTDemucs
We select Open-Unmix (UMX) Stöter et al. 2019) and Demucs (HTDemucs) (Rouard et al. 2023 for feature injection. UMX estimates the time- free-fry mask of the target source using recurrent neural network (RNN) blocks. HTDemucs is a hybrid model with waveform U-Net branch and spectral U-Net branches. We inject the extracted features into the encoder block for UMX using a down-sampling block and in cach HTDemues branch using a cross-domain Transformer (dctails in Appendices E.1 and E.3 Based on the observation in Section 4.2.1 and for simplicity, we inject only top-level features from SoNIDo.
Table 5 lists the SDR scores on the test split of MUSDB18 Rafi et al. (2017) and the hidden split of MDXDB21 Mitsufuji et al. (2022): Fabbro et al. 20233. The details of the experiments are provided in Appendix E Similar to the experiment discussed in Section 4.2.1 a faster loss decrement is observed, as shown in Figures 12 and 13 in Appendix Injecting the SONIDO features into both UMX and HTDemucs greatly boosts the separation performance for both models, even on the unseen dataset MDXDB21. also improves the separation performance of HTDemues when training on data corrupted by bleeding errors (SDXDB23 Bleeding) Fabbro et al.) (2023). However, the MUSICGEN features do not always improve the results. Injecting the MUSICGEN SMALL features improved UMX, but not for the other cases. According to the ablation study results, injecting short-term Fourier transform (STFT) signal, CLAP features, MUSICGEN features, or Jukebox features leads to unstable training. However, no such behavior is observed when injecting the SONIDO features into HTDemucs. We assume that performance can be improved instability during training is avoided. As mentioned in Section 4.2.1] interpreting information contained in MUSICGEN LARGE could cost too much capacity of the downstream model and result in performance degradation.
In summary, observed that injecting the SONIDO features into separation models not only yields faster training and better performance but also improves the robustness to dataset corruption.
4.2.3 Music Mixing: Waw---vve-U.et.. CRAFx2
Wavee---eet (Steinmetz et al. 2022 along with modified CRAFx (Martnez-Raminn et al. 2020
henceforth referred to CRAFx2, used as the baselines. The input to both networks is the stereo stems pre-processed by Fx-normalization These models do not handle high-level information relevant to mixing, such as genre, instrumentation, or mood. Conditioning these models with extracted features, which implicitly contain such information, is expected to improve mixing performance. The features are computed from the monaural downmix of the mixture, which corresponds the summation of the Fx-normalized input stems. To incorporate these features, we condition both networks using Feature-wise Linear Modulation (FiLM) layers (Perez et al.
Shartnez-Raaíres et al. (022).. and the output is the stereo mixture.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Music mixing. Evaluation results on the MDXDB21-dry and MUSDB18 test sets include mean absolute percentage error for audio effect-related features, their average, and stereo- inverannvvarant loss. More details are provided in Table 15
MDXDD21-dry test set
MUSDBIS test sot
Model
Spectral
Panning
Dynamic
Londneess
Average
Stereo
Spoctral
Panning
Dynomic
Londiness
Avg
Stereo
Invarimnt
Wave-U-Net (defiuls)
0.234
0.168
0.173
89.631
0.201
0.085
0.167
0.154
34.253
Mix Wave + Jukebux
0.260
0.154
0.175
83.717
0.206
0.157
0.158
32.575
C-NeW MusieCem Small
0:340
0.197
0.064
0.147
0.162
80.161
0.214
0.158
0.079
0.163
0.153
32.151
Mix MaveWave Wave-U-Nee MusieGen Large
0.241
0.231
0.006
0.145
0.171
81.161
0.205
0.192
0.075
0.167
0.160
32.649
Mix-Waw-U-Mo + SONIDO
0.226
0.180
0.067
0.131
0.151
78.180
0.186
0.175
0.063
0.179
0.151
30.116
CRAF32 (default)
0.193
0.179
0.070
0.152
0.148
82.095
0.193
0.154
0.081
0.165
0.118
32.850
CRAPx2 + Jukebor
0:231
0.249
0.076
0.144
0.175
87.973
0.216
0.220
0.082
0.165
0.171
36.172
CRAFX2 MusieGen Small
0.229
0.244
0.072
0.148
0.173
87.273
0:211
0.204
0.083
0.178
0.169
36.418
CRAFX2 MusiGeen Large
0.223
0.219
0.073
0.132
0.363
87.318
0.224
0.206
0.080
0.175
0.171
36.519
CRAPX2 SoxDo
0:221
0.176
0.064
0.171
0.158
79.861
0.187
0.154
0.076
0.160
0.146
30.155
2018). For Mix- Wave- U-Net, we inject features into the up-sampling and bottleneck one-dimensional (1D)
convolutional blocks. For CRAFx2, we use FiLM layers to condition both the latent-space mixer and synthesis back-end (sec Appendix
We train all models on MUSDB18, and the stereo-invarian loss, along with all training hyperparameters, remains the same, as in a previous study (Martinez- (Martinez-Ramirez et al., 2022). Due to the inherent subjectivity of the task, identifying the best model is challenging. Thus, as shown in Table [ an objective evaluation is conducted by measuring the proximity between the output mixes and larget mixes on the same test sets as in Martinez-Ramires et al. (2022). The proximity measurement is based on objective metrics (Steinmetz 2022; Martinez-Ramírez et al.) 2022). These metrics consist of spectral, panning, dynamic, and
ctt al. loudness low-level audio features, which are the key audio characteristics often manipulated during the mixing process. Further details and experiments are provided in Appendices
and F3
respectively.
As shown in Table 6 conditioning both architectures with the SoNIDO features improved objective per- formance. The training and validation curves in Figure 14 of Appendix E3 show faster loss decrease in early epochs and better generalization, respectively. Although there is no standardized objective evaluation due to the subjective nature of the task (Steinmetz et al. 2022). the presented metrics suggest that the best-performing model closely aligns with the target mixes, resembling professional human-made mixes.
Using the SONIDO features leads the best performance. The Jukebox and MUSICGEN features provide improvements but not as effective as those of SONIDO, particularly for CRAFx2, where the default model outperforms both Jukebox and MUSICGEN in various metrics. For MUSICGEN, we assume this performance gap may be attributed to its training on 32 kHz audio compared with the 44.1 kHz used for SoNIDO, which limit its effectiveness for full-band tasks, such as music mixing, that require higher sampling rates. There is no data-driven approach that used lask-agnostic features of the input stems for music mixing improvement.
Thus, we can conclude that incorporating the SONIDO features benefits both the training and performance of automatic music mixing models. This aligns with recent design studies (Lefford et al. 2021 Vanka et al. 2023), advocating for the incorporation of contextual inputs.
5 Ethical Concerns
To Irain SONIDO, we acquired an internal datasel of library music with licensing explicitly allowing machine learning training. The dataset is mostly non-vocal, biased toward orchestral and western music. A model trained on this dataset unlikely to characterize equally well for all types of music. The learnt intermediate embedding may reflect the bias. When using such a biased music foundation model as a performance booster, thorough verification is required before using such model for practical use or the decision process.
12

---

Published in Transactions on Machine Learning Research (05/2025)
6 Conclusions
We extended the use of music foundation models from MIR to generic music downstream tasks. The lask-agnostic intermediate representation extracted using proposed SoNIDO model has been applied to task- specific models music tagging, music transcription, music source separation, and music mixing. On the basis of the evaluation results, performance improvement is observed for all selected music downstream tasks. This suggests that incorporating the intermediate features extracted from pro- pre-trained auto-regrossive music foundation model should be considered as generic booster in future development of lask-specifie models. This is especially helpful when it is difficult to acquire sufficient dataset the computation resource does not allow large-scale training, A study on the bias propagation of pre-trained music foundation model to downstream task model should be conducted in another future work.
Acknowledgement
We would like to thank Takashi Shibuya, Naoya Takahashi, Marc Ferras, and Masato Tshii for many helpful comments during the preparation of this manuscript. Besides, we thank anonymous reviewers for their valuable suggestions and comments.
13

---

Published in Transactions on Machine Learning Research (05/2025)
References
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. Musiclm: Generating music from text, 2023.
Alexci Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298 1312. PMLR, 2022.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Dounbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Ti Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Gross- man, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kudi- tipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lec, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhni Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021. URL https:/ pottps://crtes h/tm. stanford. edu/ http//ssees//epport..o pdf
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodci. Language models are few-shot learners. CoRR, abs/ 2205/2005.1 14165, 2020. URL htttps://axxxi www. wwwg/rrg/////eeeeee.e
14165.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful rep- resentations for music information retrieval. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd International Socicty for Music Information Retricval Conference, ISMIR 2021, Online, November 7-12, 2021, pp. 88 28-96, ismir. wtt.://ewiiiiii...eeeaaiiiaaaaaa
2021. URL htttps://archivee
pdf
Ke Chen, Gordon Wichern, François G Germain, and Jonathan Le Roux. Pac-hubert: Self-supervised music source separation via primitive auditory clustering and hidden-unit bert. arXiv preprint arXiv:2304.02160. 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Σlya Sutskever. Gen- erative pretraining from pixels. In International conference on machine learning, pp. 1691-1703. PMT., 2020.
Kin Wai Cheuk, Dorien Herremans, and Li Su. Reconvat: A semi-superrised automatic music transcription framework for low-resource real-world data. In Procoodings of the 29th ACM International Conference on Multimedia, pp. 3918 3926, 2021a.
Kin Wai Cheuk, Yin-Jyun Luo, Emmanouil Benetos, and Dorien Herremans. Revisiting the onsets and frames model with additive attention. In Proceedings of the International Joint Conference on Neural Networks, pp. In press. IEFE, 2021b. doi: 10.100//5P..211..00144
14

---

Published in Transactions on Machine Learning Research (05/2025)
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse trans- formers. CoRR, (b8/1904.10550 2019. http://t://arxtt httppp://aaa..voooo/aa////////// org/ 10509
Joseph T Colonel and Joshua Reiss. Reverse engineering of a recording mix with differentiable digital signal processing. The Journal the Acoustical Socicty of America, 150(1):608-61) 619, 2021.
Marco Comunità, Zhi Zhong, Akira Takahashi, Shiqi Yang, Mengjie Zhao, Koichi Saito, Yukara Ikemiya, Takashi Shibuya, Shusuke Takahashi, and Yuki Mitsufuji. Specmaskgit: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond. arXiv preprint arXiv:2406.17672, 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation, 2023.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL hutps:://openrevies net/ forum?id==vvddB222R) Featured Certification, Reproducibility Certification.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.orrr org/ abs/1810 04805
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Tlya Sutskever. Juke- box: A generative model for music, 2020.
Chris Donahue, John Thickstun, and Percy Liang. Melody transcription via generative pre-training. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, 2022.
Zhiyao Duan, Bryan Pardo, and Changshui Zhang, Multiple fundamental frequency estimation by modeling spectral peaks and non-peak regions. TEEE Transactions Audio, Speech, and Language Processing, 18 (8):2121 2133, 2010. doi: 10.1109/TASL.2111 2AA.L.201..2021111
Valentin Emiya, Roland Badcau, and Bertrand David. Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle. TEEE Transactions on Audio, Speech, and Language Processing, 18(6):1643-1666 1654, 2010.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Moham- mad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders, 2017.
Giorgio Fabbro, Stefan Uhlich, Chieh-Hsin Lai, Woosung Choi, Marco Martinez-Raméres, Weihsiang Liao, Igor Gadelha, Geraldo Ramos, Eddie Hsu, Hugo Rodrigues, et al. The Sound Demixing Challenge 2023 music demixing track. arXiv preprint (xXiv:2308006977 2023.
Seth * Forsgren and Hayk* Martiros. Riffusion Stable diffusion for real-time music generation. 2022. URL https://riffusion com/about
Joachim Fritsch. High quality musical audio source separation. Master's thesis, 2012.
Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. Mt3: Multi-task multitrack music transcription. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Siddharth Gururani, Mohit Sharma, and Alexander Lerch. An attention mechanism for musical instrument recognition. In ISMIR 2019, 2019.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, lan Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onscts and frames: Dual-objective piano transcription. In Procoedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2018. URL https::////// /ttp////ww..r/////////0.0...
15

---

Published in Transactions on Machine Learning Research (05/2025)
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Ab- delrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM TASLP, 29:3451 3460, 2021.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitttion networks. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. Mu- lan: joint embedding of music audio and natural language. In Precti Rao, Hema A. Murthy, Ajay Srinivasamurthy, Rachel M. Bittner, Rafael Caro Repetto, Masataka Goto, Xavier Serra, and Marius Miron (eds.), Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, pp. 559-566, 2022a. URL https:
//archives. ismir http://miir222//pa///000000 .df
Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, Jesse Engel, Quoe V. Le, William Chan, Zhifeng Chen, and Wei Han. Noise2music: Text-conditioned music generation with diffusion models, 2023.
Zili Huang, Shinji Watanabe, Shu-wen Yang, Paola García, and Sanjeev Khudanpur. Investigating self- supervised learning for speech enhancement and separation. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 0837-6841, 2022b. doi: 10.1109/
CCASSP443022.2222.77466
Kuo-Hsuan Hung, Szu wei Fu, Huan-Hsia Tseng, -isn-TTien Chiang, Yu Tsao, and Chii-Wann Lin. Boosting Self-Supervised Embeddings for Speech Enhancement. In Proc. Interspeech 2022, pp. 186 190, 2022. doi: 10.21 2221437/Interspeeee 2022-022222020
Rec ITU-R. Itu-r bs. 1770-2, algorithms to measure audio programme loudness and true-peak audio level. International Telccommuniations Union, Geneva, 2011.
Rainer Kelz, Sebastian Böck, and Gerhard Widmer. Deep polyphonic adsr piano note transcription. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 246 250. IEEE, 2019.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A metric for evaluating music enhancement algorithms, 2019.
Jong Wook Kim and Juan Pablo Bello. Adversarial learning for improved onsets and frames music tran- scription. International Society forMusic Information Retricval Conference, pp. 670-677, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. International
Conference on Learning Representation (ICLR), 2015.
Peter Knees, Ángel Faraldo Pérez, Herrera. Boyer, Richard Vogl, Sebastian Böck, Florian Hörschläger, Mick- ael Le Goff, et al. Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Málaga, Spain. [Málga]] International Society for Music Information Retrieval, 2015. p. 364-70. International Society for Music Information Retrieval (ISMIR), 2015.
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onset and offset times. IEEE/ACM ACM Transactions on Audio, Speech, and Language Processing, 29:3707-3717, 2021.
Junghyun Koo, Marco A Martinez-Ramírer Wei-Hsiang Liao, Stefan Uhlich, Kyogu Lee, and Yuki Mitsufuji. Music mixing style transfer: A contrastive learning approach to disentangle audio effects. 2023.
Filip Korzeniowski and Gerhard Widmer. End-to-end musical key estimation using convolutional neural network. In 2017 25th European Signal Processing Conference (EUSIPCO), pp. 966 970. IEFF, 2017.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Proc. of Interspeech 2022, 2022.
Max W. Y. Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, Jitong Chen, Yuping Wang, and Yuxuan Wang. Efficient neural music generation. In Thirty-soventh Conference on Neural Information Processing Systems, 2023. URL https://opeereviee
net/forum?idrcxxa0SS5.
Edith Law, Kris West, Michael 1. Mandel, Mert Bay, and J. Stephen Downie. Evaluation of algorithms
using games: The case of music lagging. In Keiji Hirata, George Tzanetakis, and Kazuyoshi Yoshii (eds.), Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center, Kobe, Japan, October 26-30, 2009, pp. 387-392. International So-
ciely for Music Information Retrieval, 2009. URL http://ismir2000 ismiii wwemi..eee/proooo wtwee///rroeedin///S----6. pdf
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation
using residual quantization. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11523- 2022.
M Nyssim Lefford, Gary Bromham, György Fazekas, and David Moffat. Context aware intelligent mixing systems. Journal of the Audio Engineering Society, 2021.
Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, (2(2):222 535, 2019. doi: 10.1109 10.1109/TMM.201 MMMM..018.286000
Peike Li, Boyu Chen, Yao Yao, Yikai Wang, Allen Wang, and Alex Wang. Jen-1: Text-guided universal music generation with omnidirectional diffusion models, 2023a.
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Yike Guo, and Jie Fu. Mert: Acoustic music understanding model with large-scale self-supervised training, 2023b.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JML..orr, 2023.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. URL http:/tttt/:/ap..emmattttt www.ssteeaannniii.eeeeeee.
53592270
Wei-Tsung Lu, Ju- Ju-Chiang Wang, Qiuqiang Kong, and Yun-Ning Hung. Music source separation with band- split rope transformer. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 481-485, 2024. doi:
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, et al. Foundation models for music: survey. arXiv preprint arXiv:2408. 14444440. 2024.
Zheng Ma, Brecht De Man, Pedro DL Pestana, Dawn AA Black, and Joshua D Reiss. Intelligent multitrack dynamic range compression. Journal of the Audio Engineering Society, 63(6):412 144-12222,,22 2015.
Marco A Martinez-Ramírez, Emmanouil Benetos, and Joshua D Reiss. Deep learning for black-box modeling of audio elfects. Applied Sciences, 10(2):638, 2020.
Marco A Martinez-Ramírez, Daniel Stoller, and David Moffat. A deep learning approach to intelligent drum mixing with the Wave U-Net. Journal of the Audio Engineering Society, 2021.
Marco A Maméezez-Ramrrz, Wei-Hsiang Liao, Giorgio Fabbro, Stefan Uhlich, Chihiro Nagashima, and Yuki Mitsufuji. Automatic music mixing with deep learning and out-of-domain data. In ISMIR, 2022.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Matthew C McCallum, Filip Korzeniowski, Sergio Oramas, Fabien Gouyon, and Andreas Ehmann. Su- pervised and unsupervised learning of audio representations for music understanding. In ISMIR 2022, 2022.
Gabriel Meseguer- Brocal and Geoffroy Peeters. Conditioned-u-nett Introducing a control mechanism in the u-net for multiple source separations. 2019.
Yuki Mitsufuji, Giorgio Fabbro, Stefan Uhlich, Fabian-Robert Stöter, Alexandre Défosscz, Minseok Kim, Woosung Choi, Chin-Yur Yu, and Kin-Wai Cheuk. Music Demixing Challenge 2021. Frontiers in Signal Processing, 1:808395, 2022.
Mateusz Modrzejewski, Piotr Szachewicz, and Przemyslaw Rokita. Transfer learning with deep neural embeddings for music classification tasks. In Leszek Rutkowski, Rafal Scherer, Marcin Korytkowski, Witold Pedrycz, Ryszard Tadeusiewicz, and Jacek M. Zurada (eds.), Artificial Intelligence and Soft Computing, pp. 72-81, Cham, 2023. Springer International Publishing. ISBN 978-3-031-23492-7.
David Moffat and Mark B Sandler. Approaches in intelligent music production. In Arts, volume 8, pp. 125. MDPI, 2019.
Daisuke Nizuumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spoctrogram modeling using masked autoencoders for learning general-purposee audio representation. In HEAR: Holistic Evaluation of Audio Representations, pp. 1-24. PMLR, 2022.
Aditya Arie Nugraha, Antoine Liutkus, and Emmanuel Vincent. Multichannel music separation with deep neural networks. In Proc. of 24th European Signal Processing Conference (BUSIPCO), pp. 1748 1752, 2016.
Daniel S Park, William Chan, Yu Zhang, Chung- Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech 2019, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Geoffroy Peeters. A large set of audio features for sound description (similarity and classification) in the CUIDADO project. Analysis Analysis/Synthosis /Synthssis Team. IRCAM, Paris, France, 54(0))1-25,, 2004.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jordi Pons and Xavier Serra. musicnn: Pre-trained convolutional neural networks for music audio tagging. arXiv preprint arXiv:1909.06654 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Brian McFee, Eric Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. Mir eval: A transparent inplementation of common mir metrics. In ISMIR, volune 10, pp. 2014, 2014.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:11140:6, 2019. URL htttps://app semantccschola..comm org/ CorpusID:204833000
18

---

Published in Transactions on Machine Learning Research (05/2025)
Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. Musdb18-a corpus for music separation. 2017.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2 In Proc. Advances in Neural Information Processing Systems (NeurIPSS) pp. 14866 14876, 2019.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separa- tion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Mohammad Soleymani, Micheal N Caro, Erik M Schmidt, Cheng- Ya Sha, and Yi-Hsuan Yang. 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM international workshop Crowdsourcing for multimedia, pp. 1 1-6, 2013.
Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations. ar preprint arXiv:2103.09410, 2021.
Ryan Stables, Joshua D. Reiss, and Brecht De Man. Intelligent Music Production, Focal Press, 2019.
Christian J Steinmetz, Jordi Pons, Santiago Pascual, and Joan Serrå. Automatic multitrack mixing with a differentiable mixing console of neural audio effects. In ICASSP. IEEE, 2021.
Christian J. Steinmetz, Soumya Sai Vanka, Marco A Martinez-Ramirez, and Gary Bromham. Deep Learning for Automatic Mixing. ISMIR, December 2022. URL httts:///d4aa hittaam.gittub..... io/tutorial
F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji. Open-unmix reference implementation for music source separation. Journal of Open Source Software, 2019. doi: 10.21105//oss.010600 01667.
URL https:
7/doi org/10. 2e1111///os.111110
Li Su and Yi-Hsuan Yang. Escaping from the abyss of manual annotation: New methodology of building poly- phonic datasets for automatic music transcription. In Music, Mind, and Embodiment: 11th International Symposium, CMMR 2015, Plymouth, UK, June 16-19, 2015, Revised Selected Papers 11, pp. 309 321. Springer, 2016.
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chich-Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. SQ-VAE: Variational bayes on discrete representation with self-annealed stochastic quantization. In International Conference on Machine Learning, 2022.
Yuhta Takida, Yukara Ikemiya, Takashi Shibuya, Chieh-Hsin Lai, Kazuki Shimada, Naoki Murata, Naoki Murata, Toshimitsu Uesaka, Kengo Uchida, and Yuki Mitsufuji. Hq-vac: Hicrarchical discrete represen- tation learning with variational bayes. Transactions on Machine Learning Research (TMLR), 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahni Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Toannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Ruther- ford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tu Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdich, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanau, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choc, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Pi- queras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mchran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura. Culp, Lev Proleev, Yi Luan, Xi Chen,
19

---

Published in Transactions on Machine Learning Research (05/2025)
James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalic Clay, Phil Crone, Tomas Ko- cisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabricl Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Decni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Nya Kornakov, Nithya Attauuri, Tñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjic Gu, Qiao Zhang, Jordan Grim- stad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Bremnan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Calt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wor Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Alm, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontack Lim, Sarah Hodkinson, Pranay Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justín Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Al- bin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthow Rahtz, Mai Giméncz, Logg Yeung, James Koeling, Pelko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lene, Timothy Chung, Acdan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora. Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Tinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufmaan Simon Tokunine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih- Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víc- tor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Unlü, Zhishuai Zhang, Mo- hammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki,
20

---

Published in Transactions on Machine Learning Research (05/2025)
Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Pagannin, Jcan-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shiv- akumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghami, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Sid- dhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Jamus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Ålgmyr, Timothée Lot- taz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, Frangois-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixera,, Matthew Fritze, Francesco Bertolini, Liana- Elconora Marinescu, Martin Bölle, Dominik Panlus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejezuk, Dominik Ra- biej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Tsrael, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébasticn Percira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, 7J Yan, Kane Jang, Cheng- Lee, Wojeiech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yan- ping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateck Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichen- baum, Thomas Brovelli, Sahitya Polluri, Preethi Lahoti, Cip Baelu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Åhdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjic Zhou, Mchrdad Khatir, Charles Sutton, Wojeiech Readkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Al-
21

---

Published in Transactions on Machine Learning Research (05/2025)
nahlawi, Iomut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, An- drew Lee, Nino Vicillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geolfrey Irving, Ed- ward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiáski, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeck- emeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Lc, Lu Li, Chimazic Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Prascem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohamnadHHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee- Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xuc, Chen Elkind, Oliver Woodman, John Carpenter, George Papa- makarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-Fernandezz, Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann- Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganch, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Tonila, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette- Choo, Yunjie Li, TJ Lu, Abe Itycheriah,, Prakash ShrofT, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona. Robenek, Bhavishya. Mittal, Ben Albrecht,, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimezak- Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang. Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides,
22

---

Published in Transactions on Machine Learning Research (05/2025)
Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Key- van Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gclman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, War- ren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Authony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sehastian Ruder, Morgan Redshaw, Jinhyuk Leo, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madcleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlag, Xiaolin Li, Peter Danenberg, Dennís Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vincet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoe Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024. URL
223/ 11805
John Thickstun, Zaïd Harchaoui, and Sham M. Kakade. Learning features of music from scratch. In ICLR, volume abs/ bbbs/161..ot (661199877, 2016.
Keisuke Toyama, Taketo Akama, Yukara Ikemiya, Yuhta Takida, Wei-Hsiang Liao, and Yuki Mitsufuji. Automatic music transcription with hicrarchical frequency-ting transformer. In Proceedings of the 24th International Society for Music Information Retrieval Conference, pp. 2023.
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. TEEE Transactions on speech and audio processing, 10(5):293 302, 2002.
George Tzanetakis, Randy Jones, and Kirk McNally. Stereo panning features for classifying recording pro- duction style. In ISMIR, pp. 441-444, 2007.
Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based on deep neural networks through data augmentation and network blending. In Proc. of IEEE ICASSP, pp. 261 265, 2017. doi: 10.1109/ICASSP S.T...01..77997 7952158..
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proc. Advances in Neural Information Processing Systems (NeurPPS), pp. 6306 6611, 2017.
Soumya Sai Vanka, Maryam Safi, Jean- Baptita Rolland, and George Fazekas. Adoption of ai technology in the music mixing workflow: An investigation. 2023.
Soumya Sai Vanka, Christian Steinmetz, Jean- Bap-ittt Rolland, Joshua Reiss, and George Fazckas. Diff- MST: Differentiable mixing style transfer. In ISMIR, 2024.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Milia Polosukhin. Attention is all you need. In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
neurips wwerrips.c/ppppeeeeee filesspapel
2017/Eile/3f5eee2233 2o77ffiif2fee2222/eeeeessassssa2
Luyu Wang, Pauline Lue, Yan Wu, Adria Recasens, Lueas Smaira, Andrew Brock, Andrew Jaegle, Jean- Baptiste Alayrac, Sander Dieleman, Joao Carreira, et al. Towards learning universal audio representa- tions. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4593 4597. IEEE, 2022.
Julia Wilkins, Prem Seetharaman, Alison Wahl, and Bryan Pardo. Vocalset: A singing voice dataset. In ISMIR, pp. 468 474, 2018.
Yusong Wu* Ke Chen Tianyu Zhang*, Yuchen Hui* Taylor scale contrastive language-audin pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
and Shlomo Dubnov. Large-
Qingyang Xi, Rachel M Bittner, Johan Pauwels, Xuzhou Ye, and Juan Pablo Bello. Guitarset: A dataset for guitar transcription. In ISMIR, pp. 453 460, 2018.
Yuya Yamamoto, Juhan Nam, and Hiroko Terasawa. Deformable CNN and imbalance-aware feature learning for singing techmique classification. In Hanseok Ko and John H. L. Hansen (eds.), Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 2778 2782. ISCA, 2022. doi: 10.2443//NTERSPEECCH 2022 11137. URL https: 2022-11131
//doi. 10. (11477/nteerspeeeee...
Donghha Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watanabe, and Helen Meng. Uniaudio: An audio foundation model toward universal audio generation, 2023.
Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, et al. Marble: Music audio representation benchmark for universal evaluation. arXiv preprint 10555, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Trans. Audio, Speech, Lang. Process., 30:495- 100499-507, 2022.
Zhi Zhong, Masato Hirano, Kazuki Shimada, Kazuya Tateishi, Shusuke Takahashi, and Yuki Mitsufuji. An attention-based approach to hierarchical multi-label music instrument classification. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Zhi Zhong, Hao Shi, Masato Hirano, Kazuki Shimada, Kazuya Tateishi, Takashi Shibuya, Shusuke Takahashi, and Yuki Mitsufuji. Extending audio masked autoencoders toward audio restoration. In 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5, 2023b. doi: A..100/WWAPAAAS826.0000... 10248871.
A Related Work
A.1 Understanding models
Understanding models based on supervised learning (SL) has shown good performance on music tagging tasks (Zhong et al.) [2023a] Koutini et al.] 2022 McCallum et al.] 2022), but they have difficulties into addressing tasks that involve unseen annotations during inference MMcCallum al. 2022 Niizumi et al., 2022).
24

---

Published in Transactions on Machine Learning Research (05/2025)
Self-supervised learning (SSL), however, learns features without human annotation. For example, contrastive learning reflects data similarity and dissimilarity in the learned representation (Spijkervet & Burgoyne 2021
McCallum et al., 2022). Masked reconstruction asks the model to predict the missing data and has been widely applicd to language (Devlin et al., 2018), speech (Hsu et al. 2021 sound (Niizumi et al. 2022 and
music lasks (Chen et al. 2023). MERT (Li et al. 2023b Turther exlended masked reconstruction to both acoustic and music features.
SSL methods achieved high performance across a wider range of tasks than SL methods. While a comprehen- sive set of downstream tasks, including music tagging (Li et al. 2023b McCallum et al. 2022 Nizzmi et al, 2022), music source separation (Chen et al. 2023 Li et al. (023b) and music bandwidth extension (Zhong
et al. 2023b), has been examined, some tasks related to music production, such as music transcription and music mixing, have not been well-investigated.
A.2 Generative models
Most understanding models are encoder-only models, in which generation is not covered. In contrast, gen-
crative models have the ability of gencration as well as lcarning reprosentation (Chen et al. 2020 Dhariwal
et al. 2020). Some generative models are based on auto-regressive modeling, thus can execule not only
generation but tasks such as music continuation (Dhariwal et al., 2020 Agostinelli al., 2023
Copet et al.
2023 Recent diffusion-based models can even exccute music inpainting (Forsgren & Martiros 2022 Li et al.
2023a: Huang et al. 2023) and music style transfer (Liu et al.] 2023).
Multi-tasking can also be achieved by
mask-augmentation [Li et al. 2023a] or explicit design (Yang et al. 2023
Generation can be conditioned
by text, lyrics, and melody (Copet et al. 20223 Agostinelli et al. 2023 Dhariwal et al.) 2020). The use of
generative models greatly extends the coverage of applicable music downstream tasks.
A.3 Neural tokenizer in generative models
Vector quantization (VQ) van den Oord et al. 2017 has been promising step for music generative modeling. Tokens obtained by VQ can be better characterized with generative models than raw signals. Building music foundation models on top of tokenizers has become a mainstream approach AAgostinelli
et al.
2023 Copet et al., 2023).
VQ-VAE-2 (Razavi et al. 2019) first extended VQ learning to hicrarchical discrete reprosentation learning in computer vision, which prompted the emergence of the pioneering music generation model (Dhariwal et al. 2020). It was shown to learn global and local information in top and bottom levels, respectively. Another variant of VQ aimed at learning structured discrete representation is residual VQ, which assigns multiple codes to encoded vectors in a residual manner (Zeghidour et al. 2022 Loe et al. 2022). Residual VQ was initially proposed for neural audio codec (Zeghidour el. al. 2022 Délossez el al. 2023 and shown to result in coarse-to-fine representation (Lee et al. 2022)..
HQ-VAE was proposed to encompass two advanced VQ schemes including RVQ within the variational Bayes framework (Takida et. al. 2024h. The unified scheme enhances the codebook utilization of the VQ-based models due to the effect of self-annealing (Takida et al.) 2022). The authors constructed a hierarchical latent space with three levels and jointly trained the levels on an audio dataset. The model achieved local-to-global representation similar to VQ-VAE-2 thus being freed from layer collapse.
B Architecture of SONIDO
SONIDO is a generative model that generates music samples conditioned on given text prompts. We train the model using an internal datasel contains around 115k studio-quality library music tracks sampled at 44.1 kHz. Their lengths vary from 30s to 150s, their tempos vary between 50bpm to 200bpm, and the total length is around 4,000h. 90% of the dataset is non-vocal. Although there are more than 50 genres included in the dataset, it is biased toward orchestral and western music. To generate music samples, we use the three priors in SONIDO consecutively from the top level to the bottom level. Tt is important to note that
25

---

Published in Transactions on Machine Learning Research (05/2025)
during music generation, we use the token sequences sampled from top and middle priors for conditioning. For training and feature extraction, however, we rely on ground-truth tokens.
Given a text prompt, we first obtain the CLAP embedding yreat = CLAP.text embedding(x)) Conditioned on Mtezt; the top prior transformer generates a token sequence Z1 in an reto-regressive manner. Subsequently, Z is led into the middle conditional transformer along with ytert to generate Zz. The length of Zz is four times that of Z1. Similarly, we generate Z3 using the bottom conditional prior by conditioning it with Z Ze and Yeert. The length of Z3 is once again four times that of Zz. Finally, all the three token sequences Z1 are fed into the decoder of the stage-1 model for audio reconstruction.
B.1 Stage-1 model
The stage-1 model for SONIDO is based on the aforementioned SQ- SQ-VAE-2 structure, as illustrated in Figure 2 a). It is a three-level 2 (i.e., L= 3) autoencoder. It comprises three encoding blocks, denoted as encoderbottam: encoder middle, and tocodertop, where the bottom layer processes the input audio sampled at 44.1 kHz. Each encoding block consists of 1-D convolutional layers with a strided convolution for down- sampling. The down-sampling ratios are set to 8, 4, and for encoderbottom: encoder middles and encoder respectively. The stage-1 model has three decoding blocks, decoderbottoms decoder middle and decodertop Each is mirrored version of the encoding block with the same resolution. We train the stage-1 model on top of the HQ-VAE framework on 4, 000h of 44.1-kHz studio-quality library music. The loken sequence generated by the stage-1 model is used to train the prior P(Z1.)), as presented in the subsequent section.
B.2 Stage-2 2 model
For the stage-2 model, we followed Jukebox (Dhariwal al.; 2020), which generates hierarchical tokens with the same down-sampling rates the stage-1 model (i.e, a token in the hop/middlee/botonn level compresses 128 08/32//8 audio samples). Specifically, we train three transformers in auto-regressive manner to model P ((Zılyaudio), P(Zy] M,udio), and P(Zg) Z1.2, Maudio), which we call the top prior, middle conditional prior, and bottom conditional prior, respectively.
Initially, we train a sparse transformer to learn the probability distribution of top-level token sequence Z in an regr-ssgee manner, conditioned on Yaudio (i.e., Pr, (Z1 (suiio)). Subsequently, we train other sparse transformers to model Pz, (Z, Zei Daudio)- These transformers are conditioned by token sequences from upper levels. We used similar configurations of Jukebox's 's transformers, but modified some hyperparameters, as shown in Table 7 The primary distinction between them lies in the conditioning mechanism for each prior. We modified conditioning modules to use Yaudios instead of the original conditioning inputs employed in Jukebox such as artists, genres, and lyrics. In contrast to Jukebox, the bottom conditional transformer in our model is conditioned on all the upper token sequences Zei, diverging from Jukebox's 's approach, where the Ith transformer conditioned on the adjacent upper token sequences Zi- This difference is a result of the hierarchical structure obtained with SQ- SAEV,,,, which has tight interrelation between different levels, unlike Jukebox's independently trained multi-level token sequences.
We explain the entire conditioning mechanism using the bottom conditional prior as an example, which allows us to address all the detail (see Figure 2. We first obtain the hierarchical discrete representations Zi, Zz, and Zy by applying the pre-trained stage-1 1 model to an input musical audio x. The goal of training is to make the model execute next-token prediction on Z3. conditioned on Z1, Ze and the CLAP embedding Yaudko = CLAP.audio L..Paaddo_mmmeeddiggse embedding(z)) We condition the transformer in a framo-by- by-fo- manner. Since cach level has a different time resolution, each token sequence from upper level is first converted into embedding sequence and then up-sampled to the next time resolution by transposed convolution. We use two up-sampling modules: one for Zı + Z2 and the other for Zı, Zn + Zz- The conditioning module Zi + Zz up-samples the embedding sequence Z to match the resolution of Zz- Similarly, up-sampling module Z, Zz + Z3 takes Z2 as input, along with the up-sampled embeddings of Z1, generating further up-sampled embeddings tailored to Z3 s resolution. The resulting frame-level embedding is used to condition the transformer for Zg.
For the actual next-token prediction task, we shift the token sequence Zg by one position to the right and embed each token to a continuous vector using an embedding layer. The first token, which is empty, is
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: parameerr comparison on Jukebox's and SONIDO's top-level prior.
marked
with * indicates that we used different sctting from Jukebox. Otherwise we used same configuration.
Jukebox's
SoNIDo's
Toр
Middle
Bottom
Тop
Middle
Bottom
Sample length
1048576
262144
65536
1048576
262144
65536
Context length
8192
8192
8192
8192
8192
8192
Transformer width
4800
1920
1920
4800
3200
2880
Transformer self-attention layers
72
72
72
72
72
72
Attention heads
8
1
8
4
4
Factorized attention shape
(128, 64)
(128, 64)
(128, 64)
(128, 64) 7
(128, 64)
(128, 64)
Encoder-decoder attention layers
7
Up-sampling modules
1
2 1024
Up-sampling-mouule residual block width
1024
1024
1024
Up-sampling modules residual blocks
16
16
16
16
Up-sampling-men module conv filter size Up-sampling-n module conv channels Up-sampling- module dilation growth rate
3 1024
3
3
1024
1024
1024
3
Up-sampling- module dilation cycle Initialization scale
8
0.002
0.004
0.008
0.002
0.004
0.004
Encoder initialization scale
0.014
0.014
Batch size"
512
192
184
240
240
240
Training step"
310500
265000
279000
88000
152000
272000
Learning rate*
0.00015
0.0003
0.0003
0.0001
0.0002
0,0002
Adam B2
0.925
0.95
0.95
0.925
0.95
0.95
Weight decay
0.002
0.01
0.01
0.002
0.01
0.01
optionally filled with Yaudio to make the system conditioned on the CLAP embedding. With the frame-level aggregated conditioning vectors, namely, up-sampled embeddings from the upper layers, Yaudio: and the time positional embeddings, the transformer is trained to estimate Z3 an auto-regressive manner.
B.3 Evaluation on Music Generation
For evaluation, we used the MusicCaps dataset AAgostinelli et al. 2023), which includes 5,521 2 pairs of audio clips and their text captions written by musicians. Subsequently, we used SoNIDO to generate 10-s audio samples, conditioned upon textual captions of MusicCaps. Following the objective evaluation method- Alogy(Cooper et al. 2023). evaluated SONIDO's performance MusicCaps using three objective metrics:
the Fréchet audio distance (FAD) Kilgour et al. 2019 Kullback-Leible Divergence (KL), and CLAP
Wu
et al.
2023 score.
Table compares SoNIDo terms of the three metrics with other SOTA auto-regressive methods, namely MusicLM (Agostinelli et al. 2023). McLoDy (Lam et al. 2023), and MnsicGen (Copet et al. 2023
We used FAD to measure the overall quality of the generated samples. It reference-free metric that compares embedding statistics computed on an evaluation set with embedding statistics computed on the clean dataset. A lower FAD indicates higher degree of similarity between the embodding statistics from generated audio samples and the embedding statistics from the ground-truth (i.e., MusicCaps). We used the official VGGish DAD computation module to extract embeddings from audio. Since VGGish was trained using 16-kHz audio, all the audio data were resampled to 16 kHz.
We also computed KT-divergence, which compares the two probability distributions of labels. The labels are estimated by applying a pre-trained audio classifier to the generated and original audio clips. Following (Copet et al., 2023), we used a SOTA audio classifier called PaSST. A lower KL-divergence indicates that audio content generated from model aligns more closely with the characteristics of the reference. We report the mean of Kll-divergence.
We used subsct of 5,514 samples due the preseuce of expired invalid URLs in MusicCaps items. Efforts made retrieve the missing files; however, three samples ahsent despite efforts. htwpehhh..cccooooooeeeeeeeeeraaa
https://github. com/ www.oogeerrrerrcc/eeeeeeeeeeee reteec/eee/ooooeerrrrreeeeeeetee stce distance
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 8: Objective evaluation of SoNIDo and other auto-regressive generative models on MusicCaps. Model marked with indicates that we generated audio samples and computed metrics using same protocol.
MUSICCAPS Test Sct
Model
Target SR
FADaga +
KL
CLAPser f
MusicLM
24kHz
4.0
MeLoDy
24kHz
5.41
MusicGon-large
32kHz
3.8
1.22
0.31
MusicGen-large (public ++
32kHz
5.88
1.41
0.27
SoNIDO
*
4k.1
4.98
1.45
0.39
Finally, we calculated the CLAP consistency loss to provide a consistent benchmark aligned with SOTA methodologies, even though might not be entirely fair since SONIDO was trained using CLAP embeddings. We computed the cosine similarity between the audio CLAP embedding from the generated clip and text CLAP embedding from the original text caption from MusicCaps. A higher CLAP score indicates stronger alignment between the generated clip and original text description.
SoNIDO achieved an FAD (denoted as FADugg) of 4.98, a performance comparable to that of the other models. We only report the KL-divergence computed using PaSST as the audio classifier in Table SoNIDo attained a KI. Divergence (denoted as KT) 1.45, which is slightly higher than MusicGen-large. Finally, we report the CLAP consistency score (denoted as CLAPser). SONIDO's CLAP score was the highest, but should be noted that SONIDO was trained using CLAP embeddings, as we mentioned earlier.
While the evaluation result of SONIDO is slightly inferior to the SOTA, it is important to note that SoNIDO generates audio with higher sampling rate. Table 8 shows the target sampling rate of each model denoted as TARGET SR. While SONIDO generated 44.1 kHz directly, the other models generated audio with lower sampling rates. Furthermore, the evaluation protocol requires SONIDO to be evaluated in much lower sampling rate. Despite this challenge, we designed SONIDO to operate at a studio quality level (i.e., 44.1 kHz), enabling its potential use most downstream lasks, which we describe the following sections.
C Music Tagging
We implemented the attention-based aggregator described in Section 3.2 as a 1-layer standard transformer. There is only single attention head in the attention layer. Features from different priors are processed by an input normalization layer then converted to 768 dimensions with linear layer. Siverlr Caatellon et al.) 22021,, , we conducted grid search for the method of input normalization (BatchNorm or LayerNorm). and the dropout ratio in MLP (0.10, 0.25, 0.50, 0.75). For all single-label tasks, by default we set the label smoothing to 0.1. The batch size was set to 256 for MTAT and Nsynth for their large amounts of data, and 64 for others. Unless specifically mentioned, the learning rate was 55
'Publicly released MusicGen-lalll large trained on non-vocal dataset: http:///gttbcccc///eeeeer//////t htt1://oeeL.cccaa/////SDDDLLLLLD MOODEECCAOOOOCCAADD....C md
Table 9: Datasets for music tagging. Except MTAT (multi-label) and EmoMusic (2-axis regression), all other tasks are single-label. Segment length of VocalSet is after pre-processing.
Dataset
Task
Num. of Classes
Num. Segments
Segment Length
Sampling Rate
MTAT
Auto tagging
50
25 k
29
16 kHz
Pitch
128
Nsynth
~00 k
4
16 kHz
Instrument
EmoMusic
Emotion regression
n/a
45 8
44.1
Gtant
Key estimation
24
22.1
120
44.1 klz
GTZAN
Genre
10
~900
30 s
22.05 kHz
Singer
20
VocalSet
~7. k
3
44.1 kHx
Vocal technique
10
28

---

Published in Transactions on Machine Learning Research (05/2025)
Table 10: Preliminary study on impact CLAP conditioning in music tagging. Acc.: accuracy, mAP; mean average precision (bold: top score).
Features
MTAT mAP
Nsynth-pitch Acc.
CLAP
39.6
48.8
Unconditional SoNIDO
39.0
90.6
CLAP-conditional SoNIDo
39.9
91.5
Table 11: Ablation study on feature aggregation pipeline in music tagging (bold: top score)
Features
Aggregation
MTAT ROC-AUC
mAP
Top
Average pooling
91.1
39.9
'Top
Attention
91.1
40.4
Mid.
Average pooling
91.1
39.4
Mid.
Attention
91.3
40.9
Top
mid.
Average pooling
91.5
40.6
Top
mid.
Attention
91.7
41.5
C.1 Comparison of Features
Since SoNIDO is trained with an upstream auto-regressive modeling task, where both CLAP-conditional and unconditional cases are introduced, it is still unclear which setting should be used for downstrream lasks. We thus conducted preliminary study with MTAT, a dataset designed for roarse-graines tagging tasks, and Nsynth-pitch, & dataset designed for fine-grained classification tasks. In this experiment, features from the top prior were aggregated by average pooling, and the learning rate was set to le As mentioned above, the CLAP audio encoder receives the same clip of music as SoNIDO and encodes it into a single feature vector. The results in table I0 indicates that CLAP performs well for coarse concepts, while unconditional feature extraction results in better accuracy for pitch estimation. On top of them, the CLAP-conditional feature extraction achieved better scores in both tasks. We will thus use CLAP-conditional feature extraction as the default feature extraction for time-invariant retrieval tasks.
C.2 Investigation on Feature Aggregation
Next, we conducted ablation studics to verify the effectiveness of the feature aggregation pipeline. The learning rate for attention- and average pooling is 5e and 1e-4, respectively. As shown in Table I1 we have the following observations: (1) the feature aggregation pipeline outperformed the average pooing baselines; (2) applying the samee pipeline to hin-hhieaal features did not bring as much performance gain as the hierarchical case, showing the hierarchical features contain complementary information for music tagging. The bottom prior did not contribute to better performance. We assume this is because the bottom prior processes high-frequency components which are loss important for the tasks tested. It is also worth mentioning that many SOTA classification models have been working with F, 16 kHz, e.g., Niizumi el al. b McCallum et al. (2022): Wang et al. (2022), neglecting high frequency components.
D Music Transcription
Musie transcription requires accurate predictions of both pitch and temporal information for optimal per- formance. Current models (Toyama et al.) 2023 Gardner et al., 2022 rely exclusively spectrograms input. There have been attempts in applying extra feature music transcription. For example, Donahuel
ct al.
2022) used Jukcbox for melody transcription. However, the HookThcory dataset used by Donahue 2022) does not include octave information, which makes it differ from the typical music transcription.
el al.
There are two ways modeling the typical music transcription task: piano-roll-ban based music transcrip-
tion (Hawthorne et al. 2018 Kim Bello 2009 Gardder et al.) 2022 Chouk et al. 2021b Thickstun et al., music transcription (Gardner et al.) 2022). The former transforms spectrograms into piano rolls in which both have the same time resolution. The
2016 Kelz al.] 2019 Toyama et al., (2023) and
29

---

Published in Transactions on Machine Learning Research (05/2025)
log mel spectrogram
lincar
concatcnate
lincar
split
sigmoid
frame
feature token
lincar
lincar
sigmoid
onsct
Figure 5: Model architectures of linear music transcription for piano
252
14 [e4)
spstragra
128 [skas)
kp
00.000
T0 (kmm)
eddde kpg
SecDo
.....0
[alee]
2 (oles)
(a) Linear transcription model using SONIDO or Jukebox
(b) hFT- TFF--raanformer using SoxIDo Jukebox
spotinggra
spettingamm
00..00
tokon
124
MisisGen
......
MasicGa
0.0000
104
(c) Linear transcription model using MUSICGEN
(d) hFTT-Tranformer using MUSICGEN
Figure 6: Feature alignment schemes for different models in music transcription
latter transforms spectrograms into a series of tokens indicating the note pitch and note on and off locations. We selected the relatively well-established piano-roll-based music transcription to test the effectiveness of intermediate representation extracted from music foundation model.
D.1 Piano Transcription with Linear Layers
To understand the usefulness of the extracted intermediate features, started with piano transcription by probing those features with linear layers. The architecture used for this experiment is shown in Figure [
The inputs are log mel spectrogram and the SoNIDO features. Input audio signals are down-sampled to
22.05 kHz then converted to a 256-bin mel spectrogram using a 2048-point Hann window with a 512 hop
size. The aligument. between the mel rogram and extracted features is illustrated Figures 0 aa) and (c), respectively.
The mel spectrogram is 3D tensor (B, N, F), where B = 8 is the batch size, N = 256 is the number of frames, and F = 256 is the number of frequency bins. A linear layer converts the tensor into (B, N, Z) along the last axis, where Z = 512 is a common size for latent embeddings. The SoNIDo, MUSICGEN and Jukebox features are formed as a 3D tensor (B, V, G), where V is the number of tokens per N frames (2,048 for the top layer of SONIDO and for Jukebox, 8,192 for the middle layer of SoNIDO, and 299 for MUSICGEN), and G is the embedding size of features (4,800 for the top layer of SONIDO and for Jukebox, 3,200 for the middle layer of SoNIDo, 2,048 for MUSICGEN LARGE, and 1,024 for MUSICGEN SMALL). The first linear layer converts G to Z, then the second linear layer converts V to N to make the tensor into the shape of (B, N, Z). The tensors are then concatenated along the last axis. If the top layer of SONIDO, middle layer of SoNIDo, or MUSICGEN is used, the dimension of the last axis is 2Z, whercas if both layers of SONIDO are used, the dimension is 32. The following linear layer is used to convert 2Z or 3Z to 2P, where P 88 is the number of pitches. Finally, the tensor is split into two (B, N, P) tensors followed by a sigmoid activation to indicate the estimated frame and onset information. We followed Kong et al. (2021) to extract the precise timing of onsets from datasets, in which we set the ppprr-parraeter to control the target sharpness J to 3. The loss function for frame and onset is binary cross entropy.
30

---

Published in Transactions on Machine Learning Research (05/2025)
Table 12: Fl scores on different datasets when best validation checkpoint was obtained using Bach10 as validation set.
Best validation checkpoint
Dataset
Base
MUSICGEN Small
MUSICGRN Large
SoNiDo Top
Sonio Middle
SoNiDo Top+Miidle
Bach10
43.3
68.4
60.7
69.4
74.4
72.5
GuitarSet 34.8
49.7
38.6
43.0
50.1
44.9
Su
13.5
32.7
31.7
21.1
27.3
24.4
TRIOS
20.8
38.2
31.4
34.9
40.7
39.4
We trained multiple models with the following input: (1) spectrogram only, (2) spectrogram and the top layer of the SoNIDO features, (3) spectrogram and the middle layer of the SoNIDO features, (4) spectrogram and both the top and middle layers the SoNIDo features, (5) spectrogram and the MUSICGEN LARGE features, (6) spectrogram and the MUSICGEN SMALL features, and (7) spectrogram and the Jukebox features. For each model, we trained for 50 epochs on one A100 graphics processing unit (GPU), using Adan (Kingma & Ba 2015) optimizer with learning rate of 1e- PyTorch ReduceLAOnPlater
used for learning- rate scheduling with default parameters. We chose to use the checkpoint with the highest F1 score in the validation split for each model.
As listed in Table g the model using the SoNIDO features, the MUSICGEN features, or the Jukebox features outperformed the spectrogram-only bascline. In particular, the contribution of SONIDO's middle layer was higher than that of the top layer. These results suggest that the SoNIDO features are promising for music transcription tasks.
D.2 Linear Instrument Music Transcription
Following the setup described previously in D.I we also investigated the effectiveness of the extracted features with instrument-agnostic music transcription. We wanted to study if the features of music foundation model are also applicable to a multi-instrument transcription scenario. In this experiment, we reused the model illustrated in Figure and trained on the URMP dataset, which contains strings, woodwinds and brass instruments (Li et al. 2019). Unlike piano transcription, do not need to split the final output into frame and onset prediction for instrument-agnoste transcription. This is due to the fact that some musical instruments, such as violins and flutes, sometimes produce ambiguous onset attacks. Applying onset prediction to these instruments can be detrimental to the F1 score (Cheuk et al. 2021a).
We validated the model performance on the Bach10 dataset (Duan et al. 2010 and selected the best checkpoint based on the validation performance. We then evaluated the best checkpoint on GuitarSet (Xi et al. 2018), Su (Su & Yangl 2016), and TRIOS (Fritsch) [012]] datasets. Table 12 shows the F1 scores obtained using the best checkpoint. The results indicate that features of music foundation model, regardless of SONIDO MUSICGEN, boost eesrrument-agnostie transcription performance compared with the baseline model, which uses only the spectrogram as the input fcatures. This indicates that the model trained using these features has a stronger generalizability across different datasets. SoNIDO generally outperformed MUSICGEN on Bach10, GuitarSet, and TRIOS, while MUSICGEN performed better on Su. This difference may be due to the different datasets on which SONIDO and MUSICGEN were trained. We will investigate this further in the future.
D.3 hFT-TTrasformer
The detailed model architecture of hhT- Transforoerr has been described in Toyama al. 2023
Fig- ures b)) and (d) show the featuro-spectrogran alignment scheme for -FT-TTansformer. The feature tokens and spectrogram cannot be perfectly aligned due to the different sampling frequencies in TOTT-Tranforman (16 kHz),, SONIDO (44.1 kHz), and MUSICGEN (32 kHz). The N frame in the spectrogram corresponds to roughly 707 tokens of the top layer of the SONIDO features and of the Jukcbox features, 2, 828 tokens of the middle layer of the SONIDO features, and 104 tokens of the MUSICGEN features. 7 the modified hFT-Transformm that accepts the SONIDO MUSICGEN features as the additional input. Following the
31

---

Published in Transactions on Machine Learning Research (05/2025)
1 Hierarchy
2nd Hierarchy
ouppi Jst
outpud 2nd
frame
onset
ofliet
velocity
frame
onset
offect
velocity
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
linear
linear
linear
linear
linear
lincar
linear
lincar
linear
concatenate
1st
Transformer decoder
2nd
Transformer encoder
Transformer encoder
lincar
linear
Lincar
linear
conv block
position embbedddig
position embedding
position embeddiig
lineta
log mel spectrogram
feature token
[O.1....FFII
(O.I.....PP1
N-.1.....N1II
Figure 7: Modified version of hFT-TransSormer for SonIDo, MUSICGEN, and Jukebox feature tokens
setup in D.1 the SONIDO and MUSICGEN feature tokens have three dimensions (B, V, G), where B = 8 is the batch size, V is the number of tokens per number of frames N 128, and G is the embedding size of the feature tokens. The first linear layer for the feature tokens reduces G to 7' 256; the second linear layer reduces V to N then reshapes the tensor to (B, N, F" 16, Z" 16); the third linear layer changes Z" to Z = 256, then the last linear layer changes F" to F" = 128. Thus, tensor with shape (B, N, F" Z) is obtained. When using both the top and middle layers of the SONIDO features, we form such tensors for each layer following the pipeline above. The tensor(s) and output of the first encoder are then concatenated on the third axis. Finally, the size of the concatenated tensor is reduced to 256 (F in TTyyma et al. 2023) by a linear layer. These F", F", Z', and z" were determined from preliminary experiments.
We trained the following models that have different inputs: (1) spectrogram only, (2) spectrogram and the top layer of the SONIDO features, (3) spectrogram and the middle layer of the SONIDO features, (4) spectrogram and both the top and middle layer of tthe SONIDO features, (5) spectrogram and the MUSICGEN LARGE features, (6) spectrogram and the MUSICGEN SMALL features, and (7) spectrogram and the Jukebox features, the same as the experiment described in [.I] We trained the models for 50 epochs on one A100 GPU. For the other conditions, we followed Toyama et al. 2023). To confirm if these features are useful when there are less training data, we train the models using 100, 50, 25, and 10% of training data. We chose the checkpoint with the highest Fl score in the validation split for further evaluation.
Figure g shows the loss curves of training and validation. The models using the SONIDO, MUSICGEN Jukebox features reached a lower loss value at an earlier epoch compared with the bascline model using the spectrogram only as input. Table 13 lists the scores on the test set of MAPS. When all the training data were available, the models using the SoNIDO, MUSICGEN and Jukebox features, except the model using the MUSICGEN LARGE features outperformed the baseline. When the models were trained with scarce data, the performance of the models using the SONIDO, MUSICGEN and Jukebox features was superior to that of the model using the spectrogram only. When the training data size was 50 and 25%, the performance of the models using the SONIDO features was still comparable to the baseline model trained with 100% data.
32

---

Published in Transactions on Machine Learning Research (05/2025)
0.5
NTT-Tauffommr
0.50
MT Trandee
IFI- IFa-foofermmr- SoaiDo
MTT Tranddmme
SoniDo lop
0.4
MF. AFT-Tranfommrr SociDo Middle
MFI-Tnnsfommer SoniDu TopkMMsddle
0.40
MTT Wrondooeee
MFT Trandtmee
SoniDo TopaMiidee
SoniDo Middle
hF Transfmmmr MusisGin Lange
MPT Tranddooe
Musk Large
0.3
Transfomer MusizGen Small
0.35
IFT-Tranetommer
Musietien Small
bFT Transormrr+ Jakebox
NFT Tonndomer
Jukebox
0.30
0.25
0.1
0.20
0.15
10
20
30
40
50
10
30
40
20
Epoch
Epoch
(a) Training Loss (100%)
(b) Validation Loss (100%)
0.5
hF Tranfferee MFT-Trasformer SoniDo
MFT-Trandfsmme
1F7-Tranaammm SomiDo Top
bFT Tressommer SociDo Middle
Tr-araanaa SaniDo Middle
0.4
Transfomrrr
Mopae
18T Tran-aaaa SoaiDe Top&Middle
hFT: Transfomer+
MusicGen Large
Mra-Taandasmm Musiciion
Trunforrlreerr MusiGinn Smull
MFT TranTrar MusicGian Small
0.3
Trl-ffasfemer Jakchex
MFT-Jaundssmm Jukchox
02
0.1
10
20
30
40
50
10
50
40
50
Finnc
Eprcc
(c) Training Loss (50%)
(d) Validation Loas (30%)
bF Tranfforr Inmstoooorrr SoniDoo
hFT: Transoomer
0.5
0.7
SonDoo Tip
MFT: Traisfomer SoniDo Middle
bFT: Transformer SoniDo Middle
0.4
IFT-Trunffemer SociDo Tep&Millle
0.6
Trans Son Top&Miblle
IFI- Transfaaem Musisfinn Larg
hFT- TFT-Tianffommr MusicCies Large
Transfrmmer
MusicGan Small
0.5
Tormeeer MusicGen Small
hFT- Tresfurmer Jokebox
hFT: Teansoomrr Jucobux
0.2
20
30
50
20
30
40
50
Epoch
Epoch
(c) Training Loss (25%)
(f) Validation Losss (25%)
hF TFl-Tuusfemmr
0.8
HFT FT Tiaistooriie
0.5
Trastrerfeem SociDo Top
Transfere SonDo Top
MFI-Trnsfummer SuniDo Middle
hFT Toa-Tosooommr SonDo Middle
MFT Tranfforeerr SoniDo TepkMiiblle
0.7
hFT Transffemer Sunio Top&Middde
0.4
bFT- Transfrmer
MusicGen Large
hf Transoomer MasicGen Larpe
bFT- Tresfommer
MusicGen Small
bFT Transtomme MusicGen Small
0.3
Trausfomer+ Jikebxx
0.6
Tiasstomer Juiebox
0.5
0.4
0.0
20
50
20
50
Epuch
Epoch
(g) Training Loss (10%)
(h) Validation Loss (10%)
Figure 8: Loss curves for hFFT-Transformer in music transcription task
33

---

Published in Transactions on Machine Learning Research (05/2025)
Table 13: Evaluation results on MAPS in music transcription (bold: best score, underline: second-bes
score). "Note" refers note-wise estimation. First row is of Transfrnfforeer Toyama et al. (2023).
Training data Input
Frame
Note
Note Offset
Note w/ Offset&
Spectrogram
85.14
66.34
48.20
Spectrogram SONIDO
83.92
86, 45
68.27
51.34
Spectrogram SoNDDO Middle
83.47
86.13
67.93
51.24
100%
Spectrogram SONiDO Top SoniDo Middle
84.16
85.96
67.37
50.98
Spectrogram MUSICGEN LARGE
81.53
85.14
66.28
48.69
Spectrogram + MUSICGEN SMALL
82.94
85.97
68.27
50.42
Spectrogram Jukehox
84.23
86.54
68.26
50.46
Spectrogram
39.12
23.34
13.22
9.12
Spectrogram SoNIDO
83.35
85.51
65.84
47.40
Spectrogram SoNIDo Middle
82.52
85.46
65.40
47.25
50%
Spectrogram SoNIDO Top SonIDo Middle
83.37
85.33
67.04
49.32
Spectrogram + MUSICGEN LARGE
82.26
84.58
65.50
47.07
Spectrogram + MUSICGEN SMALL
82.24
85.21
65.32
46.72
Spectrogram Jukebox
83.33
85.58
64.62
46.20
Spectrogram
12.88
1.61
0.66
1.01
Spectrogram SONIDo Top
81.71
84.70
63.00
45.50
Spectrogram SoNIDO Middle
81,65
$4.59
62.19
43.91
25%
Spectrogram SoNIDO Top SonIDo Middle
81.44
83.79
62.61
14.71
Spectrogram + MUSIcGEN LARGR
78.98
82.36
58.64
39.62
Speclrogram + MUSICGEN SMALL
79.39
82.23
60.36
41.02
Spectrogram Jukebox
81.96
84.47
63.63
44.63
Spectrogram
9.83
0.59
0.17
0.46
Spectaogram SONDDo Top
65.91
66.61
39.88
25.87
Spectrogram SoNIDO Middle
70.77
74.02
45.75
29.46
10%
Spectrogram + SoNIDo Top + SonIDo Middle
71.57
75.00
46.18
30.63
Spectrogram + MUSIcGEN LaRgr
61.81
63..27
$7.03
24.01
Spectrogram + MUSICGEN SMALL
63.73
65.90
39.00
21.94
Spectrogram Jukebox
70.43
73.76
45.80
30.42
E Music source separation
E.1 Details of UMX with SONIDO
Huang et al. (022bb investigated various speech enhancement (SE) systems that self-supervised learning (SSL) features and discussed the challenge that the SSL features may have lost some local signal information necessary for estimating lower-level features (e.g., spectrograms, waveform). Following the above observation, Hung et al. 2022)) proposed to combine spectrograms with SSL features in their SE system to avoid such problem. Therefore, we hypothesize that the features extracted with a large-scale foundation model could serve as auxiliary information for a neural network on music source separation. However, it remains unclear how integrate the extracted fcatures into the network. Therefore, we investigated several intogration strategies.
Figures g and show the architectures of the original UMX and UMX with SoNIDo, respectively. UMX starts with an input audio waveform and converts it into an STFT spectrogram. The magnitude part of the spectrogram is normalized to mean of 0 and standard deviation of 1, based on statistics collected from the entire training dataset before training. The normalized magnitude spectrogram is then passed through an "encoder" block, which includes a linear layer, batch normalization (BN) layer, and hyperbolic tangent
Sequence modeling
Eicoddr
Decoder
STFF
Linear
RNN
BN- ReLU
Figure 9: Original architecture of UMX
34

---

Published in Transactions on Machine Learning Research (05/2025)
So
Narmuo Seation
Elivae
Linear
of
Lase injection
Farty ingection
Waraforn
STFT
Nomee listioo
Linear
BN- tanh
KNN x3
Cancas- exstie
Linear
ReLU
Leca-
Meiiieg
ISTFT
Woostom
Figure 10: Architecture with SoNIDO using UMX
LBOC
Extrackod festure
1800
100
Hop siz (1924 23 mg
WaveSamn
Hop size (8 10 23.2 ms)
Extractdd feature
Hep
Window don (4192)
Kernel sire (32)
Kemel
Mee pooling
Unfabling
R 2015
R 30:4
Magnitide spectiigram Ulvery hop soo)
DoAn sangled extrocted feorue (Bberr hop sioe)
Down-uamllld estracted feature (Every hop size)
(a) Original branch
(b) Average/ Max pooling
(c) Unfolding
Figure 11: Down-sampling blocks to align time resolution.
function. A "sequence modeling" block, consisting of three RNN layers along with skip connection structure is then used. It processes the encoded features and combines their input and output features. The resulting combined features are then transformed into time-frreque fine-freyeency masks in the "decoder" block, which is made up of several linear layers, BN layers, and rectified linear unit (RcLU) activations. These time- fime-freuencyy masks are used to modify the magnitude spectrogram, and a multi-channel Wiener filter (MWF) Nugraha et al. (2016); Uhlich et al. (2017) is optionally applied to the spectrogram. Finally, an inverse STFT yields the separated audio waveform.
All learnable parameters are optimized with the mean- reean-squared error of magnitude spectrograms. We propose using the SONIDO features while keeping the UMX architocture unchangod. As shown in Figure 10 SoNIDo is introduced to extract features from the input audio waveform. These features are then adjusted to have a standard statistical distribution computed across the entire training dataset similarly to the normalization used for the magnitude spectrograms in UMX. Subsequently, they are processed through a down-sampling block, followed by a linear layer. The down-sampling block is introduced because of the time resolution difference between SONIDO and UMX; SONIDO generates feature for every 128 waveform samples. In contrast, the original branch of the UMX model processes every 1024 waveform samples, as determined by the STFT hop size. We explain the design of the down-sampling block in the next paragraph. The down- sampled features are then converted from the dimension of 4800 to 512 by the linear layer and summed up with the original features from the UMX branch.
In the original UMX architecture, the input is the spectrogram of STFT. Since all modules in UMX work with the time resolution of STFT, the down-sampling block is required to align the time resolution between the SONIDO features and STFT. In the ablation study, we explored three different down-sampling operations: max pooling (MP), average pooling (AP), and unfolding (UF). Figure II illustrates the three different operations for the down-sampling block.
AP averages the multiple SONIDO features with the hop size and kernel size corresponding to the hop size and the window size of STFT, respectively. MP works similarly AP but replaces the average pooling with a max pooling operation. In UF, the dimension of the SONIDO feature sequence is first converted from
35

---

Published in Transactions on Machine Learning Research (05/2025)
Table 14: Evaluation results of music source separation on vocal extraction task
SDR [JB]
Method
Vocals
Accompaniment
Open-Unmix (UMX)
5.71
11.57
UMX with SoxiDo (MP, EI)
5.76
11.86
UMX with SoxiDo (AP, EI)
5.55
11.71
UMX with SoxiDo (UF, ET)
5.92
11.92
UMX with SoNIDo (UF, L1)
5.83
11.68
0.8
1.5
UMX UMX SoniDo
UMX
0.7
1.4
UMX SoniDo
0.6
0.5
1.2
0.4
1.1
0.30
200
400
600
R00
1.4
2000
400
600
K00
Epach
Epoch
(a) Training loss
(b) Validation loss
Figure 12: Training and validation curves for UMX experiments in music separation task
4800 to 150 with a linear layer and ReLU function. Next, the feature sequence corresponding to the window size of STFT are concatenated every STFT hop size. After the concalenation, the dimension of the feature returns to 4800 again.
As well as the design of the down-sampling block, we further evaluated two mothods for injecting the down- sampled features into UMX, as shown in Figure 10 With the "early injection" (EI) method, the SoNIDo features are injected into UMX's s encoder block. With the "late injection" " (I.I) method, the extracted features are injected into the decoder block.
We explored the optimal down-sampling and injection method for UMX on vocal extraction task using MUSDB18. We followed the original training configuration of UMX for all experiments except that we disabled data augmentation for simplicity. To purely evaluate the trained components, MWF was skipped in all experiments.
E.2 Results: Ablation study for UMX with SoNIDO
The results of the aforementioned ablation study are summarized in Table 14
For the down-sampling block, MP and AP did not improve the SDR score, whereas UF achieved a 0.34 dB improvement for vocals and 0.36 dB improvement for accompaniment. The results indicate that UF is the proper choice for the down-sampling block. We assume that the lower-level information lost during the pooling operation over the temporal axis in MP and AP, while UF preserves such information by stacking multiple tokens into one frame in the unfolding manner.
Table 14 shows that ET is superior to LI. This suggests that it is the sequence modeling block in UMX that effectively used the SONIDO features to improve separation performance. The observation also inspired us to inject SoNIDO features into the transformer block in HTDemucs.
Figure 12 shows the training and validation curves for UMX and UMX with SoNIDO (UF, EI). The loss curve of UMX with SONIDO tends to be lower than that of UMX in both training and validation. i.e., the number of epochs required to reach target loss smaller when UMX was trained with SoNIDo. The
36

---

Published in Transactions on Machine Learning Research (05/2025)
converging loss of UMX with SONIDO was also lower than that of the original UMX, which reveals that the benefit. from SoNIDO can be consistently observed in the training phase, and the performance improvement is significant.
E.3
Experiments: HTDemucs with SONIDO
We provide more details for music source separation with HTDemues (Rouard et al. 2023) and SONIDo from To investigate the effect of the SoNIDo features, we trained several models MUSDB18 (Rafi)
Section 4.2.2.
et al. 2017 We compare the following models:
HTDemuuss (default): Model with default settings (Dora signature "955717c8"). The default batch size 32 which corresponds to 4 samples per GPU as we trained in parallel on GPUs. The default number of training epochs 360.
HTDemucs SoniDo : Combination of HTDemuccs with SoNIDo. We introduced two new cross- domain transformer encoders into HTDemucs which inserted directly after the original cross- domain transformer encoder. The first encoder facilitates information exchange between the spectral and SoNIDo feature sequence, while the second one enables interaction between the waveform and SONIDO feature sequence. The SONIDo features are computed from the monaural downmix of the mixture using the top prior. Hence, the SoNIDO feature sequence has a dimension of (T = 2688) x (C 4800), reflecting the characteristics of training samples with a duration of 7.8 s. Subsequently, these features undergo normalization through LayerNorm and further projected from their original dimension of 4800 to 2048 using linear layer, giving them the same size the fealures in the sequences from both the spectral and waveform branches of HTDemucs. Each additional cross-domain transformer encoder has depth of 3, where we set cross first=True. Due to the additional SoNIDO model, we needed to reduce the batch size to 16 (corresponding to 2 samples per GPU as we trained on 8 GPUs). To keep the number samples that the models seen during training the same, the number of training epochs was increased 720. Additionally, match the random remixing augmentation of the default HTDemues model, we added 860 random mixes, generated from the 86 training songs in MUSDB18. It is important to note that this done to ensure the same augmentation and does not introduce new songs to the training set; we still exclusively trained on the train split of MUSDB18.
HTDemuus (ablation 1): Training with default settings '955717e8: where we also reduced the batch size to 16 and increased the number of training epochs to 720, together with the additional 860 random mixes used for HTDemucs + SoNIDo
Training as HTDemues (ablation 1) but where the number of layers in the cross-domain transformer was increased from to 11, matching the 2 additional transformer layers of HTDemues + SoNIDO
HTDemucs (ablation 2):
HTDemucs + STFT-2048: Same training settings HTDemues + SONIDO but where used STFT features instoad of SONIDO features. We computed the STFT with Hann window of 2048 samples and hop size of 512 from the monaural downmix of the mixture.
HTDemucs + STFT-4096: Same training settings as HTDemues + SONIDO but where we used STFT features instead of SoNIDo features. We computed the STFT with Hann window of 4096 samples and hop size of 256 from the monaural downmix of the mixture.
HTDemucs + CLAP: Same training settings as HTDemucs SONIDO but where we used the embeddings from CLAP (Wu* et al. 2023) instead of the SONIDO features. More specifically, we used the 'fne-grained embeddings" before the final AP in CLAP. This experiment enabled us to compare features obtained with two trained models (as opposed to the STFT
hhttpe://github. htttpe//ssshuucc////abbbrereeree hwtp:///eaebbbeeeeerrc//oorr
Note that SONIDO and CLAP were trained different datasets.
37

---

Published in Transactions on Machine Learning Research (05/2025)
0.40
0.25
HiDenges (deleul))
0.35
HTbemjr (shlation 1)
Cabintion
0.30
9.24
0.25
HTDammrx CIAP (sblation
HiDenses
rablation 4)
0.22
0000220 50%
Touning
HiDamgss Musasian bmall (abiation 6)
Saildatton
0.20
ITDDengcs Mustofen large Publation ?>
0.35
HiDensts Jukebok (ublation d)
0.18
HTDeiuucs BooiDo
0.30
0.05
100
200
300
400
500
60n
700
200
200
400
520
600
750
100
200
5010
400
5ã0
não
700
Epoch
Epoch
Epoch
(a) Training loss
(b) Validation loss
(c) Validation SDR (dB)
Figure 13: Training and validation curves for HTDemucs experiments in music source separation task. We conditioned HTDemucs with four different features (STFT, CLAP, MusicGen and SONIDO). We achieved highest separation scores when injecting features of SoNIDo. Interestingly, we observed instabilities when injecting STFT and CLAP features but not when injecting SoNIDo features.
HTDemucs + MusieGen Small: Same training settings as HTDemucs + SONIDO but where we used intermediate features from MusicGen (Copet et al.) 2023) instead the SoNIDo features. More specifically, we used the 'musicgen-small activations at the output the 12th layer. This experiment enabled compare features obtained with two music foundation models
HTDemucs + MusicGen Large: Same training settings as HTDemucs + MusicGen but where we used the 'musicgen-large" activations at the output of the 24 layer. This enabled us compare features obtained with two music foundation models with the same order of magnitude of number of learnable parameters. Note that, to reduce instability during training, this experiment used AdamW (Loshchilov & Hutter 2017 as the optimizer, instead of Adam.
HTDemucs Jukebox: Same training settings HTDemucs SoNIDO but where used in- termediate features from Jukebox (Dhariwal et al. 2020) instead of the SONIDo features. More specifically, we used the activations at the output of the 36th layer of the model '5b
Figure 13
displays the training and validation curves for these models.
F
Music Mixing
Music mixing is a crucial task in music production and typically conducted using audio processors or audio effects, which are signal processing systems that alter specific charactoristies of the input signal. Several signal processing and machine learning methods have been investigated to automatize this task (Steinmetz) et al. 2022), with the goal of simplifying the process for less experienced content creators and enhancing the workflow capabilities of professionals MMoffat & Sandler 2019)
Data-driven deep learning approaches for automatic music mixing have locused on two fundamental frame- works: direct transformation networks, in which the model executes the mixing in a black-box manner, and parameter estimation networks, which the mixing is carried out via differentiable audio processors. Martinez-Ramírez et al.] (2021) proposed Mix- Wave-U-Net, a modified Wave-U-Ne. for drum mixing as a direct. transformation system, while (Steinmetz et al. (2021) introduced differentiable mixing console in which neural proxies act parameter estimation network. Both systems have been acknowledged having limited performance due to the scarcity of available training data, failing to meet professional audio engineering standards. Such systems require unprocessed or dry multitrack recordings and their correspond- ing mixtures, and large datasets are not readily accessible. To address this limitation, Martinez-Ramfre
Note that SoNIDO and MusicGen trained on different datasets.
Note SONIDo
were trained on diflerent datasris.
38

---

Published in Transactions on Machine Learning Research (05/2025)
et al. (2022) proposed an Fx-normalization preprocessing method that enables the training of direct trans- formation automatic mixing systems using processed or wet multi-track audio datasets, akin to the datasets used in source separation. Building on this approach, [Koo et al. (2023) introduced a contrastive learning approach that allows a direct transformation network to execute mixing style transfer. Vanka et al. (2024) proposed a differentiable mixing style transfer system that predicts console parameters from raw tracks and a reference mix, advancing both paramcter estimation and style transfer approaches.
While Koo et al. 2023 used SSL embeddings from reference mixture guide the mixing style transfer task, to the best of our knowledge, our approach is the first data-driven automatic mixing approach that incorporates SSL features from the input stems or high-level information related to genre, instrumentation, or mood to enhance automatic mixing performance.
F.1 Details of Mix-Wave U-Net and CRAFx2 with SONIDO
Mix- Mok-Wave-U-Net extends the U-Net architccture for audio signal processing tasks. We used FiLM lay- ers (Perez et al.] 2018) to incorporate the SONIDo features into each up-sampling 1D convolutional block and the bottleneck ID convolutional block in the Mix- Wave-U-Net Following Meseguer-Bres Brocal & Peeters (2019), we positioned FiLM layers after the normalization layer and before the LeakyReLU activation func- tion.
CRAFx2 Martinez- et al. 2022 comprises (1) an adaptive front-end that learns filter bank, (2) latent-space mixer that learns a mixing mask functioning as equalizer, dynamic range compression, and reverberation transformations, and (3) a synthesis back-end that, through adaptive gains, implements londness and panning transformations for each filter-bank channel. To enhance the mixing performance of the network, we also incorporated the SONIDO features into the relevant layers the network. Following a similar approach to Mixx-Wae-UUl Wave-U-Net, we use FiLM layers to condition both the latent-space mixer and synthesis back-end. The latent-space mixer was first constructed on the basis of a temporal dilated convolutional network (TCN) followed by stacked bidirectional long short-term memory (BLSTM) layers, and FILM layers were inserted within the TCN block and before the BLSTM layers. For the TCN part, the FiLM layers were placed after the depthwise convolution operation and before the second nonlinear activation function. Before the input of the BLSTM layers, we introduced a FiLM layer. Finally, the synthesis back-end incorporated a squeezo-and-cceiiaatio block (SE) (Hu et al. 2018), which scales channel-wise information by applying adaplive gains. GroupNorm and FiLM layer are added after the second linear layer of the SE and before the sigmoid function.
F.2 Experiments: Mix-WaveUU-Net and CRAFx2 with SONIDO
We used Wave-U-UU-- and CRAFx2 to explore the effect of the SONIDO features on the music mixing task. The input to all networks is the Fx-normalized (Martinez-Rammi bass, drums, and other, with the output being the stereo mixture. Each input stem consists of 2 channels of 10-s audio frames at 44.1 kHz. The SONIDO features are computed from the summation of the Fx-normalized input stems using the top prior and unconditional extraction. Therefore, the SONIDO feature sequence has a dimension of (T = 3446) x (C- = 4800).
et al.) (022) stereo stems; vocals,
Although music mixing a Lime-varying task, we hypothesize that high-level concepts such as genre, in- strumentation, and mood, might improve mixing performance. Therefore, following the pre-processing for time-invariant retrieval described in Section 3.2. we carried out AP over the time dimension, and the tokens were time-averaged to dimension of (T 1) (C = 4800), subsequently normalized via non-trainable LayerNorm layer, and dimensionally reduced to 512 through linear layer. We found that reducing the dimension too much (e.g., 128) decreased performance. The linear layer is followed by a dropout layer with a probability of 0.25 to avoid overfitting. All networks underwent. the same token-proeessing steps.
From MUSDB18, 86 songs were used for training, and 14 and 50 for validation and testing, respectively. Since this dataset is significantly smaller than that used by Mamirez al. (2022), the number of batches per epoch was reduced from 1600 to 320, and all models were trained for 520 epochs. For the
39

---

Published in Transactions on Machine Learning Research (05/2025)
50
45-
Mix Wave-U-Nel (deSuul))
Mix Uavee-UNee (defnull)
Mix Uar Jukeboo
Mix- Wave-U-Nel Jukebox
40-
U.Ne MusicGen Small
Waree-WWWe-... U.te Music Small
Wave-U-Net MasicGen Large
Waree-WWW---aae MusicGen Large
35
Wave-U-Nel SoniDo
U-V SonDo
30
30
20
300
300
Epoch
Epach
(a) Mix- U-Ne:: Training loss
(b) Mix- W-ve-UNNt:: Validation loss
50
CRAFs2 (deSuult
CRAFKZ (default)
CRAFx2
Jukebox
40-
CRAPx2 Jukebox
CRAFxx2
MusicGen Small
CRAPx2 MusicGen Small
CRAFx2
MusicGen Largs
CRAFx2
MusicGen Large
35
CRAFs2
SoniDo
CRAPX2 Son
30
25
25
15
100
200
300
400
500
20
100
200
300
400
500
Epoch
Epoch
(c) CRAFx2: Training loss
(d) CRAFx2: Validation loss
Figure 14: Training and validation curves for War---vv---N and CRAFx2 in music mixing task
default networks, we used the suggested initial learning rate of 1e-3 For models involving SoNIDo, to ensure stability, we set the initial learning rate to 1e-1.
The loss function corresponds to the shereo-invariant loss that Martinez-Raaaann best-performing, which they referred to as Lb, and consists of A-weighting pre-emphasis and low-pass finite impulse response filters, L2-norm on the spectral magnitude, and the L1-norm on the spectral log-magni
et al. 2022 reported as the
To investigate the effect of the SONIDO features, we trained various models using features from both Mu- sieGen Small and MusicGen Large. The training and feature injection settings identical to those of Mix-Waver Wave-U-Net + SONIDO and CRAFx2 +SONIDo, respectively, and the MusicGen features were ex- tracted in the same manner as described in Appendix E.3/
To objectively evaluate the performance of all mixing systems, we used audio features related to the main audio characteristics that audio engineers manipulate during the mixing process, as shown in sev- eral works (Colonel & Reiss 2021: Steinmetz et al., 2022; Martinee-Ramiee
al., 2022 Vanka et al, 2024). We computed the following audio features, spectral: centroid, bandwidth, contrast, roll-off, and flatness (Pecters, 2004); panning: the panning root mean square (PRMS) for total panning, low, mid, and high frequencies TTanetakis et al.] 2007); dynamic: RMS level, dynamic spread, and crest factor (Ma et al. 2015); and loudness units full scale (LUFS) level and peak loudness (ITU-R 2011). All features were com- puted using a running mean of 0.5-8 (Tzaneaaiis et al.) 2007). The objective evaluation test sets are identical to Martínez- Mlartinez-Ramirez and the MUSDB18 test set, comprising 50 wet multi-track songs.
Ramírez et al. (2022). These include MDXDB21-dry, an 18-song dry test set from MDXDB21,
F.3 Results: Mix-Wave-U-Ner and CRAFx2 with SONIDO
From the training and validation curves in Figure 14 we can see that both Mix- Ware-WWWo-W----- Pavo-U-Net + SONIDO and CRAFx2 + SonIDO exhibited significant improvement. during training and enhanced generalization
40

---

Published in Transactions on Machine Learning Research (05/2025)
Table 15:
Objective metrics correspond to mean absolute percentage error per low-level audio feature. Results are presented for MDXDB21-dry test set and MUSDB18 test set.
Spectral
Panning
Dynamic
Loudness
Test set/ / Model
baad- width
centroid
contrast
roll-off Eatness
PRMS tutal
PRMS Jaw
PRMS
PRMS high
RMS
spread
crest
LUFS
peak
mid
MDXDB3I-drg
Hitx-Ware--Ul U.Net (defanlt)
0.178
0.256
0.312
0.217
0.296
0.101
0.246
0.077
0.047
0.096
0.094
0.243
Mix Wave- U-Net + Juke
0.236
0.176
0.300
0.322
0.140
0.278
0,236
0.084
0.325
0.078
0.047
0.100
0.086
0.222
Sixx-Ware-UUl U.Net MusicCen Small
0.265
0.182
0.273
0.324
0.147
0.192
0.286
0.119
0.211
0.065
0.044
0.082
0.087
0.206
Mix-Waoo- Waoo-U-Net MusieGen Large
0.264
0.191
0.286
0.323
0.149
0.262
0.243
0.130
0.288
0.065
0.044
0.089
0.083
0.207
Mixx-Wae--U.N. 1 SONIDO
0.228
0.159
0.295
0.290
0.158
0.183
0.243
0.088
0.205
0.071
0.041
0.089
0.088
0.173
CRAFs2 (defanit)
0.216
0.165
0.196
0.264
0.123
0.168
0.273
0.078
0.197
0.072
0.049
0.087
0.086
0.218
CRAFx2 + Jukebox
0.282
0.206
0.176
0.339
0.151
0.266
0.333
0.082
0.315
0.082
0.046
0.098
0.084
0.204
CRAPx2 + MusieGeu Suall
0.262
0.195
0.172
0.338
0.144
0.265
0.312
0.079
0.318
0.081
0.047
0.089
0.090
0.209
CRAFx2 + MusieGon Large
0.274
0.215
0.170
0.334
0.147
0.230
0.292
0.086
0.267
0.077
0.047
0.094
0.075
0.190
CRAPx2 + SoNIDo
0.226
0.157
0.273
0,283
0.169
0.145
0.307
0.085
0.162
0.068
0.044
0.084
0.109
0.232
MUSDD18
Mix- "Nave-U-Net (default)
0.260
0.173
0.170
0.291
0.109
0.156
0.222
0.104
0.172
0.087
0.069
0.098
0.091
0.241
Hiax-Wave-UUl U-Net + Julee
0.179
0.169
0.302
0.108
0,211
0.204
0.094
0.241
0.089
0.098
0.089
0.225
Mix- Wave-U-Net- + MusieGen Small
0.291
0.192
0.162
0.301
0.119
0.145
0.211
0.116
0.160
0.088
0.056
0.093
0.097
0.230
Mix--aaee U.Net MusicGen Large
0.176
0.170
0.299
0,203
0,232
0.108
0.225
0.079
0.057
0.090
0.096
0.237
Mix. Moos-aoe--aare U.Net 1 SoNiDO
0.240
0.152
0.162
0.265
0.110
0.179
0.217
0.103
0.200
0.068
0.050
0.069
0.089
0.269
CRAPs2 (defanit)
0.253
0.173
0,152
0.274
0.111
0.132
0.253
0.086
0.147
0.097
0.047
0.098
0.089
0.241
CRAF52 + Juisebux
0.210
0,154
0.303
0,236
0,264
0.104
0.275
0.097
0.047
0.103
0.089
0.241
CRAFX2 MusieGen Small
0.279
0.190
0.157
0.303
0.125
0.223
0.242
0.091
0.258
0.038
0.047
0.105
0.101
0.255
CRAF52 + MusieGen Large
0.212
0.163
0.317
0.136
0.209
0.277
0.099
0.237
0.093
0.048
0.098
0.099
0.251
CRAFX2
SoNIDO
0.243
0.157
0.148
0.276
0.113
0.132
0.249
0.088
0.146
0.093
0.046
0.089
0.097
0.240
during validation. These results are reflected in Table 6 where the models that included SONIDO tokens consistently displayed improved performance of the stereo-invariant loss. Regarding the audio effect-related features, Table shows that across most feature categories, Mix-Wave-U-Nel + SoNIDO outperformed the default and Mix-Wave- U-ve-
Incorporating Jukebox and MusicGen fcatures generally leads to improvement over the default network, although not as much as SONIDO. MUSICGEN SMALL performed slightly better than the large model, confirming the trend observed in previous downstream tasks. However, for both MUSICGEN and Jukebox, improvement over the default network is not always the case with CRAFx2. In the MDXDB21-dry test set, the default model performed better than MUSICGEN in terms of spectral and panning features as well as stereo-invariant. loss. Yet, MUSICGEN did enhance the dynamic and loudness features, while Jukebox only improved loudness and generally lod to overfitting. The performance gap of MUSICGEN in spectral-related metrics, and generally when compared with SoNIDo, might stem from MUSICGEN being trained on 32-kHz audio, while both our evaluation datasets and training dataset of SONIDO correspond to 44.1-kHz audio.
This discrepancy in training data sampling rates leads us to assume that the relatively lower performance of MUSICG EN may be atrributed to this difference. Further investigation is required to confirm this pothesss and explore the impact of sampling rates on music foundation models as boosters for music mixing tasks.
It is worth noting that the inherent diffcrences between the MDXDB21-dry and MUSDB18 test sets are reflected in these scores. This is expected, given that the MDXDB21-dry data consists of dry multi-tracks, presenting a more realistic mixing scenario than the wet multi-tracks of MUSDB18. Therefore, we attribute the substantial difference in reported stereo-invariant values in Table D to the fact that the target mixture has been processed with different audio processors, and the timbral characteristies cannot always be matched by the trained networks
The mismatch in performance between the stereo-invariant loss and low-level features related to audio effects reflects the challenge of objectively evaluating automatic mixing systems, which remains an ongoing and open research direction (Stables et al. 2019 Steinmetz et al., 2022). Thus, for a more in-depth analysis of such systems, a listening test is required. However, for the scope of this paper, the reported objective results support our hypothesis that incorporating the SONIDO features, i.e. high-level knowledge of the input stems to be mixed, overall improves training and generalization when these features are properly used.
41


Reviews:
[{"review_id": "mXpi8QxNve", "reviewer": "Reviewer_8o1p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written, and the architecture is described in detail. The suite of downstream music tasks is comprehensive. Pushing the frontier of using music foundation models for powerful representations and downstream music tasks is valuable for the field. In some settings, SOTA scores have been improved.\n\nWeaknesses:\nThe main weakness of this paper is that it lacks baselines for its primary claim of using extracted features from a music foundation model for downstream music tasks. The paper experimentally compares the presented MFM only against MusicGen in the experiments section (for music transcription, music source separation and music mixing). Music foundation models should not necessarily be restricted to the tokenized audio + language modeling ecosystem, and even within this particular setting, benchmarking the presented MFM solely against MusicGen doesn't provide enough context to demonstrate the full potential of the extracted intermediate representations. It mainly supports the claim that using feature extraction for certain downstream tasks can lead to performance improvements. As it stands, it's difficult to contextualize the presented methodology. \n\nThe lack of dataset availability makes it more challenging to interpret the quality of intermediate features, particularly in light of potential biases.\n\nAdditionally, in the context of this paper, the definition of a Music Foundation Model is unclear (any representation learning setting?, multi-task adaptability?, music tagging, music transcription, music source separation, music mixing and beyond?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a methodology in which intermediate features from a Music Foundation Model are used to enhance various downstream music tasks, such as music tagging, music transcription, source separation, and music mixing, resulting in performance improvements in some cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To strengthen the work, the main proposed change would be to include more baselines than just MusicGen for the Music Foundation Model, in order to better contextualize the work and demonstrate the effectiveness of the extracted intermediate features. MFMs do not necessarily need to be restricted to the tokenized audio + language modeling setting. (For a variety of examples, see: Ma, Yinghao, et al. 'Foundation models for music: A survey.' arXiv preprint arXiv:2408.14340 (2024).) Also, providing more details— even at a high level— and statistics regarding the dataset would improve the understanding of the affordances of the intermediate representations, particularly in terms of any potential biases. Moreover, clarifying the definition of Music Foundation Models would enhance the paper contextually.", "broader_impact_concerns": "Nothing in particular besides the points mentioned in the Ethical Concerns section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "avrn1zAoJ4", "reviewer": "Reviewer_ukVv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positives:\n- First application of the HQ-VAE parameterization in music foundation models\n- State of the art performance for various downstream tasks\n\nNegatives:\n- Despite SoniDo being a generative model, there are very limited results on generative audio tasks (only music mixing seems to be covered), e.g., text to audio, audio to audio, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes “SoniDo,” a new foundation model for audio tasks and thoroughly characterizes its audio encoding capabilities over various downstream music understanding and mixing tasks. Fundamentally, SoniDo differs from other foundation models through its underlying quantization setup (HQ-VAE vs. VQ-VAE / RVQ) which promises better encoding capabilities. SoniDo embeddings seem to boost downstream performance when introduced in various task-specific specialized models compared to using MusicGen embeddings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below are some follow-up questions on the paper:\n\n- MusicGen small seems to always perform better than large? See tables 2, 3, 4, 5, and 6. Is this expected?\n- Since the training dataset for the foundation models is different, how to attribute performance delta to the proposed change of HQ-VAE vs. VQ-VAE / RVQ?\n- What is the open-sourcing plan of SoniDo? Are there plans to open-source the dataset, model weights, or anything else?", "broader_impact_concerns": "While the authors do address ethical concerns about their dataset, more details on how the dataset was procured, what kind of data does the dataset have, etc. should greatly improve the ethical aspect of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "opo5UgpjBt", "reviewer": "Reviewer_Gxn5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong Elements\n- Goal of the paper is to increase performance by using hierarchical intermediate features and to verify that extracted features contain good information on music understanding.\n- - This paper achieves this goal by implementing a HQ-VAE specifically for music audio. \n- - This paper solidifies claims by running many downstream tasks and comparing to the SOTA in each respective task\n- Specifically lists out the SOTA for each downstream task\n\nWeak Elements\n- Is token out applied to the whole sequence, or for each hierarchical layer (Figure 4)?\nSmall Weak Elements \n- Position of diagram captions are not consistently on top or on bottom of diagram", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "SoniDo employs a unique approach compared to  other foundational music models such as Jukebox, MusicLM, and MusicGen. Jukebox utilizes multi-resolution VQ-VAEs, where input audio for each resolution is encoded and decoded separately. MusicLM and MusicGen rely on RVQ (Residual Vector Quantization) within an encoded space, followed by decoding. SoniDo uses a HQ-VAE architecture, which functions similarly to Jukebox but introduces a hierarchical model. Instead of having a single encoder for each resolution, SoniDo creates different encoded spaces for each resolution that are jointly decoded. This hierarchical encoding and decoding mechanism sets it apart from the others.\n\nAnother distinguishing feature of SoniDo is its custom loss function. First, there is an L2 reconstruction loss based on the input and output. Next, for each hierarchical layer, there is an L2 loss on the before and after quantized features followed by the subtraction of the entropy of the pmf. The loss is also regularized with respect to the variance of p(x|Z).\n\nIn the second stage, SoniDo trains three distinct autoregressive sparse transformers to model priors. SoniDo introduces a new data augmentation technique called token-out, which prevents overfitting. Token-out takes the concatenation of the three hierarchical layers’ features and randomly masks tokens, followed by a shuffling.\n\nSoniDo has demonstrated exceptional performance in various experiments. SoniDo has shown that the features extracted used for music understanding can be used for downstream tagging tasks. SoniDo passes state of the art for many tasks. For music transcription, it has been shown to significantly enhance transcription accuracy. In source separation tasks, combining SoniDo with HTDemucs consistently delivers superior results compared to other studies. For music mixing, adding SoniDO to existing models has been shown to reach state of the art for the majority of tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Strengthen Work\n- Change position of diagram captions so that they are all the same position relative to the diagram\n- Add some phrasing in the last paragraph of 3.2 to talk about the masking over the whole sequence rather than masking each layer independently.\n\nCritical Changes\n- No major changes", "broader_impact_concerns": "All ethical concerns, including licensing and bias in the model, have been addressed in the ethics section and introduction.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the reviewers had a recommendation that was leaning reject but the recommendation was based on novelty, which is not a key criterion for TMLR acceptance.  Another reviewer (8o1p) leaned accept but noted various things in their recommendation.  I am pasting these below:

"
RC1) "most models in other directions—such as contrastive learning and masked modeling—mainly focus on understanding tasks such as tagging" -> It doesn't necessarily mean that models focusing on music understanding do not learn useful representations for many downstream tasks including generative, for example, we have seen similar success with CLIP + BigGAN back then.

"they tend to perform significantly worse than task-agnostic models on more complex tasks like source separation". Besides source separation, in Figure 4, what authors suggest here would be a too-quick-conclusion, as there is no strong evidence for that in the Figure.

RC2) Thanks for more information.

RC3) The definition of a Music Foundation Model proposed in this work is reasonable; however, the current study does not fully meet the criteria to warrant that designation. To qualify as a music foundation model, (a) the model should arguably be trained on a large and diverse music dataset. While the training set used in this study appears to be of adequate size relative to what's available in the literature, its musical diversity and coverage is highly questionable. Additionally, (b) the model and its learned representations should demonstrate broad utility across a wide range of downstream musical applications. In this case, the downstream tasks explored are not comprehensive enough—particularly lacking in generative music scenarios such as conditioning, inpainting, and style transfer to name a few, across various modeling paradigms.

My main point is that the context of this work remains difficult to justify. While I believe this research is a valuable contribution and deserves to be shared with the community, referring to it as a "Music Foundation Model" overstates its scope. Although the authors have provided additional information regarding the dataset, there is still no clear evidence that the learned representations support general music understanding. Moreover, given the limited range of downstream musical tasks evaluated, it is difficult to argue that this is a general-purpose, one-for-many foundation model.

What the study demonstrates is that SoniDo performs well on several downstream tasks—such as music tagging, transcription, source separation, and mixing—and, in some cases, surpasses models like MusicGEN, as well as Jukebox and MERT with more limited experimental settings. However, it does not warrant the broader label of a Music Foundation Model. My primary concern with the paper lies in its contextualization and rhetoric, which could be improved to make more balanced claims and better position the work.
"

It seems the reviewer's biggest concern is the presentation of the paper more than anything.  I think this can be addressed by appropriately bringing the paper into context, and perhaps toning done some of the language, as recommended above by the reviewer.
Decision: Accept (poster)