Foundation Models for Music: A Survey
Authors: name: Yinghao Ma; name: Anders Øland; name: Anton Ragni; name: Bleiz MacSen Del Sette; name: Charalampos Saitis; name: Chris Donahue; name: Chenghua Lin; name: Christos Plachouras; name: Emmanouil Benetos; name: Elio Quinton; name: Elona Shatri; name: Fabio Morreale; name: Ge Zhang; name: György Fazekas; name: Gus Xia; name: Huan Zhang; name: Ilaria Manco; name: Jiawen Huang; name: Julien Guinot; name: Liwei Lin; name: Luca Marinelli; name: Max W. Y. Lam; name: Megha Sharma; name: Qiuqiang Kong; name: Roger B. Dannenberg; name: Ruibin Yuan; name: Shangda Wu; name: Shih-Lun Wu; name: Shuqi Dai; name: Shun Lei; name: Shiyin Kang; name: Simon Dixon; name: Wenhu Chen; name: Wehhao Huang; name: Xingjian Du; name: Xingwei Qu; name: Xu Tan; name: Yizhi Li; name: Zeyue Tian; name: Zhiyong Wu; name: Zhizheng Wu; name: Ziyang Ma; name: Ziyu Wang
Published: 2024-08-26 15:13:14

Abstract: In recent years, foundation models (FMs) such as large language models (LLMs)
and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
including music. This comprehensive review examines state-of-the-art (SOTA)
pre-trained models and foundation models in music, spanning from representation
learning, generative learning and multimodal learning. We first contextualise
the significance of music in various industries and trace the evolution of AI
in music. By delineating the modalities targeted by foundation models, we
discover many of the music representations are underexplored in FM development.
Then, emphasis is placed on the lack of versatility of previous methods on
diverse music applications, along with the potential of FMs in music
understanding, generation and medical application. By comprehensively exploring
the details of the model pre-training paradigm, architectural choices,
tokenisation, finetuning methodologies and controllability, we emphasise the
important topics that should have been well explored, like instruction tuning
and in-context learning, scaling law and emergent ability, as well as
long-sequence modelling etc. A dedicated section presents insights into music
agents, accompanied by a thorough analysis of datasets and evaluations
essential for pre-training and downstream tasks. Finally, by underscoring the
vital importance of ethical considerations, we advocate that following research
on FM for music should focus more on such issues as interpretability,
transparency, human responsibility, and copyright issues. The paper offers
insights into future challenges and trends on FMs for music, aiming to shape
the trajectory of human-AI collaboration in the music realm.