Futga: Towards Fine-grained Music Understanding through
  Temporally-enhanced Generative Augmentation
Authors: name: Junda Wu; name: Zachary Novack; name: Amit Namburi; name: Jiaheng Dai; name: Hao-Wen Dong; name: Zhouhang Xie; name: Carol Chen; name: Julian McAuley
Published: 2024-07-29 22:53:32

Abstract: Existing music captioning methods are limited to generating concise global
descriptions of short music clips, which fail to capture fine-grained musical
characteristics and time-aware musical changes. To address these limitations,
we propose FUTGA, a model equipped with fined-grained music understanding
capabilities through learning from generative augmentation with temporal
compositions. We leverage existing music caption datasets and large language
models (LLMs) to synthesize fine-grained music captions with structural
descriptions and time boundaries for full-length songs. Augmented by the
proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
changes at key transition points and their musical functions, as well as
generate detailed descriptions for each music segment. We further introduce a
full-length music caption dataset generated by FUTGA, as the augmentation of
the MusicCaps and the Song Describer datasets. We evaluate the automatically
generated captions on several downstream tasks, including music generation and
retrieval. The experiments demonstrate the quality of the generated captions
and the better performance in various downstream tasks achieved by the proposed
music captioning approach. Our code and datasets can be found in
https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.