MusiXQA: Advancing Visual Music Understanding in Multimodal Large
  Language Models
Authors: name: Jian Chen, name: Wenye Ma, name: Penghang Liu, name: Wei Wang, name: Tengwei Song, name: Ming Li, name: Chenguang Wang, name: Ruiyi Zhang, name: Changyou Chen

Multimodal Large Language Models (MLLMs) have achieved remarkable visual
reasoning abilities in natural images, text-rich documents, and graphic
designs. However, their ability to interpret music sheets remains
underexplored. To bridge this gap, we introduce MusiXQA, the first
comprehensive dataset for evaluating and advancing MLLMs in music sheet
understanding. MusiXQA features high-quality synthetic music sheets generated
via MusiXTeX, with structured annotations covering note pitch and duration,
chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks.
Through extensive evaluations, we reveal significant limitations of current
state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed
Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant
performance gains over GPT-based methods. The proposed dataset and model
establish a foundation for future advances in MLLMs for music sheet
understanding. Code, data, and model will be released upon acceptance.