FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Authors: name: Junyu Luo, name: Zhizhuo Kou, name: Liming Yang, name: Xiao Luo, name: Jinsheng Huang, name: Zhiping Xiao, name: Jingshu Peng, name: Chengzhong Liu, name: Jiaming Ji, name: Xuanzhe Liu, name: Sirui Han, name: Ming Zhang, name: Yike Guo

Multimodal Large Language Models (MLLMs) have experienced rapid development
in recent years. However, in the financial domain, there is a notable lack of
effective and specialized multimodal evaluation datasets. To advance the
development of MLLMs in the finance domain, we introduce FinMME, encompassing
more than 11,000 high-quality financial research samples across 18 financial
domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We
ensure data quality through 20 annotators and carefully designed validation
mechanisms. Additionally, we develop FinScore, an evaluation system
incorporating hallucination penalties and multi-dimensional capability
assessment to provide an unbiased evaluation. Extensive experimental results
demonstrate that even state-of-the-art models like GPT-4o exhibit
unsatisfactory performance on FinMME, highlighting its challenging nature. The
benchmark exhibits high robustness with prediction variations under different
prompts remaining below 1%, demonstrating superior reliability compared to
existing datasets. Our dataset and evaluation protocol are available at
https://huggingface.co/datasets/luojunyu/FinMME and
https://github.com/luo-junyu/FinMME.