Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
Authors: name: Xinyu Fang; name: Zhijian Chen; name: Kai Lan; name: Shengyuan Ding; name: Yingji Liang; name: Xiangyu Zhao; name: Farong Wen; name: Zicheng Zhang; name: Guofeng Zhang; name: Haodong Duan; name: Kai Chen; name: Dahua Lin
Published: 2025-03-18 17:51:34

Abstract: Creativity is a fundamental aspect of intelligence, involving the ability to
generate novel and appropriate solutions across diverse contexts. While Large
Language Models (LLMs) have been extensively evaluated for their creative
capabilities, the assessment of Multimodal Large Language Models (MLLMs) in
this domain remains largely unexplored. To address this gap, we introduce
Creation-MMBench, a multimodal benchmark specifically designed to evaluate the
creative capabilities of MLLMs in real-world, image-based tasks. The benchmark
comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous
evaluation, we define instance-specific evaluation criteria for each test case,
guiding the assessment of both general response quality and factual consistency
with visual inputs. Experimental results reveal that current open-source MLLMs
significantly underperform compared to proprietary models in creative tasks.
Furthermore, our analysis demonstrates that visual fine-tuning can negatively
impact the base LLM's creative abilities. Creation-MMBench provides valuable
insights for advancing MLLM creativity and establishes a foundation for future
improvements in multimodal generative intelligence. Full data and evaluation
code is released on https://github.com/open-compass/Creation-MMBench.