Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on
  Materials Characterization
Authors: name: Zhengzhao Lai, name: Youbin Zheng, name: Zhenyang Cai, name: Haonan Lyu, name: Jinpu Yang, name: Hongqing Liang, name: Yan Hu, name: Benyou Wang

Materials characterization is fundamental to acquiring materials information,
revealing the processing-microstructure-property relationships that guide
material design and optimization. While multimodal large language models
(MLLMs) have recently shown promise in generative and predictive tasks within
materials science, their capacity to understand real-world characterization
imaging data remains underexplored. To bridge this gap, we present MatCha, the
first benchmark for materials characterization image understanding, comprising
1,500 questions that demand expert-level domain expertise. MatCha encompasses
four key stages of materials research comprising 21 distinct tasks, each
designed to reflect authentic challenges faced by materials scientists. Our
evaluation of state-of-the-art MLLMs on MatCha reveals a significant
performance gap compared to human experts. These models exhibit degradation
when addressing questions requiring higher-level expertise and sophisticated
visual perception. Simple few-shot and chain-of-thought prompting struggle to
alleviate these limitations. These findings highlight that existing MLLMs still
exhibit limited adaptability to real-world materials characterization
scenarios. We hope MatCha will facilitate future research in areas such as new
material discovery and autonomous scientific agents. MatCha is available at
https://github.com/FreedomIntelligence/MatCha.