AutoMat: Enabling Automated Crystal Structure Reconstruction from
  Microscopy via Agentic Tool Use
Authors: name: Yaotian Yang; name: Yiwen Tang; name: Yizhe Chen; name: Xiao Chen; name: Jiangjie Qiu; name: Hao Xiong; name: Haoyu Yin; name: Zhiyao Luo; name: Yifei Zhang; name: Sijia Tao; name: Wentao Li; name: Qinghua Zhang; name: Yuqiang Li; name: Wanli Ouyang; name: Bin Zhao; name: Xiaonan Wang; name: Fei Wei
Published: 2025-05-19 03:04:50

Abstract: Machine learning-based interatomic potentials and force fields depend
critically on accurate atomic structures, yet such data are scarce due to the
limited availability of experimentally resolved crystals. Although
atomic-resolution electron microscopy offers a potential source of structural
data, converting these images into simulation-ready formats remains
labor-intensive and error-prone, creating a bottleneck for model training and
validation. We introduce AutoMat, an end-to-end, agent-assisted pipeline that
automatically transforms scanning transmission electron microscopy (STEM)
images into atomic crystal structures and predicts their physical properties.
AutoMat combines pattern-adaptive denoising, physics-guided template retrieval,
symmetry-aware atomic reconstruction, fast relaxation and property prediction
via MatterSim, and coordinated orchestration across all stages. We propose the
first dedicated STEM2Mat-Bench for this task and evaluate performance using
lattice RMSD, formation energy MAE, and structure-matching success rate. By
orchestrating external tool calls, AutoMat enables a text-only LLM to
outperform vision-language models in this domain, achieving closed-loop
reasoning throughout the pipeline. In large-scale experiments over 450
structure samples, AutoMat substantially outperforms existing multimodal large
language models and tools. These results validate both AutoMat and
STEM2Mat-Bench, marking a key step toward bridging microscopy and atomistic
simulation in materials science.The code and dataset are publicly available at
https://github.com/yyt-2378/AutoMat and
https://huggingface.co/datasets/yaotianvector/STEM2Mat.