AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

## Abstract

Abstract Machine learning-based interatomic potentials and force fields depend critically on accurate atomic structures, yet such data are scarce due to the limited availability of experimentally resolved crystals. Although atomic-resolution electron microscopy offers a potential source of structural data, converting these images into simulation-ready formats remains labor-intensive and error-prone, creating a bottleneck for model training and validation. We introduce AutoMat , an end-to-end, agent-assisted pipeline that automatically transforms scanning transmission electron microscopy (STEM) images into atomic crystal structures and predicts their physical properties. AutoMat combines pattern-adaptive denoising, physics-guided template retrieval, symmetry-aware atomic reconstruction, fast relaxation and property prediction via MatterSim, and coordinated orchestration across all stages. We propose the first dedicated STEM2Mat - Bench for this task and evaluate performance using lattice RMSD, formation energy MAE, and structure-matching success rate. By orchestrating external tool calls, AutoMat enables a text-only LLM to outperform vision-language models in this domain, achieving closed-loop reasoning throughout the pipeline. In large-scale experiments over 450 structure samples, AutoMat substantially outperforms existing multimodal large language models and tools. These results validate both AutoMat and STEM2Mat - Bench, marking a key step toward bridging microscopy and atomistic simulation in materials science.The code and dataset are publicly available at https://github.com/yyt-2378/AutoMat and https://huggingface.co/datasets/yaotianvector/STEM2Mat

## 1 Introduction

Figure: Figure 1: Overview of AutoMat. The left part shows an LLM agent how to manage four modules—pattern-adaptive denoising, template selection, atomic reconstruction (STEM2CIF), and ML-based property prediction. The right panel presents a line chart and radar plot comparing different models in terms of energy and structural errors.
Refer to caption: extracted/6452132/figures/teaser_overview.jpg

Machine learning (ML) interatomic potentials and force fields now approach  *ab initio* accuracy in predicting atomic energies and forces[1, 2, 3, 4, 5] but remain limited by experimentally validated crystal structures.
Meanwhile, scanning transmission electron microscopy (STEM) can image atoms individually at sub-ångström resolution[6, 7, 8, 9], yet translating them into quantitative crystal structures still relies on expert-driven, time-consuming annotation.
This disconnect creates a critical gap between structural representation and theoretical validation in materials science[10, 11].

Although recent advances in STEM image analysis have shown promise, most existing studies focus on individual components like denoising[12, 13, 14, 15], atom localization[16, 17, 18], reconstruction[19, 20, 21] and phase classification[22, 23]. These approaches remain fragmented and are not integrated into end-to-end systems.
Conventional image denoising techniques suppress noise and improve contrast at the pixel level but cannot yield periodic or chemically meaningful crystal structures[24, 25, 15, 12].
Atomic detection models can localize atomic peaks but cannot infer complete lattices(^1^11In crystallography, a lattice refers to the periodic arrangement of atoms in a crystal structure.) or identify atomic species.
General-purpose multimodal models like GPT-4.1mini[26] and Qwen2.5-VL[27] exhibit basic image understanding but lack the ability to produce simulation-ready formats such as crystallographic file(CIF).
Even domain-specific tools like AtomAI[28] can only predict atomic coordinates in minimal systems and do not support full structure reconstruction or property prediction.
Meanwhile, public datasets (e.g., JARVIS-STM) mainly target STM( Scanning Tunneling Microscopy) images, lack DFT-level energy labels, and are unsuitable for benchmarking structure–property pipelines.
As a result, the field still lacks a fully automated end-to-end system that can convert raw STEM images into reconstructed structures and simulated properties, along with the standardized benchmark for comprehensive evaluation.

To address this gap, as shown in Fig. [1](https://arxiv.org/html/2505.12650v1#S1.F1), we introduce AutoMat, the first agent-coordinated framework linking STEM imaging with atomistic simulation, along with a benchmark tailored to the task.
The AutoMat pipeline comprises four key stages:
1) Pattern-Adaptive STEM Image denoising: We apply a pattern-adaptive mixture-of-experts network, MOE-Denoising Inference Variational Autoencoder Super-Resolution[12](MOE-DIVAESR), to denoise and enhance raw iDPC-STEM(^2^22iDPC-STEM stands for integrated Differential Phase Contrast Scanning Transmission Electron Microscopy, a technique that enhances light element contrast in atomic-resolution imaging.) images.
The ResNet-18-based gating network elects the most suitable expert network for each input image based on its estimated noise level, enabling joint denoising, inpainting, and super-resolution.
2) Physics-Guided Template Retrieval: Enhanced images are matched to a large-scale library of simulated STEM projections. Top candidate structures are retrieved using pixel similarity and filtered by elemental contrast patterns
to produce strong structural priors.
3) Symmetry-Constrained Structure Reconstruction: Atomic peaks are detected via unsupervised clustering. We fit the lattice under symmetry constraints, assign atomic species based on the candidate, and generate the standard CIF file representing the periodic crystal structure.
4) Energy Evaluation via Machine-Learned Potential: The reconstructed structure is relaxed using the pretrained MatterSim potential to predict formation energy.
A language agent coordinates the all stages and automatically retries failed stages based on quality checks.

To support rigorous evaluation, we curated 2,143 high-quality monolayer structures from C2DB[29], Materials Project[30], and OpenCrystal[31], and simulated their corresponding iDPC-STEM images with abTEM[32]. From this pool we selected 450 representative image–structure pairs, which constitute our STEM2Mat benchmark and are used for all subsequent evaluations. Our evaluation metrics include projected lattice RMSD, formation energy MAE, and atom-wise structure matching success rate. Additionally, we introduce three fine-grained indicators to assess reconstruction quality (e.g., atomic recall/precision), robustness across noise levels, and computational efficiency.
On this benchmark, AutoMat achieves a projected RMSD of $0.11\pm 0.03$ Å, energy MAE below 350 meV/atom, and an atomic correspondence total success rate of 83.2%, outperforming GPT-4.1-mini, Qwen-VL, LLama4V, ChemVLM[33], and AtomAI by an order of magnitude.
These results demonstrate AutoMat as a reproducible and accurate end-to-end solution for microscopy-driven materials modeling.

The contributions of this paper are summarized as follows:

• Agent-driven end-to-end pipeline.
AutoMat proposes an automated agent integrating pattern-adaptive denoising, physics-informed template retrieval, symmetry-constrained reconstruction, and ML-based relaxation, enabling the conversion of raw STEM images into material property predictions.

• Algorithmic advancements. We design MOE-DIVAESR as a pattern-adaptive denoiser for diverse STEM images, enabling efficient noise reduction, defect correction, and detail enhancement. We also develop STEM2CIF to reconstruct crystal structures by identifying the minimal repeating unit using symmetry heuristics and physical constraints, then converting the result into standard CIF format.

• STEM2Mat benchmark & evaluation suite. We release a dataset of 2,143 distinct supercell structures used to simulate large-field STEM images. In addition, we provide a 450-case test split with DFT-calculated energies and establish unified metrics to evaluate lattice reconstruction accuracy, energy fidelity, robustness, and computational efficiency.

• State-of-the-art results with open access. AutoMat outperforms leading large multimodal models by an order of magnitude. All code, data, and evaluation tools are publicly released to support reproducibility and future development.

## 2 Related Work

Automated Microscopy Image Analysis.
In recent years, deep learning methods have been extensively applied to electron microscopy and scanning probe microscopy data analysis[12, 13, 14, 15, 16, 17, 19, 20, 21]. Current approaches range from unsupervised defect detection to supervised atomic column identification. AtomAI [28], for example, integrates microscopy images with computational simulations, but primarily focuses on atom segmentation and identification. It lacks a fully automated, agent-based framework that coordinates tool usage to directly recover crystal structures and predict material properties.

STEM Image to Structure Reconstruction.
Existing methods for reconstructing crystal structures from STEM images typically rely on multiple images, prior structural information, or are limited to single-element systems. For instance, De Backer et al [19] employed Bayesian genetic algorithms for 3D reconstruction, but their approach was demonstrated only for simple single-element systems. Currently, few methods can directly generate standard crystallographic information files (CIF) from single experimental images, especially for complex multi-element two-dimensional crystals[20].

Vision-Language Models in Chemistry.
Recently, multimodal large language models (such as ChemVLM[33], GPT-4.1mini[26], and Qwen-VL[27]) have started to be explored in chemistry and materials science. However, these models generally lack the capability to accurately handle detailed spatial structure tasks. Existing chemical agent tools (e.g., ChemCrow[34] and Chemagents[35]) are limited to text-based descriptions and cannot process image-based inputs, significantly restricting their applicability in microscopy-based analyses.

Material Property Prediction Models.
Advances in machine learning-based interatomic potentials (such as MatterSim[1], M3GNet[36], and MACE[37]) have significantly improved the accuracy of computational property predictions. Combining these models with experimental imaging provides a novel, digital twin-like approach for validating structural reconstructions. However, existing benchmarks predominantly evaluate models using theoretical datasets, lacking end-to-end assessments starting from experimental image inputs.

In summary, these approaches highlight progress and limitations in microscopy image analysis, structure reconstruction, multimodal modeling, and property prediction. AutoMat addresses these gaps by integrating an automated agent from STEM images to material property prediction and establishes the STEM2Mat benchmark for evaluating the robustness and scalability of automated material characterization.

## 3 Dataset Construction

### 3.1 Composition and Taxonomy

Our benchmark focuses on two-dimensional (2D) materials, whose
atomic-scale thickness allows STEM to resolve individual columns with
minimal multiple-scattering artifacts.
Starting from nearly 10,000 candidate structures harvested from
C2DB, Materials Project, and OpenCrystal,(^3^33All sources were queried in April 2025 using identical monolayer filters.)
we followed a two-stage curation process.
First, automated filters removed non-stoichiometric, partially
occupied, or 3D bulk entries.
Second, domain experts inspected symmetry, cleavage energy, and
substrate feasibility, yielding 2,143 high-confidence monolayer
crystals.
The collection spans six chemical families (Fig. LABEL:fig:data_overview_appendix):
*(i)* classic 2D materials—graphene, MoS2,
h-BN, black phosphorus;
*(ii)* emergent allotropes, e.g. silicene, borophene;
*(iii)* conductive MXenes (23 distinct formulas);
*(iv)* intrinsic 2D magnets such as CrI3 and
Fe3GeTe2;
*(v)* Janus structures typified by MoSSe;
and *(vi)* Ruddlesden–Popper–type 2D perovskites.
Elemental diversity is broad: 67 unique elements appear, producing
76 unary 1,409 binary, and 658 ternary systems.
Each structure is stored as a cif file with validated lattice
vectors and fractional atomic coordinates.

### 3.2 Image Simulation and Data Augmentation

To simulate realistic large-field STEM imaging conditions, We generated synthetic iDPC-STEM micrographs using the open-source abTEM simulation engine. For each structure, a random $12\times 12$ to $16\times 16$ supercell was constructed and projected at $0.1\text{\,}\mathrm{\SIUnitSymbolAngstrom}\mathrm{/}\mathrm{p}\mathrm{i}
\mathrm{x}\mathrm{e}\mathrm{l}$ resolution. Five electron-dose settings ($1\text{--}6\times 10^{4}\,\mathrm{e}^{-}/$\mathrm{\SIUnitSymbolAngstrom}^{2}$$) and realistic lens aberrations were sampled to mimic experimental conditions.
Poisson detector noise was injected to match reported signal-to-noise ratios. To study model robustness, we applied Gaussian blurring and dose-specific shot noise to simulate additional imaging imperfections. Ground-truth atomic coordinates were rendered into Gaussian “atom masks” to enable supervised training of localisation models. Each sample thus forms an image–structure–property triplet: (i) the noisy STEM projection, (ii) the corresponding ground-truth cif, and (iii) DFT-level formation energy (along with band gap and magnetic moment, if available). We conducted principal component analysis (PCA) on structural fingerprints, which revealed clear clustering patterns. Subsequently, $k$-means clustering ($k{=}2,6$) was used to ensure balanced train/validation/test splits across the chemical diversity of the dataset.

### 3.3 STEM2Mat Benchmark Split and Tiering

To construct a representative and tractable benchmark dataset for STEM-based crystal modeling, we applied stringent geometric and chemical screening criteria to the 2,143 collected 2D material structures, ensuring their suitability for monolayer imaging and end-to-end reconstruction. Specifically, we retained only structures containing no more than three distinct elements. For those with multiple elements, we required a minimum atomic-number span of ten, i.e., $\max(Z_{i})-\min(Z_{i})\geq 10$, to ensure sufficient imaging contrast between heavy and light atoms. To guarantee monolayer geometry, we limited the $z$-axis thickness to no more than 3 Å. Each structure’s atomic coordinates were then projected onto the $(x,y)$ plane, discretized onto a 1 Å grid, and evaluated for overlapping projections. Only structures with a projected duplication ratio below 10%, i.e., $\frac{\text{Number of overlapping grid points}}{\text{Total grid points}}\leq 0.1$, were retained to avoid ambiguity in atomic interpretation.

Following this multi-criteria filtering process, we retained 450 well-defined, unambiguous monolayer structures—21% of the original dataset—for blind end-to-end evaluation. The remaining 1,693 samples were split into training (80%) and validation (20%) sets to support model training and tuning.

Figure: Figure 2: Overview of the STEM2Mat Benchmark design, illustrating the tiered classification of STEM samples by material complexity and imaging dose, which systematically stratifies reconstruction difficulty from simple unary to complex ternary compounds.
Refer to caption: extracted/6452132/figures/benchmark.png

To analyze model performance as a function of task difficulty, we stratified the test set into three tiers based on material composition and imaging noise (see Fig. [2](https://arxiv.org/html/2505.12650v1#S3.F2)):

• Tier 1 – Unary materials acquired under high electron doses ($5\text{--}6\times 10^{4}\,\mathrm{e}^{-}/$\mathrm{\SIUnitSymbolAngstrom}^{2}$$); these images are high contrast, low noise, and represent the easiest cases (35 samples).

• Tier 2 – Binary materials or moderate electron dose conditions ($3\text{--}4\times 10^{4}\,\mathrm{e}^{-}/$\mathrm{\SIUnitSymbolAngstrom}^{2}$$); these represent intermediate complexity in both noise level and atomic diversity (456 samples).

• Tier 3 – Ternary compounds imaged at low dose ($1\text{--}2\times 10^{4}\,\mathrm{e}^{-}/$\mathrm{\SIUnitSymbolAngstrom}^{2}$$); these samples exhibit high noise, complex contrast, and are the most challenging to reconstruct (79 samples).

While the tier sizes are imbalanced, this hierarchical structure reveals
a clear gradient of reconstruction difficulty, establishing a principled
ladder for evaluating robustness and scalability. With the STEM2Mat-Bench tiers defined, we now specify the quantitative metrics used throughout the paper. For detailed definitions and formulae of the evaluation metrics, see evaluation metrics [3.4](https://arxiv.org/html/2505.12650v1#S3.SS4).

### 3.4 Evaluation Metrics

To compare methods reproducibly, we report two *primary* metrics—energy error and lattice error—and two *holistic* metrics that reflect chemical and structural validity.

#### Mean Absolute Error (MAE).

Average difference between predicted and DFT formation energies:

$$ $\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\bigl{|}E^{\text{pred}}_{i}-E^{\text{ref }}_{i}\bigr{|}\;[\mathrm{meV/atom}]$ (1) $$

#### Projected Lattice RMSD.

Deviation of in-plane lattice constants:

$$ $\mathrm{RMSD}_{xy}=\sqrt{\tfrac{1}{2}\Bigl{[}(a^{\text{pred}}\!-\!a^{\text{ref }})^{2}+(b^{\text{pred}}\!-\!b^{\text{ref}})^{2}\Bigr{]}}$ (2) $$

#### Composition Correctness (C.C.).

1 if elemental types and counts match the reference; 0 otherwise.

#### Structure Success Rate (S.S.).

A prediction is successful when it satisfies both chemistry and geometry.
First define 2-D spatial similarity

$$ $S_{\mathrm{spatial}}=\exp\!\bigl{(}-\mathrm{MSE}_{2\mathrm{D}}\bigr{)},$ (3) $$

where $\mathrm{MSE}_{2\mathrm{D}}$ is the mean-squared error of projected
atomic positions after optimal element-wise matching.
Then

$$ $\mathrm{S.S.}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\!\Bigl{[}S_{\mathrm{spatial} }^{(i)}\geq 0.8\Bigr{]}\times 100\%.$ (4) $$

## 4 Overview of AutoMat

### 4.1 Limitations of Existing Large Multimodal Models and Domain Tools

Most current large multimodal models (e.g., GPT-4.1mini, LLama4V, Qwen2.5-VL, ChemVLM) focus on general-purpose image understanding tasks such as scene recognition, OCR, and molecular structure identification, but exhibit limited capability in interpreting scientific images like electron microscopy (EM). Domain-specific chemistry agents (e.g., ChemAgent[35], ChemCrow[34]) remain predominantly text-based, executing tool calls for predefined tasks without closed-loop visual reasoning or image-guided decision-making. Specialized STEM data toolkits (e.g., AtomAI) provide atomic coordinate extraction and segmentation, yet their applicability is largely restricted to single-element nanoparticles and fail with complex, multi-element STEM images.

To date, no agent autonomously generates simulation-ready CIFs and predicts formation energies from a single real STEM image, thus completing a generalizable structure–property pipeline. To bridge this gap, we introduce AutoMat, a modular agent framework with advanced reasoning, enabling end-to-end modeling from pixel-level inputs to material property predictions.

Figure: Figure 3: AutoMat’s LLM agent orchestrates four stages—denoising, template matching, structure reconstruction, and property prediction—from STEM image to relaxed crystal with properties.
Refer to caption: extracted/6452132/figures/agent_pipline.png

### 4.2 Tool Pool

AutoMat integrates a specialized suite of intelligent tools to bridge electron microscopy characterization and material modeling. Its core comprises four modules—MOE-DIVAESR, Image Template Matching, STEM2CIF, and MatterSim—for image denoising, template matching, structure reconstruction, and property prediction, respectively(see Appendix LABEL:appendix:moe_divaesr and LABEL:appendix:template2cif for full implementation details).

MOE-DIVAESR is a structure-pattern-adaptive mixture-of-experts (MoE) model for STEM image denoising. Trained on an augmented STEM dataset, it includes multiple expert networks, each targeting distinct structural patterns. A gating network dynamically selects the optimal experts based on the features of the input image. This architecture enables MOE-DIVAESR to perform denoising, defect correction, and fine-grained detail enhancement, yielding sharply resolved atomic columns as high-quality inputs for downstream reconstruction.

Image Template Matching compares the enhanced image to a pre-stored or externally mounted database of structural templates or known materials. Using image features and elemental cues, it narrows the candidate structures and selects the one most consistent with the observed features.

STEM2CIF combines the selected structural template and enhanced image to reconstruct a high-fidelity atomic model. The module identifies atomic column positions, infers lattice parameters, and determines atomic species. It also incorporates crystallographic symmetry heuristics and physical constraints to reduce the structure to its minimal repeating unit, outputting a crystallographic CIF file.

MatterSim is a pretrained machine-learning interatomic potential model for rapid structural relaxation and property evaluation. Trained on large-scale DFT datasets, MatterSim achieves near-DFT accuracy for energy and property predictions. Integrated with ASE, it enables structure optimization and rapid assessment of properties such as total energy, formation energy, and elastic moduli, providing a fast and accurate alternative to conventional DFT.

### 4.3 LLM-Orchestrated End-to-End Pipeline

Based on the aforementioned tool pool, AutoMat deploys an LLM agent based on DeepSeekV3 to dynamically coordinate the end-to-end tool execution process, as illustrated in Figure [3](https://arxiv.org/html/2505.12650v1#S4.F3). The large language model agent is responsible for chaining the four sequential stages, managing inter-stage data flow, and making iterative decisions throughout.

The process starts from a raw STEM image. The agent first chooses to call MOE-DIVAESR to perform denoising and structural enhancement on the image. After obtaining a clearer image, the agent determines that Image Template Matching needs to be executed to retrieve candidate templates from the structure database based on atomic geometry and elemental contrast. After selecting the best-matching structure, the process proceeds to the STEM2CIF tool for structure reconstruction. This module localizes atomic peaks, fits lattice parameters, assigns elements, and finally generates a standard CIF file. To calculate the physical properties of the material, MatterSim is automatically called at the end to relax the structure and predict total energy, formation energy, and other properties. Throughout the entire process, the agent monitors the quality of intermediate results in real time and performs rollback and retries if failures occur. Logic-driven orchestration enables AutoMat to complete a closed-loop automated material modeling workflow from image to property prediction under varying input conditions.

Figure [3](https://arxiv.org/html/2505.12650v1#S4.F3) summarizes the process, showing how the agent integrates image, structure, and properties. The final output is a stable atomic model with predicted properties, achieving AutoMat’s goal of automated reasoning from STEM images to materials insights.

## 5 Experiments and Results

### 5.1 Quantitative Evaluation

#### Baseline Overview

To evaluate AutoMat, we compare it against three baselines targeting reasoning, reconstruction, and oracle performance:

- •
Vision–Language Models (VLM). GPT-4.1mini, Qwen2.5-VL (32B), LLama4V (17B) and ChemVLM (8B) receive a fixed prompt, composition hints, and a STEM image to infer material properties, assessing multimodal reasoning.
- •
AtomAI. AtomAI’s segmentation network detects atomic centers; relative coordinates plus image resolution are used to fit the lattice. This baseline measures reconstruction quality only.
- •
Ground-Truth CIF + MLIP (Oracle Upper Bound). The true CIF is fed directly to the MatterSim MLIP to benchmark formation-energy error under perfect-structure assumptions.

We summarize the performance of these baselines and our AutoMat system on the test set in Tables [1](https://arxiv.org/html/2505.12650v1#S5.T1) and [2](https://arxiv.org/html/2505.12650v1#S5.T2), which together cover samples across Tier 1–3 difficulty levels.

**Table 1: Formation-energy MAE (meV/atom) across tiers. “↓” indicates lower is better; “–” indicates the method does not provide an energy prediction.**
| Method | Tier 1 $\downarrow$ | Tier 2 $\downarrow$ | Tier 3 $\downarrow$ | Avg. $\downarrow$ |
| --- | --- | --- | --- | --- |
| GPT-4.1 Vision | 2521.9 | 2657.4 | 2817.7 | 2657.3 |
| Qwen-VL (32B) | 3488.6 | 2673.9 | 4113.9 | 2763.5 |
| LLama4V (17B) | 2549.8 | 2898.3 | 3662.6 | 2931.7 |
| ChemVLM (8B) | 3961.1 | 4237.5 | 5090.3 | 4237.6 |
| AtomAI | – | – | – | – |
| AutoMat (Ours) | 343.59 | 320.21 | 333.49 | 321.57 |
| Upper Bound | 57.377 | 47.227 | 30.945 | 48.105 |

**Table 2: Structural-accuracy metrics for methods that output atomic models. RMSDxy$\downarrow$: in-plane lattice root-mean-square deviation (lower is better); C.C.$\uparrow$: composition correctness; S.S.$\uparrow$: structure success rate.**
| Tier | Method | RMSDxy (Å)$\downarrow$ | C.C. (%)$\uparrow$ | S.S. (%)$\uparrow$ |
| --- | --- | --- | --- | --- |
| 1 | AtomAI | 43.96$\pm$0.31 | 2.70 | 0.0 |
| AutoMat (Ours) | 0.11$\pm$0.02 | 88.9 | 85.0 |  |
| Upper Bound | 0 (def.) | 100 | 100 |  |
| 2 | AtomAI | N/A | 0.0 | 0.0 |
| AutoMat (Ours) | 0.11$\pm$0.03 | 85.9 | 84.0 |  |
| Upper Bound | 0 (def.) | 100 | 100 |  |
| 3 | AtomAI | N/A | 0.0 | 0.0 |
| AutoMat (Ours) | 0.11$\pm$0.03 | 73.1 | 73.1 |  |
| Upper Bound | 0 (def.) | 100 | 100 |  |

Figure: Figure 4: Case studies on Tier 2 and Tier 3 samples comparing AutoMat, GPT-4.1mini and Llama-4-Maverick: enhanced images, best-matched templates, reconstructed unit cells and predicted energies.
Refer to caption: extracted/6452132/figures/model_results_compare.png

#### Discussion of Results.

In energy prediction, AutoMat achieves a mean formation energy MAE of $332\pm 12$ meV/atom, with tier-wise results of 343.59, 320.21, and 333.49 meV/atom for Tier 1–3. While this is higher than the MLIP oracle lower bound of 57 meV/atom, it remains significantly better than the multi-eV errors produced by vision–language models. As task difficulty increases, MAE in VL baselines rises accordingly, validating our tiered benchmark design. These results suggest that AutoMat’s residual error is mainly due to reconstruction rather than MLIP limitations, and that the predicted structures are reliable for downstream property evaluation.Figure [4](https://arxiv.org/html/2505.12650v1#S5.F4) illustrates two representative examples from Tier 2 and Tier 3, comparing AutoMat with GPT-4.1mini and LLaMA4. The figure highlights AutoMat’s superior ability to reconstruct structures and predict energy accurately, even under low-dose and multi-element conditions.

For structural reconstruction, AutoMat achieves an average projected RMSDxy of approximately 0.11 Å, two orders of magnitude lower than AtomAI (43–44 Å), with most deviation correctable through final relaxation. For composition correctness, AutoMat achieves 83% on average across tiers (88.9%, 85.9%, 73.1%), while AtomAI achieves less than 2.7%, only in simple Tier 1 cases. For structure success rate, which considers both spatial and compositional agreement, AutoMat achieves 83.2% overall (Tier-wise: 85.0%, 84.0%, 73.1%), whereas AtomAI rarely produces valid structures.

In summary, AutoMat not only outperforms all existing baselines but also maintains high performance in challenging Tier 3 scenarios involving multi-element compositions and low imaging doses, demonstrating its robustness and generalizability across the full benchmark spectrum.

### 5.2 Error Analysis

To better understand the failure modes of AutoMat, we conducted a detailed analysis of representative failure cases across all three tiers. Two primary error types were identified:

(1) Template retrieval failure (39.3%):
In these cases, AutoMat failed to retrieve the correct structure from the template database, resulting in mismatches in atomic arrangements and element types. This led to cascading errors in structure, composition, and property predictions. Incorrect atom counts further caused large energy estimation errors.
For example, in a Tier 3 case, a true structure containing U, F, and O was mismatched with a P-based template, causing severe atomic misalignment and an energy_per_atom error of up to 3.13 eV.

(2) Downstream failure despite correct template (60.7%):
Even with the correct template, downstream steps failed due to projection ambiguity or elemental confusion. In 40% of these cases, atoms appeared too close in the 2D projection, and the lack of z-axis information led to poor relaxation and inaccurate energy estimates. In 20.7%, elements with similar atomic numbers (e.g., C and O) exhibited indistinguishable contrast, causing misclassification and full breakdowns in lattice fitting and CIF generation.
For instance, in a Tier 2 case, closely spaced C and O atoms were confused, leading to complete loss of C atoms and reconstruction failure.

These findings highlight two key directions for improving AutoMat: (i) improving retrieval robustness via uncertainty-aware or multi-candidate matching; and (ii) overcoming 2D projection limits through 3D-aware modeling [38, 39, 40] and enhanced modality integration. Together, these efforts can greatly improve structural fidelity and prediction reliability in complex systems.

## 6 Conclusion

We proposed AutoMat, an end-to-end agent system that automatically reconstructs material structures and predicts properties from STEM images. By integrating pattern-adaptive vision models, symmetry-constrained reconstruction, and LLM-driven orchestration, AutoMat achieves accurate alignment between microscopy data and atomic modeling, significantly outperforming existing methods in structural and energetic evaluation. Meanwhile, we propose STEM2Mat-Bench for this task. AutoMat advances autonomous materials research and AI-driven experimentation. Future work will focus on strengthening its role as a bridge between experimental characterization and theoretical computation.

## References

- Yang et al. [2024]
Han Yang, Chenxi Hu, Yichi Zhou, Xixian Liu, Yu Shi, Jielan Li, Guanzhi Li, Zekun Chen, Shuizhou Chen, Claudio Zeni, Matthew Horton, Robert Pinsler, Andrew Fowler, Daniel Zügner, Tian Xie, Jake Smith, Lixin Sun, Qian Wang, Lingyu Kong, Chang Liu, Hongxia Hao, and Ziheng Lu.
Mattersim: A deep learning atomistic model across elements, temperatures and pressures, 2024.
URL [https://arxiv.org/abs/2405.04967](https://arxiv.org/abs/2405.04967).
- Batatia et al. [2024]
Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M. Elena, Dávid P. Kovács, Janosh Riebesell, Xavier R. Advincula, Mark Asta, Matthew Avaylon, William J. Baldwin, Fabian Berger, Noam Bernstein, Arghya Bhowmik, Samuel M. Blau, Vlad Cărare, James P. Darby, Sandip De, Flaviano Della Pia, Volker L. Deringer, Rokas Elijošius, Zakariya El-Machachi, Fabio Falcioni, Edvin Fako, Andrea C. Ferrari, Annalena Genreith-Schriever, Janine George, Rhys E. A. Goodall, Clare P. Grey, Petr Grigorev, Shuang Han, Will Handley, Hendrik H. Heenen, Kersti Hermansson, Christian Holm, Jad Jaafar, Stephan Hofmann, Konstantin S. Jakob, Hyunwook Jung, Venkat Kapil, Aaron D. Kaplan, Nima Karimitari, James R. Kermode, Namu Kroupa, Jolla Kullgren, Matthew C. Kuner, Domantas Kuryla, Guoda Liepuoniute, Johannes T. Margraf, Ioan-Bogdan Magdău, Angelos Michaelides, J. Harry Moore, Aakash A. Naik, Samuel P. Niblett, Sam Walton Norwood, Niamh O’Neill, Christoph Ortner, Kristin A. Persson, Karsten Reuter, Andrew S. Rosen, Lars L. Schaaf,
Christoph Schran, Benjamin X. Shi, Eric Sivonxay, Tamás K. Stenczel, Viktor Svahn, Christopher Sutton, Thomas D. Swinburne, Jules Tilly, Cas van der Oord, Eszter Varga-Umbrich, Tejs Vegge, Martin Vondrák, Yangshuai Wang, William C. Witt, Fabian Zills, and Gábor Csányi.
A foundation model for atomistic materials chemistry, 2024.
URL [https://arxiv.org/abs/2401.00096](https://arxiv.org/abs/2401.00096).
- Gasteiger et al. [2021]
Johannes Gasteiger, Florian Becker, and Stephan Günnemann.
Gemnet: Universal directional graph neural networks for molecules.
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 6790–6802. Curran Associates, Inc., 2021.
URL [https://proceedings.neurips.cc/paper_files/paper/2021/file/35cf8659cfcb13224cbd47863a34fc58-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2021/file/35cf8659cfcb13224cbd47863a34fc58-Paper.pdf).
- Liao et al. [2024]
Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt.
Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations, 2024.
URL [https://arxiv.org/abs/2306.12059](https://arxiv.org/abs/2306.12059).
- Lu et al. [2023]
Shuqi Lu, Zhifeng Gao, Di He, Linfeng Zhang, and Guolin Ke.
Highly accurate quantum chemical property prediction with uni-mol+, 2023.
URL [https://arxiv.org/abs/2303.16982](https://arxiv.org/abs/2303.16982).
- Zhang et al. [2018]
Daliang Zhang, Yihan Zhu, Lingmei Liu, Xiangrong Ying, Chia-En Hsiung, Rachid Sougrat, Kun Li, and Yu Han.
Atomic-resolution transmission electron microscopy of electron beam–sensitive crystalline materials.
*Science*, 359(6376):675–679, 2018.
ISSN 0036-8075.
- Zhang et al. [2023]
Hui Zhang, Guanxing Li, Jiaxing Zhang, Daliang Zhang, Zhen Chen, Xiaona Liu, Peng Guo, Yihan Zhu, Cailing Chen, and Lingmei Liu.
Three-dimensional inhomogeneity of zeolite structure and composition revealed by electron ptychography.
*Science*, 380(6645):633–638, 2023.
ISSN 0036-8075.
- Zhang et al. [2020]
Qing Zhang, Alvaro Mayoral, Junyan Li, Juanfang Ruan, Viveka Alfredsson, Yanhang Ma, Jihong Yu, and Osamu Terasaki.
Electron microscopy studies of local structural modulations in zeolite crystals.
*Angewandte Chemie*, 132(44):19571–19581, 2020.
ISSN 0044-8249.
- Ortalan et al. [2010]
Volkan Ortalan, Alper Uzun, Bruce C Gates, and Nigel D Browning.
Direct imaging of single metal atoms and clusters in the pores of dealuminated hy zeolite.
*Nature nanotechnology*, 5(7):506–510, 2010.
ISSN 1748-3387.
- Kalinin et al. [2022]
Sergei V Kalinin, Colin Ophus, Paul M Voyles, Rolf Erni, Demie Kepaptsoglou, Vincenzo Grillo, Andrew R Lupini, Mark P Oxley, Eric Schwenker, and Maria KY Chan.
Machine learning in scanning transmission electron microscopy.
*Nature Reviews Methods Primers*, 2(1):11, 2022.
ISSN 2662-8449.
- Kalinin et al. [2023]
Sergei V Kalinin, Debangshu Mukherjee, Kevin Roccapriore, Benjamin J Blaiszik, Ayana Ghosh, Maxim A Ziatdinov, Anees Al-Najjar, Christina Doty, Sarah Akers, Nageswara S Rao, et al.
Machine learning for automated experimentation in scanning transmission electron microscopy.
*npj Computational Materials*, 9(1):227, 2023.
- Yang et al. [2025]
Yaotian Yang, Hao Xiong, Zirong Wu, Zhiyao Luo, Xiao Chen, Xiaonan Wang, and Fei Wei.
Deep learning-enabled stem imaging for precise single-molecule identification in zeolite structures.
*Advanced Science*, 12(6):2408629, 2025.
doi: https://doi.org/10.1002/advs.202408629.
URL [https://advanced.onlinelibrary.wiley.com/doi/abs/10.1002/advs.202408629](https://advanced.onlinelibrary.wiley.com/doi/abs/10.1002/advs.202408629).
- Lin et al. [2020]
Ruoqian Lin, Rui Zhang, Chunyang Wang, Xiao-Qing Yang, and Huolin L Xin.
Temimagenet and atomsegnet deep learning training library and models for high-precision atom segmentation, localization, denoising, and super-resolution processing of atom-resolution scanning tem images.
*arXiv e-prints*, page arXiv: 2012.09093, 2020.
- Lin et al. [2021]
Ruoqian Lin, Rui Zhang, Chunyang Wang, Xiao-Qing Yang, and Huolin L Xin.
Temimagenet training library and atomsegnet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images.
*Scientific reports*, 11(1):5386, 2021.
ISSN 2045-2322.
- Wang et al. [2020]
Feng Wang, Trond R Henninen, Debora Keller, and Rolf Erni.
Noise2atom: unsupervised denoising for scanning transmission electron microscopy images.
*Applied Microscopy*, 50(1):23, 2020.
ISSN 2287-4445.
- Ziatdinov et al. [2017]
M. Ziatdinov, O. Dyck, A. Maksov, X. Li, X. Sang, K. Xiao, R. R. Unocic, R. Vasudevan, S. Jesse, and S. V. Kalinin.
Deep learning of atomically resolved scanning transmission electron microscopy images: Chemical identification and tracking local transformations.
*ACS Nano*, 11(12):12742–12752, 2017.
ISSN 1936-086X (Electronic) 1936-0851 (Linking).
doi: 10.1021/acsnano.7b07504.
URL [https://www.ncbi.nlm.nih.gov/pubmed/29215876](https://www.ncbi.nlm.nih.gov/pubmed/29215876).
Ziatdinov, Maxim Dyck, Ondrej Maksov, Artem Li, Xufan Sang, Xiahan Xiao, Kai Unocic, Raymond R Vasudevan, Rama Jesse, Stephen Kalinin, Sergei V eng Research Support, Non-U.S. Gov’t 2017/12/08 ACS Nano. 2017 Dec 26;11(12):12742-12752. doi: 10.1021/acsnano.7b07504. Epub 2017 Dec 14.
- Borshon et al. [2024]
Ishraque Zaman Borshon, Marco Ragone, Abhijit H Phakatkar, Lance Long, Reza Shahbazian-Yassar, Farzad Mashayek, and Vitaliy Yurkiv.
Predicting column heights and elemental composition in experimental transmission electron microscopy images of high-entropy oxides using deep learning.
*npj Computational Materials*, 10(1):275, 2024.
- Eliasson and Erni [2024]
Henrik Eliasson and Rolf Erni.
Localization and segmentation of atomic columns in supported nanoparticles for fast scanning transmission electron microscopy.
*npj Computational Materials*, 10(1):168, 2024.
- De Backer et al. [2022]
Annick De Backer, Sandra Van Aert, Christel Faes, Ece Arslan Irmak, Peter D Nellist, and Lewys Jones.
Experimental reconstructions of 3d atomic structures from electron microscopy images using a bayesian genetic algorithm.
*npj Computational Materials*, 8(1):216, 2022.
- Stoops et al. [2025]
Tom Stoops, Annick De Backer, Ivan Lobato, and Sandra Van Aert.
Obtaining 3d atomic reconstructions from electron microscopy images using a bayesian genetic algorithm: Possibilities, insights, and limitations.
*Microscopy and Microanalysis*, 31(1):ozae090, 2025.
- Huang et al. [2025]
Wenqiang Huang, Yucheng Jin, Zhemin Li, Lin Yao, Yun Chen, Zheng Luo, Shen Zhou, Jinguo Lin, Feng Liu, Zhifeng Gao, et al.
Auto-resolving the atomic structure at van der waals interfaces using a generative model.
*Nature Communications*, 16(1):2927, 2025.
- Li et al. [2021]
J. Li, M. Telychko, J. Yin, Y. Zhu, G. Li, S. Song, H. Yang, J. Li, J. Wu, J. Lu, and X. Wang.
Machine vision automated chiral molecule detection and classification in molecular imaging.
*J Am Chem Soc*, 143(27):10177–10188, 2021.
ISSN 1520-5126 (Electronic) 0002-7863 (Linking).
doi: 10.1021/jacs.1c03091.
URL [https://www.ncbi.nlm.nih.gov/pubmed/34227379](https://www.ncbi.nlm.nih.gov/pubmed/34227379).
Li, Jiali Telychko, Mykola Yin, Jun Zhu, Yixin Li, Guangwu Song, Shaotang Yang, Haitao Li, Jing Wu, Jishan Lu, Jiong Wang, Xiaonan eng Research Support, Non-U.S. Gov’t 2021/07/07 J Am Chem Soc. 2021 Jul 14;143(27):10177-10188. doi: 10.1021/jacs.1c03091. Epub 2021 Jul 6.
- Maksov et al. [2019]
Artem Maksov, Ondrej Dyck, Kai Wang, Kai Xiao, David B Geohegan, Bobby G Sumpter, Rama K Vasudevan, Stephen Jesse, Sergei V Kalinin, and Maxim Ziatdinov.
Deep learning analysis of defect and phase evolution during electron beam-induced transformations in ws2.
*npj Computational Materials*, 5(1):12, 2019.
ISSN 2057-3960.
- Gambini et al. [2023]
Laura Gambini, Tiarnan Mullarkey, Lewys Jones, and Stefano Sanvito.
Machine-learning approach for quantified resolvability enhancement of low-dose stem data.
*Machine Learning: Science and Technology*, 4(1), 2023.
ISSN 2632-2153.
doi: 10.1088/2632-2153/acbb52.
- Ihara et al. [2022]
Shiro Ihara, Hikaru Saito, Mizumo Yoshinaga, Lavakumar Avala, and Mitsuhiro Murayama.
Deep learning-based noise filtering toward millisecond order imaging by using scanning transmission electron microscopy.
*Scientific reports*, 12(1):13462, 2022.
ISSN 2045-2322.
- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Ziatdinov et al. [2022]
Maxim Ziatdinov, Ayana Ghosh, Chun Yin Wong, and Sergei V Kalinin.
Atomai framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy.
*Nature Machine Intelligence*, 4(12):1101–1112, 2022.
- Haastrup et al. [2018]
Sten Haastrup, Mikkel Strange, Mohnish Pandey, Thorsten Deilmann, Per S Schmidt, Nicki F Hinsche, Morten N Gjerding, Daniele Torelli, Peter M Larsen, Anders C Riis-Jensen, et al.
The computational 2d materials database: high-throughput modeling and discovery of atomically thin crystals.
*2D Materials*, 5(4):042002, 2018.
- Jain et al. [2013]
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al.
Commentary: The materials project: A materials genome approach to accelerating materials innovation.
*APL materials*, 1(1), 2013.
- Vaitkus et al. [2023]
Antanas Vaitkus, Andrius Merkys, Thomas Sander, Miguel Quirós, Paul A. Thiessen, Evan E. Bolton, and Saulius Gražulis.
A workflow for deriving chemical entities from crystallographic data and its application to the Crystallography Open Database.
*Journal of Cheminformatics*, 15(1), Dec 2023.
doi: 10.1186/s13321-023-00780-2.
URL [https://doi.org/10.1186/s13321-023-00780-2](https://doi.org/10.1186/s13321-023-00780-2).
- Madsen and Susi [2020]
Jacob Madsen and Toma Susi.
abtem: Ab initio transmission electron microscopy image simulation.
*Microscopy and Microanalysis*, 26(S2):448–450, 2020.
- Li et al. [2025]
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, et al.
Chemvlm: Exploring the power of multimodal large language models in chemistry area.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pages 415–423, 2025.
- Bran et al. [2023]
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller.
Chemcrow: Augmenting large-language models with chemistry tools, 2023.
URL [https://arxiv.org/abs/2304.05376](https://arxiv.org/abs/2304.05376).
- Tang et al. [2025a]
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein.
Chemagent: Self-updating library in large language models improves chemical reasoning, 2025a.
URL [https://arxiv.org/abs/2501.06590](https://arxiv.org/abs/2501.06590).
- Chen and Ong [2022]
Chi Chen and Shyue Ping Ong.
A universal graph deep learning interatomic potential for the periodic table.
*Nature Computational Science*, 2(11):718–728, 2022.
- Batatia et al. [2023]
Ilyes Batatia, Dávid Péter Kovács, Gregor N. C. Simm, Christoph Ortner, and Gábor Csányi.
Mace: Higher order equivariant message passing neural networks for fast and accurate force fields, 2023.
URL [https://arxiv.org/abs/2206.07697](https://arxiv.org/abs/2206.07697).
- Guo et al. [2023]
Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li.
Viewrefer: Grasp the multi-view knowledge for 3d visual grounding.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 15372–15383, 2023.
- Tang et al. [2024]
Yiwen Tang, Ray Zhang, Jiaming Liu, Zoey Guo, Bin Zhao, Zhigang Wang, Peng Gao, Hongsheng Li, Dong Wang, and Xuelong Li.
Any2point: Empowering any-modality large models for efficient 3d understanding.
In *European Conference on Computer Vision*, pages 456–473. Springer, 2024.
- Tang et al. [2025b]
Yiwen Tang, Zoey Guo, Zhuhao Wang, Ray Zhang, Qizhi Chen, Junli Liu, Delin Qu, Zhigang Wang, Dong Wang, Xuelong Li, et al.
Exploring the potential of encoder-free architectures in 3d lmms.
*arXiv preprint arXiv:2502.09620*, 2025b.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: - Beyond this specific system, what do the authors see as the key missing capabilities for a mature scientific agent? Is the bottleneck in tool diversity, model multimodality, or domain adaptation from natural to experimental images? (This is not a criticism, just an open question.)
- How dependent is the end-to-end reconstruction accuracy on the template retrieval step?
- In Sec. 4.3, are the branch decisions (‘repeat denoising / proceed to STEM2CIF / fallback to template matching’) implemented via hard programmatic thresholds on tool outputs, or are they purely prompt-driven?
- The workflow in Sec. 4.3 seems to follow a fairly linear sequence of predefined steps, guided by quality metrics and failure logs. It would be helpful if the authors could clarify to what extent the agent’s decisions rely on such programmed criteria versus genuine reasoning by the LLM, and what unique advantages the agent provides compared with a scripted automation pipeline.
soundness: 3
strengths: The paper tackles a meaningful problem: converting STEM images (and related textual context) into atomic structures and properties usually requires manual expert intervention. Using an agent framework with an LLM as the controller to call different domain tools is an interesting and valuable direction for the field.
1. The authors construct a benchmark covering unary, binary, and ternary materials to test the feasibility of this idea, and show that the proposed system achieves strong performance and outperforms several general-purpose vision-language models.
2. The tool pool is clearly defined and mostly self-contained: a self-trained denoising module (MOE-DIVAESR), a custom template matching algorithm, a custom STEM2CIF reconstruction module, and MatterSim for property prediction.
3. The paper shows that the LLM agent can perform multi-step reasoning, including rollback and re-calling tools when errors occur, suggesting genuine decision-making ability rather than fixed scripting.
summary: This paper proposes a framework that takes raw STEM images and textual information as inputs. It leverages an agent system to call four tools, enabling image preprocessing, atomic structure analysis, and property prediction. The authors also construct a comprehensive benchmark for this task, and the proposed method achieves SOTA performance compared with several baseline VLMs.
weaknesses: 1. The contribution is mainly twofold: (1) the benchmark, and (2) the agent framework. The benchmark construction, although based on simulated data, is fairly comprehensive and valuable. However, the agent framework feels more like an engineering implementation rather than a methodological innovation.
2. The paper’s description may give the impression that the agent itself directly interprets visual information from STEM images. In practice, however, the LLM component is text-only (DeepSeek V3), and the visual understanding is handled by separate image-processing modules. Moreover, the comparison with other VLM baselines does not clearly demonstrate that incorporating visual modalities provides any tangible benefit for this task. Clarifying this distinction—and discussing whether visual reasoning is necessary or helpful in this setting—would make the paper’s claims and design choices easier to understand.
3. To strengthen the benchmark, adding a human-expert baseline would make the comparison more meaningful, since the goal is partly to evaluate how close the agent can come to replacing manual human work. Moreover, if the dataset could include real STEM images in addition to simulated ones, it would significantly increase its impact and realism.

Minor issues:
1. Citation formatting errors, e.g., “in predicting atomic energies and forcesYang et al. (2024)” → “in predicting atomic energies and forces (Yang et al., 2024)”.
2. In Fig. 1, the text “Output: Optimized AttomicStructure” appears to contain a typo.
3. In Sec. 3.2, “To simulate realistic large-field STEM imaging conditions, We” — the “We” should be lowercase.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Necessity of the Agent: How much "intelligent" reasoning does the agent perform? How much would performance drop if AutoMat were implemented as a hard-coded pipeline (Denoise -> Match -> Reconstruct -> Predict) with a simple retry loop?

- Novel Structure Performance: How does AutoMat handle a truly de novo crystal structure that is not in its template database? Can the STEM2CIF module reconstruct a lattice from scratch without a template?

- Path to 3D: What are the primary blockers to extending this framework to 3D bulk materials? Is it fundamentally impossible to reconstruct a 3D structure from a single 2D STEM projection as used here?
soundness: 3
strengths: - Solves a Critical Problem: Provides the first end-to-end automated solution for the high-value bottleneck of converting experimental microscopy images into usable computational models.

- Specialized Toolset: The framework's power lies in its domain-specific tools which possess knowledge that general-purpose VLMs lack .

- Overwhelming SOTA Performance: AutoMat achieves order-of-magnitude improvements in both structural (RMSD) and energy (MAE) metrics over all baselines, including domain-specific and general models .

- Benchmark Contribution: The paper introduces STEM2Mat-Bench, a high-quality, tiered benchmark (Tier 1-3) that is essential for evaluating robustness and driving future research in this area .
summary: This paper introduces AutoMat, an LLM Agent-based framework to automate the conversion of raw STEM images into simulation-ready crystal structures (CIFs) for property prediction. The agent orchestrates a pool of specialized tools (including a Mixture-of-Experts denoiser, template retrieval, a symmetry-aware reconstruction module, and an ML potential for property prediction) to create an end-to-end "image -> structure -> property" pipeline. On a new benchmark, STEM2Mat-Bench, AutoMat outperforms all baselines by an order of magnitude.
weaknesses: - Agent Role Unclear: The workflow appears to be a mostly fixed, sequential pipeline (Denoise -> Match -> Reconstruct -> Predict). The "agent's" role seems more like a controller than a dynamic planner, making the "agentic" claim feel overstated.

- 2D Limitation: The entire framework and benchmark are restricted to 2D monolayer materials, which is a significant simplification of the general materials science problem.

- Template Dependency: The error analysis shows that 39.3% of failures are due to "Template retrieval failure", indicating a critical dependency on a comprehensive template database

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: Please see weaknesses.
soundness: 2
strengths: The problem from STEM images to downstream atom position and properties is very useful in connecting Experiments with simulation's.
summary: The paper introduces AutoMat, an agent-based system that claims to reconstruct atomic crystal structures directly from Scanning Transmission Electron Microscopy (STEM) images and predict material properties. The proposed workflow integrates, denoising, a template-matching retrieval module, STEM2CIF for lattice reconstruction, and MatterSim for energy prediction—managed by an LLM “agent” for orchestration. Additionally, the authors present STEM2Mat-Bench, a synthetic benchmark dataset of 450 annotated samples, used to evaluate reconstruction accuracy and formation-energy errors. Results show AutoMat outperforming other baselines methods in structural RMSD and energy MAE metrics.
weaknesses: a) Overstated novelty. Many ideas(agenticEM2CIF, agentic orchestration) are incremental extensions of existing frameworks such as SciLink, MicroscopyGPT and Mdcrow.

b) Benchmark validity. The comparison between agentic and non-agentic systems is not apples-to-apples. LLM-VLM baselines are not designed for such end-to-end tasks, making the reported “order of magnitude” improvement misleading. Also AtomAI is like python module with Neural nets like “unet” to segment images, how is it fair to compare it with agentic systems? Rather a study over different llm's orchestrating the same "tools" would be more useful. Basically answering questions like what models perform better or worse, do domain models beat bigger non domain models?

c) Lack of experimental(data collected on Microscope) validation, severely limits scientific impact.

 d) Unclear originality of released data. Since the dataset is built on existing repositories (C2DB, OpenCrystal), the “data release” claim is more of a repackaging effort.




references :
- AtomAI - https://github.com/pycroscopy/atomai, 
- SciLink - https://github.com/ziatdinovmax/SciLink
- Mdcrow - https://github.com/ur-whitelab/MDCrow
- MicroscopyGPT - notebook present at  : https://github.com/atomgptlab/atomgpt

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: * Considering the 450 test cases vs. 35/456/79 tier sizes (sum 570). Which is correct, and what numbers were used in Tables 1–2? If 570 is correct, where do the extra cases come from?
* How does AutoMat perform if the ground-truth structure is not in the retrieval library? Can you report structural metrics and energy MAE in that setting? 
* Can you add real-image evaluation on well-characterized monolayers with known CIFs? 
* Is it possible to provide module-wise ablations?
* How sensitive is retrieval to microscope parameter mismatch (defocus, aberrations, pixel scale)? Any simple calibration step before retrieval?
soundness: 2
strengths: * The image → structure → property loop is valuable and timely for AI-for-science, integrating denoising, retrieval, symmetry-aware reconstruction, and MLIP relaxation in a single agent is a meaningful system contribution.

* The presented benchmark shows that AutoMat outperforms domain toolkits and VLMs on structural accuracy
summary: The work presented proposes AutoMat, an LLM agentic pipeline that converts a single STEM image into a simulation‑ready crystal structure (CIF) and then predicts formation energy via an ML interatomic potential. The agent coordinates four tools: (1) MOE‑DIVAESR for pattern‑adaptive denoising, (2) template retrieval from a library of simulated STEM projections, (3) STEM2CIF for symmetry‑aware reconstruction, and (4) MatterSim for energy relaxation/prediction.
weaknesses: * Internal inconsistency with dataset and tiering count, its stated that there is  450-case test split, yet in section 3.3 its reported that tier sizes 35, 456, 79.
* The pipeline is motivated by experimental workflows, yet all quantitative results use abTEM simulations (fixed pixel size, dose ranges, aberration/noise models). No results are shown on real STEM micrographs
* AtomAI is a segmentation/localization library, not a CIF reconstructor; holding it to CIF-level metrics seems mismatched.
* The upper bound using the ground-truth CIF and MLIP shows an MAE of 48–57 meV/atom, substantial residual versus DFT even for perfect structures. Reported AutoMat MAEs (~320–330 meV/atom) thus reflect a mix of (i) structural error, (ii) MLIP bias vs. DFT, and (iii) sensitivity to starting geometries.
* The appendix reports agent/backbone swaps with ±3% variation and the retry mechanism, but there is no controlled ablation showing the marginal benefit of MOE-DIVAESR vs. a strong single denoiser, nor of symmetry constraints, nor of retrieval vs. no-retrieval. Given the four modules, their individual contributions should be quantified.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: + As I emphasized in the limitations, please provide experimental validation, at least the case studies, to demonstrate the framework’s effectiveness on real data.

+ It is unclear whether the structures retrieved from the database are used as ground truth in the validation process. If that is the case, then after Template Matching, the system essentially matches a noisy synthetic STEM image to the already recorded crystal structure, which is considered the “ground truth.” In that scenario, the STEM2CIF step seems unnecessary, as it only introduces additional noise and produces a structure that deviates from the retrieved “ground truth.”
This also raises another concern: if the retrieved structures are directly used as reference, then the composition correctness (C.C.) metric will trivially reach 100%, thus losing its discriminative meaning.
	
+ The validation using formation energy MAE does not provide additional insight, since MatterSim is relatively robust to small structural mismatches, making the results insensitive to moderate reconstruction errors.
 
+ The novelty of each component is rather limited. The current agent mainly relies on rule-based and prompt-driven tool scheduling, lacking any learnable policy optimization such as reinforcement learning-based tool selection, which would significantly strengthen the framework’s adaptability.
soundness: 3
strengths: The overall storyline is reasonable. The end-to-end structural analysis is decomposed into four key steps, and the agent is capable of orchestrating the tools to implement the entire workflow. The validation is multi-faceted and relatively comprehensive.
summary: This paper presents AutoMat, an agent-based framework that directly converts Scanning Transmission Electron Microscopy (STEM) images into Crystallographic Information Files (CIFs), aiming to bridge the gap toward end-to-end structural analysis. The main contribution lies in the agent framework, which consists of four components: MOE-DIVAESR, Template Matching, STEM2CIF, and MatterSim (a sota tool that is not part of the paper’s original contribution). The idea is reasonable; however, several important issues need to be addressed.
weaknesses: The current dataset is quite limited, containing only 2,143 structures, with 450 used for validation. This significantly affects the generalizability of the proposed workflow. Moreover, AI-assisted characterization is indeed a highly practical strategy; however, relying solely on simulation-based validation poses a high risk under a purely theoretical framework. Therefore, validation using experimental real-world data should be incorporated.


Decision: Reject