MatText: Do Language Models Need More than Text & Scale for Materials Modeling?

Authors: Nawaf Alampara, Santiago Miret, Kevin Maik Jablonka
Primary area: datasets and benchmarks
Keywords: Large language models, AI for science, Material representations, LLMS for Materials, Datasets and Benchmark

Abstract:
Effectively representing materials as text has the potential to leverage the vast advancements of large language models (LLMs) for discovering new materials. While LLMs have shown remarkable success in various domains, their application to materials science remains underexplored. A fundamental challenge is the lack of understanding of how to best utilize text-based representations for materials modeling. This challenge is further compounded by the absence of a comprehensive benchmark to rigorously evaluate the capabilities and limitations of these textual representations in capturing the complexity of material systems. To address this gap, we propose MatText, a suite of benchmarking tools and datasets designed to systematically evaluate the performance of language models in modeling materials. MatText encompasses nine distinct text-based representations for material systems, including several novel representations. Each representation incorporates unique inductive biases that capture relevant information and integrate prior physical knowledge about materials. Additionally, MatText provides essential tools for training and benchmarking the performance of language models in the context of materials science. These tools include standardized dataset splits for each representation across a range of dataset sizes, probes for evaluating sensitivity to geometric factors, and tools for seamlessly converting crystal structures into text. Using MatText, we conduct an extensive analysis of the capabilities of language models in modeling materials with different representations and dataset scales. Our findings reveal that current language models consistently struggle to capture the geometric information crucial for materials modeling across all representations. Instead, these models tend to leverage local information, which is emphasized in some of our novel representations. Our analysis underscores MatText's ability to reveal shortcomings of text-based methods for materials design.

Decision: reject
Reviewer scores: 3, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730093581831

thread_type:
official_review

<thread_object>
creator: reviewer_q6nl

<content>
summary:
This paper proposes a benchmark and toolkit for evaluating language models for modeling materials. By evaluating current LLMs on different text representations of materials from data sizes and inductive biases, the authors find problems in modeling materials with LLMs.

strengths:
- Several experiments from different aspects are provided such as different text representations, different data sizes, and LLaMa and BERT.
- Many recent methods are surveyed and summarized, and tasks, datasets, and toolkits are provided.

weaknesses:
- The comparison and results are meaningful for materials LLM community, but the technical novelty is weak.
- The details of tasks and results are not well explained, and only numerical scores are shown, so it is difficult for readers unfamiliar with the material's domain to follow the paper.
- The claims on the limitations of LLMs are too strong considering the grokking or emergent abilities of LLMs. 
  - Wei et al., Emergent Abilities of Large Language Models, TMLR, 2022

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723717810

thread_type:
official_review

<thread_object>
creator: reviewer_nb8q

<content>
summary:
The paper studies the potential of using LLMs for materials science, specifically in materials modeling. The authors introduce MatText, a benchmarking suite with nine text-based representations tailored for material systems, incorporating unique inductive biases to capture physical properties. The different analysis using MatText reveals that current LLMs struggle to effectively leverage geometric information, which is crucial in materials science, and instead rely heavily on local information. This suggests that text-based LLM approaches, unlike in natural language tasks, may not be inherently suitable for capturing the complexities of material properties.

strengths:
-  MatText includes different text-based representations and standardized datasets for evaluating LLMs on materials science tasks.
- The study identifies critical challenges, such as the inability of LLMs to leverage geometric information
- Provides tools for converting crystal structures into text and standardizing materials datasets.

weaknesses:
- While the authors observe that LLMs focus on local information rather than spatial relationships (which have been noticed already in similar studies), they don’t provide a mechanism or architectural change to address this.

- The authors introduce nine different text-based representations of materials, each with unique inductive biases. However, text representations may be an inherently limited approach for encoding complex 3D geometric and physical properties. Certain representations (e.g., CIF) may add complexity without necessarily improving model performance, as the LLMs struggle with numerical encoding of positional data. In my opinion evaluating LLM on handling such representations is not fair. 

- The experiments is conducted with different tokenizers for handling numerical information. However, standard tokenization methods for numbers can inpuct the model’s ability to process continuous data, such as atomic positions or lattice parameters. This point needs further clarification.

questions:
Does the Local-Env representation, by focusing primarily on local information, risk overlooking other crucial features such as long-range interactions or periodicity? If so, how can we address this limitation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730876049212

thread_type:
official_review

<thread_object>
creator: reviewer_cyjr

<content>
summary:
This paper introduces a benchmark for evaluating language models at predicting various properties of a material given its composition and structural information. To test the ability of LMs at utilizing different types of information (e.g., composition vs geometry), the benchmark provides several text representations of a material. Further, standardized data splits at different data scales are provided to test scaling behaviors.

Results suggest that LLMs, both existing ones and those pretrained on materials data from scratch, struggle at utilizing geometric information when making predictions. Further scaling the pretraining data seems to provide limited benefits.

strengths:
- Standardizing materials modeling with a benchmark providing precomputed representations and data splits is very useful. This data could  be used in a lot of follow up research.
- Evaluations point out an interesting limitation of LLMs at utilizing geometric information effectively.

weaknesses:
- One of the main claims of the paper is that scaling pretraining alone may not be sufficient for improving materials modeling performance. However, this is based only on increasing data size (which still shows improvement on 2 out of the 3 tasks). However, existing studies of scaling laws all note that all three -- model size, training compute and pretraining data size -- must be scaled in tandem for optimal performance. Hence, it is not clear if the observations on scaling would still hold if we increase model size and train for longer.

- The contributions of the paper are a bit thin -- the data itself is sourced from existing resources (NOMAD and MatBench). The main contribution are the different text representations evaluated on MatBench tasks. None of these lead to an improvement over existing methods.

- Only a small set of materials modeling tasks were evaluated -- so while the benchmark is somewhat comprehensive in the representations it uses, it is not comprehensive in the modeling abilities it tests.

questions:
- Why "datasets with 2D structures were excluded" from MatBench?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730892948243

thread_type:
official_review

<thread_object>
creator: reviewer_vmtz

<content>
summary:
In this paper the authors propose a benchmarking suite for evaluating LLMs in modeling materials, which includes evaluations as well as tools for converting materials to LLM-appropriate representations and calculating materials-specific metrics, such as sensitivity to crystal structure. The benchmark includes 9 distinct representations for encoding materials (some of which are newly proposed in the paper), with the shared task across representations of predicting material properties (e.g. shear modulus, bulk modulus, formation energy) given the input material representation. They use the benchmark to evaluate BERT pretraining and Llama (parameter efficient) finetuning (Llama-3-8B-Instruct, Llama-2-13b-chat-hf) on this task, comparing to the current state-of-the-art model according to the MatBench benchmark (variants of graph NNs). In their experiments they find that scaling pretraining (in a small BERT model) does not seem to improve performance on the benchmark, that locality (encoded into one of the input representations) is a useful inductive bias, and that current models do not seem to use geometric information.

strengths:
- **Important and interesting area of research.** Leveraging LLMs for reasoning in the space of material composition, structure, and properties is an exciting and potentially impactful area of research, and developing benchmarks and software libraries towards this end should encourage further developments in this space. The authors are spot on in terms of the key challenge in this space, which is how to effectively represent rich metadata about materials composition and structure.
- **Clearly written and easy to understand.** The paper was generally well written and easy to follow.

weaknesses:
- **Lack of engagement with prior work exploring methods for encoding structured/non-text information in language models (discussion or implementation).** The focus of the work / main findings are that LLMs are not very good at encoding/reasoning over numerical/structured values such as those important in representing and reasoning over the relationship between material composition and properties. But as far as I can tell, the paper only experiments with the most naive possible input and output representations (input: encoded representation of material composition/structure; output: numerical value of property). In this scenario, it is expected that the model will not do well in leveraging encoded structure because it is essentially forced to learn from scratch, because the representation of structure being used during finetuning/testing is likely too distinct from the representation of any similar knowledge during training, where the model was largely exposed to natural language text and code. It is known that LLMs, particularly the instruction-fine tuned models used in the paper, can struggle when provided with this type of non-natural-language formatted data, due to the mismatch between distributions observed during training, finetuning and evaluation. See e.g. [Fatemi et al. (2023)](https://arxiv.org/pdf/2310.04560), [Hegselmann et al. 2023](https://proceedings.mlr.press/v206/hegselmann23a.html), [Li et al. 2024](https://arxiv.org/abs/2403.07969), [Yin et al. 2023](https://arxiv.org/pdf/2306.01150), and other referenced work. While it would be interesting to validate this in this domain, insights from the current experimentation are limited because the experiments do not represent a realistic, good-faith setup for probing the models’ capabilities towards this end. Example methodologies that might do better would be tool use (to outsource quantitative reasoning), encoding the material structure information into code-type snippets, or even just encoding the input-output pairs as natural language questions (the Robocrys encoding gets close to this by at least encoding materials as natural language descriptions.) A single specific approach to this, RASP (Weiss et al. 2021), is discussed in the conclusions section, but not experimented with.
- **Limited contributions over prior work.** Relationship with MatBench should be clearly characterized very early in the paper, due to the similarity. MatBench is designed to benchmark any type of model on the same task, and the main contributions of this work over MatBench are: additional input representations (since this is the main challenge in using LLMs for this type of modeling) and software to help with evaluation and converting between representation formats. Because of the limited empirical results (as discussed above), this paper mostly provides a new benchmark, which is mostly composed of existing data/representations, but not many insights based on the benchmark, or other findings demonstrating its usefulness.
- **Limited analysis of results.** The appendix contains the beginning of some interesting analyses of the results. In particular, I would recommend moving discussion of the experimentation with tokenization to the main paper in future drafts, and elaborating. I’m very surprised that changes to the tokenization did not meaningfully improve performance. It would be interesting to see some examples of the tokenized inputs, under the default and modified settings. The encoded representations are so different than the text the default tokenizers were trained on, I would expect this to have a significant impact, and I wonder if just implementing Born & Manica (2023) was insufficient.
- **Discussion of limitations could be improved.** This work has many limitations not included. Breadth of models compared was limited. For example, benefits of scaling are only discussed in the context of “grokking” but it’s not clear why scaling more would not help further, since it was shown to lead to improvements in the existing results. The BERT models are incredibly small (4 layers and 8 attention heads), etc.

questions:
- If you experimented with different prompt formats and strategies for decoder-only LLMs, could you report what you tried and whether it had any impact on results?
- If not, why did you use instruction fine-tuned models, but did not fine-tune on or experiment with instruction-style input-output pairs?
- Why not experiment with larger models (potentially proprietary/closed)?
- Why did you choose the specific size of the BERT model?

Additional notes:
- Would be helpful to add some additional highlighting in Table 2 — e.g. which representation (or MatBench model) performs best for each task? Very hard to parse visually. Also, both RMSE and ROC are reported, would be nice if it was indicated for which rows higher is better and lower is better.
- Fix references to be parenthetical (e.g. \citep rather than \citet) on line 84, also various similar mistakes in the appendix.
- Typo on line 332
- It would be helpful if you added some concrete examples of input-output pairs to the main paper, perhaps including examples that do and do not encode locality, since a distinction there was one of your key findings. Including versions tokenized using the given tokenizers would be informative as well.
- Relevant concurrent work: https://arxiv.org/abs/2409.14572
</content>
</thread_object>

</thread>
<thread>
creation date:
1732655275891

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to reviewers

comment:
We thank all reviewers for their feedback. We would like to highlight the key technical and scientific contributions of our work, which we did not make clear enough in our original draft:

**Technical Contributions:**
1. First comprehensive benchmark of text-based materials modeling:
   - 405 Llama and 540 BERT finetune runs
   - 36 backbone training runs
   - Systematic evaluation across different scales (7B to 70B parameters)
   - Novel Python package with consistent implementations

2. Novel analysis methodology:
   - Hypothetical potential experiments that precisely ablate compositional and geometric contributions
   - Attention analysis showing how models process different types of information
   - Systematic evaluation of tokenization strategies

**Scientific Insights:**
1. *More information is not better:*
   - Geometric information is not effectively leveraged by current text-based approaches
   - Additional information (e.g., CIF) sometimes even degrades performance
   - This limitation persists across different:
     - Representations
     - Model architectures
     - Tokenization strategies
     - Scales (both data and model size)

2. *More scale is not better:*
   - Only marginal benefits from scaling, as shown in our comprehensive scaling experiments:

| Task | SLICES | CIF P₁ | Composition | CIF Symmetrized | Z-Matrix | Local-Env | SOTA |
|------|---------|---------|-------------|----------------|-----------|------------|------|
| **BERT Custom Number Tokenizer** ||||||||
| Shear Modulus (GPa) | 0.144±0.002 | 0.152±0.003 | 0.175±0.006 | 0.152±0.007 | 0.153±0.002 | 0.169±0.008 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.149±0.003 | 0.154±0.007 | 0.173±0.006 | 0.153±0.008 | 0.152±0.002 | 0.154±0.007 | 0.104±0.004 |
| Perovskites (eV) | 0.099±0.007 | 0.095±0.007 | 0.563±0.006 | 0.109±0.008 | 0.095±0.005 | 0.098±0.009 | 0.055±0.004 |
| Bandgap (eV) | 0.843±0.009 | 1.565±0.033 | 0.959±0.010 | 1.125±0.030 | 1.095±0.013 | 0.805±0.005 | 0.396±0.005 |
| Formation Energy (eV) | 0.349±0.017 | 0.681±0.009 | 0.407±0.008 | 0.358±0.020 | 0.826±0.017 | 0.255±0.007 | 0.048±0.06 |
| Is Metal (ROC AUC) | 0.931±0.002 | 0.920±0.005 | 0.929±0.002 | 0.932±0.002 | 0.875±0.004 | 0.934±0.004 | 0.952±0.007 |
| **BERT Regression Transformer Number Tokenizer** ||||||||
| Shear Modulus (GPa) | 0.150±0.006 | 0.136±0.004 | 0.174±0.005 | 0.153±0.002 | 0.154±0.005 | 0.146±0.002 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.144±0.003 | 0.143±0.004 | 0.173±0.007 | 0.156±0.006 | 0.154±0.006 | 0.143±0.003 | 0.104±0.004 |
| Perovskites (eV) | 0.101±0.005 | 0.098±0.007 | 0.561±0.007 | 0.108±0.010 | 0.097±0.002 | 0.096±0.005 | 0.055±0.004 |
| **Llama-3-8B-Instruct - Finetuned** ||||||||
| Shear Modulus (GPa) | 0.288±0.076 | 0.343±0.130 | 0.365±0.152 | 0.342±0.132 | 0.382±0.038 | 0.329±0.071 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.460±0.246 | 0.315±0.077 | 0.302±0.108 | 0.402±0.182 | 0.219±0.017 | 0.480±0.004 | 0.104±0.004 |
| Perovskites (eV) | 0.294±0.150 | 0.181±0.018 | 0.716±0.033 | 0.225±0.031 | 0.286±0.045 | 0.410±0.045 | 0.055±0.004 |
| **Llama-2-7b-chat-hf - Finetuned** ||||||||
| Shear Modulus (GPa) | 0.189±0.006 | 0.194±0.007 | 0.212±0.008 | 0.191±0.008 | 0.193±0.007 | 0.181±0.007 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.181±0.012 | 0.185±0.011 | 0.196±0.006 | 0.186±0.018 | 0.186±0.008 | 0.177±0.005 | 0.104±0.004 |
| Perovskites (eV) | 0.155±0.010 | 0.130±0.004 | 0.691±0.018 | 0.289±0.008 | 0.146±0.006 | 0.139±0.008 | 0.055±0.004 |
| **Llama-2-13b-chat-hf - Finetuned** ||||||||
| Shear Modulus (GPa) | 0.179±0.007 | 0.180±0.005 | 0.210±0.007 | 0.184±0.009 | 0.178±0.007 | 0.183±0.009 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.168±0.006 | 0.172±0.009 | 0.197±0.013 | 0.181±0.010 | 0.173±0.009 | 0.163±0.012 | 0.104±0.004 |
| Perovskites (eV) | 0.133±0.005 | 0.227±0.035 | 0.689±0.016 | 0.206±0.014 | 0.146±0.006 | 0.125±0.004 | 0.055±0.004 |
| **Llama-2-70b-chat-hf - Finetuned** ||||||||
| Shear Modulus (GPa) | 0.173±0.008 | 0.180±0.010 | 0.205±0.010 | 0.184±0.009 | 0.182±0.006 | 0.180±0.007 | 0.108±0.001 |
| Bulk Modulus (GPa) | 0.173±0.008 | 0.171±0.008 | 0.193±0.008 | 0.182±0.012 | 0.174±0.012 | 0.171±0.012 | 0.104±0.004 |
| Perovskites (eV) | 0.112±0.002 | 0.115±0.002 | 0.715±0.014 | 0.149±0.006 | 0.127±0.001 | 0.110±0.001 | 0.055±0.004 |

3. *Locality matters:*
   - Local-Env representation performs comparably to or better than representations with explicit positional information
   - Suggests current approaches may be most effective when problems can be solved using local contributions

These findings provide crucial guidance for the development of more effective approaches to materials modeling using language models and highlight fundamental limitations that need to be addressed.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates the application of LLMs to materials modeling, focusing on a variety of backbones, scales, and tokenization strategies. The study reveals that current LLMs encounter challenges in effectively utilizing geometric information, instead relying predominantly on local features.

Strengths:
The paper addresses a critical challenge in materials modeling using LLMs, particularly the limitations in leveraging geometric information.

Weaknesses:
The claims regarding encoding information and scaling are not sufficiently supported by the evidence presented. For instance, Reviewer VmTz notes that the discussion on encoding information does not adequately account for prior research. Additionally, Reviewer nB8q observes that the paper does not address mechanisms or architectural changes that could substantiate these claims. Furthermore, Reviewers VmTz and CyJR indicate that the arguments related to scaling lack robust logical or experimental backing. Finally, Reviewer q6NL suggests that the paper's interpretation of its limitations may be overly strong.

Conclusion:
The reviewers have unanimously rated the paper as either "Reject" or "Marginally below the acceptance threshold." Based on this feedback, which underscores the need for stronger evidence and a more comprehensive discussion of the claims, I give the rating of "Reject."