Incorporating Domain Knowledge into Materials Tokenization
Authors: name: Yerim Oh, name: Jun-Hyung Park, name: Junho Kim, name: SungHo Kim, name: SangKeun Lee

While language models are increasingly utilized in materials science, typical
models rely on frequency-centric tokenization methods originally developed for
natural language processing. However, these methods frequently produce
excessive fragmentation and semantic loss, failing to maintain the structural
and semantic integrity of material concepts. To address this issue, we propose
MATTER, a novel tokenization approach that integrates material knowledge into
tokenization. Based on MatDetector trained on our materials knowledge base and
a re-ranking method prioritizing material concepts in token merging, MATTER
maintains the structural integrity of identified material concepts and prevents
fragmentation during tokenization, ensuring their semantic meaning remains
intact. The experimental results demonstrate that MATTER outperforms existing
tokenization methods, achieving an average performance gain of 4% and 2%
in the generation and classification tasks, respectively. These results
underscore the importance of domain knowledge for tokenization strategies in
scientific text processing. Our code is available at
https://github.com/yerimoh/MATTER