NatureLM: Deciphering the Language of Nature for Scientific Discovery
Authors: name: Yingce Xia, name: Peiran Jin, name: Shufang Xie, name: Liang He, name: Chuan Cao, name: Renqian Luo, name: Guoqing Liu, name: Yue Wang, name: Zequn Liu, name: Yuan-Jyue Chen, name: Zekun Guo, name: Yeqi Bai, name: Pan Deng, name: Yaosen Min, name: Ziheng Lu, name: Hongxia Hao, name: Han Yang, name: Jielan Li, name: Chang Liu, name: Jia Zhang, name: Jianwei Zhu, name: Kehan Wu, name: Wei Zhang, name: Kaiyuan Gao, name: Qizhi Pei, name: Qian Wang, name: Xixian Liu, name: Yanting Li, name: Houtian Zhu, name: Yeqing Lu, name: Mingqian Ma, name: Zun Wang, name: Tian Xie, name: Krzysztof Maziarz, name: Marwin Segler, name: Zhao Yang, name: Zilong Chen, name: Yu Shi, name: Shuxin Zheng, name: Lijun Wu, name: Chen Hu, name: Peggy Dai, name: Tie-Yan Liu, name: Haiguang Liu, name: Tao Qin

Foundation models have revolutionized natural language processing and
artificial intelligence, significantly enhancing how machines comprehend and
generate human languages. Inspired by the success of these foundation models,
researchers have developed foundation models for individual scientific domains,
including small molecules, materials, proteins, DNA, and RNA. However, these
models are typically trained in isolation, lacking the ability to integrate
across different scientific domains. Recognizing that entities within these
domains can all be represented as sequences, which together form the "language
of nature", we introduce Nature Language Model (briefly, NatureLM), a
sequence-based science foundation model designed for scientific discovery.
Pre-trained with data from multiple scientific domains, NatureLM offers a
unified, versatile model that enables various applications including: (i)
generating and optimizing small molecules, proteins, RNA, and materials using
text instructions; (ii) cross-domain generation/design, such as
protein-to-molecule and protein-to-RNA generation; and (iii) achieving
state-of-the-art performance in tasks like SMILES-to-IUPAC translation and
retrosynthesis on USPTO-50k. NatureLM offers a promising generalist approach
for various scientific tasks, including drug discovery (hit
generation/optimization, ADMET optimization, synthesis), novel material design,
and the development of therapeutic proteins or nucleotides. We have developed
NatureLM models in different sizes (1 billion, 8 billion, and 46.7 billion
parameters) and observed a clear improvement in performance as the model size
increases.