DPLM-2: A Multimodal Diffusion Protein Language Model
Authors: name: Xinyou Wang; name: Zaixiang Zheng; name: Fei Ye; name: Dongyu Xue; name: Shujian Huang; name: Quanquan Gu
Published: 2024-10-17 17:20:24

Abstract: Proteins are essential macromolecules defined by their amino acid sequences,
which determine their three-dimensional structures and, consequently, their
functions in all living organisms. Therefore, generative protein modeling
necessitates a multimodal approach to simultaneously model, understand, and
generate both sequences and structures. However, existing methods typically use
separate models for each modality, limiting their ability to capture the
intricate relationships between sequence and structure. This results in
suboptimal performance in tasks that requires joint understanding and
generation of both modalities. In this paper, we introduce DPLM-2, a multimodal
protein foundation model that extends discrete diffusion protein language model
(DPLM) to accommodate both sequences and structures. To enable structural
learning with the language model, 3D coordinates are converted to discrete
tokens using a lookup-free quantization-based tokenizer. By training on both
experimental and high-quality synthetic structures, DPLM-2 learns the joint
distribution of sequence and structure, as well as their marginals and
conditionals. We also implement an efficient warm-up strategy to exploit the
connection between large-scale evolutionary data and structural inductive
biases from pre-trained sequence-based protein language models. Empirical
evaluation shows that DPLM-2 can simultaneously generate highly compatible
amino acid sequences and their corresponding 3D structures eliminating the need
for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive
performance in various conditional generation tasks, including folding, inverse
folding, and scaffolding with multimodal motif inputs, as well as providing
structure-aware representations for predictive tasks.