DPLM-2: A Multimodal Diffusion Protein Language Model
Authors: Xinyou Wang
Zaixiang Zheng
Fei YE
Dongyu Xue
Shujian Huang
Quanquan Gu
Venue: ICLR 2025 Poster (2025)
Keywords: protein foundation model
diffusion language model
multimodal language model
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities.
In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures.
To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer.
By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals.
We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models.
Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach.
Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs.

Published as a conference paper at ICLR 2025
DPLM-2: A MULTIMODAL DIFFUSION PROTEIN LAN- GUAGE MODEL
Xinyou Wang Zaixiang Zheng" '9 Fei Ye Dongyu Xue Shujian Huang c Quanquan Gu
School of Computer Science, Nanjing University ByteeDance Research Langzinyoo@@@mmainnjj...e
(zhengzaixiang, quanquan gu @bytedance.coom
PProeect Lead *Corresponding Author
Project Page:
github io/
ABSTRACT
Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their func- tions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships be- tween sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. this paper, we introduce DPLM-2, multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization- based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence- based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid se- quences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well providing structure-aware representations for predictive tasks.
INTRODUCTION
Proteins are macromolecules that execute crucial roles in every living organism. They are character- ized by their amino acid sequences and diree-iimensional structure, where the sequence determines the structure, which in turn governs the protein's function. Generative modeling for proteins has made significant strides in recent years. Among them, diffusion models (Ho et al., 2020; Song et al., 2020) cxhibit great success in protein structure-based generative modeling (Watson et al., 2023; Yim et al., 2023). Meanwhile, large-scale protein language models (Rives et al., 2019; Lin et al., 2022), trained on evolutionary-scale sequence database, have become one of the most important cornerstones in sequence- requence-based foundation models for protein sequence representation learning and generation. Remarkably, DPLM (Wang et al., 2024), discrete diffusion (Austin et al., 2021) based protein language models, has exhibited the state-of-the-art performance in both sequence generation and understanding, addressing a wide range of sequence-oriented applications.
(Watson al., 2023; Yim et al., 2024) and antibody design (Jin et al., 2021; Kong al., 2022: Zhou et al., 2024), require jointly determine both structure and sequence. However, the aforementioned approaches mostly employ generative models for one modality (either sequence structure) and resort separate models (Jumper et al., 2021; Dauparas et al., 2022) for the other. This highlights the pressing need for multimodal protein generative models that can integrate both sequence and structure, enabling understanding of protein behaviors and functions. This, therefore, raises the following question:
Many protein engineering applications, e.g.,
comprehensive
"This work done during Xinyou's internship ByteDance Research.

---

Published as a conference paper at ICLR 2025
A
Seructure Tokenization
B
Training and Sampling of Multimodal Diffusion Protein Language Model (DPLM-2)
seructure tokenizar
seructure
dicrete STAR tokens
de-tokentzer
(1GM))
warmup from sequenccibarre based
DPLM-2
Trandbomer Layer (Bdrestooa Malthaus Assrrice
Hun
GVP
evalutionary acale sequence data
residas indix
quinGeer (UFQ)
YATDD Swisfho (20K)
sorucoure tolenizer
amino aold tokenteer
Sipe lose
MKTVRQERLKYRA
structure
DPLM-2 www.ueu
APTI
A
JrIt
Applications
arthe-asd Bobuns
(1) uncondiional prosein generation (irrcttse
co-gineration)
(3)
struct-aware protein represerations downstreom predecive tasks classferrrrgrssio
(5) conditional protein generation (e most scaffudin with seruct soe mocd-no Inpus output)
SEruct de col.
DPLM-
DPLM3
struct de tok.
KOOOO
DOOOO
struct tokerizwr
OOODO
(3) folcing (seq cond structure generation)
(4)
imvdakding (struct-cond. sequince goneration)
DPLM.2
SEruct de-tok
PATVRQERLK
KOOOO
OODOO
OOOOO
DPLM-2
ROWA
DPLM-2
HOOOO
DOOOO
OAYOO
Figure 1: Overall illustration of DPLM-2. (A) Structure tokenization consists of GVP-based
encoder to yield invariant backbone geometric features, lookup- free quantizer (LFQ) to discretize encoded structural features into structure tokens within a codebook, and an IPA-based decoder as
de-tokenizer to convert structure tokens back to backbone atomic coordinates. (B) Multimodal
learning and generation of protein structure and sequence with DPLM-2. (C) Various applications of
DPLM-2 as a protein foundation model: (1) unconditional protein sequence-structurg mixed-modal
co-generation; (2) protein sequence-structure joint representation for predictive tasks; (3) structure
prediction; (4) bace-bbckbone sequence generation; (5) conditional protein generation with structure-
sequence mixed-modal input and output.
Can build multimodal protein foundation model to simultaneously model, understand, and generate both sequences and structures?
To pursue this goal, Multiflow (Campbell et al., 2024) is a recent effort for structure-sequence co-generation that incorporates sequences into structure-based generative models using multimodal flow matching. Despite its impressive structure generation capability, Multiflow exhibits suboptimal seneturalll-coopaatie sequences and consequently resorts instance-
performance in co-generating level knowledge distillation from ProteinMPNN (Dauparas et al., 2022). Furthermore, it completely falls short in protein folding for given sequences, showing Mulitilow's inadequacy in sequence understanding. We argue that this bottleneck arises from the absence (co=)evolutionary inductive bias derived from massive pre-training from sequence database, as prior studies have demonstrated that the coluttionarll-infoomee implicitly capture structural information enables direct structure prediction (Lin et al., 2022). As a consequence, the limitation in sequence understanding and generation renders Multiflow inadequate as multimodal protein generative foundation.
representations learned by pre-trained protein language models
Inspired by the connection between evolutionary knowledge and spatial interactions, we deem that sequence-basee based generative language models like DPLM, with their strong sequence generation and predictive abilities, hold great promise as foundation for multimodal learning for proteins. Despite its exciting potential, this approach presents two key challenges: (1) language models cannot directly handle continuous data like structure; and (2) language models heavily necessitate sufficient scale of data and compute resources while structure data is much smaller compared to sequence databases.
In this paper, we address the aforementioned questions by introducing DPLM-2, a multimodal protein foundation model that advances the oftte-of-theaan discrete diffusion- based protein language model (i.e., DPLM) to accommodate both sequences and structures. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure,
2

---

Published as a conference paper at ICLR 2025
as well as their marginals and conditionals. We present several key recipes to facilitate multimodal learning in DPLM-2: (1) the core difficulty lies in enabling the language model to learn structural information, which is challenging and remains elusive, for which we develop a lookup- zation (LFQ. Yu et al., 2023) structure tokenizer to convert 3D coordinates to discrete tokens and vice versa (Fig. A, $3.3); (2) we implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence- DPLM (Fig. IB, $3.2); and (3) we also address the exposure bias problem in discrete diffusion for sequence learning (Ranzato et al., 2016; Bengio et al., 2015) by a self-mixup training strategy that leads to enhanced generation quality and diversity.
quanti-
We highlight our main contributions and findings as follows:
(i) We present DPLM-2, multimodal protein generative language model that aims to simulta- neously model, understand and generate protein structure and sequence. We show that can be fairly efficient and effective to obtain mulitmodal protein model with moderate amount of high-quality data, a decent structure tokenizer and publicly-accessile sequence-only pre-trained language models.
(ii) As a mulitmodal generative model, DPLM-2 enables unconditional protein co-generation of both structure and sequence, which demonstrates good tructtre-sequent consis- tency (Fig. IC(I)) Our empirical evaluation shows that DPLM-2 attains competitive co-generation performance compared to structure- grusture-based generative approaches, while the proteins generated by DPLM-2 have a better alignment with the characteristics of natural proteins in secondary structure statisties ($4.1).
(iii) In addition, DPLM- 2 supports various conditional generation tasks by its multimodal nature, ranging from seeueencecconditioned folding (Fig. IC3), $4.2), prructure-conditionn inverse-folding (Fig. C(4), $4.3), to more successful motif-scaffolding given multimodal motif conditioning (Fig. IC(5), $4.4).
(iv) Last but not least, we demonstrate that the structure- aware protein representation learned by DPLM-2 brings additional benefit for range of protein predictive tasks (Fig. IC(2), $4.5). Concurrent work. During the development of DPLM-2, we became aware of the recently pro- posed multimodal generative protein language model, ESM3 (Hayes et al., 2024), which also jointly models tokenized structure and sequence using a generative masked language model. While both models aim for similar goals, DPLM-2 differs from ESM3 in several key aspects: (1) Multimodal protein generation: DPLM-2 treats structure and sequence modalities equally by design and em- phasizes the simultaneous co-generation of compatible protein sequence and structure, whereas ESM3 is a sequence- requence-first model (other modalities are subject to dropout during training) and gener- ates in cascaded modality- dedaallyy-y-moodee by-modality manner. (2) Data and compute efficiency: ESM3 seeks to perform mulimodal pre-training from scratch using a huge amount of syntheric data, with modal size ranging from 1.4B to 98B. With strict license and absence of training infrastructure, this pro- hibits community from replicating for customized purposes. In contrast, DPLM-2 leverages much smaller datasets (PDB + SwissProt) and builds on open-source, pre-trained sequence-base based DPLM (150M/650//3)), which leverages DPLM's learned evolutionary knowledge and inherits strong sequence understanding and generation capabilities. We are also committed to open- our models, training and inference code to democratize multimodal generative protein LMs to benefit the community. Overall, we believe DPLM-2 provides unique contributions to the community.
2
PRELIMINARIES
Table Generative tasks
2.1
GENERATIVE MODELING FOR PROTEIN
w.r.f. structure & sequence.
The aim of generative protein modeling is to estimate the underly- ing distribution prot (prott of the protein data of our interest by learning a probabilistic model po (prot). Here prot = (r1, (r1,72, .,...,,... (T.) denotes a protein with L residues, where each residue r; = (8,,
task
objective
folding
(x(sss) Pu(s
inv folding seq. gen. struct. gen.
Ps(S)
is represented by two major modalities, i.e., S; E {0, 1] ||| is cat- egorical variable for its amino acid type in = {I...., 20), and I; IR X3 is the real-value Cartesian coordinates of its residue
(x())
seq-struct co-gen.
Po x)
atoms (we only consider backbone atoms herein, i..,, [N, C., C, with Natoms = 4). Namely,
po (prot) (0 82, 8L, 31, ,1,02,- T2, ... .I) Po (s.
As a result, most of protein tasks can be viewed as specifying their input conditioning and output between these two modalities (Tab. 1), including (1) sequence-conditione structure prediction (fold-
3

---

Published as a conference paper at ICLR 2025
ing, Jumper et al., 2021; Lin et al., 2022; Huguet et al., 2024), (2) structure- sequence generation (inverse folding or fixed- backbone design, Dauparas et al., 2022; Hsu et al.. 2022; Zheng al., 2023b), (3) sequence learning
generation (Rives et al., 2019; Nijkamp et al., 2022; Alamdari et al., 2023; Wang et al., 2024), (4) structure generation (Yim et al., 2023; Watson et al., 2023; Ingraham et al., 2023), and (5) sequence-structure co-generation (Jin et al., 2021; Shi al., 2022; Campbell et al., 2024). These further enable various conditional applications by allowing single or mixed-modal conditioning for partial generation, e.g., motif-scaffolding and antibody design.
2.2 DIFFUSION PROTEIN LANGUAGE MODEL (DPLM)
Language models (LMs), typically parameterized by Transformers (Vaswani et al., 2017) have become the de facto choice dominating different domains with scalable and performing expressive- ness (OpenAI, 2023). Among them, protein LMs have been serving as one of the Al foundation for protein sequence learning (Rives et al., 2019; Lin et al., 2022) and generation (Nijkamp et al., 2022; Alamdari et al., 2023).
Diffusion protein language model (DPLM, Wang et al., 2024), in particular, shows excelling per- formance in both generation and representation learning of protein sequences. DPLM is grounded in absorbing discrete diffusion framework (Austin et al., 2021; Zheng et al., 2023a), which is char- acterized by a forward and backward Markov process. Let Cat Cat(x: p) be a categorical distribution on protein sequence y parameterized by a vector on (I>) 1) -dimensionnl probability simplex. The forward process discrete diffusion defines a Markov process governed by the transition kernel ;,xx(t 1) + (1 Be qmoise) that gradually perturb the data x()) g(x(")
((x()) xx(=1))== = Cat (x(*); (x,x((t1)))))
into a stationary distribution x!") ~ Quoise- For absorbing diffusion, Qnoise is the point mass with all of the probability on the absorbing (mask) state. The learned backward process po (x (x() (x)
reversely denoises the x()) towards the data distribution x("), which is typically optimized by the variational bound of the log- ligeeeidddoodd (Ho et al., 2020):
(x (0:1))
Bx(xe [log po (x [0] E ((((mm))
log
g(x 1:7) x()))
= =B log (x (x(1))))))) + Σt
KL (g((xx1111 (xe
x
(0) ) (xxx)) +oosst,
where J: is the learning objective. The learning objective of discrete diffusion can be further simplified into reweighted cross-entropies (Zheng et al., 2023a), resembling masked language modeling at arbitrary noise levels:
J. = (x(())
KL
(0) x x (lo (x (9)
= E (x)(0)
} Eisig
b() log po (1: (0)
1 (!))
where A is a weighting coefficient induced from the specific noising schedule and b.(t)
1 10 +0 For inference, DPLM is able to generate amino acid sequences by the reverse iterative
denoising process of discrete diffusion (Hoogeboom et al., 2021; Austin al., 2021) from the following distribution,
(x(x = Eggor (x(()))(((()) (po )oo (x()) x(()).
Specifically, at time 6, it first generates (0) from po(- then a less noisy ((-11) is sampled by q(- q(-1x("),,xx (9) ())) Within absorbing diffusion, the generation process can be viewed as an
iterative mask-predict approach. For sequence representation for predictive tasks, can be obtained by simply letting DPLM take the sequence input.
DPLM-2: MULTIMODAL DIFFUSION PROTEIN LANGUAGE MODEL
3.1
OVERVIEW
Fig. illustrates DPLM-2's overall architecture. DPLM-2 is built on the state-of-ther sequence- based generative protein LMM, i.e., DPLM (Wang et al., 2024), using discrete diffusion probabilistic framework to concurrently model both protein sequences and their corresponding structures. To facilitate structure learning in language models, we introduce a token-based representation for protein structure via a tokenizer that converts x E L x3 the 3D coordinates of the protein backbone 1x into a discrete structure token sequence, denoted as z = (21,22,- 21.) E {0, 1) [2|, where each token 2, represents local structural element of the i-th residue. Given tokenized structure,
4

---

Published as a conference paper at ICLR 2025
DPLM-2 processes mulitmodal input by concatenating the structure token sequence with the corresponding amino acid sequence s for the same protein. Notably, there exists a position-by- position correspondence between and s, where 2; and s; refer to the two modalities of the i-th residue, respectively. To reinforce this correspondence, we assign identical position encodings to both 2s and 8;, thereby ensuring that structural and sequence information is aligned at the residue level.
To train DPLM-2, we leverage high-quality dataset comprising 20K clustered experimental structures from the Protein Data Bank (PDB) (Berman et al., 2000) and 200K predicted structures from the AFDB SwissProt split (Varadi et al., 2022), with length < 512. During training, DPLM-2 is tasked with denoising the imput sequence across spectrum of noise levels, ranging from fully noisy to completely clean. The multimodal training objective of DPLM-2 is derived from Eq. (1) as,
J. Ex(xet ,500)) a e- (okeemio((((
A'' Eisgr,
log pa (o (0) (0) (1)) s())
(2)
where log (25, s;l.) = (z:l)( + Po (8:1.)) by assuming conditional independence (see discussion in SG). By learning pa (2(((11), s((=))/2(()) s(!)), the model enables the simultaneous generation of highly correlated protein structures and sequences. This eliminates the need for cascaded generation, allowing us to derive both the protein's structure and sequence in a single step.
To further enhance DPLM-22 2's ability to differentiate between structure and sequence, noising level for each modality is subjected to distinct scheduler, denoted as ly and (s, respectively. This facilitates more comprehensive understanding of the relationships between protein sequences and their corresponding structures. This design also allows us to explore arbitrary combinations of (tz. ts). thus providing flexible sampling options, including sampling from the marginals of each modality and conditionals between them for various applications (Fig. IC). For conditional sampling (e.g.. folding and inverse- folding), set the noise scheduler of the conditioned modality to O, which means no noise in the conditioned modality. Please refer to $A.2 for more details.
Furthermore, we also identify the exposure bias issue in discrete diffusion for sequence learning (Ran- zato et al., 2016; Bengio et al., 2015), and mitigate this by proposing a self-mixup strategy inspired by scheduled sampling, which improves both generation quality and diversity (see $A.5).
3.2 EFFICIENT WARM-UP FROM PRE-TRAINED SEQUENCE-BASBI DPLM
Protein sequences encode critical evolutionary information, reflecting co-evolutionary processes where residue pairs mutate together and often interact in 3D space, offering insights for predicting protein folding (Melnyk al., 2022b). Lin et al. (2022) further showed that protein language models trained on large-scale evolutionary data implicitly capture this information, which can facilitate structure prediction. Motivated by the link between evolutionary knowledge and structural interactions, we propose to built DPLM-2 with an efficient warmup from pre-trained sequence- based DPLM, to make the most of established evolutionary information for protein structure modeling, Since our structure dataset is significantly smaller than UniRef50 sequence database (200K 45M), enabling efficient fine-tuning of the pre-trained model. we want to keep the sequence knowledge intact and reduce the risk of catastrophic forgetting, we apply LoRA (Hu et al., 2021) to limit too much deviation to the original parameters. This approach not only lowers training costs compared to starting from scratch but also effectively transfers valuable evolutionary information.
3.3 LEARNING STRUCTURE TOKENIZATION
The core difficulty of achieving mulimodal protein LM lies in enabling the language model to learn structural information, which is challenging and remains elusive, Tokenizing continuous data modalities into discrete representations (Van Den Oord et al., 2017) has gained attraction across domains like image synthesis due to its ability to capture compact, meaningful information, enabling effective compression and efficient generation, especially with sequence based models like Transformers. Recent efforts have applied this approach to protein structure coordinates (Van Kempen et al., 2024; Liu et al., 2023; Gao et al., 2024; Lu et al., 2024). This allows language models to better learn the composition of local structural elements. However, how to learn effective structure tokenizer remains an active research question.
A meconenatiin scr toleeberrs
B- strucct tokona dary street
Structure tokenization und er a typical VQ- VAE (Van Den Oord et al., 2017) framework can be summarized as follows:
vokeriarr
size
ldlt.ca
commo 2023
soore msd
VQ-VAE-IL IK
1024
0.76
0.71
0.80
6.84
1
0177
0.86
4.15
LFQ
0.79
0:38
3,62
encoder
quantizer
decoder
0.56
0533
LPQ-15
16334
0.87
0.44
2.32
where (1) a structure encoder encodes back- Figure 2: Reconstruction and secondary structure
bone 3D coordinates x E L x3 into in-
correspondence of structure tokenizers.
5

---

Published as a conference paper at ICLR 2025
variant features R I.xdumm, (2) quantizer converts into of L discrete tokens where 2 E {0, (0.......12 | given a sinite-size codebook 2; and (3) a structure decoder reconstructs 3D coordinates x from the discrete tokens.
We utilize GVP-based (Jing al., 2020) structure encoder from pre-trained GVP-Transformer (Hsu et al., 2022) and a IPA-based (Jumper et al., 2021) structure decoder. In terms of quantizer, our preliminary experiment showed that conventional VQ- pretty much struggles in training. To mitigate this, we instead adopts Lookup-Free Quantizer (LFQ) from the currently best visual tok- enizer (Yu et al., 2023) to protein structure tokenization. Specifically, the latent space of LFQ is
decomposed as the Cartesian product of dinlee-dimensionll binary variables, as C = x <==
log, Ck.
where C2 {-1,1}. Given the encoded feature encoder(x) E f [2], each dimension (indexed by k) of the quantized representation quant ( is obtained from:
quant(e,)[M) C., = sign(e, [k]) - ( [k] & 0} + [e,[k]] > 0}.
As such, with LFQ, the token indices for z [1 22. 24, 2n is given by:
2 (ande((uant(e)))
> |2| 2 [e) 0), V4,
The LFQ-based structure tokenizer is trained on the same structure dataset as mentioned before, using combination of reconstruction, commitment, and entropy regularization losses, similar standard VQ-VAE. Here FAPE loss (Jumper et al., 2021) is used as the primary reconstruction loss. (see $B.1 for more details.)
Evaluation. As shown in Fig. 2A, LFQ significantly outperforms VQ- -QEE regarding reconstruction accuracy while training of LFQ is much faster than VQ- "Q-VAE (2 vs. 15 days on 8 A100s). Increasing codebook size leads to improved reconstruction while codebook size of 8192 achieves the best caapression-rcconstuuciio trade-off. Meanwhile in Fig. 2B, we observe strong correlation between structure tokens and secondary structures. For instance, a lot of structure tokens concentrated at the alpha helix and beta sheet vertices, while some tokens lic between regions. This suggests that structure tokens the fine-grained structural elements in backhone local environment.
4 EXPERIMENTS
In this section, we evaluate DPLM-2 on various generative and understanding scenarios, including unconditional protein generation (structure, sequence, and snvcture-sequent co-generation, $4. 1), and variety of conditional tasks, such as folding (64.2), inverse folding ($4.3) and motif-scaffolding ($4.4), and a series of protein predictive tasks ($4.5).
4.1 UNCONDITIONAL PROTEIN GENERATION
The goal of unconditional protein generation is to produce both the 3D structure and amino acid sequence. Typically, this is done using a cascaded approach: either generating the structure first and then use another model to predict the sequence, or vice versa. Here, we focus on generating structure and sequence simultaneously, We evaluate DPLM-2 on both cascaded and simultaneous generation across three tasks: unconditional structure generation, unconditional sequence generation, and structure -eeueence co- re-generatin.
Following Multiflow (Campbell et al., 2024), we evaluate the generated proteins terms of quality, novelty and diversity. Designability is measured through self-consistency evaluation and foldabil- ity (Yim et al., 2023; Watson et al., 2023; Wu et al., 2022a). Self-consistency evaluation assessed by folding the generated sequence with ESMFold (Lin et al., 2022), then using Sc-TMs core and sc-RMSD with the co- co-generated structure to evaluate similarity. Foldability is evaluated via ESMFold, with PLEDT 70 considered plausible. Novelty is assessed by comparing generated structures to known ones in PDB using T'MScore (pdb-TM), with lower values indicating greater novelty. Diversity measured by calculating pairwise TMscore (inner-TW) TM), where lower scores indicate more dissimilarity. The number of clusters identified by FoldSeck (van Kempen et al., 2023) also quantifies diversity, normalized by the total number of structures.
4.1.1 DPLM-2 ENABLES HIGH--UALITY, DIVERSE AND NOVEL PROTEIN SEQUENCE AND STRUCTURE GENERATION
Tab. and Fig. 3 present the results of DPLM-2 for unconditional protein generation. We highlight our key findings in the following aspects:
generate diverse and highly-plausite protein with simultaneous structure- sequence co-generation. We sampled 100 proteins for each length in 100, 200, 300, 400, and 500. The co-generation can be performed in simultaneous generation (co-generation) and cascaded
(1) DPLM-2

---

Published as a conference paper at ICLR 2025
designablliyy.srruuure---o compatibility
B
structure diversity
C
sequence foldability
scTMScore.
Reluste.
PLDDT.
E0-
50
20
DRUM
DPUM2
70
length
structure novelty
comparison wrt model size
long protein generation
pcb-TM.
62
0.1
15CM B00M 33
153M 60UN 30
s050 -50 argh
5950
550
a300 +350
lengt1
model to>
madel C.0
kough
kengi
H case study of structure-sequence co-generated samples
I- showcase of designing symimetric oligomers
Length 100 1C77-8.97
Longsh: 200 scRMED:
Longth: 300 sckMED:
Lengsh: 400 scRMSDD00.8
Leegth 600 <CTM-0
Length 700
3CTM-0.9 0.95
SERMS::
Figure 3: Evaluation of DPLM-2 on unconditional structure-sequencce co-generation. Here for designability of co-generated proteins, we use ESMFold to obtain refolded structure of DPLM-2- generated sequence and measure the structural similarity between IPLM-2-generatee structure and the refolded structure, which aims to measure the compatibility of the co-generated structure and sequence pairs.
first generating the structure then the sequence conditioned on generated structure (struct - seq), and the reverse way (seq -> struct), without the need of other folding inverse folding models. Fig. 3A/B demonstrates that DPLM-2 can sample sequence and structures with high designability across various lengths, with most SC-TM values exceeding 0.9, with diverse structure clusters. Fig. 3D shows that the novelty of sampled proteins, measured by pdb-TM, generally increases with longer protein lengths. In addition, DPLM-2 can generate with both modalities simultaneously or a modality-by-moddae highest SCTM, suggesting that co-modeling indeed benefits protein generation.
workflow:
As shown in Tab. 2, the co-generation performance exhibit
(2) DPLM-2 can attains competitive performance with strong baselines on co-generation, as well as backbone-only and sequence-only generation, respectively. As shown in Tab, 2, DPLM-2 achieves the strong 8 SC-TM compared to strong baselines, approaching the quality of native structures from PDB. We notice that ESM3-Open (Hayes et al., 2024), which runs in pegueenc--ten--sruccee order, fails short of unconditional generation. Compared to MultiFlow (Campbell et al., 2024), DPLM-2 achieves comparable co-generation quality. Notably, also reported in Campbell et al. (2024), Multiflow falls short of sequence generation when directly trained from structures with native sequences, resulting in greatly degraded co- co-generation performance without data distillation from external inverse folding models (ProteinMPNN) For reference, we also provide the result of Multiflow retrained using our training data, where co-generation performance remains unsatisfying and lags behind DPLM-2, which suggests that DPLM-2 has advantages of directly and effectively learning from complex structure-sequence joint distribution. Moreover, DPLM-2 can also only produce single modality if needed, where it matches the best competitive models in these settings respectively. These results demonstrate DPLM-2's i effectiveness as a mulitmodal generative model.
(3) DPLM-2 generates longer proteins beyond training data. As DPLM-2 is trained with a 512 length cutoff, we are curious about its length extrapolation, and evaluate sampled proteins at lengths of |600, 700, 800, 900, 1000]. As shown in Fig. 3F, notably, for proteins exceeding the maximum training length of 512, the PLDDT scores of sequences sampled by DPLM-2 are close to those of DPLM. This suggests that DPLM-2 largely retains its original sequence generation capability inherited from sequence pre-tnnining DPLM, leading to its capability of length extrapolation.
(4) Case study. Fig. 3H shows some generated samples of DPLM-2 up to 700 residues, while in Fig. 3I we showcase that we can manipulate DPLM-2 design symmetric oligomers by forcing to duplicate the predicted tokens with repetitive structure and sequence patterns.
(5) Abaltion study on the training strategy. We investigate the effects of warmup from the sequence- based pre-trained DPLM and data augmentation with high-quality AlphaFold predicted structures on DPLM-2. The sequence pre-training significantly improve both designability and diversity, while
UpphaFold-predicted
7

---

Published as a conference paper at ICLR 2025
Table 2: Benchmarking comparison of unconditional protein generation, in terms of structure- sequence co-generation, backbone-only generation, and sequence-only generation. For each method, we generate 100 samples for lengths [100, 200, 300, 400, 500]. denotes Multiflow variants
retrained by us using different dataset native PDB data without ProteinMPNN distillation and the same training data as DPIMM2 (i.e., PDB+SwissProt), respectively.
Structure-seqeence Consistency
Novelty
Diversity
scTM
scRMSD (
PLDDT avg.
avg. inser-M MaxCluster (f)
Structare-sequence co-generation.
Native PDB protein
0.904 2: 0.129
4.623
5.688
0.262 0.025
0.776
ESM Open seq struct)
24.180
0.220
MuldFlow w/ distilation tofficial ckpt)
0,930 1: 0.098
3.208
4.741
79.447
0.704 0.000
0.356 1: 0.032
0.500
Plow wño distillation
0.750
0.163
9.306
8,499
61.519
0.350 4: 0.038
0.490
"Multilov (retrained
training data)
0.871
0.934
6.258
62.624
0.331 0.052
D-LM-2 1650M, seq struet)
0.907 0.117
6,337
9.403
82,246
0.653
0.195
0.280 + 0.038
0.651
DPLM-2 (650M, struct
0.096
6.735
81,910
0.637 0.1595
0.308
DPL.-2 (650M, co-generation)
0.925 0.085
3,899
3,723
82.686
0.640 0.204
0.287 0.030
0 0.545
Unconditional backbone generation, (sequence predicted
ProtcinMPNN)
Natve PDB struct. (seq. from PMPNN)
0.569 0.000
0.864
0.000
0.262 +
0.782
Fia
0.818 1: 0.000
3,919 1: 0.000
0.668 0.000
0.444 :: 0.064
0.252
FoldFlow
0.540 0.000
0.000
0.286
REDiITusion
0.914 0.000
1.969 0.000
0.657 0.000
0.352 1: 0.038
0.598
DPLM-2 (650M)
0.945 0.082
4,451 5,261
0.6.7 0.195
0:297 + 0.049
0.575
Inconditional sequence generation. (structures predicted by ESMFold)
EvoDil
35,846
0.432 0.106
0.265 1: 0.025
0.990
DPLM (650M)
83,252
0.541 0.187
0.242 0.041
0.735
DPLM-2 (650M)
82,246
0.662 0.199
0,280 0.042
0.700
stats of secondary structure
of secondary
designability
Pegergan
to,mes
60 000102C330 wwt.00066655000
03 D2 43 0A 05 0.6
63 02
03 04 D5 26
C cennondttioalllyggeeeeete proteins from different models
helk milo
573051950
1000 rods
MultiFlow
PDB
DPLM2
Length: 70
Longth: 100
Longth: 200
Longch: 300
Longth: 400
Length: 500
Figure 4: Analysis regarding secondary structure of generated proteins. (A) Statistics of averaged proportions of secondary structures for proteins from different methods and PDB; (B) Secondary structure designability: (C) Samples of Multiflow, PDB and DPLM-2, well their secondary structure distributions.
data augmentation further enhance the designability, especially for long proteins. For more details of ablation study, please refer to A.66.
4.1.2 DPLM-2 GENERATES PROTEINS THAT RESEMBLES NATURAL PROTEINS
To further analyze the properties of different model, examine their secondary structure distribution against natural proteins from PDB.
Proteins sampled by DPLM-2 have secondary structures most similar to natural proteins. As seen in Fig. 4A, structure-bascd models like RFDiffusion and MultiFlow generate proteins with more helices and fewer sheets and loops than natural proteins in PDB. Protein language models like ESM3 and DPLM-2 show no strong bias towards alpha helices, but ESM3 tends to generate more loops. Among the methods, DPLM-2 produces the most natural-like secondary structure proportions, closely matching PDB proteins. In Fig. 4C, proteins generated by MultiFlow contain many helices and become more globular length increases, exhibiting idealized secondary structures. In contrast,
8

---

Published as a conference paper at ICLR 2025
proteins generated from DPLM-2 resembles natural ones have more balanced structures, with fewer helices and more beta sheets and loops. On the other hands, simplex plots in Fig. 4C shows that while MultiFlow's proteins clustered in helix-rich regions, DPLM-2's proteins span wider area similar to natural proteins, while it rarely samples proteins composed mostly of sheets and loops, which do occur in nature. Additionally, Fig. 4B shows that the loop ratio has significant impact on designability, where a higher proportion of loops will increase SCRMSD, as loops are highly flexible. Thus, proteins with long loops, which DPLM-2 often generates, tend have relatively high SCRMSD, aligning with the results in Tab. 2. 4.2 FORWARD FOLDING (SEQUENCE- DOQUENCE-CONDITIONEE STRUCTURE PREDICTION)
The goal of folding is to predict the 3D structure for the given amino acid sequence (Jumper et al., 2021). As a mulitmodal generative model, DPLM-2 spontancously enables protein structure prediction task (see Fig. 1C-3) given sequence as conditioning. We assess DPLM-2 on CAMEO 2022 and a PDB data split used by Multiflow (Campbell et al., 2024). We utilize RMSD and TMscore between predicted and ground truth structure for evaluation, while DPLM-2 adopts argmax decoding for 100 sampling iterations. Tab. 3 indicates that DPLM-2 can perform sufficiently good folding in a zero-shot man- ner. Performance can be improved after further supervised fine-tuning (SFT) using folding ob- jective (max, log po (z(s)) Overall, DPLM-2 can outperform or on par with the strong base- lines, while achieving close performance with ESMFold. Plus, we observe that DPLM-2 with larger model scales can attain better results than smaller ones. We suggest that DPLM-2 bene- fits from the evolutionary information inherited from DPLM pre-trained on the vast number of protein sequences, which can be transferred and leveraged into structure modeling.
Table 3:
Structure prediction performance com- parison between DPLM-2 and different baseline approaches on CAMEO 2022 datasets. t: PVQD results quoted from Liu et al. (2023).
CAMEO 2022
PDB date split
Models
RMSD
Iescore
RMSU
TAscore
ESMFold
3.99/2.03
0.85/0.93
2
0.930.977
TPVQD
4.08/1 95
0,81/0.88
MultiFlow
17.84/17.96
0.50V0.46
15.64/16.08
0.53/0.49
ESM3
6.33/2.98
0.85/0.92
4,94/2.28
0.87/0.93
D-LM- (150M)
9.22/7.64
0.75/0.81
8.35/5.60 60
0.76/0.82
folding SFT
7.66/4.37
0.80V0.86 86
6.00/3.41
0.83/088
DPLM-2 (650M)
7.37/4.89
0.79/0.86
5.67/3.33
0.83/0.88
folding SFT
6.21/3.78
0.84/0.88
3.40/1.78
0.8940.9
DPLM-2 (3B)
6.34/3.65
0.83/0.89
4.54/2.54
0 0.86/099
folding SFT
5.71/3.23
0.85/0.90
3.15/1.69
0.9000.5
4.3 INVERSE FOLDING (STRUCTURE-CONDITONN) SEQUENCE GENERATION)
The goal of inverse folding is to find amino acid sequence that can fold to given backbone structure. For evaluation, we employ amino acid recovery (AAR) for sequence evaluation, and we also assess the structure by self-consistency TM-score (sCTM) between the native structure and the ESMPold-predicted structure of the generated sequence.
DPLM-2 can generate reasonable sequences that fold into the given structures. Tab. 4 presents that DPLM-2 can outperform or be par with other co- generetertion models (MultiFlow, ESM3). As the model size increases, the perfor- mance in terms of sequence recovery (AAR) and structural consistency (sCTM) improves, reveal- ing the same scaling law observed in the folding task. We suggest that multimodal training effectively aligns the structure and sequence into the same space, such that DPLM-2 can yield the corresponding sequence without additional training.
Table 4: Comparison inverse folding task.
CAMBO 2022
PDB date split
Models
AAR
SCIM
AAR
SCTM
MuldiFlo
32.28/33.58
0.87/0944
37.74/37.59
0.94/0.96
ESM3
47.06/46.24
0.90N0.95
49.50/49.4 42
0.94/0.97
DPLM-2 (150M) 45. 45.22/46.12
0.81/0.93
18.83/47.96
0,89/0.95
DPLM-2 (650M)
49.01/50.10
0.88/0.93
54.8055/077
0.9180.96
DPLM- (3B) 52.36/53.72 0.89/0.95 61.67/57.91
0.92/0.96
4.4 SCAFFOLDING WITH MIXED-MODAL MOTIF CONDITIONING
The objective of motif-scafii soffil-caafollding is to generate a suitable scaffold to preserve the structure of the given motif and maintain its original function. We follow the experimental setting of Yim et al. (2024), with 24 motif-scaffolding problems and we sample 100 scaffolds for each motif, where we (1) first determine the length of scaffold, and then (2) keep the motif segment unchanged and sample the scaffold part conditioned on the motif. The scaffold length is sampled from a range provided by Yim et al. (2024), and when there are multiple motifs, the order of motif segments is consistent with Yim et al. (2024). We provide the 3D structure and sequence of motif as input of DPLM-2. As multimodal model, we evaluate DPLM-2 using sequence-based, co-generation approaches. A scaffold is considered successful if satisfies both criteria (1) overall designablity, which is successful when PLDDT > 70 (for sequence-based models) or SCTM > 0.8, and (2) mreif-preseving, which is deemed successful when the predicted motif structure matches the native one with mot i RM-D < AAA
cmucture-based and

---

Published as a conference paper at ICLR 2025
Fig. 5 reveals that DPLM-2 is capable of generate reason- able scaffolds for the given funccional motifs, In sequence- based, structure-basee and co- ro-generation evaluation, DPLM-2 can outperform or be on par with the corresponding approaches in most cases, solving more motif problem and achieving higher average success rate. We compared to sequence-based method, DPLM-2 shows better performance since allows structural in- put of motif, which is important for preserving motif's ss structure hence the functions. Remarkably, DPLM-2 - attains comparable performance with RFDiffusion when only generating scaffold structure, while achieve better performance when simultane- ously designing scaffold sequence and structure, outperforming ESM3. Despite not experimentally verified, these results sug- gest that with DPLM-2, mulitmodal conditioning and gener- ation could lead to more successful conditional protein design.
solved problems
20
25
200
TREAA bes sumples
044
0.25
0.29
0.1
ENONT DALM E31O D7U12 IIDIT DOOO ES1 DIU CROR
wepercalasse
ciegeeratio
Figure 5: Evaluation of motif- scaffolding w.r.t success rate and num. of solved problems.
4.5 EVALUATION OF PROTEIN REPRESENTATION LEARNING
Directly access to structure information is supposed to benefit downstream protein predictive tasks. To inspect this, we evaluate DPLM-2 on a variety of protein predictive tasks utilizing the dataset provided by SaProt (Su et al., 2023), where we provide tokenized protein structure tokens along with the protein sequences to DPLM-2.
DPLM-2 can perform multimodal representation learning by leveraging both structure and sequence information. Tab. 5 presents that DPLM-2 shows further improvement compared to sequence-only methods (ESM2, DPLM) on some tasks, indicating that DPLM-2 can leverage protein structures generate better representations containing multimodal information for downstream tasks. However, we find that DPLM-2 falls behind the state-of oftt----rttt-ttttt- structure-aware protein LM, i.e., SaProt, in most tasks and even lags behind DPLM in certain tasks. We hypothesize this is because the training data of DPIM-2, consisting of PDB and SwissProt, is smaller and differs from UniRef50, which DPLM is pretrained on, potentially causing catastrophie forgetting and suboptimal representation. To test this, we conducted an ablation study specific tasks where DPLM-2 underperforms compared to DPLM. We observe that without large-scale sequence pretraining. DPLM-2 outperforms DPLM significantly, suggesting that: (1) Incorporating structure information enhances performance over sequence-only models. (2) Smaller datasets can lead to catastrophic forgetting, diminishing the benefits of large-scale pretraining, Please refer to $E for more details.
Table 5: Performance on various protein predictive downstream tasks. t: benchmarked results are
quoted from Su al. (2023).
Deeploc
Thermostabllty
HumanPPI
Metal Ion Binding
EC
Models
MF
BP
CC
Subcellular
Binary
Spearman' p
(%)
Acc (%)
Fnax
Fnax
Fmax
Acc (%)
(%)
SaProt (650M)
0.724
86.41
75.75
0,884
0.678
0.356
0.414
85.57
93.55
MIF-T (Yang al., 20226)
0.694
75.54
75.08
0.803
0.627
0.239
0,248
78.96
91.76
GearNet (Zhang 2023)
73.86
71.26
0.404
ESM2 (650M) DPLM (650M)
0.691
84.78
71.88
0.866
0.676
0.344
0.402
83.68
92.28
0.695
86.41
75.15
0.875
0.680
0.357
0.409
84.56
93.09
DPLM-2 (650M)
0.714
84.44
74.28
0.878
0.680
0.359
0.411
82.98
93.64
5
DISCUSSIONS
In this paper, we introduce DPLM-2, multimodal diffusion protein language model that understands, generates and reasons over protein structure and sequence, aiming to severe as a mulimodal foundation for protein. Despite promising performance spanning protein co-generation, folding, inverse folding and conditional motif scafil-saafooldin with mulimodal input and output, there remains several limitations deserving to be addressed. (1) Structure data: Our findings indicate that while structure awareness may help with predictive tasks, the limited structure data constrains DPLM-2's ability to learn robust representations. also important to account for longer protein chains and multimers in future studies. (2) Trade-off of discrete latent representation: Tokenizing structure into discrete symbols facilitates multimodal protein language models and co-generation but may come at the cost losing fine-graing structural details and control, such precise atomic positions and Future work should aim to also integrate the strengths of data-space structure- arructur-bassdd generative models into sequence- based multimodal language models to maximize the best of both worlds.
distances.
10

---

Published as a conference paper at ICLR 2025
ACKNOVLEDGEMENT
We would like to thank Dr. Hang Li for insightful discussions on the project and feedback on the manuscript that help shape this study. We thank Yi Zhou, Jing Yuan, Yilai Li, Yuning Shen, Wesley Hsieh and Daiheng Zhang for their valuable comments.
REFERENCES
Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Xijie Lu, Nicolo Fusi, Ava Pardis Amini, and Kevin Yang. Protein generation with evolutionary diffusion: sequence all you need. bioRxiv, pp. 2023-09, 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34, pp. 17981-17993, 2021.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pp. 1692-1717. PMLR, 2023.
Inigo Barrio-Hernander Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron LM Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering predicted structures at the scale of the known protein universe. Nature, 622(7983):637-645, 2023.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Informa- tion Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. neurips cc/paper/2015//eeeeeeeeepee
1171-1179, 2015, URL /tttps://proceett Absttrr .tmml.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235- 242, 2000.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: a universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2100 2022.
Andrew Campbell, Jason Yim, Regina Barzilay. Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402. 24499.240..04997. 2024.
Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact implemen- tation through uniformization. arXiv preprint arXiv:2402.08099 2024.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023.
Alexander E Chu, Jinho Kim, Lucy Cheng, Gina El Nesr, Minkai Xu, Richard W Shuai, and Po-Ssu Huang. An all-atom protein generative model. Proceedings of the National Academy of Sciences, 121(27)e22311000)
2024.
Justas Dauparas. Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, (78(6615):499555 2022.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward understanding the language of life through sulf-supe learning. IEEE transactions pattern analysis and machine intelligence, (4(10):71227722) 2021.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.
11

---

Published as a conference paper at ICLR 2025
Zhangyang Gao, Cheng Tan, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. arXiv preprint TTXiv:2209112643 2022.
Zhangyang Gao, Cheng Tan, and Stan Z Li. Knowledge-design: Pushing the limit of protein design via knowledge refinement. arXiv preprint arXiv:2305. 2023.
Zhangyang Gao, Cheng Tan, Jue Wang, Yufei Huang, Lirong Wu, and Stan Z Li. Foldtoken: Learning protein language via vector quantization and beyond. arXiv preprint arXiv:2403.09673, 2024.
Tomas Hayes, Roshan Rao. Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024 07, 2024.
Liang He, Shizhuo Zhang, Lijun Wu, Huanhuan Xia, Fusong Ju, He Zhang, Siyuan Liu, Yingce Xia, Jianwei Zhu, Pan Deng, et al. Pre-training co-evolutionary protein representation via a pairwise masked language model. arXiv preprint arXiv:2110.15527, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
URL https: setpss:/pproeet http:///oocednnns.eeeee ieseeeddnnn.euuripp... wwtpppper//////////ecpp oarr Paper.pdf.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465 2021.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8946-8970. PMLR, 17-23 Jul 2022. URL https: http///oooceeeen/eeeeeeeee
mlr. www.ppess//ee..ee.
mwr..press/a v 62 h/s2//hsu2aa html.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.000 09685, 2021.
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, and Minlie Huang. On the learning of hon-autoregreesive transformers. In International Conference on Machine Learning, pp. 9356-9376. PMLR, 2022.
Guillaume Huguet, James Vuckovie, Kilian Fatras, Erie Thibodeau-Lauffe Pablo Lemos, Riashat Islam, Cheng- Hao Liu, Jarrid Rector-Brooks Tara Akhound- -aabuundSSadehh, Michael Bronstein, et al. Scquence-augmmented se (3)-flow matching for conditional protein backbone generation. arXiv preprint wwXXh:2405.20313.. 2024.
John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana Lord, Christopher Ng-Thow-Hing, Erik R Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature, pp. 1-9, 2023.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi S Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. International Conference on Learning Representations, 2021.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. International Conference on Learning Representations, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvumkkool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58))
2021.
12

---

Published as a conference paper at ICLR 2025
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv.org/ abs/ / 412..... 6980.
Xiangzhe Kong, Wenbing Huang, and Yang Liu, Conditional antibody design as 3d equivariant graph translation. arXiv preprint arXiv:2208. 06073, 2022.
Jin Sub Lee, Jisun Kim, and Philip M Kim. Proteinsgm: Score-based generative modeling for de novo protein design. bioRxiv, pp. 2022-07, 2022.
Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and Marie-Francine Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps. arXiv preprint arXiv:2305.1558: 2023.
Yeqing Lin and Mohammed AlQuraishi. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. arXiv preprint arXiv:2301. 12485, 2023.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
Haiyan Liu, Yufeng Liu, and Linghui Chen. Diffusion in quantized vector space generates non- idealized protein structures and predicts conformational distributions. bioRxiv, pp. 2023-11, 2023.
Amy X Lu, Haoran Zhang, Marzych Ghassemi, and Alan Moses. Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, pp. 2020-09, 2020.
Amy X Lu, Wilson Yan, Kevin K Yang. Vladimir Gligorijevic, Kyunghyun Cho, Pieter Abbeel, Richard Bonneau, and Nathan Frey. Tokenized and continuous embedding compressions protein sequence and structure. bioRxiv, pp. 2024 08, 2024.
Ali Madani, Ben Krause, Eric Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, pp. 2021-07, 2021.
Weian Mao, Muzhi Zhu, Zheng Sun, Shuaike Shen, Lin Yuanbo Wu, Hao Chen, and Chunhua Shen. De novo protein design using geometric vector field networks. arXiv preprint arXiv:23 1800, 2023.
Matthew McDermott, Brendan Yap. Harry Hsu, Di Jin, and Peter Szolovits. Adversarial contrastive pre-training for protein sequences. arXiv preprint arXiv:2102. 000000000062...220 2021.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances Neural Information Processing Systems, pp. 29287-29303, 2021.
Igor Melnyk, Vijil Chenthamarakshan, Pin- Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, and Devleena Das. Reprogramming large pretrained language models for antibody sequence infilling. arXiv preprint arXiv:2210.071444 2022a.
Igor Melnyk, Aurelie Lozano, Payel Das, and Vijil Chenthamarakshan. Alphafold distillation for improved inverse protein folding. arXiv preprint arXiv:2210.03488 2022b.
Seonwoo Min, Seunghyun Park, Siwon Kim, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Pre-training of deep bidirectional protein sequence representations with structural information. IEEE Access, 9:1239 12-3992.. 2021.
Ananthan Nambiar, Maeve Heflin, Simon Liu, Sergei Maslov, Mark Hopkins, and Anna Ritz. Transforming the language of life: transformer neural networks for protein prediction tasks. In Proceedings of the 11th ACM international conference on bioinformatics. computational biology and health informatics, pp. 1 1-8, 2020.
13

---

Published as a conference paper at ICLR 2025
Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. arXiv preprint arXiv:2206. 13517, 2022. 2022.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint arXiv:2308. 2023a.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias diffusion models. International Conference on Machine Learning, pp. 09245-262655... PMLR, 2023b.
Esmaeil Nourani, Ehsaneddin Asgari, Alice C McHardy, and Mohammad RK Mofrad. Tripletprot: deep representation learning of proteins based on siamese networks. JEEE/ACM Transactions on Computational Biology and Bioinformatics, 19(6):374--3755 2021.
OpenAl. Gpt-4 technical report, 2023.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:/ / arxiv. www.v.orgga.a..aaa..oo.o..oowoot aab///b////// 1111/11111111 06732.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. PNAS, 2019. doi: 10.1 101/62200.. URL https: : www. bi orxi v. org/ content/ /10. 1101 /220013222
Chence Shi, Chuanrui Wang, Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08761, 2022.
Yang Song. Jascha Dick--ickstten, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2020.
Nils Strodthoff, Patrick Wagner, Markus Wenzel, and Wojciech Samek. Udsmprot: universal deep sequence models for protein classification. Bioinformatics, 36(8):2401-2400 2020.
Pascal Sturmfels, Jesse Vig, Ali Madani, and Nazneen Fatema Rajani. Profile prediction: alignment- baseed pre-training task for protein sequence models. arXiv preprint arXiv:2012.00195. 2020.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan, Saprot: Protein language modeling with structure-award vocabulary. bioRxiv, pp. 2023-10, 2023.
Haotian Tang, Yecheng Wu, Shang Yang. Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arxiv:2410. 108/2, 2024.
Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206. 04119, 2022.
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Dogan.
Learning functional properties of proteins with language models. Nature Machine Intelligence, (3):227-245 2022.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Michel Kempen, Stephanie Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology, pp. 1-4, 2023.
14

---

Published as a conference paper at ICLR 2025
Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseck. Nature biotechnology, 42(2):243-246. 2024.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1 D:D4499-4444, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, volume 30, pp. 5998-6008, 2017.
Robert Verkuil, Oni Kabeli, Yilun Du Basile IM Wicky, Lukas F Milles, Justas Dauparas, David Baker, Sergey Ovchinmikov, Tom Sercu, and Alexander Rives. Language models generalize beyond natural proteins. bioRxiv, pp. 2022-12, 2022.
Xinyou Wang, Zaixiang Zheng, Fei Ye. Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners, arXiv preprint 2rXiv:2402.18567, 2024.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 62097976)1188911100 2023.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint ar arXiv:2410.13844 2024.
Kevin E Wu, Kevin K Yang, Rianne van den Berg, James Y Zou, Alex X Lu, and Ava P Amini Protein structure generation via folding diffusion. arXiv preprint arXiv:2209. 15611, 2022a.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022 07, 2022b.
Yijia Xiao, Jiezhong Qiu, Ziang Li, Chang- Yu Hsieh, and Jie Tang. Modeling protein using large-scale pretrain language model. arXiv preprint arXiv:2108.00 074335 2021.
Kevin Yang, Alex Lu, and Nicolo Fusi. Convolutions are competitive with transformers for protein sequence pretraining. bioRxiv, pp. 2022-05, 2022a.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, pp. 2022-05, 05, 2022b.
Kai Yi, Bingxin Zhou, Yiqing Shen, Pietro Lio, and Yu Guang Wang. Graph denoising diffusion for inverse protein folding. In sevethh Conference on Neural Information Processing Systems, 2023. URL https: opwnrevieww net forum? id 4XK.x44XXXXK....
Jason Yim, Brian L. Trippe. Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint ar arXhv:2302.02277, 2023.
Jason Yim, Andrew Campbell, Emile Mathieu, Andrew YK Foong, Michael Gastegger, José Jiménez- Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling. Frank Noé, et al. Improved scaffl-caffoldin with se (3) flow matching. arXiv preprint arXiv:2401.04088 2024.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu. Alexander G Hauptmann, et al. Language model beats diffusion diffusion-tokemizer tokenizer is key to visual generation. In The Tweifth International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2025
Cheng Zhang. Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma, Tom Minka, Nick Pawlowski, et Understanding causality with large language models: Feasibility and opportunities. arXiv preprint 0005::23304.0552,, 2023.
Zikun Zhang, Zixiang Chen, and Quanquan Gu. Convergence of score-based discrete diffusion models: A discrete-time analysis. arXiv preprint arXiv:2410.02321, 2024.
Jiangbin Zheng and Stan Z Li. Progressive multi multi-modality learning for inverse protein folding. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1 1-6. IEEE, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.0553 057377 2023a.
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei YE, and Quanquan Gu. Structure-informed language models are protein designers. In International Conference on Machine Learning. 2023b.
Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang, and Quanquan Gu. Antigen-specific antibody design via direct energy-based preference optimization. Advances in neural information processing systems, 2024.
Yiheng Zhu, Jialu Wu, Qiuyi Li, Jiahuan Yan, Mingze Yin, Wei Wu, Mingyang Li, Jieping Ye, Zheng Wang, and Jian Wu. Bridge-if: Learning inverse protein folding with markov bridges. arXiv preprint ar 2rX00:241..02120, 2024.
16

---

Published as a conference paper at ICLR 2025
A D TRAINING
A.1 IMPLEMENTATION DETAILS
DPLM-2 takes the discrete structure token sequence and amino acid token sequence as input. As demonstrated in Fig. I, we concatenate the two sequences into one sequence of double length. DPLM-2 employs an efficient warm-up strategy by initializing with pre-trained sequence-based DPLM ($3.2) to leverage the evolutionary information learned by DPLM for protein structure modeling. Considering that the vocabulary of DPLM only consists of amino acids, we expand the vocabulary of D with discrete structure tokens. The embeddings for these new tokens are initialized using the mean and standard deviation of the learned amino acid embeddings. This embedding initialization keeps the distributional statistics of the embedding space consistent with the pre-trained DPLM, ensuring stable early-stage training (for learning struture-sequentt alignment) and reducing the risk of extreme gradients that could cause training instability.
A.2 DISTINCT NOISE SCHEDULER OF TRAINING
We introduce a distinct scheduler to control the noise level of structure and sequence flexibly during training ($3.1). Different combinations of structures and sequence schedulers (denoted as lg and (g, respectively) imply training for different applications. Specifically, we mainly focus on (I) sequernce-conditioned structure generation (e.g. folding). (2) structure-condittoned sequence generation (e.g.. inverse-folding), (3) sequence generation, (4) structure generation, (5) structure- sequence co- vo-generation, as shown in Tab. For conditional generation tasks (e.g., folding and unveree-foddigg we set the noise scheduler of the conditioned modality to 0, e.g., no noise in the conditioned modality. Specifically, in the folding task, the l, is always set to O, while in the inverse- folding tasks the l- is always set to O. In the structure-sequeece co-generation task, we keep the lz consistency in co oo-generation. The structure
and to for the same, enhancing the cruuture-sequeenee or sequence generation tasks do not depend on another modality, so set the noise scheduler of another modality to T, c.g., 100% noise in another modality. For example, in structure generation task, the t. is always set to T. During training, we train the above tasks simultaneously. We divide the training data in batch into 5 parts according to a preset proportion, and each part is used for a specific task training. In our experiment, the proportion for each task is the same, which is 20% After training. we can further enhance specific generation task by supervised finetuning (SFT). This involves continuing training for the specific task with a proportion of 100%, while the proportion for other tasks is set to 0%. For example, in Tab. 3, the folding supervised finetuning is performed by continue training with folding task based on a pre-trained DPLM-2 with 100% proportion of training data.
A.3 HYPERPARAMETER
We train all models using A optimizer (Kingma & Ba, 2015) with 31 = and 33 = 0.95. We use a weight decay of 0.01 and gradient clipping of 0.5. We employ 2K warmup steps until reaching the maximum learning rate, and utilize a linear decay scheduler to decay LR to 10% of the maximum learning rate by the end of training. The maximum learning rate is Ic-4, and the overall training step is 100,000. We utilize the pretrained DPLM as the parameter initialization, and the diffusion timestep is set to 500. We train 150M DPLM-2 with 8 A100 GPUs for 3 days, while 650M with 16 A100 GPUs for 3 days and 3B with 16 A100 GPUs for a week.
A.4 DATASET
The training set of DPLM-2 is composed by experimental data, i.e., PDB (Berman et al., 2000), and high quality synthetic data, i.e., SwissProt (Varadi et al., 2022). We filter the SwissProt data by PLDDT > 85. After filtering, the overall training set contains approximately 200,000 proteins. We limit the maximum length of the training set to 512. For proteins longer than 512, we randomly crop it to 512. We crop the low PLDDT (pLDDT < 50) segments located at the both ends of proteins in the SwissProt dataset. These segments are typically non-structural and may negatively impact the training results. Moreover, we find that the length distribution of the training set is not balanced, where the number of proteins with length less than 100 is relatively small, leading to suboptimal diversity among the short proteins. Therefore, during training, we randomly crop long proteins to short proteins with a probability of 50% for each batch to improve the diversity.
A.5 TACKLING EXPOSURE BIAS IN DISCRETE DIFFUSION WITH SELF-MIXUP TRAINING STRATEGY
The exposure bias problem. which is described as the input mismatch between training and sampling, has already garnered attention in the research of continuous diffusion (Ning al., 2023a;b; Li et al.,
17

---

Published as a conference paper at ICLR 2025
2023) and NLP (Ranzato al., 2016; Bengio et al., 2015). We find that the discrete diffusion model also encounters this issue. According to the Eq. the model trained to be tasked with po (x |x()). essentially doing masked-prediction. During training, the model makes prediction conditioned on the
x(1), which is mixture of ground-truth and mask tokens as noise: x(1) axx (0) + (1 02)9nosse
However, during inference, the model predicts po(x; (0) (*) conditioned on the previously generated
sample x which is mixture of model prediction and masks, essentially requiring denoising and masked-prediction. The difference between (f) and (1)) causes a discrepancy between p=(x; (0) (x())
and po ( (x(()), potentially leading to error accumulation since the model trend to be over-confident on its predictions (as in training the model is always exposed to ground-truth, hence the name exposure bias), and negatively impacting the generation performance.
To mitigate this, we propose to bridge this gap by training model to make predictions conditioned on
its own predicted results: (1) predict x0)) conditioned on the ground truth sample x(*), (2) construct
the generated sample: x(1) x0) + (1
ai Amoise
(3) compute self- self-mixup loss according to Eq. 1:
J. E (
A
Σ ISiSA.
bt(t) log (x. (0) (1)
We can illustrate this more clearly with break-down example. Let the ground-truth x()) be ABCDE, and the x(1) be [m] [m] [m] DE as in masked discrete diffusion, where [m] represents the mask token. The training process is as below:
(i) Call a model forward to obtain model prediction x which is abcDE (with the ground truth token DE preserved for masked positions), where abc represent model prediction by
argmax decoding.
(ii) Construct self self-mixup (()), In our experiments, we always replace the ground-truth token in
(0) (DE in this case) with the mask token [m]. Therefore x(() becomes abc [m] [m].
(iii) Compute self-mixup loss, which is essentially cross entropy loss between po (x: (0) (01)) and
(") all positions.
More specifically, this can be seen as mask positions are applied masked language modeling loss while mon-masked positions are applied denoising autoencoder loss. Moreover, this also improves sample-efficiency compared to typical masked discrete diffusion where training loss is only applicd to mask positions.
In our experiments, we first train DPLM-2 with the original loss Eq. 1 for 50,000 steps to ensure the prediction quality. This step is crucial; otherwise, the model's 's predictions might be poor, leading to an excessively large self-mixup loss and causing training instability. After this initial phase, we continue training with self-mixup loss to mitigate the exposure bias issue. Tab. 6 shows that the self-mixup training strategy effectively enhances the diversity of samples. We conduct experiments with the DPLM-2 650M model on the unconditional generation task. We sample 100 proteins within each length interval and calculate ScTM for structure-scquence compatibility and the number of clusters for diversity. We attribute this to the model producing more accurate logits during inference, leading to more diverse reasonable sampling paths instead of converging on the sampling paths with the highest probability, which results in more diverse proteins.
Table 6: Ablation study on the self-mixup training strategy.
Mixup strategy
length 100
length 200
length 300
length 400
length 500
scTM
clusters
scTM
clusters
scTM
clusters
scTM
clusters
scTM
clusters
0.9237
44
0.9180
53
0.9147
48
0.9059
42
0.8896
33
0.8812
62
0.8820
62
0,9172
59
0.9099
54
0.8845
38
A.6 ABLATION STUDY ON THE SEQUENCE PRE-TRAINING AND SYNTHETIC STRUCTURES
In DPLM-2 training, we start with a warmup from the sequence-base pre-trained DPLM exploit established evolutionary information and augment the data with high-quality AlphaFold Spphafold-rredicted predicted structures from SwissProt (around 200K) and clustered PDB structures. This section evaluates the effects of sequence pre-training and data augmentation on unconditional protein generation.
We investigate the effect of sequence pre-training by randomly initializing DPLM-2 instead of using DPLM parameters, while for effect of synthetis structures we leverage PDB structures only for training. We conduct experiments on 150M DPLM-2, for each DPLM-2 variant we sample
18

---

Published as a conference paper at ICLR 2025
Table 7: Ablation study on the sequence pre- pre-training and training data augmentation.
sequence
synthetic
length 100
longth 200
longth 300
length 400
length 500
pre-training
scTM
clusters
scTM
clusters
scTM
clusters
scTM
clusters
scTM
clusters
0.9241
0.8674
34
0,7667
33
0.5016
0,4511
0.9610
0.9349
47
0.9169
38
0.8643
0.7673
0.8988
27
0.9182
15
0,9343
13
0.8518
21
0.8288
31
Table 8: Results of codebook size, codebook utilization and 1ddt on the training set and valid set of different structure tokenizers.
training set
valid set (CAMEO 2022)
tokenizer
codebook size
codebook utilizzaion
1dát_ca
Iddt_ca
TM-score
RMSD
VQ-VAE-IK
1024
63.50%
0.76
0.71
0.80
6,14
LFQ-IK
1024
100.00%
0.82
0.77
0.86
4.35
LFQ-2K
1024
100.00%
0.84
0.79
0.88
3.62
LFQ-4K
4096
100.00%
0.86
0.82
0.91
3.31
LFQ-8K
8192
99.50%
0,92
0.86
0.93
2.58
LFQ-16K
16384
98.60%
0.92
0.87
0.94
2.32
100 examples for each length in 100, 200, 300, 400 and 500. We compute scTM and the number of difference clusters in each length. Tab. 7 demonstrates that sequence pre-training and data augmentation can significantly improve the designability and diversity, especially generating long proteins (length > 300). We hypothesize that the limited number of long proteins in PDB leads to insufficient training. contrast, sequence pretraining, which includes evolutionary data, essential and can be transferred to improve protein structure modeling and generation quality. Additionally, this evolutionary information boosts sampling diversity. While increasing the amount training data improves designability, it is less effective in enhancing diversity compared to sequence pretraining. By combining both strategies, we achieve the best overall performance, which forms the core of our training strategy.
B DISCUSSIONS ON THE STRUCTURE TOKENIZATION
B. IMPLEMENTATION DETAILS
We utilize GVP-based (Jing al., 2020) structure encoder from pre-trained CVP-Transformer (Hsu et al., 2022) with its parameters frozen during training. The structure encoder transforms backbone structures into geometric features, which projected onto latent embedding using MLP layer. The structure decoder follows the IPA-based modules from AlphaFold2 (Jumper et al., 2021), using EvoFormer layers without MSA row attention, following ESMFold (Lin al., 2022), to generate atomic positions from the structure tokens. We train structure tokenizer using the same structure data as our mulitmodal language model, containing both experimental and high-quality structures. The training objective of structure tokenizer includes reconstruction loss, codebook commitment loss, and entropy regularization loss ensure effective codebook utilization. For the reconstruction loss, we adopt the FAPE loss, violation loss, and distogram loss from AlphaFold2 (Jumper et al., 2021), measuring the difference between predicted and native structures. To further enhance the training, we introduce a sequence prediction head on top of the structure decoder's 's final representation and minimize the cross-entropy against the native sequence.
B.2 THE UTILIZATION AND INTERPRETABILITY OF STRUCTURE TOKENS
In the Fig. 2A have shown the reconstruction accuracy and an interpretation analysis the correspondence of structure tokens and local structural elements in terms of secondary structures. We also calculate the codebook utilization Tab. 8. We find that LFQ-based tokenizers always achieve nearly 100% codebook utilization with more evenly distributed code usage, while vanilla VQ-VAB struggles with codebook collapse. The Fig. 2B demonstrates the interpretability of structure tokens with informative simplex plot of structure tokens vs second structure. We can observe strong correlation between a vast majority of the structure tokens and structured local environments, where the alpha helix and beta sheet vertices, while some tokens lie
a lot of structured tokens concentrate between regions or the loop vertice. There are also a subset of structure tokens having less clear clues to specific secondary structures. This suggests that structure tokens mostly capture clear secondary elements, some may correspond to structured local environments (in bewteen helic and sheet), while others could be high-level abstract entities.
19

---

Published as a conference paper at ICLR 2025
Figure 6: The histogram of ctttuurllloooiiiii motifs v.s. motif motif.clusters for each struc- ture token. We randomly sampling 500 out of 69192 structure tokens for readability.
Figure 7: Visualization of mapping structure tokens to structural motifs.
struct_token:
struct token: 8397
struct_token: 2916
B.3 VISUALIZATION OF MAPPING STRUCTURE TOKENS TO STRUCTURAL MOTIFS
We provide more fine- graingri insights into what structure tokens learn by mapping structure tokens to structural motifs. Specifically, as structure tokens are residue-wiss representations, aim to map each structure token to structural motifs defined as the hearest-neighbor local structural environment of residue in the training dataset. For efficiency, used only the PDB dataset. The process is as follow:
(1) For each structure in the PDB dataset (approximately 20K in total), we first tokenize the structure into structure tokens and save the pair of structure tokens and ne-nearrst-nnighborss structural motifs for each residue. We use 30 nearest neighbors because the pre-trained GVP EVPTransformerEnnoder which we used as the structure encoder, employs 30 nearest neighbors as the hyperparameter for geometric features.
(2) After processing all structures, we obtain table where each row corresponds a structure token and its associated structural motifs (i.e.,
wwr.uuurummmoiii.... Lmot ifs)
(3) To analyze whether a structure token tends to occur in a similar local structural environment, we use Foldseek (TM-threshold 0.5) to cluster the structural motifs for each structure token (i.e., mot motif.clusters) Although Foldseek may not be entirely accurate in clustering such short and discontinuous structural regions, it provides a reasonable comparative sense the similarity or difference among all structural motifs associated with each structure token.
In Fig. 6, we plot the histogram of num_st www.stuuuuuurrr. iottiimismooif v.s. motif-cluster for each structure token (randomly sampling 500 out of 8,192 structure tokens to ensure readability). From the visualization, we observe that many structure tokens correspond to highly similar structural motifs (evidenced by a small ratio of the number of motif clusters to the number of total structural motifs), while others exhibit high degree of ambiguity.
Additionally, we visualize the mapping between structure tokens and structural motifs in specific cases. In Fig. showcase structure tokens and their corresponding similar structural motifs
20

---

Published as a conference paper at ICLR 2025
across four different PDB structures, illustrating the diversity or consistency in the mapped local structural environments.
B.4 LIMITATION ON THE STRUCTURE TOKENIZATION
Our approach can be seen as decoupling the learning of the structurally invariant topology to the tokenizer and the language model, and the geometric reasoning on 3D coordinates to the structure decoder parameterized by triangular modules and IPAs from AlphaFold2 with FAPE loss. This shares the similarity to AF2 and ESMfold, where the sequence encoding modules like Evoformer (co-evolution encoding) and ESM (amino-acid encoding) provide invariant features (in the form of single and pair embeddings) to the structure decoder that learns to convert invariant features into 3D coordinates. The AF2-style structure decoder does not enforce strict equivariance to rigid transformations, Instead, it relies on the FAPE, loss to ensure structural consistency, which minimizes coordinate errors in a manner that is invariant to global rotations and translations.
As such, we suggest that the primary trade-off when using invariant structure tokens instead of 3D coordinates mainly lies in the potential loss of fine-graing structural details. Albeit being the key enabler to multimodal PLMs, structure tokenization is essentially clustering similar local structured environments, which results in lossy compression and the absence of lineggraieed structural variation. The primary principle of the solution is that we need to "recover" and preserve the high-frequenne variation that gets lost during quantization. We propose some potential directions for mitigation:
Separate structure encodings for DPLM-2. We can introduce different structure encoders for encoding and generation purposes, respectively. For parts of protein where atomic coordinates are already provided, lossy tokenization may not be necessary. Instead, we can use robust features from powerful structure encoders like EVp-Transformer while continuing use structure tokens for generating the remaining parts. To achieve this, the model can he trained to alternate between these two types of encodings. A similar approach has been applied successfully in recent vision-language MLLMs (Wu et al., 2024), as the vision-language community has also recognized that understanding and generation often require different types of representations.
Modeling continuous structure features with hybrid tokenization. In the structure tokenizer, the vector quantizer module converts encoder features into discrete structure token features, but rssiduals-differeneees the residuals- differences between the original and quantized features-are are lost, removing fine- grained structural details. To address this, can using continuous generative modeling, such as iiffusion/flow fllowbaawwwddd models, to learn to recover these residuals. This would work by conditioning on the structure tokens and possibly the final hidden states of DPLM-2. The protein structure generation process would involve first generating discrete structure tokens that capture the overall topology. then using those tokens to generate the missing residuals. These residuals would be added up to the structure token embeddings to recover more complete and accurate structure representation, closer to the features produced by the structure encoder. This approach could significantly improve structure generation. By combining this idea with hybrid structure encodings. DPLM-2 could not only interpret given structures at atomic accuracy but also generate structures that include the missing fine-grained variations. Similar strategies have shown significant success in visual autoregressive generation with visual tokenizers (Tang et al., 2024).
C ANALYSIS ON THE SAMPLING STRATEGY
We utilize argmax decoding for conditional generation tasks (e.g.. folding and inverse folding) to maximize generation accuracy and ensure a fair comparison with DPLM. On the other hand, stochas- tic sampling was employed for unconditional generation or notif-scaffolding tasks to encourage generation diversity while maintaining good generation quality.
Specifically, utilize temperature- based stochastic approach. We mainly focus on the temperature- annealed version based on the sampling procedure of DPLM (Wang et al., 2024) for better sampling diversity. The overall sampling approach is shown in algorithm 1. The temperature annealing sampling approach introduces more randomness during the initial stage of sampling by using a large temperature, and more fidelity during the final stage of sampling by using a small temperature. This method improves generation diversity while maintaining generation quality.
Moreover, we observe that stochastic sampling could also improve the generation diversity in conditional tasks (e.g., inverse folding) while keeping the quality. As shown in Tab. 9, the argmax decoding strategy picks the token with highest probability at each timestep. yielding sequence with high probability and resulting in high amino acid recovery (AAR). On the other hand, we employ a sampling strategy with annealing temperature from 2.2 to 0.1 to improve diversity, and the generated
21

---

Published as a conference paper at ICLR 2025
Table Ablation study on the sampling strategy in inverse folding task.
CAMEO 2022
Model
AAR
SCTM
MultiFlow
32.28/33.58
0.87/0.94
ESM3-Open
47.06/46.24
0.90/0.95
DPLM2 650M (argmax decoding)
49,01/50.10
0.88/0.93
DPLM2 650M caapperature-anncale sampling)
43.15/42.24
0.88/0.93
sequence has lower AAR while maintaining the same ScTM argmax decoding. This demonstrates that temperature annealing sampling strategy is capable of generating more diverse sequences that, while not similar to the ground truth, still meet the given structural conditions.
Algorithm Cemperatuur-anneallee stochastic sampling
Input: trained network fe (-), the total sampling steps T, the minimum temperature Tmin and the maximum temperature Tmax-
Output: generated sample x (0).
for n 1,2,... ..,, N do
Initialize XT,, quoisei
Initialize bT = O;
end for
for f T , do
Determine the temperature T of the current timestep
Tmin 1 (Tmax Tmin)
for 1, 1,2, ... No do
Draw Xg.n ~ Categorical (Jo (xe,a)(T)
Generate Vm-1,n according to log n
if br,n = 1 then
(1)
Draw u Quoises
Xt- = (1)
1i-1,nxt,, + (1
(1) (1),
1/1111 (1-0/_1m)) u
else
Draw u ~ Gnoise (x1,n): (
(2)
X = 1,1,n Xo,n + (1 11 1 (1-0
(2).
Xi,
end if
Let 11-1.1 = bin A 1/11 V u
(1)
end for end for Return No.1:N.
D ANALYSIS ON THE EVALUATION METRICS IN THE UNCONDITIONAL GENERATION
In Tab. 2, we observe that DPLM-2 achieves high scTM score while the scRMSD score is bit higher than other baselines, e.g., MultiFlow. We will make detailed discussion of this.
We first highlight that the generated samples from DPLM-2 share similar scTM (0.925) and scRMSD (3.9) as native PDB samples, which also exhibit good scTM (0.904) with a little bit higher scRMSD (4.623). Moreover, Additionally, DPLM-2 maintains balanced structural composition (helix: 0.4, strand: 0.2, coil: 0.45), closely resembling natural distributions. In contrast, for MultiFlow, the officially released model with distillation attains much lower scRMSD (3.2), while the performance of our retrained version (on the same DPLM-2 training set) degrades in both se scTM (0.871) and scRMSD (6.58). Lower scRMSD MultiFlow with distillation, appears be driven by overrepresentation of structured elements (Fig. 4A), i.c., significantly biasing towards proteins with more helices, with less strands and loops (Fig. 4C). This overrepresentation drives the observed ScRMSD improvement but deviates from natural protein diversity.
Then we delve into the insight and purpose of the TM-score and RMSD metrics. TM-score emphasizes global topology, while RMSD is sensitive to local structural error.. As such, although scTM and ScRMSD are generally correlated, discrepancies can arise. The purpose of TM-score is to solve this
22

---

Published as a conference paper at ICLR 2025
Table 10: Analysis on the performance degradation in the representation learning task, including HumanPPI, MetailonBinding and DeepLoc Subcellular.
Exp id Model
Training set
Hum
MetallonBinding
Deeploc Subcellular
Acc(%)
ASc(%)
Aco((6)
AFDB
86.41
85.57
DPLM
PDB + swissprot (200K)
73.33
62.25
63.49
2 DPLM-2
PDB + swissprot (200K)
77.22
69.47
66.77
DPLM w/ fully peetraining UniRef50
UniRef50 (45M)
86.4]
75.15
84. 56
DPLM-2 finetuning from DPLM
PDB swissprot (200K)
MPL.-22 w/ finetuning from DPLM
Aeps PDB a swrisspoot (1.5M)
87.78
83.42
sensitivity of RMSD, because RMSD is an average distance of all residue pairs in two structures, local error (e.g. misorientation of the tail) will raise a big RMSD value although the global topology correct. In TM-score, however, the small distance is weighted stronger than the big distance, which makes the score insensitive to the local modeling error. As shown in Fig. 4B, some samples from DPLM-2 with higher loop proportion are more conformationally flexible, hence may show high scTM (>0.9) but worse scRMSD (>2.0), similar to natural protein. However, this does not necessarily indicate a limitation in generation quality but reflects differences in metric sensitivity. As a result, the in-silico designability of protein generation should be evaluated comprehensively using both scTM and scRMSD, as each metric offers distinct insights and serves different purposes. For users aiming to generate samples with accurate global topology, scTM serves as reliable indicator, whereas scRMSD may occasionally exclude reasonable structures. Conversely, for applications requiring structurally rigid and stable proteins, such as functional designs (e.g., binder design), scRMSD has been shown to correlate more strongly with in vitro success rates, as suggested by RFDiffusion.
E ANALYSIS ON THE PERFORMANCE DEGRADATION IN REPRESENTATION LEARNING
In Tab. 5, we find DPLM-2 demonstrates a performance degradation compared with the DPLM, which is used for parameter initialization for DPLM-2, in some tasks (e.g.. DeepLoc Subcellular). We hypothesize two potential causes for the observed degradation: (1) DPLM-2 needs to accommodate additional structural representations given the same model capacity (parameters), which could negatively impact the representation learning performance. (2) As continuous training on smaller magnitude of structure data, DPI.M-2 may experience catastrophic forgetting of the representation power gained during DPLM's large-scale sequence pretraining.
To explore (1), eliminated pretraining factors by retraining both DPLM and DPLM-2 with random initialization on the SwissProt and PDB datasets for 100K training steps. Additionally, we evaluated performance across all three tasks (HumanPPI, Metallon and Deepl Subcellular) where DPLM-2 underperformed compared to DPLM. As shown in the Tab. 10, when large-scale sequence pretraining removed, DPLM-2 significantly outperforms DPLM (exp 2 vs exp 1). This indicates that incorporating structural information enhances performance rather than harming it, which rejects the hypothesis (1).
However, when DPLM undergoes large-scale pretraining and DPLM-2 is subsequently trained from the pretrained DPLM, the performance of DPLM-2 on certain tasks diminishes (exp 4 exp 3). Given the relatively smaller structure data for DPLM-2 training, this suggests that catastrophic forgetting occurs during DPLM-2's multimodal training, reducing the advantages of large-scale pretraining. To verify and mitigate this, we curate additional 1.3M predicted structures from AFDB representative (Barrio-Hernandez et al., 2023), and trained DPLM-2 on this larger data. The experimental results show that the amount of structure data is indeed a key factor for better multimodal protein representations, leading to significantly improved performance over the original data (exp vs exp 4). In particular, on HumanPPI, enlarging data from 200K to 1.5M helps DPLM-2 attain 2.3% improvement, and also outperforms SaProt, a strong multimodal PI.M trained with 40M Foldseek tokenized AFDB data.
F MORE EMPIRICAL RESULTS
F.1 COMPREHENSIVE EVALUATION ON THE UNCONDITIONAL. SEQUENCE GENERATION
In addition to the Tab. 2, we have conducted more comprehensive evaluations the unconditional generation in terms of protein sequence, including: (1) sequence and structural diversity: we conduct MMseqs2 clustering and Foldseek structural clustering different thresholds. For MMseqs2
23

---

Published as a conference paper at ICLR 2025
Table 11: Comprehensive analysis on the protein sequence generation. We evaluate the performance in terms of pLDDT, sequence and structural diversity, sequence naturalness and sequence novelty.
MultiFlow
MultiFlow
Evaluation Metric
DPLM
DPLM-2
(official w/ distillation) (retrained on DPLM-2 d data)
structural plausibility (1)
PL.DDT
79.4
62.6
84.0
83.7
sequence diversity (1)
MMscqs2 cluster at seq-id=0.3 & plddt >> 70
0.804
0.204
0.740
0.745
MMsegs2 cluster at seq-id=0.5 & plddt
0.860
0.294
0.745
0.755
MMseqs2 cluster al seq-id=0.7 & plddi
0.294
0.815
0.795
MMseqs2 cluster at seq-id=0.9 & piddt
0.862
0.294
0.885
0.895
structural diversity (?)
Foldseck TMscore=0.3 & scTM > 0.5
0.030
0.080
0.198
Foldscek T5score=0.5 & scTM > 0.5
0.500
0.440
0.545
Foldseek at TMscore=0.7 & scTM > 0.5
0,962
0.830
0.646
Foldseek T'Mscore=0.9 & scTM > 0.5
0.990
0.910
0.746
sequence naturalness (1)
ProGen2 ppl
8.11 2.08
9.15 2.77
4.33 +: 2.51 4.08 d: 2.00
sequence novelty (U)
MMscq2 search against PDB+swissprot
0,306
0.312
0.304
0.475
Table 12: Unconditional generation from the empirical length distribution.
Length
8CTM
SCRMSD
PLDDT
Length interval:
[100, 200,
500]
0.925 0.085
3.899
3.723
82.686
Training set (PDB+Swir) Swissprot) length dist. 0.929 0.086
3.967
3.257
83.698
clustering, we cluster samples with PLDDT > 70, while for foldseek clustering we cluster samples with scTM > 0.5. This quality threshold for diversity is inspired by MultiFlow, which more informative by avoiding diverse but messy sequences. Then, we divide the number of clusters by the total number of samples to measure the diversity. (2) sequence naturalness: we calculate perplexity as measure of naturalness with ProGen2-large (Nijkamp et al., 2022). (3) sequence novelty: we calculate novelty through sequence identity to the nearest neighbor the training set.
All models generate 100 samples per length the range of 100, 200, 300, 400 and 500 for evaluation, with the results demonstrated in the Tab. 1.. One particularly insightful observation is the distinet behavior of MultiFlow (w/ distillation) and DPLM-2 regarding structural diversity. Specifically, DPLM-2 exhibits greater diversity under strict TM-score thresholds (S 0.5). while MultiFlow achieves better diversity at higher TM-score thresholds (2 0.7). Combined with the average inner- TM scores (DPLM-2: 0.275, MultiFlow: 0.356). this suggests that DPLM-2 excels at generating diverse structures in terms of global topologies but exhibits limited structural variation within each cluster. This finding highlights a key limitation of the current structural tokenization approach: the loss of fine-grained structural variations, emphasizing the need for future improvements in this area. Additionally, DPLM-2 achieves the lowest ProGen2 perplexity, while its sequence identity to training data (0.475) is higher than that of DPLM and MultiFlow. This indicates that the sequences generated by DPLM-2 align more closely with the natural distribution.
F.2 UNCONDITIONAL GENERATION FROM THE EMPIRICAL LENGTH DISTRIBUTION
In our paper, we follow the setting in the MultiFlow and sample within length intervals in the unconditional generation, ensuring fair comparisons with previous models under the similar settings to better assess the strengths and limitations of our models.
Meanwhile, DPLM-2 is capable of generating proteins from the empirical length distribution. Specifically, we sample 2048 sequences with length sampled from the length distribution of PDB and SwissProt datasets. Tab. 12 demonstrates that DPLM-2 can generate highly plausible proteins from empirical length distribution, which is consistent with sampling with length intervals.
F.3 REPRESENTATION LEARNING EVALUATION WITH MORE BASELINES
We have added more recent strong baselines, such as GNN- based methods (e.g., GearNet), in addition to Tab. to make comprehensive comparison the representation learning tasks, as shown in Tab. 13. This demonstrates that DPLM-2 capable utilizing both protein structure and sequence generate more informative representations for series of downstream tasks.
24

---

Published as a conference paper at ICLR 2025
Table 13: Representation learning performance on various protein predictive tasks, comparing between DPLM-2 and more recent strong baselines. i means results are quoted from SaProt paper, while * means results are quoted from their respective paper.
Model
Themestability
HumanPPI
Metal Ion Binding
EC
GO
Subcellular
Deeplucc
Binary
3pearnan's
Azc (%)
Acc (%)
Emax
Fmax
2maax
2max
Rcc (%)
Acc (%)
SuPool (650M)
0.724
86.41
75.75
0,882
0.682
0.486
0.479
85.57
93.55
Neatret- (650M)
0.660
85 80
74.44
0.889
0.678
0.522
0.508
84.16
93.63
MII-ST (Ying et nl., 2022b)
0.694
75.5
75.08
0,807
0.633
0.375
0.322
78.96
91.76
GGaurett (Zhing et 2023)
0.571
73.60 23.86
71.26
0.871
0.641
0.181
0.176
69.45
89,18
GGarNet updated (Zhang et 2023)
0.681
CCupleNet[1 [1]
0.866
0.669
0.467
0.494
"CDCow [2]
0.820
0.654
0:453
0.479
"BSM2-650M-S [3]
"VABS-NET [4]
0,900
0
0.531
0.579
15MM--GaNNN--NR-MM
0.896
0.683
0.518
0.501
ESM2 (650M)
0.691
84.78
71.88
0.868
0.670
0.473
0.470
83.68
92.28
DXP (650M)
0.695
86.41
75.15
0.875
0.480
0.478
84.56
93.09
DPLM-2 (650M)
0.714
84.44
74.28
0.881
0.682
0.193
0.181
82.98
93.64
F.4 INVERSE FOLDING EVALUATION WITH MORE BASELINES
For inverse folding task, we mainly focus on the comparison with other multimodal generative models (MultiFlow, ESM3) in the Tab. 4. We have also added more recognized baseline methods in inverse folding evaluation, as shown in Tab. 14.
We conduct experiments on CATH 4.2 testset. We ob- serve that DPLM-2 is able to achieve close results with the strong baselines despite slightly lower scTM. To fur- ther improve scTM to bridge the last gap, there are sev- eral potential directions: (1) inverse folding SFT training: DPLM-2 conducts this task in a zero-shot manner while other systems purpose-bul built models, thus lrikntreetted SFT training could help as we have observed in folding; (2) better structure modeling includes introducing separate structure encoders for structure encoding and generation purposes, or hybrid tokenization for recovering the lost fine-grain structural variations, as discussed in the $B.4.
Table 14; Inverse folding performance
comparison between DPLM-2 and other baselines on the CATH 4.2 testset. t means results are quoted from their re- spective paper.
Model
AAR
SCT3
Knowledge Design (Ciso al. 2023)
60.77
"raDe IP al., 2023)
"MMDesign (Zheng & Ld, 2024)
54.88
IVEN IEE (Mao et al. 2023)
62.67
PiFfidd (Gao al.. 2022)
$1,66
"Bridge
2024)
ProsiiMPNN (Duuparas et 2022)
45.96
0.87
LM Design (Zheng et al., 20236)
54,41
0.88
DPI M-2 w/ argaux decoding
42.70
0.84
DPLM-2 w/ lemperature ameelled sampling
36,30
0.84
F.5 MOTIF SCAFFOLDING
Evaluation Pipeline. We evaluate DPLM-2 in sequence-based, sruuttre-based and co-generation ways. The overall illustration is shown in Fig. 8. We focus on the two aspects: overall quality and motif part consistency. The assessment of overall quality varies across different approaches. Specifically, (1) For sequence-based method, we only take the generated sequence and utilize ESMFold to obtain the predicted structure, and the PLDDT score provided by ESMFold is used to assess overall quality.
(2) For structure- truse method, we only take the generated structure, and then leverage ProteinMPNN to predict the sequence, followed by ESMFold to predict the structure, where overall quality is assessed by SCTM. (3) For co-generation method, we take both the generated structure and sequence, and predict structure given generated sequence with ESMFold, where s SCTM is calculated between generated structure and ESMFold predicted structure to evaluate overall quality. Considering that the ground truth motif structure is given, we only utilize the ESMFold predicted structure to calculate mot i F-MSSD
Result of Each Problem. Tab. 15 presents the result of each motif-scaffolding problem. DPLM- 2 achieves the best average success rate in each evaluation. Compared with ESM3, DPLM-2 shows better results in 12 problems in co-generation evaluation and 10 problems in sequence- based evaluation. Meanwhile, DPLM-2 outperforms RFDiffusion in 14 problems in structure-based evaluation. This demonstrates that DPLM-2 can achieve strong performance under various evaluation methods. We also find that taking the best result from 8 samples can bring significant improvement compared to 1 sample, especially in terms of success rate. In the co- co-generation evaluation, DPLM2 with sampling 8 times improves the success rate of most of the problems by large margin. We hypothesize that sampling eight times largely alleviates errors caused by randomness in the sampling process, thereby producing a more suitable scaffold for the given motif.
25

---

Published as a conference paper at ICLR 2025
Figure 8: scaffolding.
Sequence-basee based, structure-based and co-generation evaluation pipeline of motif-
sequence-based
prediction seqpredt structpred:
motif-preserving RMSD ( (ESMFold (aSSsoollsseeoonoooo
designability
PLOOT (EM )>70
structure-based
prediction
seqpred: x
structpreat
motif-preserving
RMSD (ESMFold PMPNN
designability
(EScore (WNoieootto(MM((ssttttttttmm)) structpre)=0.8
co-generation
prediction
seqpredt /
struCtpredi /
motif-preserving
RMSD (ESMEold (seqpres) [notif), (ntlu
designability
(iMscore(EMMoldd(seee
structpre)>*0.8
Table 15: Motif-scaafolding results of each problem. means best result from 8 samples.
sequence bused
structure based
generation
EvaDiff
DPLM
ESM3
DPLM2
*RFDillusion
1DPLM2
ESM3
DPLM2
*DPLM2
IBCF
0.00
0.00
0.01
1.00
0.23
0.01
0.05
IPRW
0.61
0.83
0.96
0.86
0.08
0.96
0.54
0.84
0.95
IQIG
0.00
0.00
0.02
0.03
0.00
0.00
0.03
0.02
0.05
IYCR
0.02
0.38
0.77
0.74
0.93
0.18
0.53
0.98
2KL8
0.04
0.08
0.11
0.47
0.88
0.94
0.11
0.57
1.00
3IXT
0.25
WJHW
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
4ZYP
0.40
SIUS
0.00
D.00
0.00
0.00
0.012
D.00
0.00
0.00
0.00
STPN
0.00
0.00
0.61
0.00
STRV long
0.00
0.00
0.19
0.00
0.37
0.08
0.19
0.00
0.07
STRV med
0.00
0.00
0.16
0.03
0.24
0.07
0.16
0.02
0.19
STRV_short
0.00
0.00
0.01
0.07
0.04
0.10
0.01
0.03
0.11
SWN9
0.00
0.00
0.02
0.00
0.00
0.20
0.00
0.00
0.00
SYUI
0.00
0.00
0.02
0,00
6e6R_long
0.01
0.65
0.07
0.91
0.86
0.92
0.04
0.78
1.00
6EGR med
0.03
0.94
0.24
0.93
0.89
0.77
0.97
6E6R short
0.07
0.87
0.09
0.86
0.39
0.78
0.06
0.64
0.99
6EXz_long
6EX7_med
0.00
0.00
0.31
0.66
0.49
0.63
0.31
0.55
0.96
6bxz.sbort
0.00
0.66
0.39
0.41
0.28
0.58
0.87
7MRX long
0,00
0.02
0.36
0.23
0.09
0.32
0.37
0.20
0.73
7MRX med
0.00
0.31
0.65
0.28
0.11
0.59
0.22
0.70
7MRX short
0.00
0.34
0.68
0.26
0.02
0.41
0.74
0.24
0.88
rate
7/24
11/24
21/24
18/24
20/24
20/24
20/24
18/24
19/24
avg.
0.35
0.40
0.53
G DISCUSSION ON THE CONDITIONAL INDEPENDENCE ASSUMPTION
In the Eq. 2, we make a conditional independence assumption between the protein structure and sequence. However, conditional independence is special assumption made by DPLM-2, is fundamental assumption made by diffusion models in general and their multimodal extensions, derived from the nature of their forward and backward processes. Previous theoretical studies on diffusion models have shown the convergence between generated samples distribution and data distribution g guaranteed under such conditional independence. In this paper, we have empirical between co- co-generated structures and sequences (c.g..
evidence showing the consistency/compatiiie scTM for co- co-generation), and we believe a mathematical proof this is beyond the scope this paper and can refer to the established theoretical results diffusion. Nevertheless, do love to elaborate on our thoughts and understanding of this as follows.
Conditional independence in diffusion models in general. Conditional independence over the ele- ments of high-dimensional data, i.e., 1x--18 (x:) 1 (o(xx.1.11) (xx) is prevailing assumption in diffusion probablistic models, both continuous and discrete variants, thanks to their iterative nature of probabilistic modeling, For example, in continuous diffusion models for vision generation, the denoising networks learn to reconstruct a denoised image at each timestep t by simultaneously and independently operating over all pixels conditioned on the previous noisier pixels of the image
26

---

Published as a conference paper at ICLR 2025
and the current timestep (or equivalently noise level) t. So for discrete diffusion, where discrete diffusion for text or protein sequence treats tokens of a sequence of x;-1 independently given Xt- Several recent works have established the theoretical foundations both continuous diffusion (Chen et al., 2023) and discrete diffusion (Chen & Ying, 2024; Zhang et al., 2024), showing that there are theoretical guarantees of the convergence of the generated sample distribution of the diffusion models and the data distribution, which means that a well-learned diffusion models can preserve the statistical structure of the data (in other words, the consistency between the elements x = (z1,..,,Zd (d) of the generated samples).
the convergence analysis of
Conditional independence in multimodal diffusion models. Multimodal diffusion mod- els aim to accommodate two or more modalities using a unified models. In this case, conditional independence between modalities is generally made 1t- x1, Y:) ( X,, Yt-)) Po [o() Xx,, For instance, UniDiffuser (Bao et al., 2023) is a multimodal continuous diffusion model that handles text and image modalities independently at each timestep, conditioned
the predictions from the previous timestep. Multiflow (Campbell et al., orientation, and
2024), on the other hand, factorizes protein data into three modalities--translatiinn conditional independence. It establishes a multimodal diffusion//foow-
amino acid type-assumin based model by combining three types of stochastic processes over Euclidean, SO(3), and categorical spaces for these modalities. In DPLM-2, we adopt unified discrete diffusion approach where structure tokens and amino acid tokens are treated as conditionally independent. While theoretical guarantees for the convergence of mixture diffusion processes are still under-explored, existing discrete diffusion theory (Chen & Ying. 2024) ensures that a vell-trained DPLM-2 can converge to the tokenized sequence tokens.
data distribution, supporting consistency between structure and
Additionally, theoretical studies on non-autoregresssvv Transformers for text generation, which are akin to masked discrete diffusion, indicate that the learning difficulty of such models can be evaluated through conditional total correlation. This dataset-dependent measure captures the discrepancy between a joint data distribution and a fully factorized distribution under conditional independence (Huang et al., 2022). These studies suggest that simplifying the target data, for instance, by using tokenized structure instead of 3D coordinates, reduces conditional total correlation, thereby enhancing both learning and generation quality.
Given the consistency of structure tokens and amino acid can be ensured to learn in DPLM-2 by previous theoretical results, the overall structure and sequence consistency can be achieved with decent structure tokenizer, such as the one proposed in this paper, which accurately maps structure tokens to their atomic coordinates.
H RELATED WORK
H.1 PROTEIN LANGUAGE MODELS
There is growing interest in developing protein LMs at the scale of evolution, such as the series of ESM (Rives et al., 2019; Lin et al., 2022). TAPE (Rao et al., 2019), ProtTrans (EInaggar et al., 2021). PRoBERTa (Nambiar et al.. 2020). PMLM (He et al., 20.1). ProteinLM (Xiao et al., 2021), PLUS (Min et al., 2021), Adversarial Masked LMS (McDermott et al., 2021), ProteinBERT (Brandes et al., 2022), CARP (Yang et al., 2022a) in masked language modeling (MLM) paradigm, Prot- GPT2 (Ferruz et al., 2022) in causal language modeling paradigm, and several others (Melnyk et al., 2022a; Madani et al., 2021; Unsal et al., 2022; Nourani et al., 2021; Lu et al., 2020; Sturmfels et al., 2020; Strodthoff et al., 2020). These protein language models exhibit remarkable generalization ability on various downstream tasks and be able to capture evolutionary information about secondary and tertiary structures from sequences alone. Mcanwhile, recent study shows these models' po- tency in revealing protein structures (Lin al., 2022), predicting the effect of sequence variation on function (Meier et al., 2021). ) antibody infilling (Melnyk et al., 2022a) and many other general purposes (Rives et al., 2019). Simultancously, Verkuil et al. (2022) demonstrate that the large scale protein LMs can generate de novo proteins by generalizing beyond natural proteins, both theoretically and experimentally validating their hypothesis exhaustive detail, in which protein LMs demonstrate competency in designing protein structure despite being exclusively trained on sequences.
H.2 PROTEIN STRUCTURE GENERATIVE MODELS
Diffusion models have become popular tools in structural biology for protein generation, and their utility has been demonstrated across a range of generative tasks in recent years. Trippe et al. (2022), along with others, have introduced several diffusion model variants, each with its unique
27

---

Published as a conference paper at ICLR 2025
approach. For instance, while some models focus generating the protein backbone by diffusing over protein coordinates, others, such as those proposed by Wu et al. (2022b), target inter-residue angles. Lin & AlQuraishi (2023) and Yim et al. (2023) have developed models that handle both the position and orientation of residue frames. RFDiffusion (Watson et al., 2023) is a model that assists in designing protein structures for specific functions, such enzymes. is versatile in protein design and has been used to create therapeutic proteins, with some designs being confirmed in the laboratory. ProteinSGM (Lee al., 2022) model that uses 2D matrices, which represent the distances and angles between protein parts, to create 3D protein structures for novel protein designs. FoldingDiff (Wu al., 2022a) a model that generates protein sequences expected fold into specific structure. These sequences are verified with prediction tools, although they have not been experimentally confirmed yet. Chroma (Ingraham al., 2023) model designed for creating large proteins and protein complexes, considering various constraints like distances and symmetry. It transforms a collapsed polymer into protein backbone and sequence more quickly than older methods, thereby allowing for the efficient generation of large structures. Multiflow (Campbell et al., 2024) develop mulitmodal flow matching for protein cructtre-sequente co-generation (Jin et al., 2021; Shi al., 2022). ProtPardelle (Chu et al., 2024) propose all-atom generative approach for co-design.
28


Reviews:
[{"review_id": "iR1XwUrD8b", "reviewer": "Reviewer_f8By", "summary": "This paper introduces DPLM-2, a multimodal discrete diffusion protein language model that can simultaneously generate both protein sequences and their 3D structures. The model extends the DPLM protein language model by incorporating structural information through a lookup-free quantization-based tokenizer that converts 3D coordinates into discrete tokens. DPLM-2 is trained on both experimental structures from PDB and synthetic structures from AFDB-SwissProt.", "questions": "1. The model does not learn the distribution of protein lengths. Have you tried to overcome this limitation?\n\n2. The ablation results presented in 4.1.3 and table 3 are controversial. Could you please clarify the procedure, how many samples was used for evaluation and so on? \n\n3. On page 9, line 442 is stated that DPLM-2 adopts argmax decoding. In the original DPLM paper argmax did not work. Can you elaborate on this?\n\n4. The experiment on DeepLoc is not described in sufficient detail. Why you chose to use only one tiny dataset? Could you provide experiments that show the described catastrophic forgetting issue, which is of high importance?\n\n5. One of the main claims of the paper states that the co-generation guarantees consistency between structure and sequence. This is a strong statement that requires strong evidence. However on line 223 the assumption of conditional independence is made. Can you provide a rigorous mathematical proof that garantees such consistensy?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written.\n\n2. The model achieves simultaneous structure-sequence generation without requiring a two-stage training approach.\n\n3. The experiments on learning structure tokenization are valuable for the community.", "weaknesses": "1. The DPLM-2 model is trained starting from the weights of DPLM model. As evident from e.g. Table 2 (poor diversity) and Table 5 (high AAR), DPLM suffers from mode collapse. Training DPLM-2 starting from such a checkpoint  leads to severe mode collapse in DPLM-2. Generating the same result over and over again makes a generative model useless. Training both models from scratch rather than finetuning might help and provide the insight on how adding structural information to the DPLM architecture affects the generation ability. Even better would be a thourough ablation study of the training procedure.\n\n2. Authors compare DPLM and DPLM-2 only with EvoDiff on unconditional sequence generation. Adding more baselines of different architectures (AR transformers, CARP, continuous diffusion, flow-matching, etc.) would greatly improve the work.\n\n3. There are some issues with model evaluation. First, the model is evaluated only structurally, but it is langugae model after all. Using sequence-based evaluation metrics, including sequence clustering for the diversity should benefit the soundness of the work. Second, it is not clear, why the Authors use pdb-TM (claculate TM-score against PDB), if the model is trained also on the synthetic data. Third, the designability metric used in this work evaluates the consistensy between the generated structure and the prediction of ESMFold on the generated sequence. It does not measure protein \"quality\".\n\n4. Authors compare DPLM-2 on inverse folding task with weak baselines. Adding recognized IF models would greatly benefit the work. The same goes to other tasks, e.g. representation learning.\n\n5. The dataset preparation included random cropping. It is not clear if it has a detrimental effect on the model behaviour.\n\n6. The paper lacks analysis on the trained structural tokens. Additional exploration of the interpratability of the tokens themselves and untilization of the codebooks would greatly benefit the paper. Do the tokens correspond to some local environments in structure, or are they just abstract entities?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QliHgzQ8M", "reviewer": "Reviewer_4A34", "summary": "The paper presents DPLM-2, a multimodal protein foundation model designed to represent and generate protein sequences and structures. DPLM-2 introduces a token-based approach to protein structures, converting 3D backbone coordinates into discrete structure tokens. The model then processes structure token sequences alongside amino acid sequences, with aligned position encodings to reinforce residue-level correspondence between them, and is trained using a denoising objective.\n\nAddressing the limitations of previous models like Multiflow, which lack sequence-based pretraining to capture co-evolutionary relationships, DPLM-2 leverages pretraining on unlabeled sequences and finetuning on structures, to learn the joint distribution of sequences and structures. This allows DPLM-2 to capture both modalities effectively and enables it to model joint, marginal, and conditional distributions.\n\nAdditionally, DPLM-2 demonstrates competitive performance across tasks such as unconditional generation, folding, inverse folding, and motif-scaffolding, showing its capability as a comprehensive multimodal protein modeling tool.", "questions": "- Sequence tokens use a smaller vocabulary than structure tokens. Are the corresponding structural embeddings in DPLM-2 trained from scratch during fine-tuning?\n- While DPLM-2 enables flexible generation, what are the trade-offs in structural invariance when using structure tokens instead of 3D coordinates?\n- In Table 2, what is meant by DPLM-2 (seq → structure) or (structure → seq)? If these indicate modality-by-modality generation, could you clarify how this is implemented?\n- For the protein representation learning evaluation, it might be useful to include a broader range of baselines, such as GNN-based models, for a more comprehensive comparison.\n- In Section 3.3, you state, “Recent efforts have applied this approach to protein structure coordinates (Van Kempen et al., 2024; Liu et al., 2023; Gao et al., 2024; Lu et al., 2024). This allows language models to better learn the composition of local structural elements. However, how to learn an effective structure tokenizer remains an active research question.” Given the similarity of your method to prior approaches, particularly with the use of LFQ as in Gao et al., 2024, could you elaborate on how your method contributes to addressing this active research question?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The work's originality lies primarily in (1) the DPLM framework that combines sequence tokens and structure tokens with a lookup-free quantization (LFQ) VAE, and (2) the use of sequence pretraining followed by LoRA fine-tuning. These contributions provide valuable insights to the field. The authors also address relevant concurrent work effectively.\n\n- Clarity: The paper is generally easy to follow, though some sections would benefit from further explanation and technical details.\n\n- Empirical Performance: DPLM-2 demonstrates strong empirical performance across a wide range of generative tasks, including sequence-structure co-generation, folding, inverse folding, and motif-scaffolding. The paper also includes valuable ablation studies on sequence pretraining and data augmentation, offering insights into the model’s effectiveness and robustness.", "weaknesses": "-Missing Details: Key details are missing in certain sections. For instance:\n\n     (1)A core contribution is the DPLM-2 model itself, but essential details, such as how sequence and structure tokens are combined and how the structure tokenizer is trained, are not included in the main text. Moving these details from the appendix to the primary text, with clear explanations including the frozen pretrained structure encoder and the sequence prediction head on top of the structure decoder, would significantly improve clarity.\n\n     (2)The distinct noise level for each modality could be better explained, as this aspect is currently underdeveloped.\n\n     (3)In Section 4.2, the authors mention that performance can improve with supervised fine-tuning using a folding objective; however, the paper lacks details on this fine-tuning process.\n\n- Clarity and Consistency Issues: Minor inconsistencies reduce clarity, such as inconsistent bolding of best results in Tables 2 and 6. In Table 4, DPLM-2 is presented as performing well in zero-shot folding, yet its RMSD is high compared with ESM3, PVQD, and ESMFold, achieving competitive performance only after fine-tuning (SFT). Additionally, clarifying the presentation of mean and median values could help with data interpretation.\n\n- Mixed Results: The results on certain tasks are not fully convincing. In particular, while DPLM-2 exhibits high scTM in unconditional protein generation, it does not achieve lower scRMSD compared to other methods, indicating potential limitations in generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hs0SFi50YQ", "reviewer": "Reviewer_FqCq", "summary": "The paper presents a multimodal diffusion protein language model that integrates information from both sequence and structure modalities. The model is built upon a pre-trained DPLM model, using LoRA technology to extend the original sequence-only model’s capability to process structural knowledge. This approach not only reuses the knowledge embedded in the existing sequence-based protein language model but also reduces training costs and expenses. Compared to traditional autoregressive language models, the diffusion language model offers more flexible generation capabilities, making it better suited for modeling protein data with extensive non-unidirectional semantic dependencies. Overall, this work represents a natural extension and generalization of the DPLM model, exploring the performance of such diffusion language models on multimodal protein data and demonstrating the potential advantages of multimodal protein language models.", "questions": "1. **Regarding structure tokens:** Figure 2B shows that most structure tokens are concentrated in alpha helix and beta sheet regions, with lower density in loop regions. However, in line 409 of the paper, you state that \"loops are highly flexible.\" Could this pose a potential issue? More variable regions typically contain richer and more diverse information, which may require more structure tokens to model this diversity effectively. Yet, your experimental results do not align with this. Moreover, your results in Figure 4B also indicate that the model’s designability is notably worse in regions with a higher proportion of loops. Could one possible reason for this issue be that the structure tokens in loop regions are not well-learned, resulting in weaker modeling capability?\n\n2. **Regarding the pLDDT metric:** Table 2 shows that MultiFlow (retrained on our training data) and DPLM-2 (co-generation) perform similarly on scTM and scRMSD metrics but diverge significantly on the pLDDT metric. A similar trend is observed when comparing Figures 3A and 3C. In Figure 3A, scTM and scRMSD metrics remain relatively stable as sequence length increases, with scRMSD even increasing for sequences of length 500. However, Figure 3C shows a notable upward trend in pLDDT as sequence length increases (from 100 to 300), without a significant drop in pLDDT even for sequences of length 500. These phenomena indicate that the scTM and scRMSD metrics convey somewhat contradictory information to the pLDDT metric. Generally, a significant drop in pLDDT should correspond with a decrease in scTM and an increase in scRMSD. My question here is whether this inconsistency might indicate that the pLDDT metric is unreliable. I raise this because, in previous experiments, we observed that pLDDT, being predicted by a neural network (e.g., ESM-2) trained on natural protein datasets, sometimes fails to handle model-generated proteins that deviate significantly or exhibit severe irregularities, leading to inflated pLDDT values. This phenomenon is quite common in unconditional generation. Therefore, I believe using a more diverse set of metrics is essential. However, you did not provide evaluation results for MultiFlow (retrained on our training data) on the Novelty and Diversity dimensions, which makes it harder to assess the model’s performance on this part of the task.\n\n3. **Regarding scTM and scRMSD metrics:** In Table 2, under the unconditional backbone generation task, DPLM-2 achieves the highest scTM metric while also showing a significantly worse scRMSD metric, both in terms of mean and variance, which are at high levels. Is this phenomenon somewhat unusual? Given that these two metrics should have a certain level of correlation.\n\n4. **Regarding the inverse folding task:** Comparing the performance of ESM3 and DPLM-2 in Table 5 reveals a pattern where DPLM-2 often exhibits a higher AAR while frequently showing a lower scTM. Could this possibly be due to biases in the protein structure prediction model (ESMFold) or because the sequences generated by DPLM-2, despite having higher ARR, may deviate further from the target protein’s data distribution?\n\n5. **Regarding repetition:** In practice, we observed that diffusion protein language models tend to generate a large number of repetitive amino acids. I noticed that DPLM employed a resampling scheme to address this issue (code link: [here](https://github.com/bytedance/dplm/blob/5545c6d4166f515b4eb66ada41d0ab3178dfe6ca/src/byprot/models/lm/dplm.py#L279)). Therefore, regarding the DPLM-2 model, I would like to know whether it also encounters similar repetition issues. If so, what approach did you adopt to resolve it? If not, I would like to know what prevented this issue in DPLM-2.\n\n6. **Regarding VQ-VAE:** I am curious about the vocabulary usage rate of VQ-VAE when the vocabulary size is 1024. I noticed a significant performance difference between VQ-VAE and LFQ in this case, so I wonder if the vocabulary usage rate might be a contributing factor to this issue.\n\n7. **Regarding catastrophic forgetting:** I would like to confirm whether DPLM-2 in Table 7 still utilizes structural pre-training, given that neither of these models used large-scale sequence pre-training. If structural pre-training was indeed applied to DPLM-2, then there are two differences between DPLM and DPLM-2: one is the difference in model architecture, and the other is that DPLM-2 was pre-trained on structural data. With these two variables present, how can we determine that the performance decline in DPLM-2 is due to catastrophic forgetting?\n\n8. **Regarding the structural information:** Similar to ESM-3, you also use discrete structure tokens to represent structural information. This approach offers scalability benefits for the model, yet it also limits the model’s ability to capture finer structural details accurately. This trade-off represents an important challenge in the current field of multimodal protein language models. I am interested to know whether you would consider any methods to mitigate this issue when applying DPLM-2 to more structure-related tasks, as you described in Section 5.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is the first multimodal diffusion protein language model, which effectively explores the application of such models in multimodal scenarios. Considering the potential of diffusion language models, I believe this is a highly meaningful research question.\n\n2. The paper has a clear logical structure, and its content is straightforward and easy to understand.\n\n3. The paper demonstrates the performance of DPLM-2 across a variety of tasks, providing substantial and comprehensive content that analyzes the model’s performance from multiple perspectives.", "weaknesses": "1. The main results reported in this paper are based on unconditional generation tasks; however, some evaluation metrics may have inherent limitations.\n\n2. The model in this paper is trained on a relatively small protein structure dataset, lacking exploration on larger-scale structural data. I believe that extending multimodal protein language models to larger protein sequence and structure datasets could further unlock the potential of this approach.\n\n3. There are some minor spelling and formatting errors: the equation on line 197 is missing a right parenthesis, and $ b_{i}(t) $ introduced in equation (1) (on line 192) is undefined in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)