Elucidating the Design Space of Multimodal Protein Language Models
Authors: Cheng-Yen Hsieh
Xinyou Wang
Daiheng Zhang
Dongyu Xue
Fei Ye
Shujian Huang
Zaixiang Zheng
Quanquan Gu
Venue: ICML 2025 spotlightposter (2025)
Keywords: Multimodal protein language model
structure tokenization
improved structure generative modeling
bitwise modeling
hybrid data-space modeling
structure-aware architectures
representation learning
multimer
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We elucidated the design choices of multimodal protein language model to address its limitations on structural modeling.

Abstract: Multimodal protein language models (PLMs) integrate sequence and token-based structural information, serving as a powerful foundation for protein modeling, generation, and design. 
However, the reliance on tokenizing 3D structures into discrete tokens causes substantial loss of fidelity about fine-grained structural details and correlations. 
In this paper, we systematically elucidate the design space of multimodal PLMs to overcome their limitations. 
We identify tokenization loss and inaccurate structure token predictions by the PLMs as major bottlenecks.
To address these, our proposed design space covers improved generative modeling, structure-aware architectures and representation learning, and data exploration. 
Our advancements approach finer-grained supervision, demonstrating that token-based multimodal PLMs can achieve robust structural modeling.
The effective design methods dramatically improve the structure generation diversity, and notably, folding abilities of our 650M model by reducing the RMSD from 5.52 to 2.36 on PDB testset, even outperforming 3B baselines and on par with the specialized folding models.
Project page and code: https://bytedance.github.io/dplm/dplm-2.1.

Elucidating the Design Space of Multimodal Protein Language Models
Cheng-Yen Hsieh
Xinyou Wang" 00 Daiheng Zhang i 10 Dongyu Xue 9 Fei Ye Shujian Huang
Zaixiang Zheng
Quanquan Gu
Abstract
Jumper et al., 2021)) that fail to capture the interplay be- tween them. This limitation hinders the ability to jointly model, understand, and generate proteins in unified frame- work, which is essential for tasks like protein design, folding, and functional annotation (Lee et al., 2022b; Senior et al., 2020; Lees et al., 2011).
Multimodal protein language models (PLMs) in- tegrate sequence and token-based structural in- formation, serving
powerful foundation for protein modeling, generation, and design. How- ever, the reliance on tokenizing 3D structures into discrete tokens causes substantial loss of fidelity about fineggrained structural details and corre- lations. In this paper, we systematically eluci- date the design space of multimodal PLMs to overcome their limitations. We identify tokeniza- tion loss and inaccurate structure token predic- tions by the PLMMs as major bottlenecks. To ad- dress these, our proposed design space covers improved generative modeling, structure-awar architectures and representation learning, and data exploration. Our advancements approach finer- grained supervision, demonstrating that token- based multimodal PLMs can achieve robust struc- tural modeling. The effective design methods dramatically improve the structure generation di- versity, and notably, folding abilities of our 650M model by reducing the RMSD from 5.52 to 2.36 on PDB testset, even outperforming 3B baselines and on par with the specialized folding models. Project page and code: byt edance. github. io/ / dplm/ dplm-2.1.
Recent efforts in multimodal protein language models such as ESM3 (Hayes et al., 2024) and DPLM-2 (Wang et al., 2024a) have demonstrated the potential of integrating se- quence and structure within a single language model as unified generative framework. In particular, DPLM-2 is a mulitmodal extension of diffusion protein language model (DPLM, Wang et al., 2024b) with discrete diffusion framework (Austin et al., 2021), which naturally aligns with the discrete nature of protein sequences, enabling it to bene- fit from large-scale pre-training on sequence databases crucial factor for accurate structure prediction (Lin et al., 2022). Beyond sequence modeling, DPLM-2 extends its capabilities by tokenizing 3D coordinates into discrete to- kens, thereby enabling direct language modeling of both modalities, hence comprehension and generation of them.
Despite the success, the structure tokenization process intro- duces structural information loss oescuring fine-grained geometric relationships critical for accurate protein model- ing. Consequently, even such state-of-thhe-an multimodal PLMs struggle to generate biologically plausible structures for complex tasks like structure folding (Lin et al., 2022) or motif scaffolding (Watson et al., 2023; Yim et al., 2024), where precise structural correlations are crucial. The loss of nuanced variations due to tokenization also degrades the structure diversity in unconditional generation.
Introduction
Proteins are the molecular machinery of life, encoded by amino acid sequences that fold into intricate three- dimensional structures to perform their biological functions. Existing approaches often treat sequence and structure as separate modalities, relying on disjoint models (e.g., ESM for sequences and AlphaFold for structures (Lin et al., 2023;
In this paper, we systematically explore the key pitfalls and the design space token-based multimodal protein language models to bridge their limitations on structural modeling. In addition the structural information loss from tokenization, the inaccuracies in language model's capability in structure (tokens) prediction, which could not be simply resolved by improving the reconstruction accuracy. We find that mndex-based structure tokens as supervised labels ignore correlations between semantically similar structure tokens, making the learning process particularly challenging. In response, we build upon DPLM-2 to advance the design
identify the primary challenges as
'Equal contribution Core Contributor Project Lead O School of Computer Science. Nanjing University Deept. of ECE, Rutgers University 9 ByteDance Seed (this work was done during Xinyou Wang and Daiheng Zhang's internship at ByteDance Sced), Corre- spondence to: Quanquan Gu <quanquan. gu @ bomm
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Elucidating the Design Space of Multimodal Protcin Language Models
space spanning improved generative modeling, structure- aware architectures, representation learning, and data ploration (ref. Table 12). We achieve a finer-grained su- pervision through bitwise discrete modeling and a hybrid approach for data-space modeling. While these methods effectively guide the design of supervision targets, language model-based architectures still lack geometric inductive biases and structural learning objective. To mitigate this, we introduce geometry-aware modules and representation alignment techniques to refine the modeling of higher-order relationship between residues, which is essential as evi- denced in protein folding (Jumper et al., 2021). As existing multimodal PLMs are often trained solely on single-chain proteins, we explore the effects of multi-chain proteins (mul- timer). which introduces richer structural interactions cru- cial for robust modeling.
guage models. The latter is our main focus in this paper, and we will elaborate on this as follows.
2.1 Multimodal generative learning for proteins with language models and structure tokenization
Language models (LMs), parameterized by large-scale Transformers (Vaswani et al., 2017) have become the de facto choice dominating different domains with scalable and performing expressiveness (OpenAI, 2023). Among them, protein LMs have been serving as one of the Al foun- dation for protein sequence learning (Rives et al., 2019; Lin et al., 2022) and generation (Nijkamp et al., 2022; Hayes et al., 2024).
DPLM. Diffusion protein language model (DPLMM, Wang et al., 2024b), in particular, shows excelling performance in both generation and representation learning of protein sequences, and even structures thanks to its recent multi- modal extension D (Wang et al., 2024a). The family of DPLMs grounded in absorbing discrete diffusion frame- work (Austin et al., 2021; Zheng et al., 2023a), which is char- acterized by forward and backward Markov process. Let Cat (x: p) be a categorical distribution on protein sequence y parameterized by a vector on (IV) 1 1)-dimensional probability simplex. The forward process of discrete diffu- sion defines a Markov process governed by the transition ker- nel q(x(*)
We summarize our main contributions as follows:
We conduct a comprehensive study revealing key pit- falls in structure token-based multimodal protein lan- guage models, and systematically elucidate their design space for robust structural modeling.
Utilizing improved approaches such as bit-wise dis- crete modeling offers finer-grained supervision, signifi- cantly improving structure generative capability. Introducing representation-lvee learning and architec- tural innovations infuses geometric inductive biases and effectively refines generation diversity.
((x("|x(())) (t) = Cat (x("; (xx(-=1) + (1 (noise) that gradually perturb the data x q(x) into station- ary distribution xT) Anoise- The learned backward pro- cess pa (=(x(==)(xx(0 reversely denoises the x(T) towards the data distribution x, which is typically optimized by the variational bound of the log-likelihood (Ho et al., 2020). The learning objective of absorbing diffusion can be sim- plified into weighted cross-entropiess, resembling masked language modeling arbitrary noise levels:
We find that multimer and monomer modeling are deeply interconnected and leveraging multimer data advances the structural modeling for both single and multi-chain proteins.
Our design methods allow multimodal PLMs to achieve robust structural understanding, improving the folding RMSD from 5.52 to 2.36 on the PDB date dataset, outperforming 3B folding baselines with only 650M parameters.
T. -x) KL
KL [[((x(t-)1)) (1tttt1))x)0 (0(0 x) (xx 1) (x)]
1 (10)) Eisigt bi()) log (o(o))((()))
2 Revisiting Multimodal Protein Language Models: Capabilities & Constraints
where (1)) is a weighting coefficient induced from the spe- cific noising schedule. For inference, DPLM is able to gener- ate amino acid sequences by the reverse iterative denoising process a mask-predict manner, which starts from an all- mask sequence and iterates towards synthesized sequence. At time f, it first generates x() from pe(-|x(")) then a less
The aim of generative protein modeling is to estimate the underlying distribution prot ~ ((prot) of all associated moralities of the protein data by learning a probabilistic model po (prot). Here prot- = (r1, "r.., denotes protein with L residues, where each residue r. (St, X.,) is represented by two major modalities, i.e., 8, E {0, Si is categorical variable for its amino acid type in S = {I...., 20). and I; E R Nanm x3 is the real-value Cartesian coordinates of its residue atoms (we only consider backbone atoms herein, ie.. [N, Ca, C, O] with Naumss = 4). Namely, po(s, x) = Po (31, $2....................... SL: FT, T2, .L)
noisy x (t-1) is sampled by g(- (()) x(() (0)
x x())).
DPLM-2: A multimodal extension of DPLM. To facilitate structure learning in language models, DPLM-2 (Wang et al., 2024a) extends DPLM by introducing token-based latent representation for protein structure. This is achieved via two-stage approach: (1) a structure tokenizer firstly learns to convert x E Lx X3, the 3D coordinates of the protein backbone into a discrete structure token sequence, denoted as z = (21, b2....,......................... 2L) E [0.../Z|}' where each token 2; represents a local structural element of the i-th
Mutiimdal generative approaches that jointly models struc- ture and sequence can be mainly categorized into two paradigms, i.e., Atructure-enttede diffusion//low-baat models (Campbell et al., 2024b) or sequence-centered lan-
2

---

Elucidating the Design Space of Multimodal Protein Language Models
A Multimodal PLMs w/ structure tokenization
B Improved structure prediction for DPLM-2
(Al) structure takenber
(81) learning
quantization residuals
(B2) bit-wise modiling of structure tokens
tokertaation comeerts strecture into Usseno strict cokens
seruct. decoder
scruct docodor
SErUCt LDAUT
Spe New
ercoe-weriyy
ditccoo
DPLM-2
Seruct
DPLM-2
D
OOoOO NKTVRQERLK
Nichooed
NKT VAQEBLI
hite harad dicress tokans
(83) composing struct tokenizer & binuage model anables data-space structure modaling
(A2) sequence-baser based language modeling wi tokenized structure
Spewod
C Improved structure-aware architecture and representation learning
decoder
Struct decoder
index hovel
OOOOO
PKTURDERUK
reprosentation
sinfe
pet rept.
Folding Model
PNTURQERLH
sarpping mn Fiw bared anpler
RM
crow-cntregy
DPLM-2 w/ sequence-based based architecture
DPLM-2
OOOOO NATURQELLK
GLLDPLM w/ structure aware modules
abornoo stret tchara
scruct encoder
JEPAE schora
#TBR0-Is di zokora
K, E
Figure 1. Illustration of multimodal protein language models with structure tokenization, and our proposed improved approaches. (A): Structure tokenizer and multimodal protein language modeling within sequence- saguence-based architecture. We use DPLM-2 (Wang et al., 2024a) as case study herein; (B) Improved structure prediction approaches for DPL, including quantization residual learning, bit-wise discrete modeling and combining structure encoder, decoder and language model as a structure denoising model for data-space structure modeling. (C) Improved structure-an aware model architecture and representation learning aligning to folding models.
residue and |2] is the codebook size; (2) given tokenized structure, D-LM-2 processes mulitmodal input, and then performs joint language modeling of the structure token sequence z with the corresponding amino acid sequence s for the same protein. The training objective of DPLM-2 hence becomes:
2.2 The pitfalls of modeling over tokenized structures
Albeii being the key enabler to multimodal protein language models, using discrete structure tokens to represent struc- tural information also limits the model's ability to capture structural details accurately. This trade-off represents an im- portant challenge in the current field of multimodal protein language models. To understand this, we conduct in-depth study regarding structure tokenization and structure predic- tion by language models. We highlight our (O)bservations along with their implications as follows:
Tt = Ex(x,x) E, (
log po (oggott))))))
Structure tokenization. Any vector-quantization based approach can be studied for tokenizing protein atomic struc- ture into structure tokens (Van Den Oord et al., 20)7 Specifically, DPLM-2 employs an LFQ-based structure to- kenizer (Yu et al., 2023), which is the state-of-the-arr ap- proach for visual tokenization. This LFQ tokenizer can be summarized follows:
(O1): Structure tokenization results in information loss. Vector quantization converts latent features of continuous structure tokens (Zcoat) from the structure encoder into dis- crete structure tokens (Zqqum)), discarding residual informa- tion (Zeout Zquant.. As shown in Table 1, however, apply- ing quantization significantly amplifies reconstruction er- rors (RMSD 1.31 / 1.98 & TMscore 0.97 > 0.93). This indicates that quantizing continuous tokens into discrete tokens inevitably results in loss of fidelity hence detailed structural accuracy. This suggests that learning to recover the lost caanduals-pattiullaaie as a refinement ctup-could enhance structure prediction accuracy.
X encoder,
dimension-wise
decoder
Zeont
Zquant
+ Zindex
X,
quantize
where (1) structure encoder encodes backbone 3D coordi- nates x E R RxxNNxsxx3 x3 into invariant features as continuous structure tokens Zeom E LxD, (2) an LFQ module quan- tizes Zoont independently dimension-wise into bits-based (binary) discrete structure tokens Zquant E (-1,+114
Table 1. Effects of feature quantization structure tokenizer reconstruction.
which can be converted to decimal index- discrete [k] structure tokens Zindex Σ 1( (zquant > 0) 2*-1 E
Latent feature
Struct taken type
Reconstruction
RMBDL
TMe core"
{0),, and (3) a structure decoder reconstructs 3D coordinates from the discrete tokens.
Zonni
(pre-quantized) continuous token
1.3127
0.9733
(quamtized) discrete token
1,9806
0.9385
3

---

Elucidating the Design Space of Multimodal Protein Language Models
(O2): High reconstruction accuracy does not guaran- tee better structure generative performance in language models, while a significant gap remains in between. We compare the impact of different protein structure tokenizers on reconstruction and generation tasks. Specifically, we se- lect tokenizers from DPLM-2 and ESM3, training separate DPLM-2 variants with the same architecture but using their respective structure token codebook. These models are eval- uated on the CAMEO 2022 test set for both reconstruction and protein folding performance. As shown in Table 2, the FSM3 tokenizer achieves superior reconstruction accuracy (RMSD: 0.72, IMscore: 0.99), outperforming the DPLM-2 tokenizer. However, the model trained with the DPLM-2 tokenizer's codebook exhibits stronger protein folding per- formance. This suggests that while reconstruction accuracy sets an upper bound on generation quality, the substantial gap between the two highlights the critical role of the lan- guage model's generative capability in structure prediction. This suggests that, given that mild improvement in recon- struction do not necessarily translate into better generation, greater emphasis should be placed on improving structure- aware generative modeling and architectural design.
Table 3. Language model structure token prediction accuracy.
Index-based vs. bits based evaluation on structure folding.
Struct Taken Acct
Stract Eval Metric
Model
Testset
index
bil
RMSDJ
[Ksuuref
DPLM-2 2 index-based
CAMEO 2022
0.0864
0,7720
7,7025
0.7936
PDB spli
0.1188
0.7932
5,3071
0.8306
DPI M-2 BrT-hased
CAMEO 2022
0,1258
0.7958
6,4028
0.8380
PDB date split
0.2641
0.8648
3.2213
0,9043
architectural designs with better geometric awareness and representation learning ($4).
3 Improved Structure Prediction
In this section, we present several improvements aimed at enhancing the accuracy and detail of protein structure mod- eling. These approaches build upon the initial structure tokenization and aim to improve predictions by addressing challenges by introducing methods for recovering tokeniza- tion losses, bridging discrete and continuous tokens, and enabling direct data-space modeling.
3.1 Recovering Tokenization Loss by Learning Quantization Residuals with RESDIFF
In the structure tokenizer, the vector quantizer module con- verts encoded structure features into discrete structure token features, fundamentally clustering similar local environ- ments into identical token. However, according to 01, this process inherently introduces lossy compression, the residu- als-the differences between the original and quantized fea- tures-are lost during this process, eliminating fine- fine-grained structural details. The primary principle of the solution is that we need to recover and preserve the high- frequeryy variation that gets lost during tokenization.
Table 2. Tokenizer reconstruction rs. language model genera- tion. Evaluation folding on CAMEO 2022.
Reconstruction
Generation
Tokenizer
ERMSD4
TTMscoret
RXSD|
TMscoref
DPLM-2
1.9806
0.9385
7.7025
0.7936
ESM3
0.7248
0.9912
8.4424
0.7924
(03): Index- structure tokens? Multimodal PLM gets them miserably wrong in structure prediction. Ta- ble 3 shows that direct index prediction is highly inaccu- rate (0.0864 accuracy on CAMEO). However, the structural evaluation metrics (RMSD, IMscore) indicate that the gen- erated structures do not completely collapse, suggesting that despite the coarse -garine supervision, the model still captures some underlying relationships between indices. This learning process, however, remains highly challenging: since each index is derived from multiple quantized bits, even small changes at the bit level can result in drastically different indices. This issue becomes even more problem- atic as the codebook size increases, further exacerbating the difficulty of direct index prediction. In contrast, when evaluated at the bit-based level, prediction accuracy reaches 0.7720 on CAMEO, which aligns more closely with struc- tural evaluation metrics. This suggests that while the model struggles to recover exact indices, it effectively captures structural patterns at the bit level.
To address this issue, we can utilize continuous generative modeling, such as diffusion or flow-based models, to learn to recover these residuals. Specifically, inspired by Tang et al. (2024), we introduce a hight-weight diffusion module, Le., RESDIFF, to predict the residual information. Formally, let r Zcont Zquant represent the quantization residuals, we aim learn light-weight generative model (Ho et al., 2020; Nichol & Dhariwal, 2021) to predict the residual conditioned on the hidden states of language model h and discrete structure tokens z. The training loss is to minimize
L. = - [[le-co [ole-collo Cr t, h, Ziuuami 2
As illustrated in Fig. I(BI), the protein structure generation process now begins by generating discrete structure tokens Ziodex that capture the overall topology, and then these tokens and the hidden states of language model h are fed into to DISD to generate the missing residuals r. These residuals would be added up to the structure token to recover continuous structure tokens, i.e., Zeont = Zquamt + r, closer to the features produced by the structure encoder. Finally this Zcom is decoded to atomic structure.
Concluding Remarks. Given these observations, we iden- tify the primary bottlenecks of token-based multimodal pro- tein language models as tokenization loss (O1) and inef- fective structure modeling in sequence-basee architectures (32&03). To address these, we introduce improved genera- tive approaches for structure prediction ($3) and enhanced
4

---

Elucidating the Design Space of Multimodal Protcin Language Models
Table 4. Evaluation of improved approaches for structure pre- diction based upon DPLM-2. Folding SFT: supervised fine- tuning with folding objective.
flected in improved RMSD and TM-score), particularly on the PDB date test set. This suggests that the fine-grained bit-level supervision signals are more suitable for model to learn, enabling the model to capture structural patters more effectively, which enhances the latent structural modeling.
Models
CAMEO 2022
date split
RXSD)
"Nacore
RMsSD]
INscoref
ESMPold (3R) (1 al., 2022)
3,9000
0.8500
2,8400
0.9300
Muleilow (Camphell et al., 2024a)
17,8400
0.5000
15,6400
0.5300
3.3
A Hybrid Generative Approach Enables Direct Data-space Modeling
ESM3 (1.4B) (Hayes et al., 2024)
6.3300
0.8400
4,9003
0.8653
M-L (650M)
7,2025
0.7936
5,3071
0.8306
DPLM-2 RESDUP
7,2881
0.8087
5.1072
0.8430
Table highlights key limitation of discrete structure tok- enization: while it efficiently captures high-level topology and enables co-generation, the transition to a latent-space language model inevitably sacrifices atomic-level details. However, this transition inherently disentangles geometric modeling from sequence-based generative modeling--wth the structure tokenizer serving as an encoder-decoder in data space and the language model operating in the la- tent space of structure tokens. While this separation in- troduces information loss, it also presents an opportunity: the combination of the structure encoder, language model, and decoder as whole effectively functions as a denois- ing model, capable of refining structure in atomic coordi- nates. In such a way, we define a structure denoising model X9 (x,, t) X2 + Xdenoised decoder o PLM o encoder(x;, This insight allows us to seamlessly integrate this structure denoising model into a generative framework. Inspired by AlphaFlow repurposing folding models as flow-based gen- erative models (Jing et al., 2024), we incorporate this newly composed structure denoiser Xy into bowebased sampler with Euler integrator, where each Euler step interpolates X, + 1 xx(,,,1 L) + + X; up to a Kabsch alignment of &denoised against Xt, treating it as a denoising process on data data-space structure generation. We finetune such a model with flow matching (FM). This hybrid approach enables di- rect sampling in data space while preserving the scalability of discrete tokenization, ultimately improving atomic-level accuracy in protein modeling.
DPLM-2 based)
6.4028
0,8380
3,2213
0.9043
DPLM-2 (BIT-based) RESDUFF
6.1781
0.8428
3,0168
0.9076
DPLM-2 (BiT-bosed)- - FM
6.1825
0.8414
2,8697
0.9099
DPLM-2 (Bri-bosed) FM RusDue
6.0765
0.8456
2.7884
0.9146
w/ Folding SFT
5.8472
0.8442
2.3698
0.9270
DPL (3B) w/ folding SFT
5,9832
0.8443
3.1.00
0:9012
Results. We evaluate the structure prediction performance on the folding task. As shown in Table 4, the residual diffu- sion module is capable of improving the structural predic- tion accuracy by refining fine-grained structural variations based on language model predictions. Moreover, we ob- serve that the residual diffusion module is model-agnostic, showing consistent performance improvements across dif- ferent DPLM-2 variants. The Fig. 7 demonstrates that the residual diffusion module performs fine- teeeggaaied refinements on the local structure, optimizing interatomic distances to facilitate the formation of plausible secondary structures.
3.2 Bridging Discrete and Continuous Structure Tokens with Brr-based Language Modeling
Discretizing protein structures into badex-based tokens en- ables multimodal PLMS to perform structural modeling but introduces significant challenges, as discussed in 03. Since DPLM-2 employs LFQ tokenizer, its bit-level represen- tation provides more informative supervision signals, as reflected by 03 and Table 3, where bit-level accuracy is more indicative of generation quality. To bridge this gap, we aim to perform language modeling of the bit-based fea- ture of structure tokens instead of their indices, as illustrated in Fig. 1(22). To this end, inspired by Han et al. (2024), we make the most of DPLM-2's LFQ tokenizer, which already operates at the bit level, which quantizes each dimension independently, preserving more structural details while re- maining compatible with PLMs' discrete supervision. It hence becomes K binary classifications to predict each bit of K-bit structure token, instead of the original 2 classifications. This greatly reduces the learning challenges and thus improves generative accuracy. As such, the training objective with bits-based structure modeling is accordingly modified as:
Results. Table 4 demonstrates that direct data-space sam- pling with flow matching is capable of enhancing the struc- ture generation on the folding task, while supervision with the folding objective can bring further improvement. We observe that the folding performance is on par with or even outperforms the strong baseline ESMFold, particularly on the PDB date split. This demonstrates that the hybrid struc- ture modeling method can take the both worlds of the scala- bility of discrete tokenization enhanced by language model and the accurate atomic-level geometric modeling, resulting in a superior protein structure generation performance.
4 Improved Structure-aware Architecture and Representation Learning
J.th = Bgix..) E log + E. log po (0(:!)lmm
Results. As shown in Table 3 and Table 4, the bit-level supervised DPLM-2 achieves significant accuracy improve- ments across both index-level and bit-level, while substan- tially reduced structural deviation from ground truth (re-
As evidenced in protein folding (Jumper et al., 2021; Lin et al., 2023), the intricate nature of protein structures demands methods that capture higher-order relationship between residues beyond simple sequence-basee models.
5

---

Elucidating the Design Space of Multimodal Protcin Language Models
4ctertion
desct nesurrston
Sea mesurattinn
SAIT- Artention
Sugstruct
Attunciion
Sea monurattee
Figure 3. Structure attention module. We introduce geometric- aware operations such as triangle operations and logit bias (Jumper et al., 2021). L denotes number residues and D is the feature dimension, where we select 128 for pair representation and 1280 for structure representation.
Figure Integrate geometry-aware modules into multimodal protein language models (PLMs). We integrate geometric mod- ules into encoder blocks PLMs, including structure attention (top) and SeqStruct attention (bottom right). We provide archi- tecture details Figures and and the appendix.
Paireepreentittoo O., D)
While bit-based modeling offers effective guide the design of supervision targets, sequence-basee models still (1) lack the geometric inductive biases and (2) structural learning objective that might be needed to capture the complex- ity of residual interactions. To address these limitations, we introduce geometric modules and representation align- ment (REPA) (Yu et al., 2024) to enhance the understanding of residue-residue interactions and structural diversity.
Strant
Seq, mpr. [.
Figure 4. Seqstruct attention module. We concatenate structure and sequenee representations along the feature dimension. Resid- ual connections between inputs and outputs are omitted here.
4.1 GeoDPLM: Geometry-aware Model Architecture
on CAMEO2022 for each ablation variation in Table 5. We report the average training speed every 10k training steps, and use marker colors to reflect diversity. Among all op- erations, triangle operations are the most computationally intensive, with triangle update (TU) and attention (TA) dra- matically reducing the training efficiency by 3.9x and 6.8x, respectively. The attention mechanism involving pair rep- resentations as logits bias in GeoDPLM (Base) causes a slight decrease on training speed compared to DPLM-2. Using transition layers for structure representations, which significantly boosts both structure modeling and generation diversity, has minimal impacts training speed.
Inspired by PairFormer in AlphaFold3 (Abramson et al., 2024), we introduce GeoDPLM with newly added geomer- ric modules that operates on compact 2D pair representa- tions to capture pairwise spatial dependencies of residues. As shown in Figure 2, we use structure attention to in- dependently refine structure (single) representations and pair representations through transition and triangle oper- ations, followed by the Seqstruct attention to blend pair representations with both sequence and structural (single) representation.
Component-wise analysis of GeoDPLM on structure folding. Table provides a component-win study on each geometry-based module, which could be referred in Fig- ures 3 and 4. Introducing 2D pair representation maps into DPLM-2 effectively improves structure prediction, reduc- ing the folding RMSD from 7.703 to 7.244 RMSD and increases the TMMcorr to 0.8339. When models are trained without the folding SFT objective maxę log po (x|s). the inclusion of transition layers for structure representations proves critical for achieving better structure predictions. Se- qStruct attention, which combines pair representations with sequence information, achieves minimal improvement in metrics, indicating that solely incorporating pair informa- tion into structure representations is sufficient. In contrast to the common practice in structure folding, we find that triangle update and attention operations not yeild notable benefits. For simplicity, in subsequent experiments, we re- fer the model with logits bias and pair representation transition layers GcoDPLM (Base).
4.2 Representation Alignments to Folding Model
REPA benefits structure generation potentially by (1) over- coming the limitations of discrete tokens and (2) addressing the key challenge in training diffusion models. Unlike dis- crete supervision that enforces sharp targets, REPA enables smooth, informative and nigh-dimensional learning, preserv- ing finer structural nuances. Meanwhile, Yu al. (2024) has identified the primary challenge of training diffusion models as learning high-quality representations. To address this, we adopt REPA by aligning the representations of the protein language model to transfer meaningful structural semantics from specialized folding model, where we use ESMFold (Lin et al., 2023) in this paper for efficiency of massvie inference while other models such as AlphaFolds are also compatible. Originally successful in vision tasks, find that REPA effectively improves the structural diver- sity of generated proteins (Figure 6).
Training efficiency. Figure 5 presents the correlation be- tween training speed and structure prediction performance
REPA setup. We select ESMFold (Lin et al., 2023) as the target representation encoder due its computational
6

---

Elucidating the Design Space of Multimodal Protcin Language Models
Table Ablation: geometry-aware modules. We ablate each component in Figures 3 and 4 by studying their effects on folding. P.Bias & Tran: pair bias and transition layer for pair representation. S. Tran: transition layer for structure representation. Tri. Up. & Attn.: triangle update and attention layers. We name each ablation variation in the parenthesis such as (base). SFT: supervised fine-tuning using folding objective.
Methods
Structure Attention
Seqstruct
SFT
PDB date split
CAMBO 2022
P. Bias & Tran.
S. Tran.
Tri. Up.
Tri. Attn.
Alin.
RMSDL
TMscore1
RMSDJ
TMscore1
DPLM-2
X
5.521
0.8287
7.703
0.7936
GcLDPLM (Basc)
4.823
0,8521
7.244
0.8128
(ST)
3,883
0.8857
6,550
0.8339
(TU)
V
4.837
0,8598
7.197
0.8255
(TA)
V
4,415
0.8690
6,973
0.8210
(SSA)
V
4.040
0.8841
7.158
0.829
DPLM-2
X
x
X
X
X
3.347
0,9008
6.612
0.8233
GeoDPLM (Base)
3.165
0,9046
6.227
0.8414
(ST)
3.021
0,9062
6,288
0.8393
(TU)
3.639
0.8903
6.877
0.8322
(TA)
3,863
0,8790
6,393
0.8340
(SSA)
3.134
0,9054
6,329
0.8379
Table 6. Representation alignment improves structure predic- tion. REPA is compatible with both language model-based archi- tectures (DPLM-2) and geometric designs (GeoDPLM).
CEUM2
0.54
2.5
Folcing PMGG on Cameo
7.4
0.80
Methods
PDB date split
CAMEO 2022
(SA))
3.78
RMSDL
TMscoref
RMSDL
TMscoref
Uiverslo
DPLM-2
5.521
0.8287
7.703
0.7936
2.2
REPA
4.919
0.8508
7.344
0,8046
0.4
GeoDPLM
4.823
0.8521
7.244
0.8128
w REPA
4.340
0,8671
7.058
0.8217
6.2
20
40
30
REPA is versatile, being compatible with both language model-based architectures (DPLM-2) and those incorpo- rating geometric designs (GeoDPLM). Applying REPA on DPL.M-2 effectively increases both folding RMSD and TM- score, suggesting that additional learning signals can play a critical role in improving architectures that lack geometric inductive biases.
Time per 1Ck Steps (hours)
Figure 5. Training efficiency of geometric designs. We analyze the correlation between training speed and folding RMSD for each ablation variation in Table 5. Color represents generation diversity. while marker size indicates model parameters. Adding transition layers for structure representations (ST) improve diversity and folding without reducing much training speed.
Better structure-aware model architecture and repre- sentation learning boosts structure generation diversity. Protein language models often suffer low diversity in gener- ated structures (Wang et al., 2024a), potentially due to the lack of inductive biases to capture structural interactions. Figure shows structure generation diversity, where for each protein length, we generate 40 samples and quantify the diversity by the normalized number of sample clusters identified by FoldSeek (Van Kempen et al., 2024). These results show that appropriate geometric architecture designs (GeoDPLM) could effectively improve the generation di- versity. Aligning representations to structure folding model further significantly diversifies generated structures, show- ing its effectiveness in refining generation process.
efficiency and open-source availability compared to other folding models (Jumper al., 2021; Abramson et al., 2024). Using the protein sequences in our training data, we precom- pute the structure and pair representations from the folding trunk of ESMFold. We run three recycling iterations within the folding trunk to ensure the quality of representations. Instead of aligning embeddings from a certain layer as done in Yu et al. (2024), we apply a learnable nn. Pa rameter followed by a softmax function as the weight to ensemble representations across all layers. We follow Yu et al. (2024) to apply a 3-layer MIP to project the representations be- fore aligning them with the target representations through negative cosine similarity.
Compatibility with different architectures. Table 6 presents the effects of REPA on structure prediction. REPA improves structure folding on both PDB date split and Cameo datasets, highlighting its ability to go beyond re- fining the generation process. Importantly, we observe that
5 On the Orthogonality of Design Methods
Building on the individual analysis of each design method in the preceding sections, we now examine the interactions of these designs by combining them in unified setting, as
7

---

Elucidating the Design Space of Multimodal Protcin Language Models
1.02
Table 8. Statistics of PDB-Multimer. We curate a dataset of multi- chain proteins from PDB to analyze their effects structural modeling.
DRUM2
Goo CPLM
Diso DPPM RENK
3.90
0.80
1.2
1645
05:5
Dataset
# proteins
# chains
Protein Longth
Chain
Train Val
Length
0.75
2.02
0.75
0.75
0.76 0.70
PDB- Multimer
11614/211
2.88 1.66
661.57
416.37
229.39
167.00
0.70
3.0
0.70
0:36
Table 9. Effeets of monomer data on tokenizer reconstruction for multimer. Scaling monomer data significantly improves the structure tokenizer reconstruction on DullMultieer, suggesting the relevance between multimer and monomer modeling. We also provide results of monomer on CAMEO dataset.
182
0.55
035
Figure 6. Effects on generation diversity. Geometric designs and structure representation alignment with the folding model (Lin et al., 2023) significantly improve the low generation diversity of the multimodal PLM. Diversity is normalized by the number of generated samples.
Length
Training Data
Size
PDB-Multiner
CAMPO 2022
RMSDL
TMscoret
RMSnj.
TMscoret
SwissProt
200K
9.973
0.694
2.589
0.930
AFDB_R
+1.2M
6.873
0.784
2.245
0.938
discrete tokens, hence their non-orthogonal effects.
Table 7. Analysis of orthogonality. We analyze the compatibility of design methods when combined, with the recommended setting highlighted. All" denotes the combination of all methods: Gco, Bit, FM, REPA, ResiDiff, and SFT.
Hybrid modeling and geometric modules. FM effectively improves folding, but the benefits diminish with geometric modules, and can reduce generation diversity due to its ODE nature. However, benefitting from the same nature, FM accelerates the sampling process by requiring 10x fewer sampling steps.
PDB date split
CAMEO 2022
Uncond. Gen.
Models
RMSD|
TMscoref
RMSD]
TMscore"
Diversisy?
DPLM-2 (650M)
5.307
8306
7.703
.7936
0,700
3,221
.9433
6.403
.380
0.825
ResDiff. Similar to the results in the paper, ResDiff does not bring sigmificant boost to folding metrics. The major benefit of ResDiff is to provide a finer local structure as discussed in the Figure 7.
Bit + FM
2,870
9099
6.183
8418
0,525
Bit + FM+ ResDiff
2.788
.9146
6.077
8456
0.525
SFT
2.370
.9700
5.847
8442
Goo Bit
2.551
.9544
5,955
.8500
0,900
Geo + Bit + FM
2.443
.261
6.172
.8404
0.575
Geo Bit+ + REPA
2.507
.9264
6.192
.8412
0.875
We include a summary table linking to the motivation and findings of all design methods in Table 12 of the Appendix.
w/ SFT
2.404
9322
5,754
8424
All"
2.379
.9977
6,200
.3398
6 Structure Data: Multimer Exploration
shown in Table 7. For models fine-tuned with the folding SFT objective, we skipped the evaluation of unconditional co-generation. We discuss the final recommended setting and the orthogonality of designs below.
Despite the advancements in modeling, the scarcity of pro- tein structure data remains a challenge for developing ro- bust multimodal protein language models. In this sec- tion, we extend our data coverage to include multimer, proteins that consist of multiple chains, since multimer data presents diverse structural arrangements and interac- tion scenarios, which are essential for developing a more general multimodal model. Notably, most existing protein language models have been trained solely on single-chain proteins (monomer). In Table 8, we introduce our PDB- Multimer dataset. We examine the relevance and gap be- tween monomer and multimer data in the following analysis. Scaling monomer data improves reconstruction for mul- timer. Table 9 presents the reconstruction performance of the structure tokenizer. We observe that increasing the monomer data from 200K to 2M leads to a substantial im- provement in both reconstruction RMSD and TM-score on the PDB-Multimer validation set. This indicates that monomer modeling is closely related to multimer modeling and could provide direct benefits to it.
SFT. Folding SFT improves the structure folding but sacri- fices the model's ability for multimodal co- co-generation, as it is fine-tuned specifically for the folding task.
The recommended setting: Geo + Bit-based modeling. Geometric architectures are compatible with bitwise model- ing and their combinations achieve comparable results with models finetuned with folding SFT on structure folding, and further obtains an effective improvement on unconditional generation quality & diversity. This setting is also effec- tive in terms of training efficiency as it avoids additional computational overhead from other methods like FM and REPA.
REPA and Bit-hased modeling, Both REPA and bit-da modeling enhance structure folding and generation diver- sity. Meanwhile, their combinations do not lead to further improvements. We suggest that this is because REPA and bit-based modeling both help through enabling smooth and high-dimensional learning signals compared to index-
Chain linker and position offset. We attempt to identify and bridge the gap between multimer and monomer data in Table 10. Noting that chains are typically spaced farther
8

---

Elucidating the Design Space of Multimodal Protein Language Models
Table 10. Applying chain linker and position offset in multimer modeling. We present the folding results on PDB- Multimer and report the reconstruction performance of structure tokenizer. ESM- Fold used G-linker of length 25 by default in multimer folding.
monomer Swissprot data (200K) helps learn the high-level 3D structural patterns due its larger dataset size compared to PDB-Multimer (3.5K), as reflected by consistently higher TMscores that measure global structural similarity.
PDB Multimer
Method
7 Conclusions
RMSDL
IMscoret
In this work, we identity the limitations in structural mod- eling for multimodal protein language models and propose an effective design space to bridge the gap. We demonstrate that tokenization quantization loss can be effectively miti- gated with bit-label supervision and flow-matching, which significantly improve the structure prediction accuracy. We introduce geometric inductive biases through architectural design and leverage representation learning to refine genera- tion diversity. Building on the strengths of each component, we further investigate their orthogonality, which informs the final recommended setting. Lastly, to tackle the scarcity of structure data, we explore the data coverage to include multimers, ensuring broader 3D structural understanding. Our results show that these effective designs allow multi- modal models to achieve on-par or even superior folding accuracy compared to larger, specialized folding models. Despite these, we also notice that there remain several limi- tations and future work directions deserving to be explored. We provide discussions on the limitations and potentials in $B. We believe this work will contribute to advancing the development of more effective multimodal protein language models.
Tokenizer Reconstruction
D (monomer) tokenizer
6,873
0.784
w/ Pos. Offset
5,886
0.812
Folding
ESMFold
17.297
0.850
DPLM-2
19.110
0.768
Chain Linker
17.966
0.771
Pos. Offset
18.338
0.767
Table 11. Fine-tuning with multimer and monomer data. We evaluate the effects of fine-tuning with PDB-Multimer and Swis- sprot on structure prediction. Incorporating multimer data im- proves both monomer and multimer folding. SFT: supervised fine-tuning with folding objective.
Training Data
PDB Multimer
CAMBO 2022
SFT
PDB -Multimer
Swissprot
RMSD4
TMscore"
RMSD4
TNscoref
17.966
0,771
7,703
0.793
19.615
0.823
16.146
0.775
10.989
0,686
16.674
0.798
6.410
0,831
apart than individual connecting residues, we apply a posi- tion index offset to each residue, which is calculated as the product of chain index and a predefined offset value. The offset is incorporated into the relative position embedding of the structure detokenizer (Wang et al., 2024a). We further examine the effects of connecting chains using glycine (G) linkers of varying lengths under the folding scenario. These linkers not only introduce a position offset but also serve between protein chains. Our findings
A Acknowledgements
We thank anonymous reviewers for their valuable feedback. We would like to especially thank Dr. Hang Li for insightful discussions on the project and feedback on the manuscript that help shape this study, as well as Yi Zhou, Jing Yuan and Yilai Li for their valuable comments.
as pseudo-connectoo suggest that chain linkers and position offsets both improve the metries. These results highlight the difference between multimer and monomer and suggest that properly differenti- ating chains in sequence and positional space are essential for effective multimer modeling.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning and its applications on protein modeling, There are many potential societal consequences of our work. Our work on protein generation and representation learning can be used in developing potent therapeutic macromole- cues such as antibodies and accelerate the research process of drug discovery. Our method may be adapted to other scenarios of computer-aides aided design, such as small molecule design, material design, and chip design. It is also needed to ensure the responsible use of our method and refrain from using it for harmful purposes.
Finetuning on PDB-Multimer. We study the effects of multimer and monomer data by fine-tuning DPLM-2 in Table 11. We excluded multi-chain proteins with lengths outside the range of [60, 512). resulting in 3462 training samples in PDB-Multimer We observe that incorporating PDB-Multimer into the training data effectively improves the structure folding for both multimer and monomer. This highlights that multimer data might be essential for robust structural modeling. Additionally, fine-tuning with multi- mer data is essential for reducing RMSD on PDB-Multimer. Since multimer chains are typically more spatially separated than neighboring residues in monomers, models trained solely on monomer data tend to yield a higher RMSD, which captures local atomic deviations. Meanwhile, the
References
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J.. Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.

---

Elucidating the Design Space of Multimodal Protcin Language Models
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in dis- crete state-spaces. In Advances in Neural Information Processing Systems, volume 34, pp. 17981-17993, 2021.
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Steinegger, M., and Rost, B. Prostt5: Bilingual lan- guage model for protein sequence and structure. biorxiv. bioRxiv, 2023.
Back, M., DiMaio, F., Anishchenko, I., Dauparas. J., Ovchinnikov, S., Lee, G. R., Wang. J., Cong. Q., Kinch, L. N., Schaeffer, R. D., et al. Accurate prediction of pro- tein structures and interactions using a three- ttrck-track neural network. Science, 37346557))8771887766 2021.
Ho, J. Jain, A., and Abbeel, P. Denoising diffu- sion probabilistic models. Advances New- ral Information Processing Systems, 33:6840- 6851, 2020. URL https://pron attps://procecings.
4 4cboffec@54aaf005511100000022220 pdf.
Behjati, A., Zare-Mirakabad, F., Arab, S. S., and Nowzari- Dalini, A. Protein sequence profile prediction using pro- talbert transformer. Computational Biology and Chem- istry, 99:107717, 2022.
Ingraham, J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., Ng- Thow-Hing, C., Van Vlack, E. R., et al. Illuminating protein space with a programmable generative model. Nature, pp. 1 1-9, 2023.
Brandes, N., Ofer, D., Peleg, Y.. Rappoport, N., and Linial, M. Proteinbert: a universal deep-learning model of pro- tein sequence and function. Bioinformatics, 38(8):2102- 2022.
2110, 2022.
Jing, B., Berger, B., and Jaakkola, T. Alphafold meets flow matching for generating protein ensembles. In Forty-first International Conference on Machine Learning, 2024.
Campbell, A., Yim, J.. Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint 24X.:::402.049977. 2024a.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-589, 2021.
Campbell, A., Yim, J.. Barzilay, R., Rainforth. T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint 24Xiv:240..04997, 2024b.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. Au- toregressive image generation using residual quantiza- tion. In 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 11513-11522. IEEE, June 2022a. doi: 10.1109/cvp:52688.200 01123. URL http: /xx doi. ott.oogg/1..11110000
Chen, R., Xue, D., Zhou, X., Zheng, Z., Zeng, X., and Gu, Q. An all-atom generative model for designing protein complexes. arXiv preprint arXiv:2504.11 13075, 2025.
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. Prottrans: Toward understanding the language of life through self-supervisee learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):111-7122) 2021.
0ERR52688.20222 01123.
Lee, J. S., Kim, J., and Kim, M. Proteinsgm: Score-based generative modeling for de novo protein design. bioRxiv, pp. 2022-07, 2022b.
Lees, J., Yeats, C., Perkins, J., Sillitoe, I., Rentzsch, R., Des- sailly, B. H., and Orengo, C. Gene3D: a comain-bassee resource for comparative genomics, functional annotation and protein network analysis. Nucleic Acids Research, 40(51):1465-D441, 12 2011. ISSN 0305-1048. doi: 10. 10.1093/naa/gkr11181. URL https:: /wo.///oiii.o org//0. 1093 1093/nar/gkr118)
Geffner, T. Didi, K., Zhang, Z., Reidenbach, D., Cao, Z., Yim, J., Geiger, M., Dallago, C., Kucukbenli, E., Vahdat, A., et al. Proteina: Scaling flow-based protein structure generative models. arXiv preprint 20X 2025.
Han, J., Liu, J., Jiang, Y., Yan, B., Zhang. Y., Yuan, Z., Peng, B., and Liu, X. Infinity: Scaling bitwise autoregres- sive modeling for high- image synthesis. arXiv preprint ar arXiv:2412.044: 04404111,, 2024.
Li, M., Tan, Y., Ma. X., Zhong, B., Yu, H., Zhou, Z. Ouyang. W., Zhou, B., Hong. L., and Tan, Prosst: Protein language modeling with quantized structure and disentangled attention. bioRxiv, pp. 2024 04, 2024.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q. Deaton, J., Wiggert, M., et al. Simulating 500 million years evolution with a language model. bioRxiv, pp. 2024-07, 2024.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al. Language models of protein sequences the
10

---

Elucidating the Design Space of Multimodal Protcin Language Models
scale of evolution enable accurate structure prediction. BioRxiv, 2022.
diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Lin, Z. Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., al. Evolutionary- scall prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123-1130, 2023.
Senior, A. W., Evans, R., Jumper, J. M., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Zidek, A., Nel- son, A. W. R., Bridgland, A., Penedones, H., Pe- tersen, S., Simonyan, K., Crossan, S., Kohli, P., Jones, D. T., Silver, D., Kavukcuoglu, K., and Hassabis, D. Improved protein structure prediction using po- tentials from deep learning. Nature, 577:706 710, 2020. URL https: /appi.:///ppppppp org/ /Ct:/CCpuI:::1022110000
Meier, J., Rao, R., Verkuil, R., Liu, J., Sereu, T., and Rives, A. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances in Neural Information Processing Systems, pp. 29287- 29303, 2021.
http:///ap....a.t..eaaee/e/ semant cscholar.
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023-10, 2023.
Nambiar, A., Heflin, M., Liu, S., Maslov, S., Hopkins, M., and Ritz, A. Transforming the language of life: trans- former neural networks for protein prediction tasks. In Proceedings of the I 1tth ACM international conference on bioinformatics, computational biology and health in- formatics, pp. 1-8, 2020.
Tang, H., Wu, Y., Yang, S., Xie, E.,, Chen, J., Chen, J.,
Zhang, Z., Cai, H., Lu, Y., and Han, S. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410. 10812, 2024.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference Machine Learning, pp. 3162-8171 PMLR, 2021.
Tian, K., Jiang, Y., Yuan, Z., Peng. B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865, 2024.
Nijkamp. E., Ruffolo, J., Weinstein, N., Naik, N., and Madani, A. Progen2: exploring the boundaries of pro- tein language models. arXiv preprint arXiv:2206.13317. 2022.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017a.
OpenAL. Gpt-4 technical report, 2023.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017b.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rX1v:2304.07193, 2023.
Van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L., Söding, J., and Steinegger, M. Fast and accurate protein structure search with foldseck. Nature biotechnology, 42(2):243 246, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-420S 2023.
Vaswani. A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, volume 30, pp. 5998-6008, 2017.
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song. Y. Evaluating protein transfer learning with tape. Advances in neural informa- tion processing systems, 32, 2019.
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., and Fergus, R. Bi- ological structure and function emerge from scaling unsu- pervised learning to 250 million protein sequences. PNAS, 2019. doi: 10.1101//22803. URL https: /ww. biorxiv org/ http://oneett/tttt 1101/ 1......111/2222322222
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. T., Milles, I. F., Dauparas, J., Baker, D., Ovehinnikov, S., Sercu, T., and Rives, A. Language models generalize beyond natural proteins. bioRxiv, pp. 2022-12, 2022.
Wang. X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410. 13782, 2024a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Ommer, High-resolution image synthesis with latent

---

Elucidating the Design Space of Multimodal Protcin Language Models
Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q- Diffusion language models versatile protein learners. arXiv preprint arXiv:2402.18550 2024b.
Zheng, Z., Deng, Y., Xue, D., Zhou, Y., YE, F., and Gu, Q- Structure-informeed language models protein design- ers. In International Conference Machine Learning, 2023b.
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, L.., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L.. F,, al. De novo design of protein struc- ture and function with rfdiffusion. Nature, 620(7976): 1089-1100, 2023.
Wu, R., Ding. F., Wang. R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q. Berger, B., al. High- resolution de novo structure prediction from primary quence. BioRxiv, pp. 2022-07 07, 2022.
Xiao, Y., Qiu, J., Li, Z Hsieh, C.-Y., and Tang. Modeling protein using large-scale pretrain language model. arXiv preprint arXiv:2108.074440
2021.
Yang. K. K. Lu, A. X., and Fusi, N. Convolutions are competitive with transformers for protein sequence pre- training, bioRxiv, pp. 2022-05, 2022.
Ye, F., Zheng, Z., Xue, D., Shen, Y., Wang, L.,, Ma, Y,, Wang. Y., Wang. X., Zhou, X., and Gu, Q. Proteinbench: holistic evaluation of protein foundation models. ICLR, 2025.
Ye. J. Zheng, Z., Bao. Y., Qian, L., and Gu. Q. Diffusion language models can perform many tasks with scaling and instruction- citrrcttoonfinetuninngg arXiv preprint arXiv:2308.1223 2023a.
Ye, J., Zheng, Z., Bao, Y., Qian, L., and Wang. M. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302. 100.5.0..000 2023b.
Yim, J., Campbell, A., Mathieu, E., Foong, A. Y., Gastegger, M., Jiménez-Luna, J., Lewis, S., Satorras, V. G., Veeling, B. S., Noé, F., et al. Improved motif-scaffolding with se (3) flow matching, arXiv preprint arXiv:2401.04082, 2024.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2023.
Yu, S., Kwak, S., Jang, H., Jeong. J., Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers easier than you think. arXiv preprini arXiv:2410.0668 2024.
Zheng, L.,, Yuan, J., Yu, L., and Kong. A reparameter- ized discrete diffusion model for text generation. arXiv preprint N1.::233222005333 2023a.
12

---

Elucidating the Design Space of Multimodal Protcin Language Models
A Taxonomy of the Design Space
We present the taxonomy of our design methods in Table 12, summarizing the corresponding space of each design choice with discussion on the underlying motivation. The findings from each design method are presented in the rightmost column.
Table 12. The taxonomy of design choices. their orthogonality, and systematic synthesis of findings.
Design Space
Design Choice
Traditional
Motivation
Findings
Choice
Small changes at the bit level can result in drastically different indices, making the learning process labels challenging.
Intuition:
Bit-level supervision improves prediction accuracy across index and bit bit-level, while substantially reducing structural deviation from ground truth (RMSD and TMscore).
Bit-based modeling
Index-based modeling
Empirical Analysis: Direct index prediction highly inaccurate (8.64% on Cameo).
Improved generative
Quantizing continuous tokens into discrete amplifies reconstruction errors, suggesting that recovering the lost residuals might enhance structure prediction accuracy.
ResDiff module performs fine-grained refinements the local structure (Fig, 7), showing consistent performance improvements across different DPLM-2 variants.
Decode tokens without residuals
modeling
ResDiff
Hybrid approach- data- space sampling and predicting discrete tokens
A hybrid approach with flow matching improves structure generation for the folding task and speeds up the sampling process by requiring 10x fewer sampling steps.
Discrete structure tokenization inevitably sacrifices atomic-level details.
Predict discrete tokens
Geometric modules enhance structure folding performance and generation diversity. Unlike typical folding models, analysis shows that triangle layers provide little benefit while greatly slowing down training.
Geometry- aware architecture
Sequence-based transformer architecture
Protein structures require capturing higher-order relationships between residues, as evidenced in folding.
Representation alignment (Yu et al., 2024) considerably boosts generation diversity and is compatible with both language model-based and geometric architectures.
Structure- aware approaches
Discrete token supervision might less effective for capturing finer atom-level details. Unlike discrete supervision that enforces strict and sharp targets, representation alignment enables smooth, informative and high-dimensionn learning, preserving finer structural nuances and improving generation diversity.
Alternative approaches:
Representation alignments folding model
Contact map supervision (esm2);
Discrete token supervision only
The quadratic complexity O(L*)
results in extreme inefficiency in prelim. study.
Direct coordinate supervision: Need to carefully take into account se3 symmetry. Recent progress on this: Proteina (Geffner al., 2025).
Multimer and monomer modeling are deeply interconnected and leveraging multimer data advances structural modeling for both single and multi-chain proteins.
Multimer data presents diverse structural arrangements, while most existing PLMs are trained solely on monomer.
Multimer data exploration
Monomer data only
Data
13

---

Elucidating the Design Space of Multimodal Protcin Language Models
B Discussions and Limitations
While our approach significantly improves structural modeling in multimodal protein language models, several limitations remain.
First, despite advancements in structural tokenization and generative modeling, our method still relies on discrete representations of 3D structures, which inherently introduce information loss. While bitwise discrete modeling mitigates some quantization errors, does not fully capture continuous geometric variations, limiting the fidelity of fine- fine-grained structural features. Future research could explore hybrid approaches that combine discrete and continuous representations to enhance structural expressiveness. Additionally, atomic-level precision remains fundamental challenge, current tokenized representations primarily operate at residue backbone levels, missing finer atomic interactions crucial for protein folding and function. Bridging the gap between token-based modeling and direct atomic-coordinate generation is essential for achieving true atomic-resolution structure modeling.
Second, our framework, like other multimodal PLMs, lacks explicit physical constraints and energy-based priors, which are crucial for generating physically plausible protein structures. While representation- learning helps refine structural understanding. incorporating differentiable physies-based priors or energy functions could further improve structural realism and biological validity.
Third, our analysis is conducted on relatively small protein language models (up to 3B), and the scalability of these design choices remains uncertain. We chose this setup to prioritize the analysis of our design space. Further study into how these designs scale with larger models could further enhance multimodal protein language modeling, especially given the proven benefits of scaling in DPLM-2.
Finally, our training dataset, while diverse, remains constrained by available high-quality structural data, particularly for multimers. While we demonstrate that leveraging multimeric structures enhances modeling, the sparsity of curated multimer datasets poses challenges for generalization. Future efforts should prioritize data augmentation techniques and larger-scale multimodal datasets to improve robustness across different protein families.
Ultimately, the future of multimodal protein foundation models should enable direct atom-level data-space structure modeling; however, this remains an unsolved challenge. Token-base based multimodal protein LMs provide a scalable, practical, and effective approach to generative protein modeling. Advancing this approach while addressing its limitations such as structural information loss, lack of physical constraints, and limited multimodal generalization-wii be key unlocking their full potential and provide crucial insights that help us move closer achieving atomic-resolution protein modeling in unified multimodal framework.
B.1 Discussion on Alternative Approaches
We recap our methods to address 01-03 (see Section 2.2) and discuss potential alternative approaches below. Structure tokenization results in information loss.
01:
Our approach: We train an additional continuous diffusion module ResDiff to recover the lost residuals of discrete tokens.
Training DPLM-2 on continuous tokens. However, it's unclear whether this approach would work for joint modeling of structures and sequence, which is inherently discrete.
Alternatives:
02: High reconstruction accuracy of structure tokenizer does not ensure better structural generative results.
Our approach: We primarily improve the designs of language modeling, including structure-aware generative modeling, architectural designs, and representation alignments.
Alternatives: Adopt direct modeling in the data space following a similar manner as Proteina (Geffner et al., 2025). Similarly, it's unclear if direct modeling is robust for joint modeling of discrete sequences and continuous 3D structures. 03: Multimodal PLM gets index- structure tokens miserably wrong. Small bit-wise changes could result in dramatically different index label. This challenge intensifies as codebook size grows, making direct index prediction even more difficult.
Our approach:
Finer-grained token prediction might resolve such challenge. We achieve finer-grain prediction on the "dimension-level" using bit- btsedaedd labels in contrast to index-based labels.
Alternatives: Another potential direction is to hierarchically tokenize the structures to achieve fine-grained tokens at the "resolution" level following methods like RQ-VQVAE (Lee et al., 2022a) and VAR (Tian et al., 2024). However, it remains elusive how to well define "resolution" of proteins as a natural choice.
14

---

Elucidating the Design Space of Multimodal Protein Language Models
Implementation Details
C.1 Residual Diffusion Module
We use another light-weight diffusion module on the top of DPLM-2 to predict the residuals information Zcont Zquant conditioned on the language model output discrete tokens Zquant and hidden states h. We employ linear projection on the discrete structure tokens and hidden states of each layer to obtain the condition information c:
N
Zequant W quamt
where N represents the number of layers and a, represents the weight of hidden state of layer i, which is obtained by a learnable vector w = (W1, W2,..., WN) ) E RN and ag = Σ e"j The condition information is subsequently processed by an adaptive layer norm block, similar to Peebles & Xie (2023). During generation, the language model first predicts the discrete structure tokens Zquani while providing the hidden states of each layer h, then the residual diffusion module takes the Zquant and h as conditions and generates the residuals information r. Finally, we add the residuals and the discrete structure tokens to obtain the continuous structure feature Zeont. which is decoded to 3D protein structure.
The lighh- diffusion module consists of 6 layers of MLP with hidden size of 1024. During training, the learning rate warmed up over the first 2,000 steps to a peak value of 1 x 10-1 and then linearly decayed to 1 x 10-5, We train the residual diffusion module for 100,000 steps with batch size of 240.
C.2 Bit-level Supervision
As shown in Figure 1, we explore the bit-level supervision instead of index-level supervision. The language model takes the K-dimension quantized feature as input, and leverage K binary classifiers to predict each bit of the structure feature in parallel, considering that the each bit of the quantized feature is either - or +1. For the input quantized structure feature Zquant E [f1,++1 1. K we employ a linear projection Wipput E *x*" where H represents the hidden size of language model, obtain the input embedding for the transformer block. We utilize another linear projection W Wonput E RR H22 to obtain the logits of the output quantized feature. During sampling, the logits with dimension (L, 2K) are reshaped to (L. K, 2), and we take the softmax operation at the last dimension to get the probability of each bit. The bit-level training loss is calculated on each bit by binary cross coss-entropy. Considering that the number of index-level vocabulary is 2 and if we use index-level supervision we would need to train the input and output projection with 2 * H parameters, the bit-level supervision only needs to train the input and output projection with K H parameters, significantly reducing the training cost.
In the training stage, we follow Wang et al. (2024a) and utilize the pre- preitrdined DPLM (Wang et al., 2024b) as the parameter initialization to inherit the evolutionary representations learned from the massive protein sequence data, which implicitly captures the structural information that benefits structure modeling (Lin et al., 2022). In addition to the parameters inherited from DPLM, we randomly initialize the structure input-output linear layers. The bit-level DPLM-2 is grounded in the absorbing discrete diffusion framework (Austin al., 2021; Zheng et al., 2023a). To adapt to the absorbing discrete diffusion framework, we also introduce learnable absorbing embedding for the absorbing state of structure. We employ 2,000 warmup steps until reaching the maximum learning rate x 10 and utilize linear decay scheduler to decay LR to 1 x 10-5, The overall training process consists of 300,000 steps.
C.3 Hybrid Approach for Direct Data-space Sampling
We fine-tune the bit- DPLM-2 to denoise continuous latent structural features, encoder(x:). following strategy similar to flow matching (FM). The structure encoder, DPLM-2, and structure decoder together form our structure denoising pipeline. During training, DPLM-2 predicts denoised quantized fcatures Zdenoised, which are decoded into structures Rdenoised at inference time. Instead of using the FM loss, we employ bit-wise cross-entropy loss between Zdemoised and the ground truth bit labels Zquant, avoiding the computational overhead of running the structure decoder during training.
C.4 Geometric Designs
As shown in Figure 2, we integrate geometry-aware modules into the encoder blocks of PLMs. To initialize the pair representations, we follow Multiflow (Campbell et al., 2024b) to cross concatenate the input hidden representation with dimension (L., D) into a 2D representation map with dimension (I., L,, D), followed by a 3-layer MLP. We use I. and D to denote the length and feature dimension. The feature dimension D is selected as 1280 for structure representations and 128 for pair representations. We adopt the encoder block of PairFormer (Abramson et al., 2024) as the structure attention module in Figure 3. The hidden dimension in triangle update operations is selected as 128, while the dimension of hidden pair representation in triangle attention operations is 32. We use 4 and 16 heads in the triangle attention and
15

---

Elucidating the Design Space of Multimodal Protcin Language Models
attention-withh paihh--aa-biss modules respectively. Transition layers consist of gated mechanisms and a 2-layer MLP which expands the dimension of hidden representations by ratio of 4. We select the hidden dimension as 320 and use 16 heads in the attention attention-withh-aarr-be with--aa module of the SeqStruct attention module in Figure 4.
C.5 Representation Alignment
The process of REPA has been discussed in Section 4.2, we provide additional details below.
Overview. Representation alignment (Yu et al., 2024) is originally proposed in vision tasks which aligns the hidden embeddings with meaningful representations from the self-supervised pretrained encoders like DINOv2 (Oquab et al., 2023), dramatically improving the visual generation quality the the training efficiency of diffusion models (Rombach et al., 2022). Here we extend this technique to protein modeling by aligning representations from protein language models with representations from the folding model. Formally, we define the folding model as F = ʃ2 o f1, where F(s) x, with s denoting the sequence and x denoting the structures. The model first computes the semantic embeddings fi - y R4xD which encode rich 3D structural information that allows for precise structure predictions. Additionally, extract the hidden representations (s x*) = h from our protein language model, where s" and x* represent sequence and structure tokens with masked entries. REPA aligns (h) E Lx with the target representation. where ha (h) is a trainable 3-layer MLP that ensures the alignment of feature dimension. This alignment is achieved by maximizing the similarity between two representations:
(OE = Σ simmy! (,))),
where is the residux index, and sim -) is the cosine similarity function.
Precompute pair and structure representations. Using the protein sequences in our training data, we precompute the structure and pair representations of the folding model. We select ESMFold (Lin et al., 2023) as the target encoder due to its computational efficiency and open-source availability in comparison to other folding models (Jumper et al., 2021; Abramson et al., 2024). In particular, we extract the output representations of the folding trunk ESMFold. We run three iterations of recycling mechanisms within the folding trunk.
Multi-layer ensemble. Instead of aligning embeddings from a specific layer as in (Yu et al., 2024), we apply a learnable nn. Pa Parameter, followed by softmax function, to weight and aggregate representations across all layers. We refer to this approach as multi-layer ensemble. We use multi-layer ensemble as it achieves slightly better performance in folding RMSD and more importantly, eliminates the need select the alignment layer as hyperparameter.
C.6 Folding Supervised Finetuning
The folding SFT is to supervise the DPLM-2 to generate structure tokens given sequence. During training, we keep the sequence tokens unmasked and only mask the structure tokens. In Tables 4 and 7, we perform folding SFT based on the pretrained model to further enhance folding performance.
D
More Empirical Analysis
D.1 Structure-aware Generation and Predictive Tasks
Inverse folding (structure-conditioned sequence generation). We report the amino acid recovery (AAR) self-consistency TMscore on the Cameo dataset in Table 13. Our results show that the 650M-parameter DPLM-2 variant, with bit-based modeling, outperforms the 3B-parameter baseline. Incorporating geometric architectural designs and representation alignments leads to further improvements. These findings indicate that these design methods contribute positively beyond structure generation.
Table 13. Structure-condittonee sequence generation.
CAMEO 2022
AART
TMSCOREt
DPLM2 650M DPLM2 3B
0,4962
0.8816
0.5236
0.8900
DPLM2 BITWISE GEO + BITWISE
0.5586
0.8907
0.5665
0.8886
GEO + BITWISE + REPA 0.5681
0.8909
Structure-aware protein representation learning. In Table 14, we evaluate the performance of the models in learning structure-aware protein representations, following the evaluation protocol of SaProt (Su et al., 2023). Bit-based supervision
16

---

Elucidating the Design Space of Multimodal Protcin Language Models
further improves the representation learning performance of DPLM-2. We attribute this to the finer granularity of bitwise labels compared to conventional index-based ones, allowing the multimodal protein language model to capture the structural semantics in latent space more effectively.
Table 14, Structure-aware representation learning.
HUMANPPI
DEEPLOC SUBCELLULAR
MODEL
AcC (%)
ACC (%)
SAPROT
86.41
85.57
DPLM-2
84.44
82.98
DPLM-2 BITWISE
88.89
83.39
D.2 Case Study of Residual Diffusion
We present a visualization of the protein structures generated by the protein language model (PLM) and their refined counterparts through residual diffusion, with structural alignment shown in Figure 7. In this visualization, gray structures represent the initial PLM predictions, while blue structures denote the residual diffusion refinements. The dark blue regions highlight specific areas where residual diffusion has effectively adjusted the local structural details, as reflected in the secondary structure transition from flexible loops beta strands. These visualizations demonstrate residual diffusion's capability supplement fine-graine greiggr structural variations, rectify the potential local prediction errors from the PLM, and facilitate the formation of plausible secondary structures.
Figure 7. Case study of residual diffusion. We visualize the difference between the PLM generated structure and residual diffusion refined structure. The gray and blue structure correspond to the PLM prediction and the residual diffusionn refined one, respectively.
D.3 Representaion Alignment
Table 15. REPA alignment target. We analyze the impact of aligning different representations folding performance and generation diversity using GeoDPLM with 150M parameters. All models are trained for steps. Aligning structural representations improve both tasks.
TARGET
CAMEO 2022
DIVERSITY
STRUCT
PAIR
RMSDJ.
TMSCORE
CLUSTERS?
9.410
0.757
0.563
9.069
0.766
0.756
9.467
0.764
0.769
9.448
0.761
0.700
Alignment target ablation. We use GeoDPLM with 150M parameters to study the effects of aligning different types of representations Table 15. TMscores improves across all settings, suggesting that representation alignment guides the model to capture high-level 3D structural semantics. Aligning pair representations notably refines the generation diversity but slightly degrades the folding RMSD, which emphasizes precise local atom positions. Additionally, incorporating pair representations intensifies the 1/O bottleneck, slowing training by approximately five times compared using only structure representations. Aligning structural representations improve both the generation diversity and folding tasks, while ensuring similar training efficiency as the baseline.
Ensemble weight distribution of multi-layer ensemble. We present the learned ensemble weight distribution across
17

---

Elucidating the Design Space of Multimodal Protcin Language Models
Learnable Ensemble Weight of Each Layer
0.10
0.00
0.06
Ensemble
0.04
Layer
Figure Ensemble weight distribution layers. We visualize the learned ensemble weight after applying the softmax function. By leveraging multi- layer ensemble, we avoid the need for manually selecting the alignment depth hyperparameter in REPA.
layers in Figure 8. The weights mostly concentrated in the last four layers, with the highest value occurring at the second-to-last layer. However, the value (0.1069) is not significantly dominant over other values.
D.4 Multimer Exploration
Chain linker and position index offset. We present more comprehensive study by varying the lengths of chain linkers and position offset values in Tables 16 and 17, As shown in Table 16, the reconstruction accuracy steadily improves the offset increases from 0 to 5, and eventually reaching saturation at 25. In Table 17, chain linkers provide similar benefit to position offsets, with folding RMSD improving as the linker length increases, reaching optimal performance at length of 25.
Table 16. Position offset improves multimer reconstruction. We apply an offsct to the position indices in relative position embedding of the structure detokenizer to differentiate between chains, where the offset is determined by the product of chain index and a predefined value.
PDB-MU EDB-MULTIMER
OFFSET VALUE
RMSDL
TMSCORE
6.873
0.784
6,498
0.790
6.112
0.799
5.967
0.805
10
5.937
0.811
25
5.886
0.812
50
5,891
0.812
18

---

Elucidating the Design Space of Multimodal Protcin Language Models
Table 17. Apply chain linker and position offset multimer structure prediction. We use glycine (G) of various lengths as the linker connect different chains, and employ position offsct in the position indices within both DPLM-2 and the detokenizer during structure folding, ESMFold used G-linker of length 25 by default in multimer folding.
PDB-Multimer
Length/Valus
RMSD4
TMscoref
ESMFold
25
17.297
0.850
DPLM-2
0
19.110
0.768
1
19.044
0.770
3
18.757
0.770
5
18.579
0.773
w Chain Linker
10
18,808
0.768
25
17.966
0.771
50
18.208
0.770
1
18.848
0.769
5
18,700
0.770
w Pos. Offset
10
18.338
0.767
25
18.608
0.770
D.5 Analysis on Pretraining
Bit-based pretraining for hybrid modeling. As shown in Table 18, the "bitwise & FM" variant initialized from the bit-based DPLM-2 outperforms the model trained from scratch across both datasets. This demonstrates the potential of using bit-based DPLM-2 initialization to support hybrid generative modeling. This benefit might be attributed to: (1) DPLM-2 benefits from sequence pretraining that improves structural modeling (Lin et al., 2022); (2) Sequence data vastly outnumbers experimental structures, making pretraining more effective.
Table 18. Effects of pretraining hybrid modeling.
Camco2022
PDB date
RMSD
TMScore
RMSD
TMScore
Bitwise & FM init. from DPLM-2
6.1825
0.8414
2,8697
0,9099
Bitwise & FM maining from scratch
10.9815
0.7090
6,4453
0.8070
D.6 Training and Sampling Efficiency
Bit-based vs. hybrid approach. We report the training time of DPLM-2 variants with either bit-based modeling or hybrid approach on 16 H100s for 300k training steps in Table 19. The increased training time of flow matching (FM) primarily results from the on-the-fly computation of noisy structure x, using structure encoder, as we can precompute the discrete structure tokens when FM is not applied. FM remains highly effective when inference efficiency is priority, as it accelerates the sampling process by 10x by requiring fewer sampling steps.
Table 19, Efficiency comparison. We compare the training and sampling efficiency of the DPLM2 variants using cither bit-based or hybrid modeling.
TRAINING TIME
# SAMPLING STEPS
(300K STEPS)
w/o FM
100
46 HRS
FM
10
HRS
E
Related Work
Protein language models. Impacted by the success of large language models (LLMss), similar practice has been extended to the development of PLMs. For instance, sequence-base based protein language models operate by taking each amino acid sequence as a sentence. ESM-1b (Rives et al., 2019) utilizes self-supervised masked language modeling on 250 million protein sequences spanning evolutionary diversity, later leading to the development of ESM-2 (Lin et al., 2023) that further extends the scaling laws. DPLM (Wang et al., 2024b) adopts the bi-directional receptive field and scalable discrete diffusion
19

---

Elucidating the Design Space of Multimodal Protein Language Models
pre-training process (Ye et al., 2023b;a) which allows DPLM to exhibit a strong generative capability. ProtTrans (EInaggar et al., 2021), ProteinBERT (Brandes et al., 2022), PRoBERTa (Nambiar et al., 2020), ProtAlbert (Behjati al., 2022), TAPE (Rao et al., 2019), ProteinLM (Xiao et al., 2021), and CARP (Yang et al., 2022) involve several other representative masked language modeling (MLM) paradigm. These sequence-baser PLMs perform competitively with classic methods that rely on multiple sequence alignments, indicating that PI.Ms have captured some of the evolutionary information from sequences alone. In particular, these protein language models achieve powerful generalization on various downstream tasks involving the secondary and tertiary structures. Recent findings further showcase their capabilities in predicting protein functions (Meier et al., 2021), structure folding (Lin al., 2023), and de novo designs (Verkuil et al., 2022).
Multimodal protein language models. Proteins are defined by amino acid sequences and three-dimensiones structures, where the sequence dictates the structure, and the structure, in turn, determines the protein's 's function. Due to this intrinsic relationship, there has been increasing interest in developing multimodal PLMs that integrate both sequence and structural information. Many recent research has explored structure-enhance PLMs, building upon the success of sequence- PLMS (Zheng et al., 2023b). The key foundation of this direction is the adoption of VQ-VAE (Van Den Oord et al., 201 7b), which encodes the 3D structures into discrete tokens that represent the geometric conformation of each protein residue. For instance, SaProt (Su et al., 2023) constructs a structure-aware vocabulary that integrates both 3D and amino acid tokens. ProSST (Li et al., 2024) models the correlations between amino acids and structure tokens through a disentangled attention mechanism. ProstTS (Heinzinger et al., 2023) extends the sequence-basee based Pro'T5 (EInaggar et al., 2021) by training it to predict structure and sequence tokens conditionally. ESM-3 (Hayes et al., 2024) treats sequence and structures as separate tracks and applies an all-to-all masked language modeling approach. DPLM-2 (Wang et al., 2024a) further advances the joint distribution of sequence and structures through discrete diffusion mechanism, enabling it to effectively denoise the masked sequence or structure tokens. This method allows DPLM-2 to learn meaningful representations that enhance predictive tasks, while achieving strong generative capabilities, such as conditional sequence generation and co-generation of sequence and structures.
Protein structure prediction. Leveraging deep learning approaches, research in structure folding and structure generation has advanced significantly. One key direction end-to-end structure prediction, where models map amino acid sequences to dine--imensional structures. Notable breakthroughs include the AlphaFold series (Senior et al., 2020; Jumper et al., 2021), ESMFold (Lin et al., 2023) and RoseTTAFold (Baek et al., 2021). As structure folding does not require sequence generation, these methods typically operate directly in the original data space, as seen in AlphaFold3 (Abramson et al., 2024). In addition, many of these methods innovate architectures by incorporating geometric properties of proteins. For example, OmegaFold (Wu et al., 2022) introduces Geoformer, which iteratively refines the sequence and pairwise representations to minimize the geometric inconsistencies. AlphaFold2 (Jumper et al., 2021) employs Evoformer to model 2D pairwise representations and update them through series of triangle operations. These designs highlight the importance of capturing higher-order relations for accurate structure prediction.
Protein structure generation. Another growing line of work involves structure generation, which aim to generate novel protein structures and functional design (Ye et al., 2025). In this direction, diffusion approaches have demonstrated remarkable success. For example, RFDiffusion (Watson et al., 2023) excels at guiding protein structure designs to meet specific functional constraints like enzymes. Chroma (Ingraham al., 2023) enables the generation of proteins and protein complexes and allows flexible property-based prompting. This approach efficiently generates large structures by transforming collapsed polymers into protein backbones and sequences. ProteinSGM (Lee et al., 2022b) facilitates novel 3D protein design using 2D maps of distances and angles as conditions. Multiflow (Campbell et al., 2024b) achieves crructur-sequeene co-generation by developing multimodal flow matching that allows the hybrid modeling of continuous structural data and discrete sequences. Despite the promising success in structure generation, these approaches often struggle in sequence-conditioned structure prediction, primarily due to the lack of large-scale training on sequence databases. Recent work further includes sidechains and extend to modeling all-atom structures for generating protein complexes and function design (Chen al., 2025).
20


Reviews:
[{"review_id": "RyOTbG1OfY", "reviewer": "Reviewer_4dFc", "summary": "This paper aims to systematically improve current multimodal protein language models in the following respects: (1) generative modeling, where the authors argue that structural information loss caused by index-based structure tokenization cannot be resolved by improving reconstruction accuracy, and opt for a finer-grained supervision through bit-wise cross entropy loss; (2) geometry-aware architecture and representation alignment to improve higher-order structural modeling, (3) extended data coverage on multimer besides monomer. By doing so, the authors show that the enhanced model built upon DPLM-2 achieves better structural modeling abilities, e.g. reducing the RMSD from 5.52 to 2.36 on PDB test set.\n\n## update after rebuttal\nThe authors addressed all my concerns. I raised my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback! \n\n> W1: the paper would benefit from a more structured presentation of its contributions (e.g. a taxonomy)\n\nThank you for your suggestion. We present a comprehensive taxonomy of design choices in a table and discuss their orthogonality and synthesis, as compiled in this [TABLE](https://anonymous.4open.science/r/icml-BF61/Table1.pdf).  \n\n> Q1: What is the final recommended design choice? (e.g., combining GeoDPLM + FM + RESDiff + folding SFT + REPA altogether.)\n\nWe structured the paper to clearly present the impacts of each design. To provide a complete picture, we now include **new results** that combine all design methods.\n\n|  | PDB Date |  | Cameo2022 |  | Uncond. Gen. |\n|---|---|---|---|---|---|\n|  | RMSD↓ | TM↑ | RMSD↓ | TM↑ | Diversity↑ |\n| DPLM-2 650m | 5.307 | .8306 | 7.703 | .7936  | 0.700 |\n| Bit | 3.221 | .9043 | 6.403  | .8380  | 0.825 |\n| Bit + FM | 2.870 | .9099 | 6.183 | .8418 | 0.525 |\n| Bit + FM + ResDiff (w/o SFT)  | 2.788 | .9146 | 6.077  | .8456 | 0.525 |\n| Bit + FM + SFT + ResDiff | 2.370  | .9270 | 5.847  | .8442  | N/A |\n| New results |  |  |  |  |  |\n| Geo + Bit | 2.551 | .9254 | 5.955 | .8520 | 0.900 |\n| Geo + Bit + FM | 2.443 | .9261 | 6.172 | .8404 | 0.575 |\n| Geo + Bit + REPA | 2.507 | .9264 | 6.192 | .8412 | 0.875 |\n| Geo + Bit + REPA + SFT | 2.404 | .9322 | 5.754 | .8424 | N/A |\n| Geo  + Bit + FM + SFT + REPA + ResDiff | 2.379 | .9297 | 6.200 | .8398 | N/A |\n\n**SFT**: Folding SFT improves the structure folding but sacrifices the model's ability for multimodal co-generation, as it is fine-tuned specifically for the folding task. For models finetuned with the folding SFT objective, we skipped the evaluation of unconditional co-generation. \n\n**The recommended setting** — Geo + Bitwise. Geometric architectures are compatible with bitwise modeling and their combinations achieve comparable results with models finetuned with folding SFT on structure folding, and further obtains an effective improvement on unconditional generation quality & diversity. This setting is also effective in terms of training efficiency as it avoids additional computational overhead from other methods like FM and REPA.\n\n**REPA and bit-based modeling**. Both REPA and bit-based modeling enhance structure folding and generation diversity. Meanwhile, as shown in the table, their combinations do not lead to further improvements. We suggest that this is because REPA and bit-based modeling both help through enabling smooth and high-dimensional learning signals compared to index-based discrete tokens, hence their non-orthogonal effects. \n\n**Hybrid modeling (w/ FM) and geo module**. FM effectively improves folding, but the benefits diminish with Geo modules, and can reduce generation diversity due to its ODE nature. However, benefitting from the same nature, FM accelerates the sampling process by requiring 10x fewer sampling steps.\n\n**ResDiff**. Similar to the results in the paper, ResDiff does not bring a significant boost to folding metrics. The major benefit of ResDiff is to provide a finer local structure as discussed in the Fig. 7 of our Appendix.\n\n> Q2: Clarifications on experimental configurations in Table 4 (+ResDiff, +FM, and w/SFT).\n\nWe elaborate on our experiment settings, as compiled in this [TABLE](https://anonymous.4open.science/r/icml-BF61/Table2.pdf)\n\n> Q3: Hybrid structure modeling method seems to rescue the skewed structure modeling ability of DPLM-2, which adopts the pre-trained sequence model DPLM and may be suboptimal compared to training from scratch. \n\nThanks for raising this insightful discussion. We conducted experiments as you suggested\n\n|  | Cameo2022 |  | PDB Date | |\n|---|---|---|---|---|\n|  | RMSD↓ | TMScore↑ | RMSD↓ | TMScore↑ |\n| Bit & FM from pretrained DPLM-2 | 6.1825 | 0.8414 | 2.8697 | 0.9099 |\n| Bit & FM from scratch | 10.9815 | 0.7090 | 6.4453 | 0.8070 |\n\nAs shown, DPLM-2 initialization outperforms training from scratch (see table), likely because: (1) sequence pretraining improves structural modeling (cf. ESMFold); (2) Sequence data vastly outnumbers experimental structures, making pretraining more effective. \n\n> Q4: On REPA boosting diversity and accuracy? Is it related to the unbalanced training set for the original DPLM?\n\nWe agree that the unbalanced DPLM training set may contribute to REPA's benefits. REPA further aids structure generation by:\n\n- ​Overcoming discrete token limitations: Quantization loses finer details (Table 1), while representation alignment helps preserve structural nuances via smooth, informative and high-dimensional learning signals. \n\n- ​On learning diffusion model: REPA paper points that high-quality representations are key to diffusion models; we bridge this gap by transferring structural features from folding models.", "claims_and_evidence": "The claims in this paper are supported by convincing evidence.", "methods_and_evaluation_criteria": "This paper focuses on structure modeling ability of multimodal PLMs, so I think evaluation on structure prediction in RMSD / TMscore suits it fine.", "theoretical_claims": "N/A (no theoretical claims in this paper)", "experimental_designs_or_analyses": "I have checked the experimental designs, but some settings are unclear to me. Please see Q2.", "relation_to_broader_scientific_literature": "The work is a meaningful attempt at extending multimodal diffusion protein language model. Previous work DPLM-2 incorporates structural tokens, where discretization brings the benefit of reduced noise and the potential of scaling. This paper builds on top of DPLM-2, and proposes several improvements in advancing its structure modeling ability.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n- This paper explores a range of design choices for multimodal protein language models, specifically DPLM-2, which is informative and complementary to the previous study.\n- The authors conduct an in-depth preliminary study on structure tokenization and prediction by LM.\n- Detailed and insightful ablation studies.\n\nWeakness:\n- This paper would benefit from a more structured organization of its contributions to the design space of multimodal protein language models. Rather than presenting incremental improvements in a fragmented manner (as \"DPLM-2 + X\" throughout different sections), the authors should consider providing a comprehensive taxonomy of the design choices explored, highlighting how these choices are orthogonal and can be composable, and positions these innovations within the broader context of multimodal protein models. Given the authors' ambitious goal of elucidating design spaces for multimodal PLMs, a more systematic synthesis of findings would substantially enhance the paper's impact and utility to the community.", "other_comments_or_suggestions": "- L92 \"DPLM-2: An multimodal extension of DPLM\" -> \"A multimodal\"\n- Table 4 caption is a bit confusing.", "questions_for_authors": "1. What is the final version of the recommended design choice? It seems to me that the authors separately propose specific improvements to DPLM-2, and the entire paper seems like an extensive ablation study for it. But if all the designs are modular and complementary to each other, why didn't the authors show the overall performance of combining GeoDPLM + FM + RESDiff + folding SFT + REPA altogether? \n2. I did not quite understand the setting in Table 4. As far as I know, DPLM-2 already incorporates a diffusion LM in it, so did \"+RESDIFF\" add another continuous diffusion to it? And how about \"+FM\"? L240 states that the authors \"finetune such a model with flow matching\", does that (DPLM-2 + FM) mean the original diffusion LM is finetuned by FM generative objective (both seq and struct), and \"w/ folding SFT\" denotes the generative objective of structure only (i.e. denoise structure given sequence)? \n3. Following this, the so-called \"hybrid structure modeling method\" seems to rescue the skewed structure modeling ability of DPLM-2, which adopts the pre-trained sequence model DPLM and finetunes for additional structure tokens. This raises the question that direct fine-tuning approach on DPLM may be suboptimal - why didn't authors train from scratch instead of inheriting DPLM-2 parameters? Wouldn't it yield even better performance if both modalities are jointly considered throughout training?\n4. I am curious about how the representation alignment to folding model boosts structure diversity as well as accuracy. Can the authors elaborate on that? Does this have something to do with the potentially unbalanced training set for original DPLM in terms of protein sequence length?\n\nI would consider raising my score if the authors solve these questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PwpCg31CVY", "reviewer": "Reviewer_i971", "summary": "This paper performs an exploration of how to improve multimodal protein language models that jointly model both protein sequences and structures. The paper identifies limitations in the existing literature of token-based multimodal PLMs and propose (and explore) many design choices for such PLMs. The paper identifies two issues in token-based multimodal PLMs: (i) information loss when converting continuous 3D structures into discrete tokens and (ii) difficulty in accurate structure token prediction. They develop several design choices to address this and show that their design methods improve the structural modeling capabilities of multimodal PLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!\n\n> The bit-level and index-level evaluation is really stark. Is there any broader intuition behind this that you could check experimentally? \n\nIndex-level prediction is highly challenging because small changes at the bit level can result in drastically different indices, as shown in the example below. This issue becomes even more problematic as the codebook size increases, further exacerbating the difficulty of direct index prediction.\n\n| Continous struct token | Quantized struct token (bit level) | Index |\n|---|---|---|\n| **+0.1**, -1.5, -3.2, +0.7 | **+1**, -1, -1, +1 | **9** |\n| **-0.1**, -1.5, -3.2, +0.7 | **-1**, -1, -1, +1 | **1** |\n\n\n\n> W1: Some design choices feel somewhat ad-hoc due to the lack of discussions on motivation. It could benefit from greater clarity with a summary table linking the whole design choices. \n\nThank you for this comment. To make our presentation more structured, we provide a summary [TABLE](https://anonymous.4open.science/r/icml-BF61/Table1.pdf) with greater discussion on all the design choices, traditional methods, motivations, and findings.\n\n\n> W2: It would benefit to have more uncertainty estimates in the tables, especially because some of the values highlighted seem to have small gains.\n\nThank you for your comment. We are actively launching three more independent runs of the following selected models to provide uncertainty estimates using the mean and the 95% confidence interval (CI). We will include the uncertainty estimates of other variants in the final version.\n\n|  | Cameo2022 |  | PDB Date |  |\n|---|---|---|---|---|\n|  | RMSD↓ | TMScore↑  | RMSD↓ | TMScore↑  |\n| Bit + FM | 6.210 ± 0.100 | 0.840 ± 0.005 | 2.811 ± 0.052 | 0.914 ± 0.010 |\n| Bit + FM + ResiDiff | 6.088 ± 0.085 | 0.844 ± 0.001 | 2.777 ± 0.161 | 0.916 ± 0.005 |\n\n> W3: It would be beneficial to provide a summary table with greater discussion about the novelty of the design space.\n\nThank you for your suggestion. We provide the summary table above in our response to W1.\n\n> The overall analysis is done on relatively small PLMs (up to 3B). I don't see this as an issue and I understand the constraints of computation resources, but it would be great to acknowledge more explicitly.\n\nThanks! We acknowledge this statement and will make it explicit in our final version as you suggested. While scaling is effective as shown in DPLM-2, we'd like to reassert that correct design choices are critical. In our work, we show that our 650M multimodal PLM could achieve on par results with the 3B specialized ESMFold on structure folding, showing high parameter efficiency.\n\n> Some missing benchmarks (RoseTTAFold, OmegaFold, and AlphaFold3) on structure prediction tasks could be useful, even though I'm happy for the authors to claim that the focus is on protein language models, which differ from the nature of these benchmarks.\n\nThanks for your suggestions. We will make a clear claim on our focus on PLMs and will add these benchmarks to the table in the final version.", "claims_and_evidence": "The paper explores many design choices and evaluates them empirically. Most claims and design choices are given appropriate empirical evaluation which seems to be well supported. I'll provide three examples of this. First, bit-wise modeling seems to improve structure prediction accuracy as showcased in Table 3 and Table 4. Second, representation alignment does seem to enhance model performance (shown in Table 6).  Third, position offset and chain linkers help in multimer modeling (Table 9, Table 10).\n\nThe overall analysis is done on relatively small PLMs (up to 3B). I don't see this as an issue to the claims and I understand it's hard to scale up the model size if compute is limited, but it would be great to acknowledge more explicitly that the tests are done on smaller PLMs and that there is uncertainty in how it scales. This is especially true because there are no evaluations about how this performance and gains scale with the size of the PLMs.", "methods_and_evaluation_criteria": "The paper's methods for evaluating different design choices is nice and convincing. It provides limitations of existing token-based multimodal PLMs (especially information loss from structure tokenization). The proposed methods are quite broad, they target multiple aspects of the problem at once (tokenization, architecture, representation learning, data) with the experiments showing improvements from the design choices. Their methods primarily build on DPLM-2.\n\nGiven that this is a protein structure prediction task, it seems some benchmarks are missing (RoseTTAFold, OmegaFold, AlphaFold3). I'm happy for the authors to generally claim that your goal is to work on protein language models that might differ from the nature of, say, AlphaFold3, and that your model sizes are different, such an analysis would still be useful to see by how far off the existing research is.", "theoretical_claims": "There are no proofs. The formulations of diffusion process, learning objectives for DPLM-2 and residual diffusion models, are clear.", "experimental_designs_or_analyses": "There are quite a few experimental designs in this paper (it's even a bit hard to follow all of them). Two main questions.\n\nFirst, on structure tokenization analysis. The bit-level and index-level evaluation is really stark. How come? Is there any broader intuition behind this that you could check experimentally?\n\nSecond, the ablation study comparing different improved methods is nice and well presented. For some improvements (e.g. in Table 4), it's hard to know what's a genuine improvement because there are no uncertainty intervals (e.g. standard deviations for splits).", "relation_to_broader_scientific_literature": "The paper fits into several areas. First, the paper identifies information loss during structure tokenization and propose ways to deal with this by connecting it to a few different literature areas: (i) vector quantization literature; (ii) latent diffusion model literature; and (iii) the more recent bit-wise discrete modeling literature, connecting it to image synthesis by Han et al. (2024). Second, the geometric structure-aware architecture design uses many ideas from AlphaFold2 and AlphaFold3. Third, the representation alignment connects well with knowledge distillation.\n\nOverall, this paper aims to bridge the gap between sequence-based and structure-based models, and demonstrates how techniques from various fields (e.g. computer vision) can be adapted to protein modeling.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other strengths: Many different design choices are proposed as well as evaluated.\n\nOther weaknesses: In some cases, the design choices feel somewhat ad-hoc (i.e. why did you exactly decide to choose to test this design choice as opposed to another one?). It is quite difficult to follow structure-wise and could benefit from greater clarity in terms of the presentation structure (e.g. a summary table linking the whole design choices). Lastly, it would benefit to have more uncertainty estimates in the tables, especially because some of the values highlighted seem to have small gains.\n\nIt would also be beneficial if you could provide a greater discussion about whether and how the design space exploration is novel relative to other work (what other work has explicitly done/explored and what is completely novel), especially within a summary table.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Described above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yfknhgOU68", "reviewer": "Reviewer_3QKf", "summary": "The manuscript systematically explores the design space of multimodal PLMs, aiming to identify and address their existing limitations. The authors highlight tokenization loss and inaccurate structure token predictions as major bottlenecks in structure prediction performance. To mitigate these issues, they propose multiple strategies, including bit-wise prediction, quantization residue modeling, and a hybrid generative approach that integrates the pretrained encoder-decoder with the PLM. The study is supported by comprehensive analyses and experiments, which empirically validate the conclusions and demonstrate notable performance improvements.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!\n> W1: The current manuscript does not address structure-conditioned protein generation or structure-aware protein predictive tasks.\n\nThanks for your suggestion. We provide additional results on structure-conditioned protein generation and protein predictive tasks below.\n\n**(1) Inverse Folding (structure-conditioned sequence generation)**. \n\nWe report the amino acid recovery (AAR) and self-consistency TMscore on the Cameo dataset. Our results show that the 650M-parameter DPLM-2 variant, with bit-based modeling, outperforms the 3B-parameter baseline. Incorporating geometric architectural designs and representation alignments leads to further improvements. These findings indicate that these design methods contribute positively beyond structure generation.\n\n|  | Inverse Folding - Cameo 2022 |  |\n|---|---|---|\n|  | AAR↑ | TMscore↑ |\n| DPLM-2 650M | 0.4962 | 0.8816 |\n| DPLM-2 3B | 0.5236 | 0.8900 |\n| DPLM-2 Bit | 0.5586 | 0.8907 |\n| Geo + Bit | 0.5665 | 0.8886 |\n| Geo + Bit + REPA | **0.5681** | **0.8909** |\n\n**(2) Structure-aware protein representation learning.**\n\nIn the table below, we provide the results of structure-aware protein representation learning, following the setting of SaProt. Due to the limited time, we only evaluate on the HumanPPI and DeepLoc Subcellular tasks, and we will provide results of more protein predictive tasks in the final version.\n\nAs seen, the proposed bit-based modeling is able to further improve the representation learning performance of DPLM-2. We hypothesize that the bit-based modeling, which is finer-grained, is more suitable to learn and enables the multimodal PLM to capture the structural pattern in latent space more effectively, thus enhancing structure-aware representation learning.\n\n| Model | HumanPPI | DeepLoc Subcellular |\n|---|---|---|\n|  | Acc (%)↑ | Acc (%)↑ |\n| SaProt | 86.41 | 85.57 |\n| DPLM-2 | 84.44 | 82.98 |\n| DPLM-2 Bit | 88.89 | 83.39 |\n\n> W2: The model fine-tuned without PDB multimer data achieved a higher TM-score but a lower RMSD. A more in-depth analysis would provide valuable insights.\n\nThanks for bringing up this discussion. RMSD captures local atom deviations, while TM-score measures global structural similarity. In multimers, chains are typically spaced farther apart than individual connecting residues, leading to a higher RMSD for models trained on only monomers. This highlights the importance of finetuning with multimer data to reduce RMSD. \nMeanwhile, fine-tuning on Swissprot (200K) helps learn the global protein structures better due to its larger dataset compared to PDB-Multimer (3.5K). As shown in Table 10 of our paper, incorporating Swissprot consistently achieves higher TM-score on both PDB-Multimer and Cameo.\n\n> Q1: Does the hybrid generative approach with flow matching significantly increase training time? \n\nWe report the training time of the following DPLM-2 variants on 16 H100s for 300k training steps. \nThe increased training time of flow matching (FM) primarily comes from the on-the-fly computation of noisy structure $\\mathbf{x}_t$ using structure encoder, as we can precompute the discrete structure tokens when FM is not applied. FM remains highly effective when inference efficiency is a priority, as it accelerates the sampling process by 10x by requiring fewer sampling steps.\n\n|  | # Sampling steps | Training Time (300k steps) |\n|---|---|---|\n| w/o FM | 100 | 46 hrs |\n| w/ FM | 10 | 81 hrs |", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The claims in the manuscript are supported by empirical results.", "experimental_designs_or_analyses": "Expriments are well designed and alalyses are logically reasonable.", "relation_to_broader_scientific_literature": "Multimodal protein language models have been shown in previous research to be highly effective for protein representation. However, their performance in structure prediction remains suboptimal. This paper makes key contributions by identifying the major limitations that restrict performance and proposing targeted strategies to overcome them, leading to significant improvements in folding prediction. Enhancing structure prediction not only increases modeling accuracy but also improves structure generation, ultimately advancing protein design.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-written, with a clear structure and comprehensive experiments.\n\nWeaknesses:\n\n1. As mentioned in the supplementary material, the manuscript does not currently address structure-conditioned protein generation or structure-aware protein predictive tasks, which are crucial for functional protein design.\n\n2. The performance on multimer data is particularly intriguing. When trained on SwissProt without PDB multimer training data, the TM-score is higher than when fine-tuned with PDB multimer data, yet the RMSD is worse. A more in-depth analysis of this phenomenon would provide valuable insights.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Will the hybrid generative approach significantly increase training time? Providing detailed training time measurements would be helpful for assessing its feasibility and computational cost.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rqa0yUWhkH", "reviewer": "Reviewer_RSLA", "summary": "The paper discusses the design space of multimodal protein language models through the lens of discretizing protein structures. The paper attempts to address a key challenge with information loss of fine-grained structural details upon tokenization. Multimer structures are considered, and BIT-based modeling is used to improve structural tokenization and generation from multimodal models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your comments that have greatly improved our manuscript! We address your concerns as below. We sincerely thank you once again and welcome any further feedback!\n\n> Q1: Can the authors comment on any simple baselines or alternative approaches they pursued to resolve the identified problems (O1, O2, and O3)?\n\nWe recap our approaches to address O1-O3 and discuss potential alternative approaches below.\n\n**O1: Structure tokenization results in information loss**\n\n**Our approach:** We train an additional continuous diffusion module ResDiff to recover the lost residuals of discrete tokens.\n\n**Alternatives:** Training DPLM2 on continuous tokens. However, it's unclear whether this approach would work for joint modeling of structures and sequence, which is inherently discrete. \n\n**O2: High reconstruction accuracy of structure tokenizer does not ensure better structural generative results.**\n\n**Our approach:** We primarily improve the designs of latent structure modeling, including structure-aware generative modeling, architectural designs, and representation alignments.\n\n**Alternatives:** Adopt direct modeling in the data space following a similar manner as Proteina [1]. Similarly, it's unclear if direct modeling is robust for joint modeling of discrete sequences and continuous 3D structures. \n\n**O3: Multimodal PLM gets index-based structure token prediction miserably wrong.**\nSmall bit-wise changes could result in a dramatically different index label. This challenge intensifies as codebook size grows, making direct index prediction even more difficult.\n\n**Our approach:** Finer-grained token prediction might resolve such challenge. We achieve a finer-grain prediction on the \"dimension-level\" using bit-based labels in contrast to index-based labels.\n\n**Alternatives:** Another potential direction is to hierarchically tokenize the structures to achieve fine-grained tokens at the \"resolution\" level following methods like RQ-VQVAE [2] and VAR [3]. However, it remains elusive how to well define \"resolution\" of proteins as a natural choice. \n\n> Q2: The paper discusses SaProt but does not explicitly compare to it or other related \"structure-aware\" pLMs beyond ESM3. Can the authors comment on their reasoning for including/not including specific comparisons?\n\nThanks for your question. In our submitted version, we focus on the generative capabilities of protein structure (e.g., folding). However, SaProt focuses on the protein representation learning ability, which utilizes structure tokens as the representation for structure input and is not capable of generating structure.\n\nIn the table below, we also provide the results of structure-aware protein representation learning, following the setting of SaProt. Due to the limited time, we only evaluate on the HumanPPI and DeepLoc Subcellular tasks, and we will provide results of more protein predictive tasks in the final version.\nAs seen, the proposed bit-based modeling is able to further improve the representation learning performance of DPLM-2. We hypothesize that the bit-based modeling, which is finer-grained, is more suitable to learn and enables the multimodal PLM to capture the structural pattern in latent space more effectively, thus enhancing structure-aware representation learning.\n\n| Model | HumanPPI | DeepLoc Subcellular |\n|---|---|---|\n|  | Acc (%)↑ | Acc (%)↑ |\n| SaProt | 86.41 | 85.57 |\n| DPLM-2 | 84.44 | 82.98 |\n| DPLM-2 Bit | 88.89 | 83.39 |\n\n[1] Geffner et al. Proteina. ICLR 2025.\n\n[2] Lee et al. RQ-VAE. CVPR 2022.\n\n[3] Tian et al. VAR. NeurIPS 2024.\n\n\n> Essential References Not Discussed: Lu, Amy X., et al. \"Compressing the Latent Space of Single-Sequence Protein Predictors for Multimodal Generation.\" \n\nThanks for pointing this out to us, and we will elaborate on our discussion below in the final version. \n\nThis work compresses the latent space of ESMFold to form a joint latent space of protein sequence and structure, and learn a continous generative model to sample compressed latent embeddings, which is then decoded by the ESMFold structure head and a learned sequence head. On the other hand, the multimodal protein language models like DPLM-2, use discrete diffusion generative framework to generate the discrete structure token, then decode using the learned structure detokenizer.\n\nThe CHEAP embedding introduced by this work is an effective compression of ESMFold latent space. Integrating the CHEAP embedding with multimodal protein language models holds great potential for effective structural modeling. This is a very exciting direction and we will leave it for future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Well contextualized and motivated with respect to challenges in the field, specifically MultiFlow and ESM3.", "essential_references_not_discussed": "May be of interest to the authors:\n\nLu, Amy X., et al. \"Compressing the Latent Space of Single-Sequence Protein Predictors for Multimodal Generation.\" ICML 2024 Workshop on Efficient and Accessible Foundation Models for Biological Discovery.", "other_strengths_and_weaknesses": "Considering representations of multimeric systems is an important problem in protein design and progress in this area could have significant downstream applications. Addressing limitations in current structure tokenization methods is similarly important and well-motivated.\n\nCan the authors comment on any simple baselines or alternative approaches they pursued to resolve the identified problems (O1, O2, and O3)?\n\nThe paper discusses SaProt but does not explicitly compare to it or other related \"structure-aware\" pLMs beyond ESM3. Can the authors comment on their reasoning for including/not including specific comparisons?", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores ways to improve multimodal protein language models (PLMs) that combine protein sequences and structures. The authors identify two key challenges: (1) loss of structural information when converting 3D structures into tokens and (2) difficulty in accurately predicting these tokens. To address these, they propose finer-grained supervision using bit-wise cross-entropy loss, better alignment between sequence and structure representations, and the use of data that includes both single proteins (monomers) and complexes (multimers). 

All reviewers vote for accept.
Decision: Accept (spotlight poster)