Structure Language Models for Protein Conformation Generation
Authors: Jiarui Lu
Xiaoyin Chen
Stephen Zhewen Lu
Chence Shi
Hongyu Guo
Yoshua Bengio
Jian Tang
Venue: ICLR 2025 Poster (2025)
Keywords: protein
conformation generation
conformation sampling
generative models
language models
diffusion models
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Protein conformation generation via conditional language modeling in the discrete latent space of protein structures.

Abstract: Proteins adopt multiple structural conformations to perform their diverse biological functions, and understanding these conformations is crucial for advancing drug discovery. Traditional physics-based simulation methods often struggle with sampling equilibrium conformations and are computationally expensive. Recently, deep generative models have shown promise in generating protein conformations as a more efficient alternative. However, these methods predominantly rely on the diffusion process within a 3D geometric space, which typically centers around the vicinity of metastable states and is often inefficient in terms of runtime. In this paper, we introduce Structure Language Modeling (SLM) as a novel framework for efficient protein conformation generation. Specifically, the protein structures are first encoded into a compact latent space using a discrete variational auto-encoder, followed by conditional language modeling that effectively captures sequence-specific conformation distributions.  This enables a more efficient and interpretable exploration of diverse ensemble modes compared to existing methods. Based on this general framework, we instantiate SLM with various popular LM architectures as well as proposing the ESMDiff, a novel BERT-like structure language model fine-tuned from ESM3 with masked diffusion. We verify our approach in various scenarios, including the equilibrium dynamics of BPTI, conformational change pairs, and intrinsically disordered proteins. SLM provides a highly efficient solution, offering a 20-100x speedup than existing methods in generating diverse conformations, shedding light on promising avenues for future research.

Published as a conference paper at ICLR 2025
STRUCTURE LANGUAGE MODELS FOR PROTEIN CONFORMATION GENERATION
Jiarui Lu Xiaoyin Chen , Stephen Z. Lu Chence Shi Hongyu Guo Yoshua Bengio & Jian Tangl.6
Miila- - Québec AI Institute, Université de Montréal, "MeGill University, "University of Ottawa, National Research Council Canada, CCIAR AI Chair, "HEC Montréal
ABSTRACT
Proteins adopt multiple structural conformations to perform their diverse biologi- cal functions, and understanding these conformations crucial for advancing drug discovery. Traditional physics-based simulation methods often struggle with sam- pling equilibrium conformations and are computationally expensive. Recently, deep generative models have shown promise in generating protein conformations as a more efficient alternative. However, these methods predominantly rely on the diffusion process within 3D geometric space, which typically centers around the vicinity of metastable states and is often inefficient in terms of runtime. In this pa- per, introduce Structure Language Modeling (SLM) novel framework for efficient protein conformation generation. Specifically, the protein structures are first encoded into compact latent space using discrete variational auto-encoder, followed by conditional language modeling that effectively captures sequence- specific conformation distributions. This enables more efficient and interpretable exploration of diverse ensemble modes compared to existing methods. Based on this general framework, we instantiate SLM with various popular LM architec- tures as well as proposing the ESMDiff, a novel BERT-like structure language model fine-tuned from ESM3 with masked diffusion. We verify our approach various scenarios, including the equilibrium dynamics of BPTT, conformational change pairs, and intrinsically disordered proteins. SLM provides highly effi- cient solution, offering a 20-100x speedup than existing methods in generating diverse conformations, shedding light on promising avenues for future research.
INTRODUCTION
Protein structure dynamics fundamental to understanding the biological functions of proteins. The ability of proteins to adopt multiple conformations is crucial for their function in influencing interactions with other biomolecules and the environment. Traditional computational methods, such as molecular dynamics (MD) simulations, have long been used to explore these dynamics. However, these methods are computationally expensive and time-consuming. Structure prediction models, such as AlphaFold 2 (Jumper et al., 2021) and RosettaFold (Baek et al., 2021), have made significant strides in predicting static protein structures, yet often fail to accurately capture the dynamic nature of proteins and their multiple conformations (Chakravarty & Porter, 2022).
Recently, significant progress has been made by adopting generative models to efficiently explore the complicated protein conformational space. For example. Noé et al. (2019) adopts normalizing flow to match the underlying Boltzmann distribution by learning from simulation data. Despite their potential, normalizing flow-based methods (Noé et al., 2019; Klein et al., 2023) face challenges in modeling large protein systems with hundreds of amino acids, as the invertibility constraint be- comes major obstacle when scaling up model parameters. As a remedy, denoising diffusion can efficiently learn from structural data (Jing et al., 2023; Lu et al., 2024b; Wang et al., 2024; Zheng et al., 2024), achieve good generalization, and perform amortized inference. However, modeling
EEuual contribution. Code available https: / wtt.hu/woooom//tiuuuii.iaa Correspondence to: jiarui 1u@mi 1 qusbeec, waa....angeee....

---

Published as a conference paper at ICLR 2025
nigh-dimensional protein structures explicitly in their 3D Euclidean space can demand intensive computation and usually requires accounting for special equivariant properties (Köhler et al., 2020). Furthermore, L2-based training objectives such as denoising score matching (Song et al., 2020) tend to predict local perturbations rather than capturing remote modes of alternative conformations (Wang al., 2024). Consequently, these models may overallocate their capacity to learn structural noises in the training data instead of focusing on low-frequency structural changes (Chou, 1985).
In complement with existing approaches, we present Structure Language Modeling (SLM), novel framework for protein conformation generation that performs generative modeling in the latent space of protein structures. Inspired by the recent progress in developing structural vocabularies for pro- tein representation learning (Su et al., 2023; Hayes et al., 2024), our approach first encodes structural flexibility into a distribution over latent tokens using discrete variational autoencoder, as illustrated in Fig. 1. The discrete latent encoding removes high-frequency details of protein structures, form- ing "structure languages" that effectively capture the uncertainty of complex protein conformations (Fig. 2a); Conditional language modeling is then applied to these latent structure tokens, using amino acid types as context to capture sequence-specífic conformation distributions (Fig. 2b); Protein con- formations finally be reconstructed by mapping structure tokens into 3D space with learned decoder (Fig. 2c). By leveraging generative language modeling in the discrete latent space, SLM by- passes the complexity of equivariant constraints associated with geometric symmetries and benefits from enhanced model capacity. As general framework, SLM is fully compatible with any existing language model (LM) architectures and shows promising scalability. To further demonstrate the versatility of our approach, we introduce ESMDiff, a novel BERT-like structure language model in- stantiation fine-tuned from ESM3 (Hayes al., 2024) with masked discrete diffusion (Austin al., 2021; Zhao et al., 2024) grounded in the SLM framework. Experimental results across various con- formation generation scenarios demonstrate the state-of tfte-of--the-ar thet-ar performance of SLM including the representative ESMDiff model, achieving orders of magnitude faster speeds compared to existing generative methods. The proposed framework paves the way for new research avenues addressing the protein conformation sampling challenge.
tall
We summarize our key contributions as follows.
GLY12
We comprehensively explore innovative conformation generation framework based on language modeling in the latent space, which opens up potential research avenues.
00000 Flexibility
t.- GLU49
00000 Flexibility
We introduce ESMDiff, a novel fine-tuned variant of a state-of-the-art protein language model, built on masked discrete diffusion.
Figure 1: Residue flexibility (BPTI clusters, Shaw al. (2010)) reflected by the categori- cal distribution over latent structure tokens. Dif- ferent tokens (colored in different shades ) are used to encode varying local structural patterns.
We demonstrate the superior capability of structure language models by evaluating them on various conformation generation settings and comparing them with existing methods.
2 RELATED WORK
Protein language models. In recent years, several language models of protein sequence have been built. Among these, the ESM-series (Rives et al., 2021: Lin al., 2023; Hayes et al., 2024) and other similar models (Einaggar et al., 2021; Alley et al., 2019) have garnered great attention because of their wide range of downstream applications such as protein engineering (Meier et al., 2021). On the other hand, auto-regressive protein language models, based on either recurrent neural networks (AI- ley et al., 2019), or Transformer including ProGen (Madani et al., 2023) and ProtGPT2 (Ferruz et al., 2022), able to generate de novo sequences with input controlling tokens. Specially, inverse fold- ing models (Ingraham et al., 2019; Jing et al., 2020; Hsu et al., 2022: Dauparas et al., 2022; Gao et al., 2022) learn to perform structure-based protein design with geometric-awane encoders.
Generative conformation sampling. Given the intensive computation of traditional MD simula- tions, generative models have been used to learn conformation distributions in data-driven fash- ion. The Boltzmann generator (Noé et al., 2019) uses normalizing flow to fit the Boltzmann dis- tribution from target-specific simulation data. Arts al. (2023) extends this by using denoising
2

---

Published as a conference paper at ICLR 2025
dVAE for Structure
Conformation
Structure
CV
Decoder
Codebook
Encoder
Decoder
Structure tokens
Residue embeddingg
Structure tokens
Encoder
Stracture Language Model
2
Neighhorbond- aware invariant
Sumple it
Language Model
encoding
AA-type tokens
3D structure coords
AA-type tokens
Inpur
Structure Language
(a) Encoding
(b) Training
(c) Inference
Figure 2: An illustration of the proposed SLM framework.
diffusion models for coarse-grained protein conformations. Furthermore, EigenFold (Jing et al., 2023), Str2Str (Lu al., 2024b), AlphaFlow (Jing al., 2024a), ConfDiff (Wang et al., 2024), and DiG (Zheng et al., 2024) leverage diffusion or flow matching to conditionally sample protein con- formations by learning from PDB data. Recently, AlphaFold3 (Abramson et al., 2024) revised the structure decoder of AlphaFold2 to a diffusion- module for diversified structure prediction.
Quantized representation for protein structures. Beside the prevailing diffusion models for pro- tein structure, representation learning of protein structures using discrete variational autoencoders (dVAF) has gained increasing attention in recent years. FoldSeek (van Kempen et al., 2022) one of the earliest attempt to build dVAE for fast structure search and alignment. Based on this, SaProt (Su et al., 2023) constructs learned representations with both sequence and structure tokens as input, while Pro[T5 (Heinzinger al., 2023) fine-tuned an existing language model to accept structure to- kens as input. PVQD (Haiyan et al., 2023) applied latent diffusion the embedding space dVAE for conditional protein structure generation. ProSST (Li et al., 2024) trained an autoencoder with K- means clustering applied in the latent space. Gaujac et al. (2024) and Gao et al. (2024) respectively build dVAE with large vocabularies for learning protein structure representations.
Remarks: Our work is closely related to these concurrent research directions by leveraging LMs to model and efficiently perform conformation generation over the quantized representation of protein structures. We refer to this framework as "structure language models" and describe them in detail.
PROTEIN CONFORMATION GENERATION WITH LANGUAGE MODELING
Notation. A protein with residues is identified by its sequence of amino acid types E |S|- where S is the vocabulary of 20 standard amino acids. The protein (backbone) structure is represented by its composing 3D atom positions X' I 1x44x including all backbone heavy atoms.
3.1 LEARNING THE SEQURNCE-STRUCTION DISTRIBUTION
To address the conformation generation problem, we start with modeling the sequence-to-ssruutuue translation distribution p(z|c) of interest and derive the learning objective in this section. To cir- cumvent explicitly learning in the structure space, the roto-translation invariant" latent represen- tation z is introduced Given this, the target distribution p( c) can be derived by marginalizing the joint distribution p(ælcc) J= p(æ, 200,2000 c). We further factorize this joint distribution according the Bayes' rule by isolating the latent variable (x, z|c) Ps (x/c, (1pppp(eee) where Pa denotes the (de-
encode 3D atomic protein structure, where (21...., zL) V
*For example, features like distance and angle are roto-tramslation invariant. This relationship can be for- mally written as q=1/T oz) e q(>) Ro + t) (<)l=),V7
3

---

Published as a conference paper at ICLR 2025
coding) distribution over the 3D protein structures given the structure token and sequence, and pe denotes the conditional distribution over the structure tokens, respectively modeled by neural net- works with parameter set o, 0. This gives rise the evidence lower bound on the likelihood of model distribution over protein structures conditioned on sequence:
log Pg, (ar e) 2 (=(a) (aree =)] DxL (go (zl=)) - L(b,0). 0),
(I)
where ab is introduced to parameterized the posterior distribution over latent representation z. Please refer to the Appendix G.2.1 for the full derivation of Eq. (I). Directly optimizing the right-hand side of Eq. (1) can be intractable and difficult since we have unknown posterior Zot- In practice one may adopt an one-step expectation maxiimnzaaiiiiiiiin (EM) approach (Dempster et al., 1977) by first jointly learning P& and qu with a simple and parameter-fie freee prior distribution p(z|c), followed by optimization on Po with the learned P/ and t similar to Van Den Oord et al. (2017). This yields the overall two-stage and separable training pipeline:
I. Learning quantized representation for structure. With the prior p(z|c) fixed, we begin by maximizing the ELBO (( 0) with respect to the encoder v and decoder Φ, using protein structure samples D {cc, })}. In the context of discrete latent spaces, this process is analogous to training a discrete VAE (dVAE) (Van Den Oord et al., 2017) to learn quantized representations for protein structures. Here, the encoder qu (=(xx)) maps structures latent tokens, while the decoder p& p | c) reconstructs structures from these tokens'. The prior p(z|c) is fixed to be uniform during this stage.
II Learning the prior over latent tokens. In this stage, we fix the learned parameters 6- and "i, and train the prior pe by maximizing the ELBO: arg maxe C(.",0). Since both d² and s are fixed, the reconstruction term in the ELBO cancels out, and training reduces to minimizing the KI. divergence DxI( (ge- (po)) This is equivalent performing maximum likelihood estimation, as B(c.a))-DBR>> (x (o) (2 c) with respect to Pa- Given that both z and e are categorical variables, (=
this formulation resembles translation task, allowing pe be parameterized by language models.
Optimizing Eq. (1) provides a general learning framework for conformation generation. In practice, exploring new "conformations" in an invariant latent space via learning a sequueneeto--struccure (seq2str) network while offloading the complicated geometric modeling to the structure nuto-encoder. This allows the practitioners to choose from popular architectures of structure encoders/decoders and modern language models.
can approach this objective in a demystified view:
3.2 STRUCTURE LANGUAGE MODELING
The prior learned in the previous stage is now applied to conformation generation, which can be framed as a conditional generative modeling problem for the seq2str translation. Given an input condition c E |S) which determines the molecular topology, the goal is to sample conformation ensemble from p(æ: c). To do this, we first sample a set of latent variables from the prior distribution learned earlier, decoder is jointly trained with the encoder (2/x) in the first stage, ensuring that the sampled latents align with the reconstruction. This framework supports roto-translation invariant inference and is described in Algorithm 1. Next, we illustrate this approach with two straightforward examples of structure language models (SLM): the encoder-decoder and decoder-only architectures.
~ po(z|c), and then decode these latents using the decoder P& (=( z). The
Encoder-devoder: Given the conditional nature of translation, the prior po(z/c) can be explicitly modeled by encoder-decoder architecture like T5 (Raffel et al., 2020). The decoder conditions on the context and factorizes the structure tokens sequentially: p(z|c)
I [2 e), where E Z represents the quantized structure tokens. The training objective is the negative log- likelihod (NLL) loss conditioned on [(O)- -B(c.))- Σ log c), .<1 Ø.
Decoder-only. Alternatively, the latent prior po(z|c) x (c(c, can be modeled autore- gressively using a decoder-only architecture. such as GPT (Radford et al.. 2019), where serves as the "prompt". We define y A [c, training involves maximizing the likelihood over y via the NLL minimization: L())
.......... ,I,..,,,,,2!..... =1). and the
BB(..))DD Eiz log po (yrly<t). where c, D is the i.i.d. samples from the data dis-
WWe assume that æ is conditionally independent of given latent variable for simplicity. In practice, this pa( [x) z, c) As p=())2) such in Hayes al. (2024).
leads structure decoder pa
4

---

Published as a conference paper at ICLR 2025
attention
Decoder
AA-type tokens
Structure tokens
AA-type tokens
Structure tokens
(a) Encoder-decoder
(b) Decoder-only
Figure Autoregressive prior modeling the for latent structure tokens discussed Section 3.2.
tribution over structures each with the associated amino-acid sequence as condition c. In practice, add an additional special token [sep] differentiate between these two modalities.
Inference involves sampling with a left-to-right decoding order, as defined by the autoregressive factorization of both language models. Fig. 3 briefly illustrates these two modeling strategies.
4 ESMDIFF: MASKED DIFFUSION INSTANTIATION
Building on the foundation of SLM, we here propose ESMDiff an instantiation based on discrete diffusion models (Austin et al., 2021). ESMDiff incorporates the inductive bias of seq2str translation and leverages the protein foundation model ESM3 (Hayes et al., 2024) through masked diffusion fine-tuning. The effectively fine-tuning of ESMDiff also exemplifies how a large pretrained BERT- like masked language model can be adapted to acquire conditional generative capabilities, making well-suited for broader downstream tasks such as conformation generation.
4.1 CONDITIONAL MASKED DIFFUSION LANGUAGE MODEL.
The discrete diffusion models can be generally defined by sequential process of progressive noisy discrete variables 2 E V from the categorical variable 2g E V Masked diffusion (Austin et al., 2021: Lou et al., 2023; Shi et al., 2024; Sahoo al., 2024) represents special case in which the transition includes an "absorbing state", denoted as [MASK] In this formulation, the stationary distribution assigns all probability mass the unique special token [MASK], such that P(z [MASK]] = and P(2 * [MASK]] ) For convenience, we define PM E (0, 13 (V V [[MASK] as the one-hot vector representing [MASK]. In masked diffusion, the stochastic forward process maps 2g + [MASK] and remains this state thereafter (i.e., "absorbing"). Conversely, the reverse process gradually unmasks (denoises) the [MASK] token to produce the data sample 2o. where s < (see Appendix G.2.2 for derivation):
q(2, L2, z0) = Cat (zzi [B(s, t)
Ams ((s,t))] + AM - ((s,t)) (2)
T-a(r) and (z.) = (PM; 2.). Eq. (2) implies when 2, # [MASK], the back- ward process simply copies the unmasked token by + i.e ( =0) = Cat (z.i 2;); otherwise the probability mass interpolates between PMr and zy. The posterior (z,, [2t, zu) can be approximated by Po 2.) using re-parameterization: po(2,12;) = a( 0(2,/20 u1 z;)), where the neural net ug IVI is neural network that outputs probability vector that remains in A (V) Un- like open-ended text generation, protein conformation generation is well-defined within the discrete diffusion models, conditions the input amino acid sequence, allowing each output token to correspond uniquely to a position in the input and thus enjoy a fixed-length context window $
where B(s, 1-a(e)
For conformation generation, we consider the conditional amino acid types c, our goal is to sample structure tokens through Eq. (2), utilizing a conditional posterior (2, Zo: e). This posterior can be re-parameterized similarly by incorporating the con- dition into the backbone model, resulting in pa( 2, k;; c) = q(zz 2z, us ut 2e, c)). To achieve this
of masked diffusion. Given the
*See Appendix G.1 for more details of the discrete diffusion models. *This indicates that eig and 2(1) aligned at the same position index i.
5

---

Published as a conference paper at ICLR 2025
Structure tokens
Zt
Zo
X+
Bidirectional Transformer A
PolZa
[z. C)
Time encoding
0
Denoising
T
Zy [MASK]
AA-types
(Zes C)
Figure 4:
Illustration for conditional denoising network u0 c) where the masked (structure) tokens are colored grey. The unmasked tokens are carried over to the output without update, while the masked tokens are under random transition into either unmasked (V) or still masked (X) state.
goal, the reverse process simulated by po(zu|zz; e) must effectively approximate the data distribu- tion p(zlc). A feasible training objective to optimize the estimation of the conditional ELBO within the continuous-tinn integral, resulting in the following loss (see Appendix G.2.4 for details):
(JETOL)) Exing(xe so)
da(() Ans log(us (1, 21, c), zo) dt
(()) Ee,z0
(3)
a(t) Øt
where 2p is sampled from the learned encoder (z(x)) with the corresponding amino acid condition c from the data distribution p(æ, c), and (z,) implies the loss is only applied for the latents Vt, s.t. 22
[MASK]. In practice,
employ Monte Carlo estimation to compute the integral.
Algorithm Inference: Conformation Generation of SLM
1: Require: amino acid types (condition) generic conditional language models Po(z/c), struc- ture decoder pa paxlee ee, 2), sampling temperature T O, sample size N 0.
2: Initialize an empty set of samples X = 2
3: for 1 to do
4: 2(0 pa (z(c)
Sample latent with temperature T > Decode the sampled latent vector
5: (+) ~ Palal z(0)
6: X X U (z)1))
Add the decoded structure
7: return X
4.2 BIDIRECTIONAL ENCODER AS DENOISING NETWORK
We now discuss the implementation of the conditional denoising network using bidirectional en- coder language models, such as BERT (Devlin, 2018). First, consider the sequential generalization of masked diffusion with a sequence of categorical variables. Let 2; now be sequence of discrete
structure tokens 2r.p1]- - [t,ci]] where 2r,[6 E Vi 1,..., L. Due the interpo- lation scheme of Eq. (2), we assume conditional independence and factorize the posterior distri- bution Pe 22 kz5 c) across the L output tokens, such that (= c) = 11 p0(2s, | c)
l g (z., [[ "5,,(0).440000( (t. Zt. c)) where ug (f, Zz, 2),c) c) represents the i-th output channel of neu- ral network. This implement coincides with the BERT-style transformer architecture and allow us to take advantage of existing protein foundation model, for example ESM3 (Hayes et al., 2024). For sequence of tokens, the masked log-term in the training objective from Eq. (3) replaced by the summation: E (zr log(ua.jjg) 1g(6 (1, Zz, c), Zo,11)) with notations the same as defined above.
Modifications. The following are special considerations for the network: (1) Position-coupled en- coding. Unlike general translation problem, SLMs maintain strict position-to- to--tiioo correspon- dence between amino acid types and the latent tokens" This inductive bias enables us to construct
Underlying co-evolutionary relationships between residues are shared across both amino acid types and its spatial patterns.

---

Published as a conference paper at ICLR 2025
et(t)] R?, where c2 IV r TR. ee |S] + RD, 6 IR - RD are the embedding functions and fo is linear transformation. (2) Copying. The unmasked tokens =t,[e| # [MASK] remain the same in spite of the model output. (3) Zero-out [MASK]. Since us parameterize the approximated clean data 2o (fully unmasked), the [MASK] token cannot present in the output and its probability should be zero-out. This is equivalent to adding -o to the logit. In our study, the pre-trained LM head of ESM3 is replaced with randomly initialized head with augmented vocabulary (V) during fine-tuning.
the imput embedding for all position i as follows: e(:)
cc(c)t) (ole)(2.(1)) +
5
EXPERIMENTS
Base settings. We start with the pre-trained dVAE established in Hayes et al. (2024) as the structure tokenizer (frozen). The structure quantization is perform residue-level with receptive field over local geometric neighborhoods of protein structure. The structure language models as described in Section 3.2 are based off offte-of-the-arl language models as follows: (1) S-T5 (384M) adopts the T5 (Raffel et al., 2020) architecture with bidirectional encoder and an autoregressive decoder. (2) S-GPT (961M) models the joint distribution with an uni-directional decoder like GPT2 (Radford et al., 2019). (3) ESM3 (zero shot) is a pre-trained BERT model over multi-modal data and perform zero-shot inference by the iterative decoding. (4) ESMDiff is a fine-tuned variant of (3) on the PDB data using the masked diffusion objective in Eq. (3). For S-T5 and S-GPT, we embed the sequence tokens with the pre-trained ESM3- encoder to provide model condition. For ESMDiff, two different paradigms: Iterative Decoding (ID) and DDPM are considered. See Appendix B for details.
Table 1: Evaluation results of different conformation generation methods on generating the BPTI conformations. JS.* represents the lensen-Shannon (JS) divergence between the sampled ensemble and long MD ensemble on pairwise distance (PwD), time-lagged independent components (TIC), and radius of gyration (Rg). The validity as the frequency of clash-free samples in the ensemble is also included. Moreover, the ensemble TM-score and ensemble RMSD are reported with respect to the five kinetic clusters in Shaw et al. (2010). The best results among all methods are bold while the best results among the SLM family underlined (similarly applied to all tables below).
Method
JS-PwD (L)
JS-TIC (4)
JS-RG (1)
Validity (1)
TM-ens (1)
RMSD (
MSA-based
MSA-Subs. AlphaFlow
0,593
0.482
0.742
0,990
0.840
1.526
0.503
0.462
0.777
1.000
0.845
1.441
Eigenfool
U.536
0.466
0.824
0.620
0.840
1.473
Seq-based
Str2Str (PF)
0.561
0.590
0.325
1.000
0.705
2.155
Sur2Str (SDE)
0,506
0.552
0.302
0,960
0.664
2,480
ESMFlow
0.524
0.439
0.804
0.970
0.839
1.462
S-T5 GPT
0.410
0.446
0.528
0,740
0.845
1.434
0.841
SLM
ESM3 (zero shot)
0.406
0.445
0,561
0,800
0.842
1.450
ESMDifE (ID)
0.422
0.432
0,510
0.910
0.844
1.421
ESMDiff (DDPM)
0.372
0.420
0.439
0.940
0.843
1.437
Baselines. We consider multiple open-source models evaluation baselines for the protein multiple conformation generation, which are mainly categorized into (1) MSA-based methods that includes MSA-Subsamplinn (Del Alamo et al., 2022: Bryant & Noé, 2024) and AlphaFlow (Jing et al., 2024a). These methods rely on inference-time retrieval of multiple sequence alignments (MSA) (2) Sin- gle sequence-based methods: EigenFold (Jing al., 2023) leverages harmonic diffusion process conditioned on OmegaFold (Wu et al., 2022) embeddings to generate protein structures, Str2Str (Lu al.. 2024b) simulates a round-trip local diffusion conditioned on input structure to explore hy- pothetical conformations, and ESMFlow (Jing et al., 2024a) replaces AlphaFlow with ESSFoold as the backbone; (3) Specially tailored for intrinsically disordered proteins (IDPs) generation of idp- GAN (Janson et al., 2023). Results reported for baselines are obtained by re-running the inference pipeline and based on their open-source codes. The detailed pipeline can be found in Appendix A.
Training data. The training data for structure language models are controlled to contain only PDB entries on or before May 1st, 2020. This cutoff is aligned with previous works (Jing et al., 2023; 2024a; Lu et al., 2024b; Hayes et al., 2024) trained on PDB data make fair comparison. The training set is further filtered to include all monomeric structures with a max resolution of 5.0A, length ranging from 10 to 1000, which forms total size of [D] 112.4k 4k as the training data.

---

Published as a conference paper at ICLR 2025
Participating benchmark sets. To discover the potential of SLMs in conformation sam- pling, several relevant datasets are considered for benchmarking purpose: (1) simulation dy- namics of BPTI (Shaw et al., 2010), (2) conformational changing pairs including the fold- switching (Chakravarty & Porter, 2022) and ligand-induced apo/holo states (Saldaño et al., 2022), and (3) intrinsically disordered proteins (IDPs) deposited in the protein ensemble database (PED) (Lazar et al., 2021). These benchmarking tasks reflect different characteristics and challenges of the conformation generations, which provides comprehensive evaluation for models.
5.1 STRUCTURAL, DYNAMICS OF BPTI
In the first experiment, we evaluate by generating the conformations of protein bovine pancreatic trypsin inhibitor (BPTT). The structural dynamic patterns of BPTI are well acknowledged in Shaw et al. (2010) with I mss-oon MD simulations, based on which five kinetic clusters have been revealed. Similar to Lu et al. (2024b), we report the Jensen Shannon (JS) divergence for distributions of pairwise distance (PwD), time-lagged independent components (TIC), and radius of gyration (Rg); clash-free validity and the ensemble TM-score and noo--men--squar-ddviiile the kinetic clusters. The benchmark results shown in Table 1. Following Wang al. (2024), also evaluate the best distance of the generated samples to each cluster, as shown in Table 2. The RMSD and T'M-score are both caleulated using the TM-score binary (Zhang & Skolnick, 2004) with structural alignment. Note that the Cluster 3 is a difficult remote folding mode (Wang et al., 2024), yet SLMs achieve a significant improvement by modeling with a smaller matching RMSD.
(RMSD) w.r.t.
Table Evaluating on the best matching RMSD with respect to each of the kinetic clusters reported in Shaw et al. (2010) with an non-exhaustive ensemble size of N 100. RMSD (i = 1, 3, 4,,4,.5 5) are the lower the better (Q). Among them, the Cluster the most challenging case to model (Wang et al., 2024) and highlighted in red of which the hest sample from ESMDiff is visualized.
RMSDe:
RMSDez
RMSD
RMSD
RMSDes
MSA-based
MSA-Suhs.
0.953
1.669
2412
1.616
0.982
AlphaFlow
0.882
1.693
2.418
1.380
0.915
FigenFold
1.600
2.478
Soq-based
Str2Str (PF)
1.968
I 18881
2,480
2.064
1,900
Str2Str (SDE)
2.334
2.295
2.924
2.271
2,240
ESMH-oow
0.883
1.720
2.385
1.361
0,960
S-T5 S GPT
1.628
2.285
0.968
0.922
1.687
2.357
1.462
0.953
SLM
(zero shot)
2.301
0.977
RMSD 2.198
ESMDIT (ID)
0.872
1.620
2.270
1,415
0.927
Cluster 3
ESMDIT (DDPM)
0.924
1.628
2.198
1,435
1,000
Table 3: Evaluation on the conformation changing pairs (Saldaño et al., 2022; Chakravarty & Porter,
2022): (I) Pearson correlation r between sampled diversity and ground-truth diversity measured by the residue flexibility (ResFlex, absolute deviation after alignment), and (2) the ensemble TM-score (TM-ens). For the residue flexibility, both global (gl.) and per-target (pt.) mean/median correlations are reported; for the 7M-ens, both mean/median are reported. Metrics are the higher the better (t).
Apo/holo
Fold-switch
Method
ResPllex (gl-)
ResFlex (pf.)
TM-ens
RosFlex (gl.)
ResFlex (pf.)
TM-ens
MSA based
MSAASSub. AlphaPlow
0.398
0.401 1.401/0.20 0.337)
0,856 0.899
0,350
0.320/ 0.3/03.333
0.714 / 0.765
0.864/
0.385
0.376
Eigenfold
00400
0.830/0.870
0.225
0.279 00.255
0.614 0.653
Seqgbased
Str2Str (PF)
0.174
0.326 0.300
0.731 0.728
0.161
0.246 0.233
0.615 0.644
Str2Str (SIE)
0,148
0.349 / 340
0.659/ / 0.681
0.111
0.224 /0.220
D.521 5.54
ESMFlow
0.856 0.893
0.269
0.700 D.755
S-T5 GPT
0.144
0.726
0.313
0:437
0.112
0.134/0.111
0.571 0.562
0.207
0.075 0.078
0.349 /0.300
SLM
ESMS (zero shoc)
0.312
0.473 0.466
0.839/ / 0.876
0.388
0.323 / 0.3300
0.627 0.717
ESMDiff
00.517
0.851
0.720
ESMD (DDPM)
0.420
0.489 0.515
D.K3%/ 0.877
0.402
0 341 758
0.625 D.685
8

---

Published as a conference paper at ICLR 2025
5.2 CONFORMATION CHANGING PAIRS
We continue the study on the task of modeling and predicting conformational changes in structural proteins. The authors of Jing including (1) 77 pairs of fold-switching proteins (Chakravarty & Porter, 2022); and (2) 90 apo/holo pairs with ligand-induced conformational change (Saldaño et al., 2022) to evaluate the modeling capacity of conformation diversity. Following the setting and evaluation metrics of Jing et al. (2023), we randomly sample a few-shot ensemble with five structures per target and evaluate them based on the correlation metrics of residue flexibility and the ensemble TM-score (Zhang & Skolnick, 2004). The evaluation results for both test sets are shown in Table 3. We find that MSA-based methods generally achieve better performance than other model families, which highlights the importance of using MSA for generating stable conformation changing protein targets.
(2023) have curated two benchmarking sets of pairing data
Table 4: Mean absolute error (MAE) for the IDP test set measured in different characteristics for each method, where both mean median are reported over all targets. All metrics are the lower the better (L). The contact map of P PED00247 (PDB: 2MTF) is shown to the right example.
Method
Pairwise distance
Radius of gyration
Contact map
MSA-based
MSA-Subs.
7.250 4,664
4.381 2.633 4.880 1.64
0,181 0.120 0.228 0.161
AlphuFlow
7.129 3,533
Esparimental
EigenFold
11,419/ / 6.404
8 8.663 /4 448800
1,309 / 0,560
idpGAN
0.447
Sey bused
Str2Str (PF)
9.838 7,301
6.369 4,226
0.264 0,203
Str2Str (SDE)
8,793 5,891
5.444 3,082
0.227 0.165
ESMFlow
7.692 4,429
5.110/ 1.636
0.257 0.179
S-T5
9.009 5.733
5.448 2,596
0.467
S-GPT
9,221 6414
5.634 2 2.634
0.607 0,529
SIM
shot)
6.606
2.129
ESMDHIT
ESMDiff (TD)
6 888 / 4,689
4.333 2.
0.295 / 0.239
ESMDiff (DDPM)
7.010 4,665
4,435 1,974
0.354 0,284
5.3 INTRINSICALLY DISORDERED PROTEINS
Different from structural proteins, intrinsically disordered proteins (IDPs) do not have fixed or stable tertiary structure under normal conditions. IDPs possess inherent flexibility and usually ex- ist as dynamic ensembles of conformations, allowing them to adapt to different binding partners or cellular environments. We have curated in total 114 entries from the protein ensemble database (PED) (Lazar et al., 2021) benchmarking set. In specific, we only select the experimentally validated (eg. NMR spectroscopy) structure ensembles and excluding similar protein records with training set avoid data leakage (see Appendix A.4). Due to the disordered structural character- ization for IDPs, alignment-bert based metrics such as TM-score is not applicable. Because different targets have different sizes of ensemble (from ten thousand), et al. (2023) to evaluate the mean absolute error (MAE), specifically the pairwise distance, radius of gyration, and contact map between the predicted ensemble and the ground-truth ensemble.
follow the metrics used in Janson
5.4 RUNTIME ANALYSIS
MSA Subsampling (93M)
Wallclock time (sec)
10f
AlphaFlow (95M 10 steps, w/o resampling)
EigenFold (800M, w/ OmegaFold)
10a
Str2Str (17M 1000 steps, T=0.3)
ESMFlow (3.58 steps, w/o resampling) S-T5 (380M)
104
S-GPT (960M)
10
ESMDiff (1.48 ID, 25 steps)
L=
L= 100
L= 200
L 500
ESMDIff (1.48 DDPM, 25 steps)
Residues
Figure 5: Runtime profiling for SLMs and other baseline methods. The number parameters and necessary configurations for each model are also remarked for better reference.

---

Published as a conference paper at ICLR 2025
To demonstrate the efficiency of SLM, we benchmark the runtime of each SLM and compare them with diffusion-basse baselines. The measurement is based on the elapsed wall clock time for sam- pling ensemble of N 100 across different protein lengths (see Appendix A.2). As shown in Fig. 5, SLMSs exhibit superior scalability with respect to protein size and are 20-100x faster than diffusion models like AlphaFlow, highlighting their potential for real-world applications.
5.5
CASE STUDY: STRUCTURAL INPAINTING OF NANOBODY
The position-wise masking and unmasking nature in- herently enables ESMDiff to perform the inpainting task effecttively. Intuitively, the partially masked in- put (eg. nanobody framework) can be interpreted as an intermediate state 2s, 0 in the masked diffu- sion, where the unknown positions to be modeled are designated [MASK] naturally in accordance with the formulation. Then partially reversing allows us to re- strict the conformation sampling to the specific sub- regions of the protein, analogous to the concept of "impainting" for images. Here we name the forward- backward inference as the round-trip masked diffu- sion inference (see Algorithm 5) for the inpainting task. Here, we illustrate the conformation inpaint- ing capability through a case study that focuses on regions
CDR1
(T28-
CDR2 (M49-R57)
N-term
CDR3
(699-W104)
C-term
Figure 6: Visualization of Llama VHH domain. The ESMDiff sampled conforma- tions (in khaki) superimposed the NMR structures (Renisio et al., 2002) col- ored in transparent rainbow.
generating the complementarity-deeeriii (CDRs) for a nanobody example. In specific, we ex- plored the inpainting capabilities of ESMDiff using nanobody derived from Llama (PDB entry: 1G9E).
As shown in Fig. 6, the loop structures generated by the round-trip diffusion process of ESMDiff resemble the native conformations observed in NMR structures (both N 20). The inference is conducted in a very efficient speed by only taking 5.3 + 0.3 seconds on single NVIDIA A A100-SXM4-40GB GPU, highlighting the effectiveness of SLMs in performing structural inpainting and promising potentials the real-world scenarios such as structural analysis or virtual screening.
6 CONCLUSION AND LIMITATIONS
In this work, we propose a novel conformation generation framework of structure language models (SLMs). The overall inference divided into two stages: conditional sampling of latent structure tokens and roto-translation invariant structure decoding. We develop and train a variety of con- ditional language models, especially the masked diffusion-based ESMDiff, which fine-tuned variant to enhance the capabilities of ESM3 adapting for conformation generation. Unlike existing methods, SLMs perform amortized distribution learning within an invariant latent space, leading to more efficient inference. By alleviating the need for geometric modeling, SLMs can fully exploit the scalability of modern language model architectures and take advantage of advanced hardware optimizations. Benchmarking results across various conformation generation tasks demonstrate the compelling performance and application potential of SLMs. In summary, the proposed method opens up intriguing and novel research direction for related communities to explore.
Limitations. The current study presents several limitations worth exploring for future works. Firstly, one design more advanced d'VE architecture balance between structure disentanglement and reconstruction fidelity. In addition to the discrete latent space, continuous latent space can also be considered, for example using the latent diffusion models (Rombach et al., 2022). Secondly, is worthwhile exploring alternative SLM instances specially tailored for the sequence-to-structue translation in consideration of proper inductive biases. Additionally, modeling protein side chains and other relevant modality including ligand and RNA by building an atomic structural auto-encoder also a very appealing research direction. Lastly, the outstanding performance MSA-based methods also indicates the potential to build MSA-conditioned structure language models.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For reproducibility, we provide detailed implementation specifics, including the baseline inference pipeline and training/fine-tuning procedures in Appendix A. The sampling pipelines for ESM3 and ESMDiff are thoroughly described in Appendix B, while Appendix D outlines the evaluation meth- ods and their references. The theoretical results mentioned in the main text are derived or proven in Appendix G.2. The source training and inference code for structure language models in this study are made publicly available at https:/ / / t.ub. com /lmm jiaarrrrri.. arui/ www./eeseeee /esddissi.....///////////. ff, as we believe fine- tuning foundation protein language models can have broader applicability in various downstream tasks including the protein conformation generation.
ACKNOWLEDGMENTS
We thank Can Chen and Chenqing Hua for helpful feedback well as anonymous reviewers for their constructive suggestion and comments. This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Tencent AI Lab Rhino-Bing Gift Fund and a NRC Collaborative R&D Project (A14D-132) This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3588 139727. YB acknowledges funding from NRC AI4D, CIFAR and CIFAR AI Chair. The computation resource of this project is supported by Mila, the Digital Research Alliance of Canada, and NRC.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence- based deep representation learning. Nature methods, 16(12):1315-13) 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference machine learning, pp- 233-242. PMLR, 2017.
Marloes Arts, Victor Garcia Satorras, Chin-Wei Huang, Daniel Zuegner, Marco Federici, Cecilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg. Two for one: Diffusion models and force fields for coarse-grained molecular dynamics. arXiv preprint arXiv:2302. 00600, 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 77977-1799999 2021.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871 876, 2021.
Patrick Bryant and Frank Noé. Structure prediction of alternative protein conformations. Nature Communications, 15(1):7328, 2024.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-2888 2282666-2279,,.. 2022.
Devlina Chakravarty and Lauren Porter. Alphafold2 fails predict protein fold switching. Protein Science, 31(6))e333, 2022.
Kuo-Chen Chou. Low-frequence motions in protein molecules. beta-shect and beta- barrel. Biophys- ical journal, 48(2):289-297, 1985.
11

---

Published as a conference paper at ICLR 2025
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):49-566 2022.
Diego Del Alamo, Davide Sala, Hassane Mchaourab, and Jens Meiler. Sampling alternative conformational states of transporters and receptors with alphafold2. Elife, 11:e75751, 2022..
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ar 1EXx:/8/00000003 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward un- derstanding the language of life through self-supervised learning. IEEE transactions pattern analysis and machine intelligence, ((100)711227122) 2021.
Noclia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.
Zhangyang Gao, Cheng Tan, Pablo Chacón, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. arXiv preprint arXiv:2209. 12244:22909.1663,,, 2022.
Zhangyang Gao, Cheng Tan, Jue Wang, Yufei Huang. Lirong Wu, and Stan Zi. Foldtoken: Learn- ing protein language via vector quantization and beyond. arXiv preprint arXiv:2403.0m 09600, 2024.
Benoit Gaujac, Jérémic Donå, Liviu Copoiu, Timothy Atkinson, Thomas Pierrot, and Thomas D. Barrett. Learning the language of protein structure, 2024. URL /tttpp://ariiv.. abs/ 2405. 15840.
Liu Haiyan, Liu Yufeng, and Chen Linghui. Diffusion in a quantized vector space generates non- idealized protein structures and predicts conformational distributions. bioRxiv, pp. 2023-11, 2023.
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024-02 2024.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure. bioRxiv, pp. 2023-07, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neurul Information Processing Systems, 33:6840-6851, 2020.
Moritz Hoffmann, Martin Scherer, Tim Hempel, Andreas Mardt, Brian de Silva, Brooke E Hu- sie, Stefan Klus, Hao Wu, Nathan Kutz, Steven L Brunton, et al. Deeptime: a python library for machine learning dynamical models from time series data. Machine Learning: Science and Technology, 3(1)0015009, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of neural text degeneration. arXiv preprint 1XX4:1994.00775 2019.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie. Tom Sereu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. International con- ference machine learning, pp. 8946-8970. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in neural information processing systems, 32, 2019.
Giacomo Janson, Gilberto Valces,Garcia, Lim Heo, and Michael Feig. Direct generation of protein conformational ensembles via machine learning. Nature Communcattoons, 14(1):774, 2023.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.02198, 2023.
Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles, 2024a. URL https::/p//.... /ate::///rxiiii /arrivivvvv./////v wrg. /240/ 04845.
Bowen Jing, Hannes Stärk, Tommi Jaakkola, and Bonnie Berger. Generative modeling of molecular dynamics trajectories. arXiv preprint arXiv:2-09.17800; 2024b.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Leon Klein, Andrew YK Foong, Tor Erlend Fjelde, Bruno Mlodozeniec, Marc Brockschmidt, Sebas- tian Nowozin, Frank Noć, and Ryota Tomioka. Timewarp: Transferable acceleration of molecular dynamics by learning time-coarsened dynamics. arXiv preprint arXiv:2302.01 170, 2023.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pp. 5361-5370. PMIR, 2020.
Tamas Lazar, Elizabeth Martínez-Pérez, Federica Quaglia, András Hatos, Lucía B Chemes, Javier A Iserte, Nicolás A Méndez, Nicolás Garrone, Tadeo Saldaño, Julia Marchetti, et al. Ped in 2021: major update of the protein ensemble database for intrinsically disordered proteins. Nucleic acids research, 49(D1):D404-D411, 2021.
Marie-Paule Lefranc, Christelle Pommié, Manuel Ruiz, Véronique Giudicelli, Elodie Foulquier, Lisa Truong, Valérie Phoouvenin-Contee and Gérard Lefranc. Imgt unique numbering for im- munoglobulin and cell receptor variable domains and ig superfamily v-like domains. Develop- mental & Comparative Immunology, 27(1):55-77, 2003.
Mingehen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Liang Hong, and Pan Tan. Prosst: Protein language modeling with quantized structure and disentangled attention. bioRxiv, pp. 2024 04, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Esolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637) 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.1688: 2023.
Jiarui Lu, Zuobai Zhang, Bozitao Zhong, Chence Shi, and Jian Tang. Fusing neural and phys- ical: Augment protein conformation sampling with tractable simulations. arXiv preprint arXiv:2402. 10433, 2024a.
Jiarui Lu, Bozitao Zhong, Zuobai Zhang, and Jian Tang. Str2str: A score-based framework for zero-shot protein conformation sampling. In The Twelfth International Conference on Learning Representations, 2024b.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. 1099 1106, 2023.
13

---

Published as a conference paper at ICLR 2025
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction the effects mutations on protein function. Advances in neural information processing systems, 34:29287-29303, 2021.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679-683 2022.
Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365(6457) 1447, 2019.
Guillermo
Fabian Paul, Toni Giorgino, Gianni De Fabritiis, and Frank Noé. Iden- tification of slow molecular order parameters for markov model construction. The Journal of chemical physics, 139(1), 2013.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot totto-iiimge generation. International conference on machine Pmlr, 2021.
learning, pp. 8821-8831
Jean-Guillaune Guilluuulllmm Renisio, Janice Pérez, Michael Czisch, Mare Guenneugues, Olivier Bornet, Leon Frenken, Christian Cambillau, and Hervé Darbon. Solution structure and backbone dynamics of an antigen-free heavy chain variable domain (vhh) from llama. Proteins: Structure, Function, and Bioinformatics, 47(4):546 2002.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):2016233115 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524 2024.
Tadeo Saldaño, Nahuel Escobedo, Julia Marchetti, Diego Javier Zea, Juan Mac Donagh, Ana Julia Velez Rueda, Eduardo Gonik, Agustina García Melani, Julieta Novomisky Nechcoff, Martín N Salas, et al. Impact of protein conformational diversity on alphafold predictions. Bioinformatics, 38(10):2742-2748, 2022.
David E Shaw, Paul Maragakis, Kresten Lindorff-Larsen. Stefano Piana, Ron O Dror, Michael Eastwood, Joseph A Bank, John M Jumper, John K Salmon, Yibing Shan, al. Atomic-level characterization of the structural dynamics of proteins. Science, 330(6002)3341-446 2010.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and gener- alized masked diffusion for discrete data. arXiv preprint arXiv:2406.04322 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13466, 2020.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(1 2017.
14

---

Published as a conference paper at ICLR 2025
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure- aware vocabulary. bioRxiv, pp. 2023-10, 2023.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.167555 2022.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseck: fast and accurate protein structure search. Biorxiv, pp. 2022-02, 02, 2022.
Yann Vander Meersche, Gabriel Cretin, Aria Gheeraert, Jean-Christophe Gelly, and Tatiana Ga- lochkina. Atlas: protein flexibility description from atomistic molecular dynamics simulations. Nucleic acids research, 52(D1):D344-3392 2024.
Alex Wang and Kyunghyun Cho. Bert has a mouth, and must speak: Bert as a markov random field language model. arXiv preprint arXiv: 1902.04094. 2019.
Yan Wang, Lihao Wang, Yuning Shen, Yiqun Wang, Huizhuo Yuan, Yue Wu, and Quanquan Gu. Protein conformation generation via parce- se(3) diffusion models, 2024. URL https: / www.aiiwwgggaaaa.aaaaoaaaaaoaeae abs /2403. 10..0..1188... 14088.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL httph:tp/////m /wwww aclweb. org httg///tthoogy// 2020. emnlp demos. 6.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004.
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discrete and continuous-tinn time discrete denoising diffusion. arXiv preprint arXiv:2402.03701, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023a.
Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi. Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang. Jian- wei Zhu, Yaosen Min, et al. Towards predicting equilibrium distributions for molecular systems with deep learning. arXiv preprint arXiv:2306. 35604445, 2023b.
Shuxin Zheng, Jiyan Hc, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al. Predicting equilibrium distributions for molecular systems with deep learning. Nature Machine Intelligence, pp. 1-10, 2024.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 BASELINE EVALUATIONS
For baselines, we compare our structure language models against: EigenFold (Jing et al., 2023), Str2Str (Lu et al., 2024b), MSA subsampling (Del Alamo et al., 2022), AlphaFlow & ESM- Flow (Jing et al., 2024a), and idpGAN (Janson et al., 2023). For MSA subsampling, we leverage the official repository of Del Alamo et al. (2022) under AlphaFold v2.3.2 with an MSA depth of 32, i.e. 32 maximum extra MSAs and 16 maximum MSA clusters (half the depth value (Del Alamo al., 2022)). For AlphaFlow and ESMFlow, we use the base PDB model (no distillation) with no resam- pling. The imput MSAs for test proteins are queried via the ColabFold server (Mirdita et al., 2022). All other parameters are set to the recommended values according to the original implementation by the authors (Jing et al., 2024a). For FigenFold, we follow the pipeline in Jing et al. (2023) by using OmegaFold (Wu et al., 2022) to make embeddings for the test set and sampling via the pre- pre-trained diffusion model with default parameters, where a = and = 3. For Str2Str, we use their public codebase to with initial structure guess generated by the ESMFold (Lin et al., 2023) using the script provided in the repository. We use a T = 1000 diffusion sampling steps for both probability flow and SDE (noise scale is set to 1.0 for SDE runs) and other parameters default. For idpGAN, run the inference pipeline following the Jupyter notebook in their open-source repository with the default inference settings for custom proteins.
A.2 RUNTIME PROFILING
To profile the inference time for each method, we set a ladder of tasks of different protein lengths including 50, 100, 200, and 500 residues, with an ensemble size of N 100 for each benchmark task. The profiling is carried out on a single NVIDIA A100 SXM4 GPU with 40GB memory, we measure the wall clock time elapsed for generating the target ensembles for each model. Note that for the MSA-based methods, the time-consuming MSA search is conducted only once and is amortized the overall computational cost for multiple batches. For autoregressive SL.Ms namely S-T5 and S-GPT, the past computed key-value pairs for each layer are cached for fast inference.
A.3 TRAINING DETAILS
S-T5 training. For S-T5, we build upon the implementation from the Transformers package (Wolf et al., 2020). Table S1 shows the hyperparameter settings for training. The total number of trainable parameters is 384M. The model is trained without learning rate scheduler for up to 30 epochs. The final tested model and the reported epoch number in Table S1 come from the best checkpoint, selected according to the NLL of structure tokens on a hold-out validation set. To leverage the ESM3 sequence embeddings, we bypass the encoder's input token embedding layer and feed the precomputed embeddings directly into the transformer encoder.
S-GPT training. For S-GPT, we also use the implementation from the Transformers pack- age. The total number of trainable parameters is 961M. We insert a special [sep] token in between the sequence and structure tokens when they are fed into the model, with its embed- ding being randomly initialized and learnable. In practice, we find it useful to add a hyper- parameter À balance between the NLL of sequence and structure prediction, i.e., L(0)
D logpe(yrllea)) + -Z2, log (yr (ynl where the first half of y consists of the sequence tokens and the second half consists of the structure tokens, as defined in Sec. 3.2. Table S2 shows the hyperparameter settings for training. The model is trained and up to 30 epochs but the best checkpoint is evaluated and reported.
Masked Diffusion fine-tuning of ESM3. The masked diffusion is scheduled with a log linear noise schedule (Austin et al., 2021) for clipped time domain 1. [0, with infinite small time = 0.001, such that or (t) exp|-o E (0. 1]. We leverage this schedule for both training and inference. When fine-tuning, replace the pre-trained LM head in the structure track of the ESM3 with randomly initialized head for the augmented vocabulary V = V [MASK] by setting output dimension to 4101. We learned embedding module to encode timestep via the sinusoidal time embeddings followed
log(1 - t) while the coefficient is parameterized a(t)
16

---

Published as a conference paper at ICLR 2025
Table S1: Hyperparameters of S-T5
Parameter
Value Adamw
Description
optimizer
Optimizer type
1r
1 x 10-5
Learning rate
batchhsiiz
Batch size
epochs
28
Number of training epochs
betas
(0.9, 0.999)
Coefficients for computing running averages of gradient
weight_decay
x 10-2
Weight decay coefficient
num_layers
12
Number transformer layers in the encoder and decoder
num heads
16
Number of attention heads
d.ff
2048
Feedforward network dimension
d_model
1280
Model hidden dimensionality
dropout rate
0.1
Dropout
feed forward proj
gelu
Activation function
Table S2: Hyperparameters of S-GPT
Parameter
Value
Description
optimizer
Adamn
Optimizer type
1r
1 10
Learning rate
batch size
12
Batch size
epochs
11
Number of training epochs
betas
(0.9, 0.999)
Coefficients for computing running averages of gradient
weight decay
1 10-2
Weight decay coefficient Number of transformer layers
num_layers
48
num. heads
16
Number of attention heads
n inner
5120
Feedforward network dimension
n_embd
1280
Model hidden dimensionality
dropout rate
0.1
Dropout rate
act activation oontiinnnfntttio on
gelu
Activation function
seq pred weight
0.01
Weight of loss for sequence prediction
simple 3-layer MLP coupled with SiLU activation function. We the AdamW optimizer with betas=(0.9, 0.999) and weight decay=0.01, The training is scheduled with the constant scheduler with warm up steps of 2,500.
Algorithm 2 Masked diffusion fine-tuning of ESM3
Require: Pre-trained masked language models pe (-(c, 2). training data D {c, 2), masking schedule (<()) > 0, learning rate
2: for i 1,2..... | do
3: Get sample (c, x) + D[j
4: Sample time indices E [0,
5: Sample masked probability p(t) based on the masking schedule (<t)
6:
Create masked input 2, by masking z with probability p(t)
7:
Compute the negative ELBO loss L(B: 21, based Eq. 3
8: Update model parameters: &- 0 Vo((0)
9:
return pa
A.4 CURATION OF IDP TEST SET
We curated the test set for intrinsically disordered proteins (IDPs) by downloading data from the Protein Ensemble Database (PED) (Lazar et al., 2021) August 10, 2024, using the official API, which provided a total of 481 raw entries. To prepare the evaluation set, we filtered the data to include proteins with sequence lengths between 20 and 500 and ensemble sizes of N 10. To ensure balance the dataset, we performed clustering using MMseqs (Steinegger & Söding, 2017) with the flags -s 7.5 --min-seq
9. 0. We further excluded the records with more
17

---

Published as a conference paper at ICLR 2025
than 90% sequence identity by conducting another search against the PDB training data to avoid data leakage. Finally, we select the experimentally validated (eg., NMR) ensembles by filtering with respect the measurement method, which excludes computationally generated ensemble by models such as idpGAN (Janson et al., 2023).
A.5 TRAINING DATA OF BASELINE MODELS
We list the training data and the corresponding cutoff for all the mentioned models in our study for reference, as shown in Table S3.
Table S3: Training data and base model of different methods in this study.
Method
Basc/Embedding model
Data source
Cutoff date
MSA subs.
AlphaFold2 (93M)
PDB
May 1, 2018
AlphaFlow
AlphaPold2 (93M)
PDB
May 2018
ESM
ESMFold (3B)
PDB
May 1, 2020
Eigenfold
OmegaFold (900M)
PDB
Apr. 30, 2020
Str2Str
/
PDB
Jun, 9, 2023
idpGAN
DisProt50 + MD
Jun., 2021
ESMDiff
ESM3-open (1.4B)
PDB
May 2020
B INFERENCE OF BIDIRECTIONAL SLMS
B. ZERO-SHOT CONFORMATION GENERATION OF ESM3
The sampling method for the BERT model was first introduced by Wang & Cho (2019), which treats the bidirectional encoder as a Markov random field graphical model and employs ID sampling to generate new sequences. In Hayes et al. (2024), the author propose to use iterative decoding to enable simultaneous decoding of multiple positions rather than processing them one at a time. Following the authors (Hayes et al., 2024), refer to this latter method as the iterative decoding (ID) and take advantages of this sampling strategy for zero-shot conformation generation.
In specific, leveraged the open-source 1.4B ESM3 protein language model (Hayes et al., 2024) for zero-shot conformation sampling. ESM3 is pre-trained on multi-modal data related to protein and contains several tracks of language model heads for prediction well as input embeddings, including sequence (amino acid types), structure, secondary structure, SASA and functions. In our main experiment, only the sequence and structure tracks are turned on and the input tokens for other tracks are set to the default value (i.c. [MASK] in each vocabulary). For sampling configurations, we use a sampling schedule of 25 steps and set the sampling temperature to be 1.0 without specially pointing out. During sampling, we also adopted the nucleus (top-p) sampling strategy (Holtzman al., 2019) to improve the quality of samples with a probability threshold 0.95. Throughout the experimental Section 5, we use the entropy-ranked ID sampling to obtain the results for ESM3 (described below).
B.2 RANKED ITERATIVE DECODING
Unlike one-shot decoding, which maps all masked structure tokens [MASK] to their predictions in a single feed-forward pass, the iterative decoding can adopt various strategies to iteratively select where to unmask at each sampling step from the masked structure tokens of the target protein. These strategies are governed by different ranking functions f(.) applied over the masked positions, detailed below. The overall inference pipeline (with ranking) is outlined in Algorithm 3. We investigate and compare these ranking strategies Table S4, using ESM3 as the base model.
Entropy ranking. The entropy for categorical distribution quantifies the predicted uncertainty for the current evaluation of probability. Formally, the ranking score of the masked position indexed by i (1 i < N) is the position-specific entropy:
f(1)=
Σ
P| = 2²) log Piy z²),
18

---

Published as a conference paper at ICLR 2025
Algorithm 3 Iterative Decoding with Positional Ranking
1: Require; amino acid types (condition) masked language models po(zlc, 2), ranking function
[(;; c, z), sampling temperature T >> 0, the number of decoding steps K, mask token [MASK] Target protein length
2: I= len(c)
3: Initialize . +- {0, 1..... L- 1]. U t- Ø
4: Initialize z|1] <- [MASK] Vi E M
5: Get schedule } (R: -....... K) with En nh L per-step number of positions to unmask D iterations
6: for k=
1 to K do
7: Rank / M using [(;; c, z)
8: Select top nA positions M2 + Ranked (M;key = f)l: my] > rank and select top-nk
9:
Sample latent components z ~ pa (2 c,2), i Ma with temperature T
10:
Assign 21(] + 4 Vi E M₁
11: Update the unmasked set: U + U U M
Update the masked set: M M \ M₁
13: return
where PM] = Moft is the predicted probability mass function of position i over the pre-defined vocabulary of valid structure tokens V. The entropy of each candidate position is ranked ascending, i.e. positions with smaller entropy are firstly decoded. We also adopt the adaptive entropy calculation, which means the logits prediction in the current decoding step is based on the unmasked structure tokens from the last step and calculated on-the-fly.
Maxlogit ranking. Beside the uncertainty-base based ranking, we similarly consider the maximum of logits (maxlogit) from prediction as the ranking score. This indicates we choose the positions with the top (highest) logit score as candidates. The maxlogit reflects the maximal confidence of the model among its choice well as the probability of the selected token during greedy decoding. Correspondingly, the ranking score of maxlogit for index is defined as:
f(i) max EEEV
(1n z')-
Secondary structure ranking. The ranking scores discussed above can be applied to arbitrary data for masked decoding. Here, we introduce novel decoding schedule designed for proteins, ordered by predicted secondary structure:
f(i) = ) where si = arg max pso [c, 2, 2)2,,
(6)
where Pss is a predicted categorical distribution over all eight secondary structures (SS8) defined by the Dictionary of Protein Secondary Structure (DSSP), including helix of 3-turn, 4-turn, 5-turn, and hydrogen bonded turn; extended strand of 3-sheel and isolated 3-bridge; bend and coil, and g(-) e (O....,7) 7) maps the SS8 type to the finite ordering score. Given the predicted distribution over SS8, we use greedy selection to assign the SS8 label (with the max logit) for all masked positions. The positions are further ranked from the most structural to the most disordered, or the same order as above. The intuition behind is that we want to predict the structural region in the first place, followed adopt the secondary structure prediction
by modeling the disordered region (loop). In practice, heads of pre-trained ESM3 (Hayes et al., 2024) for the prediction of SS8 types for each masked candidate positions.
B.3 INFERENCE OF CONDITIONAL MASKED DIFFUSION MODEL
After the fine-tuning stage, ESMDiff is able to perform conditional generation from sequence to structure either using (1) the iterative decoding described in Algorithm or (2) the vanilla DDPM ancestral sampling (Ho et al., 2020). The former sampling pipeline can readily accommodate the ESMDiff backbone (fine-tuned ESM3 Transformer module) without any changes. For experiments, fine-tune the ESM3 using the objective defined in Eq. 3 w/o time conditioning. The DDPM sampling of masked diffusion model is simple and straightforward by progressively unmasking all the [MASK] in fixed number of time steps = 25. This inference pipeline is shown in Algorithm 4.
19

---

Published as a conference paper at ICLR 2025
Table S4: The performance of different ranking fuunctions on the three benchmarking tasks described in Appendix B.2.
Method
JS-PwD (1)
JS-TIC (J)
JS-RG ())
Validity (f)
TM-ens (1)
RMSD-ens (U)
Uniform (wo ranking)
0,414
0.428
0.602
0.840
0.845
1.460
Entropy
0,411
0.402
0.582
0.740
0.844
1.478
Maxlogit
0.411
0.425
0,576
0.890
0,850
1.436
Secondary structure
0.411
0.423
0.628
0.930
0.849
1.441
Apo/holo
Fold-switch
Method
ResFlex (gl.)
ResFlex (pt.)
TM-ens
ResPlex r (gl.)
ResFlex r (pt.)
TM-ens
Uniform (w/o ranking)
0.223
0.400 0.384
0.796 0.848
0.328
0.276 0
0.531 00.522
Entropy
0.318
0.447 0.470
0.840 0.876
0.407
0.343 0.366
0.629/ 0.692
Maxlogit
0,386
0.480 / 0.481
0.843 0.876
0.391
0.333 / 0.309
0.629 0.715
Secondary structure
0.237
0.444 / 0.444
0.826 O. 0.868
0.377
0,309 / 0.352
0.589 0.665
Method
Pairwise distance
Radius gyration
Contact map
Uniform (wo ranking)
6.661 / 4.810
4.017 / 1.950
0.328 / 0.264
Entropy
6.706 4.726
2.112
0.249 0.170
Maxlogit
6.743 5.037
2.303
0,252 0.
Secondary structure
6.365 4.767
3.833 2.011
0.298 0.228
The DDPM sampling conducted for fine-tuned model w/ time conditioning. Throughout our study, we neither control the temperature (equivalently set to 1.0) nor use the ranking function introduced in Appendix B.2 during each reverse step in DDPM sampling, which is worth exploring in the future works.
Algorithm 4 DDPM Ancestral Sampling for Conditional Masked Diffusion
Require: amino acid types (condition) c, bidirectional model (w/o softmax head) fø(zz) |zz, c), noise schedule {az), ), initial latent ~ q(rr), the number of denoising steps T, mask token [MASK]
> Target protein length >> Initialize with noisy latent vector from prior 4: Set mask tokens: ZT, <- [MASKI, Vi E (1,2....,
2: L == ccn(cc)
3: Initialize ZT + [0. Iv|L
5: for t = T do
Ancestral denoising process Compute log probability matrix from the logits > Sample candidate tokens from categorical distribution Sample from the posterior according to Equation 2
6: Pt PixiVi +-
c))
7:
For each i, [ ~ Cat(:; P* [i.:))
8:
z/11 ~ (( ((2 11, z²)
9:
for i = 1 to L do
10:
if 21,1]] [MASK] then
11:
11 t 2, 1
b Update using sampled x7-1
12:
else
13:
21/1 [t]
Preserve the unmasked token from 2, > Return the fully unmasked latent vector
14:
return 20
B.4 COMPARISON BETWEEN ESM AND ESMDIFF
Different from the MLM pre-training of ESM. in ESMDiff, the amino acid types are fully condi- tioned while the structure tokens are trained using masked diffusion with loglinear schedule. Except for the time-conditioning and loss reweighting, both MLM and masked diffusion similarly use ran- dom masking as training objective. However, the ratio of masking should depend on different down- stream tasks to "mimic the types of inputs" during inference, indicated in Hayes et al. (2024). Thus, in the task SLMs, we reason that it benefits conformation generation more when using less-to-none sequence masking and properly scheduled structure masking. For other fine-tuning application of ESM (eg., sequence to function tokens), we recommend practitioners to make the fine-tuning objective suitable for specific input/output inference-time usage.
20

---

Published as a conference paper at ICLR 2025
C DETAILS OF CONFORMATION INPAINTING
The inherently position wiie masking and unmasking nature of masked diffusion enables ESMDiff to perform inpainting tasks effectively. Intuitively. the partially masked imput can be interpreted as an intermediate state (8 > 0) within the masked diffusion process, where the unknown positions that need be modeled are designated as [MASK] in accordance with the masked diffusion formulation. This allows to restrict the sampling of conformational changes to specific sub-regions of protein structures, analogous to the concept of "inpainting" in image processing.
Specifically, the conformation inpainting can be formulated as round-trip masked diffusion process described below:
the initial state, we first simulate the forward diffusion process to an intermediate state 2s. Instead of using the stochastic forward kernel ((z,|zu) defined in Eq. 9, 2g is obtained in deterministic way, i.c., set 2 [MASK] if Vi AM and =x,[0] f 20.14] otherwise, where AM is the residue indices for which we want to sample. Note that the ideal time is induced from the noise schedule (tt) by matching the expectation of the number of the masking positions under the forward marginal, or formally, s* - - Mr Intuitively, we shall see on average n = A [MASK] tokens in x; we simulate the stochastic kernel z, q--120). In practice, however, s 0 can be chosen tunable hyperparameter.
Forward. Given input sequence of N tokens zo E |VI
Backward. We then evolve the reverse process of masked diffusion starting using the learned posterior po c) from 25 according to Eq. 2. Thanks to the "absorbing" nature of the masked diffusion process, the copied variables {2. [() Vi & Aus are all preserved during the backward steps until return to ån (we add hat to distinguish it from input zg). This means that sampling is conducted exclusively for the target indices in Aas
Since the sampling transitions from the fully unmasked state zn 2, (s 0) and then back ên. we refer to this as the "round-trip" diffusion process. This approach is reminiscent of the forward- backward (FB) dynamics introduced by Lu al. (2024b), although the latter defined for local exploration within the SE(3) geometric space and not specifically intended for inpainting. We describe this process in Algorithm 5.
Algorithm 5 Round--Tip Trip Diffusion for Conformation Inpainting
1: Input: Initial structure æ, amino acid tokens c, the indices of the sampling sub-region A M. noise schedule (t)), denoising network ug, structure encoder qo and structure decoder ps.
2: Output: The inpainted conformation &
3: Encode Zy ~ Pv (2)/z))
4: Length L Ien( len(2)) (2)
5: // Deterministic Forward Diffusion
6: Infer the ideal intermediate time step ="= anN AAm))
7: Set 2 + 20
8: for each i AM do
9:
- + [MASK]
> Mask specified indices
10: // Generating Reverse Diffusson
11: 20 +- Reverss( C, up)
b Either in Algo. 3 or
12: Decode ~ P& (=lc, 20)
13: return ab
In Fig. 6, we explored the inpainting capabilities of ESMDiff using a nanobody derived from Llama (PDB entry: IG9E). To sample the conformations of the CDR loops, we began with a PDB structure, fixed the nanobody framework, and applied the inpainting pipeline exclusively to the three CDR loops, which is numbered by the canonical IMGT numbering scheme (Lefranc et al., 2003). Then simulate the round-trip diffusion process using ESMDiff (ID. T=1.0) with 25 steps for the CDR regions. The sampling is performed on a single NVIDIA A 4100-SXM4-40GB GPU, Both NMR and the generated structure ensembles contain N 20 conformations.
21

---

Published as a conference paper at ICLR 2025
D EVALUATION METRICS
Jensen-Shannol divergence (JS). The quality of a set of conformations is assessed by comparing the distributional similarity between the reference ensemble and the generated ensemble, akin to how the Fréchet inception distance (FID) (Heusel et al., 2017) is used for evaluating synthetic images. We use the Jensen-Shannon (JS) divergence because of its symmetry, which penalizes the model's distribution for both lack of ground truth coverage and biased coverage. To ensure compatibility with baseline models, only C* -atoms are considered in calculating divergence metrics. We use three key iroo-translation invariant features accurately capture ensemble characteristies, following Lu al. (2024b):
Pairwise distance (PwD): pairwise distances are calculated between atoms, skipping three atoms between pairs. To transform these continuous values into distribution, we construct histograms with Niin 50 bins to represent the pairwise distribution, and JS divergence is calculated over these. A pseudo-count of 10-6 6 added to zero frequencies for slight smoothing.
The two slowest components of time-lagged independent component analysis (TIC) (Pérez- Hernández et al., 2013): Pairwise distances for each protein conformation are computed and flattened. TICA projections are then fitted using the reference full MD trajectories, ap- plying the Deeptime library(Hoffman et al., 2021). The first two components selected after TICA dimension reduction for each ensemble. Histograms are constructed for both components similarly to the pairwise distances.
Radius of gyration: This measures the root mean square distance of atoms from the center of mass. The same histogram approach is used described for the other features.
Validity. following Lu al. (2024b), the validity is defined as the ratio of clash-free conformations in the generated ensemble. It is computed by dividing the number of conformations without steric clashes by the total size of ensemble. A steric clash occurs when two atoms are too close to each
other. For a given ensemble of conformations, it is expressed as: Validity(fx(1)) m)) 1.0 1
(i) N Em 1(3 j, k such that [.."[]] -c,[[ < 5), where Tc- E R represents the C* -atom coordinate of residue j in the conformation sample (1) and 5 = 3.0A.
TM-ens (RMSD-ens). First introduced in Jing et al. (2023), the ensemble TM-score (TM-ens) quantifies how well the generated ensemble matches the observed conformational states in the refer- ence data. Specifically, it is defined as the maximum TM-score between each generated sample and the reference state, averaged over all reference states. This provides a measure of structural simi- larity, with higher scores indicating better coverage between the generated and true conformations given a limited sampling budget. Similarly, RMSD-ens calculates the average minimum RMSD value between the generated and reference conformations, providing another metric to evaluate the structural fidelity of the ensemble. They are respectively defined as follows, with x represents the generated ensemble and y denotes the reference ensemble:
(( Σ
TM (f2: (,((0)))))10)))) = HybDH
max TM(x y()), and
My
RMSD ems (1)) (. (x(0))) =
Σ
min RMSD( (y00).
Hyo'h
The TM-score and RMSD are both calculated using the compiled binary executive developed by Zhang & Skolnick (2004).
Residue flexibility. Following Jing et al. (2023), we evaluate the Pearson correlation for global and per-target residue flexibility (ResFlex), which is defined the absolute deviation of the center
22

---

Published as a conference paper at ICLR 2025
C atom coordinates after structural alignment, or formally for index
N
1
ResFlex((m*))d_)) N
Align( (k) 10,
Align(a
a )c
N
k=1
1=1
where Align(-) means the Kabsch alignment as in RMSD among all N conformations in the en- semble. Afterward, the correlation calculated between the generated ensemble and the reference ensemble.
Mean absolute error (MAF). Since the experimental ensemble contains a varying number of samples for different targets in the PED database (Lazar et al., 2021), we balance by using the mean absolute error (MAE) to evaluate the performance on intrinsically disordered proteins (IDPs). For each feature, calculate the ensemble average for both the reference and generated samples, then compute the MAE between them and average this value over the channels. The mean and median MAE reported across all 114 targets.
Contact map. We calculate and visualize the contact map as the probability distribution over 2D grid of pairwise distances. First, the pairwise distance between residues is calculated, and distance threshold of 8.0A is applied to determine if two residues are in contact (Janson et al., 2023). Finally, the frequency of contacts across all conformations in the ensemble is normalized create probability distribution over the entire distance map. Formally, the contact probability between residues and j given by:
1 P Peontaet (i. = N
Σ (d (1.)) (800),
where (i,.) E I is the pairwise distance between residues and in conformation k, and N the number of conformations in the ensemble. This normalized contact map provides straightfor- ward insights into residue interactions within the generated ensemble.
E ABLATION STUDIES AND ADDITIONAL EXPERIMENTS
E. FINE-TUNING CONHIURRATION
In Table S5, we list different fine-tuning configurations in experiments. Note that using the sequence track dropout, masking, or prediction similar to the pre-training stage of ESM3 do have positive improvement on the BPTI task, we discover the model accuracy will decrease for the apo/holo or fold-switch evaluation. In specific, the track dropout rate is set to be 25% the masking rate is 10% while the sequence prediction loss has the equal weight of structure prediction.
E.2 CHANGING OF TEMPERATURES
In this section, examine the effect of altering the temperature parameter T during the sampling process. Temperature scaling influences the diversity and accuracy of the sampled conformations, with higher temperatures encouraging more deterministic outputs. We experimented with different values of ranging from 0.25 to 5.0, to observe how this affects the balance between diversity and accuracy in our conformation generation for different SLMs (for ESMDiff, only the ID sampling is applied). To make the result straightfor- ward, we evaluate on the BPTI samples (N=100) with the validity metric well as the TM-diversity. The TM-diversity is defined as the average pairwise inverse TM-score (i.e., 1 - TM(2, y)) among conformations in the sampled ensemble. As shown in Fig. S1, we observe that higher temperature generally leads to decreased validity and increased diversity. Note that the pre-trained ESM3's (w/o fine-tuning) performance degenerated significantly at the low temperature. After fine-tuning, its validity improves significantly without compromising the diversity. As the temperature increases, ESM3 and ESMDiff maintain a high validity score. However, there is minimal improvement in di- versity, suggesting that these models might not be sensitive high sampling temperatures. S-T5 and S-GPT also achieve strong validity when T < 1.0. In contrast to ESM models, these two models quite sensitive the temperature. As be seen, their validity scores quickly deteriorate as
exploratory behavior, while lower temperatures lead to
23

---

Published as a conference paper at ICLR 2025
ESMOI!
ESM3
S-T5
S-GPTT
ESMDIT
ESMS
S-T5
88-PTT
1.00
1.00
0.75
0.75
TM-diwersity
Valiity
0.50
0.50
0.00
0.00
1.25
1.00
5.00
1.00
1.25
5.00
Temperalure T
Temperalure T
(a) Validity
(b) TM-diversity
Figure S1: Moving (a) Validity and (b) TM-diversity along with temperature for SLMs.
the temperature moves beyond 1.5. Notably, at T S 1.0, S-GPT not only maintains high validity but also achieves significantly greater diversity, showcasing its superior ability to balance between diversity and accuracy.
E.3 CHANGING OF SAMPLING STEPS
This section focuses on the impact of varying the number of sampling steps in the ID or DDPM sampling. As the number of steps increases, the model has more opportumities to refine the sampled conformations, potentially improving accuracy. However, increasing the total number of sampling steps K also incurs a higher computational cost (growing linearly (K))). We explored different step counts, ranging from 5 to 100, and evaluated their performance similarly. As shown in Fig. S2, only ID sampling significantly benefits from an increased number of sampling steps. For the other two methods, accuracy improvement plateaus after 10 steps. The diversity remains largely unaffected by the number of steps for all methods, Overall, DDPM sampling consistently yields the highest validity across nearly all step counts, with the exception of at 100 steps.
ESMDAI (ID)
ESMOI (DDPM) ESM3
ESMOIf (ID)
ESMOIfI (DDPM) ESM3
0.100
0.75
Valddy
TMdiverrlly
0.078
0.90
0.25
0.020
0.00
0.000
Sampling steps
Sampling steps
(a) Validity
(b) -M-diversity
Figure S2: Moving (a) Validity and (b) TM-diversity along with the number of sampling steps for BERT-like SLMs.
E.4 EVALUATING ON THE ATLAS MD ENSEMBLES
Unlike Boltzmann generators (Noé et al., 2019; Arts et al., 2023), recent studies have focused on developing generative models that learn from PDB data plus multiple molecular dynamics (MD) trajectories (Zheng et al., 2023b; Jing al., 2024a:b; Lu et al., 2024a). We further evaluate per- formance on the ATLAS MD ensemble dataset (Vander Meersche et al., 2024), which comprises 100ns all-atom molecular dynamics simulations across 1,390 monomeric protein targets. In Jing
24

---

Published as a conference paper at ICLR 2025
Table S5: Ablation results for ESMDiff fine-tuned using different strategies across three tasks in our
study, By default, the DDPM sampling (Algorithm 4) is used to generate samples for evaluation.
Method
IS-PwD
JS-TIC (1)
JS-RG ())
Validity (1)
TM-ens (t)
RMSD-ens (L)
ESMDiff (DDPM)
0.372
0.421
0.440
0.940
0.843
1.477
w/o time condition
0,385
0.421
0.456
0.930
0.849
1.454
w/ sequence mask
0.387
0.414
0.461
0.940
0.849
1.423
w/ sequence dropout
0,358
0.408
0.431
0.910
0.846
1.407
w/ sequence prediction
0.366
0.442
0.474
0.860
1.389
Apwholo
Fold switch
ResFlex r (gl.)
ResFlex (pt.)
TM-ens
ResFlex r (gl.)
ResPlex (pt.)
TM-ens
ESMDiff (DDPM)
0.420
0.489 0.515
0.838 / 0.877
0.402
0.341 / 0.228
0.626 0.685
w/o time condition
0.419
0.479 0.487
0,847 / 0.883
0.346
0,340 0.357
0.633 0.685
w/ sequence mask
0.371
0.474 00.411
0.843 0.885
0.381
0,336 0.329
0.624 0.687
sequence dropout
0.345
0.833 0.858
0.373
0.308
0
w/ sequence prediction
0.419
0.456 0.445
0.828 0.866
0.360
0.324 0.341
0.598 0.626
Method
Pairwise distance
Radius of gyration
Contact
ESMDiff (DDPM)
7,010 / 4,665
4,438 1.974
0.354 / 0.284
w/o time condition
7.399 / /4.855
4.854 1.838
0.366 / 0.283
w/ sequence mask
7.027 /4.654
4.523 2.221
0,363 / 0.293
w/ sequence dropout
7.140 4.904
4.629 2,502
0.369 0.303
w/ sequence prediction
8.122 /4.764
5,414 2.422
0.433 /0336
al. (2024a), the authors established a comprehensive benchmark and dataset split evaluate how well different conformation generation methods capture various statistical properties when learn- ing from MD ensembles. As shown in Table S6, ESM3-based models, including both pre-trained ESM and ESMDiff, struggled to accurately reconstruct the MD ensembles from the ATLAS test set. This limitation may stem from ATLAS's relatively short 100ns simulation timescale, which rarely captures slower, substantial conformational changes (as noted by Jing et al. (2024a)). Such brief timescales particularly challenge tokenized structure and language modeling approaches, as the cat- egorical nature of structure tokens makes it difficult to represent subtle structural changes in token relationships. Despite the efficiency of SLMs, developing better methods to leverage these SLMs for learning from MD simulation ensembles represents promising direction for future research.
Table Statistical metrics on MD ensembles of ATLAS test set (Jing et al., 2024a).
Metrics
AlphaFlow-MD
ESM3 (ID)
ESMDiff (ID)
Pairwise RMSD r
0.48
0.08
0.18
Global RMSF
0.60
0.19
0.49
Per target RMSF r
0.85
0.67
0.68
RMWD
2.61
7.27
7.48
RMWD trans
2,28
5.22
5.18
RMWD var
1.30
4.35
3.37
MD PCA W2
1.52
2.06
2.29
Joint PCA W2
2.25
5.97
6.32
PC sim 0.5 %
44
22
23
Weak contacts J
0.62
0.45
0.52
Transient contacts
0.41
0.26
0.26
Exposed residue
0.50
Exposed MI matrix rho
0.25
25

---

Published as a conference paper at ICLR 2025
F STRUCTURE AUTO-ENCODERS
F.1 QUANTIZED REPRESENTATION OF STRUCTURES
Tokenizing protein structures can differ from the cases of image pixels (Van Den Oord et al., 2017; Ramesh et al., 2021). The encoded structure should not only provide informative representations for accurate reconstruction but also ensure a well-structured latent space that enables efficient language modeling. In order build better structure language models, discuss several rules that ideal structure quantization methods should follow.
Disentangled quantization. Firstly, we ask the disentangled property of the latent 2 In specific, each position of the codes 2; E [V| has only limited receptive field of the local structure and the interaction between different codes should be minimized. In other words, specific token z; is specific pattern of local structures, where small noise from inputs can be redacted or inferred from other tokens. Disentangled representation can enable efficient language modeling and the interpretability of the latent space. Formally, we assume conditional independence of 2; given input structure, i.e. (z) = q(z1, 22.......... ZN x)) g(z) [))(2)(z).....)) .t(p/))).....((/))) (z)
expected to correspond
Roto-translation invariant encoding. ping) indicates that applying (global) affine transformations T o æ R o + t. composed of 3D rotation matrix R E 1333 and translation vector t E IR3 to the input (of Euclidean space) should have no effects on the final output value. Formally, the encoder ((2(0) : X - 2 with RT-Inv property can be described as: q(2/T æ) = q( R x 1) q( A T. RT-Inv property is essential for effective structure quantization since encoding global rotation or translation into rep- resentation can make unnecessary assumption for network architectures. It is also non-trivial to encode orientation or translation in the categorical codes 2.
Ioto-translation invariance (RT-Inv) of a function (map-
Complete coverage of conformation space. The (approximately) complete coverage property of quantized representation guarantees successfully reconstruction between and ã. Good latent space Z should properly configured according to the distribution of structure characteristics. To achieve good reconstruction, latent codes should be able to catch and distinguish various subtle structure patterns from the input space X' for high-quality decoding. On the other hand, excessively large vocabulary cause sparse coding space and unnecessarily encode the noises in structure, which can thus lead to memorization (Arpit et al., 2017) and make the generalization difficult.
F.2 RECONSTRUCTION ON TEST DATASETS
To investigate whether the structure tokens by the dVAE (Hayes et al., 2024) can encode different conformations in distinguishable way, evaluate the reconstruction accuracy of the encode- decoder as a preliminary study via the notemmaan--uuur-ddvviaat (RMSD) on the benchmarking test datasets used in this study. The results are presented as histogram shown Fig S3. Note that a non-negligible ratio of the fold-switching Chakravarty & Porter (2022) and IDPs (Lazar et al., 2021) targets for which the encoder-decoder cannot reconstruct the input protein structure very well (RMSD > 0.5A) passing through the quantization process.
Furthermore, we plot the histogram for pairing structures of each target the Apo/holo (Saldaño et al., 2022) and fold-switching Chakravarty & Porter (2022) for closer investigation. As shown in Fig. $4 (a) and (b) respectively, the dVAE can have a better accuracy for the folds in the Apo/holo dataset than those of fold-switching, which may explain the relatively inferior performance of SLMS on the fold-switching test data.
'To avoid ambiguity, we confine the meaning of " "disentangle" to he local-to-global instead of semantic- level description of the structure (eg. double-helix). Thus, the latent 2; should have restricted receptive field. "Note that conditional independence of z given does not necessarily imply such relation given though these variables, as protein sequence and structure, possess high mutual dependence.
26

---

Published as a conference paper at ICLR 2025
BPTI Clusters
Apa/holo
Fold-Switching
IDPs
2.0
1.5
Density
1.0
0.5
<0.5
0.5-1.0
1.0-2.0
2.0-5.0
RMSD (unit: angstrom)
Figure S3: Histogram of RMSD of the targets from each participating test set used in this study. Frequencies has been normalized density due the unequal sizes of different datasets.
Apa
Holo
Fold
Fold 2
1.5
Density
Density
0.5
0.25
0.00
RMSD (unit: angstrom)
RMSD (unit: angstrom)
(a) Apo/holo
(b) Fold-switching
Figure S4: Histogram of RMSD of the targets in the (a) Apo/holo pairs; (b) Fold-switching pairs.
27

---

Published as a conference paper at ICLR 2025
G EXTENDED METHODOLOGY DISCUSSION
G.1 REVISITING DISCRETE DIFFUSION AS DISTRIBUTION INTERPOLATION
The discrete diffusion models (Austin al., 2021; Lou 2023; Sun al., 2022; Camphell al., 2022: Zheng et al., 2023a) can be generally defined by a sequential process of progressive noisy variables 2, E V from the categorical variable 20 E V. Denote the one-hot (row) vector of 2 as 2, {0, 1I1,, in the discrete-time case (Austin et al., 2021), the forward marginal probability of 2 at time has the following form as a composition of Markov kernel defined by Q (1 (1= 1,2 1,2.... T):
q (2) = Cat (zri z Q - Cat (zr; =0
Qt),
where Q: indicates the transition probability matrix for time / represented by [Q:] = g(zt
j = i). and Cat(:; p), p A indicates the categorical distribution with probability and A is the |-immpleex Eq. also induce the form of the marginal distribution for Vt > is q(z (2,) =
Cat (z,; z, Qt(s) Cat (z 2, Q0Q++1 ..... Qt). Correspondingly. the posterior ((2,|z,, z0) can be obtained by the reverse process (Austin et al., 2021):
a(=el=,, z0) = g(x 2.. 20 20)g(zz Cal Zzi 2n Q12
2tQ ② 20
V
(8)
((2,|20)
Both Zhao et al. (2024) and Shi et al. (2024) discuss how the discrete-time diffusion process can be generalized the time domain E [0, 1J. akin the diffusion over continuous space (Song al., 2020), by demonstrating the continuous-timg limit as T + so. Notably, when the stationary distri- bution is explicitly specified (denoted as p E AV),, we can choose a state- state-independett transition kernel in the simple form: Qtj- &
+ - a(s)) -a(s)=aa(t)) (a)))] (TTpp) p]. thus simplifying the
continuous-time forward marginal to:
a(t) (r()) 2s + +
a(t) a<8) p vO
(<z1|z,) = Cat zii
where a(l) E 00, is strictly monotone decreasing function with cpp = and 01 + 0. The equation above demonstrates that the discrete diffusion, when defined with an explicit stationary distribution, can be viewed as interpolation between two categorical distributions controlled by (t). According to Eq. 8, the reverse process of diffusion defined in Eq. 9 takes the following form for the posterior distribution, where 0 I < t< 1:
[u(t, s)2, + (1 - p(t, Az,, (p)1) @ (a(s) + (1- (1-a(s))p) a1())2, (2n) + (1 a(()) (p)
g(z,|Zr, 20) = Cal
Zz;
(10)
where p(t, s) _ ass) aa())) > and indicator function Az (.) - (zz, .) for concision. Readers are referred to Appendix G.2.2 for more details on deriving the Eq. 9 and 10.
Unlike open-ended text generation, protein conformation generation well-defined within discrete diffusion models, as it conditions on the input amino acid sequence, allowing each output token to correspond uniquely to a position in the input and thus enjoy a fixed-length context window
G.2 PROOFS AND DERIVATIONS
G.2.1 DERIVATION OF THE EVIDENCE LOWER BOUND EQUATION 1
We want to prove the inequality of ELBO of the log-likelihood over structure tokens z conditioned on input c, which is:
log ao (x 2 (x)() [log loggoo x) - DxI (p)(
Proof: Consider the log likelihood of structure x and sequence c, which parameterized by param- eters 0 do:
log pa, (m/c) log / pa.v 2|c) dz
thhs indicates that eij and z(z) are aligned at the same position index i.
28

---

Published as a conference paper at ICLR 2025
where is introduced latent variable, and pe,el (z, 2/c) is the joint distribution over and x. Write the variational distribution (z æ) as an approximation to the true posterior over the latent variables conditioned on æ, yielding the following log- liellikeihhoo::
log Po, v (x/c) log / p0, (2, z/c)
pa.
qv (z)x)) qu qu [x) dz,
Here we assume positive conditional distribution of 2(.)) By applying Jensen's inequality to the logarithm, we obtain the following inequality:
Pe.v (ax, (x, 2|c)
log (x)c) 2 (xu()
log
go (x(2)z)
We can simplify the terms inside the expectation and separates the R.H.S. into two expectations:
(=(m)) log p ar, 2|c) = E ( Po x)] + (x log po c)
(z(xx)
(x(x)x))
The second term is the negative Kullback-Leiiber (KL) divergence, and thus:
E (x)aa) [s]]ec =)]
DxI (xa,(q=((())))]00d
Since the latent tokens encode the protein structures, we assume that æ is conditionally indepen- dent of e given z. Thus, we obtain:
log e 2 Baolal Plogoo(zlcc 2) -
G.2.2 DERIVATIONS OF MARGINAL AND POSTERIOR IN G.1
Based the theoretical results in Austin al. (2021), derive the marginal and poste- rior distribution corresponding
the state-independent transition kernel defined in as Qt/= - [a(s)==a(t{I+ + (1 a(s) (()))) p]. with att) [0, 1] being some strictly monotone decreas- ing function s.t. ag = 1 and 01 + 0.
Forward Marginal. For ((2(22), we plug in the transition matrix into the definition of forward marginal:
q(21 = Cat (z1iz.Q.1) Q1(s)
Cat (2rj 2, [a(s) aa) + (1 as)) (a())) pl)
a(t)
a(t)
Cat
#ti ((8))
2, (1
)2,(1 (1 p)
(18)
a(t)
a(t)
Cat
#ti ((8))
2, (1
(z,, 1
a(s)
a(t)
a(t)
Cat
#ti a(s)
2, (1
ass) )p
(11)
where the second to last line the associative property is used and z.1T = (z,, 1) = R, Vt due to the fact that by definition Vs, be positive function defined on [0, 1],
{0, 1 is an one-hot (row) vector, i.e. (z., z.) 1.
Let pls,t) t) A as q(2,(x,) = Cat (2zp(ss) t)x, + (1 -1-10 μ(ss, t)) which can be viewed as an "interpolation" be- tween categorical distribution Cat z.) and stationary prior Cat. (::p).
write forward marginal
29

---

Published as a conference paper at ICLR 2025
Posterior distribution.
Similarly derive the posterior follows. Note that by definition: Q:
Q110= = [a(0))'a(t)<+)
(1-
p] = I (1 a(t)) p.
0):
= 1 3 20
(2, Ie, zo) = Cat
Zs:
zo 212
2t [a(s))+((()) a(() + a1 a()) ass)) p ② 20 [a(s) + (1 - a(s))
Cat
Zzi
20 [a((t)) (1
2,
((s))=a(()2z) + (1-o(0)=(((()>pp))
(a(s)zu + (1 - a()))p)
Cat
Zsi
(a(t)=0 + (1- a())pp) 2 (p(s, z) + (1 (1-p(s, (z. p) (a(()= + (1 (1-a(s))p) {(t))p)
= Cal
(12)
al()(2o, z1) + (p, zi)
G.2.3 DERIVATION OF CONDITIONAL MASKED DIFFUSION IN 4.1
The conditional masked diffusion follows the modeling of forward and reverse processes derived
above with the special case that p РM. Let pM {0, (0,1)11) be the one-hot vector for [MASK] token. The forward marginal is directly obtained as
( z.) = Cat (z:; puss, z) + (1 - p v0 Then we plug this in Eq. 12, which is:
(13)
(m(s, f) + (1 ()s, pm | (a(s) + +1 - a(s)
qzz,/22, 20) = Cat Zz;
(tt 21) + (1 (1- a(t) xn)
(14)
Since multiple inner-product terms appear between one-hot vectors 20, 21, Pas which only takes binary value {0, 1). discuss this by cases and derived the form in Eq. 10:
2, = [MASK]. In this case, we immediately note that (2t, PM1 = and (2,, z0) = 0 because
2o V represents the data and thus 20 # [MASK] 24. Plug in Eq. 14, we have:
[p(s,t)pM + (1 u1s 1 @ (a(s)= + (1 - ((s)))))))) pM)]
(x2, |2,, zo)
Cat
2si
(()
()(s + (1 (u(s, 1))1 @ (a(s) (0 + (1 - Pp)
Cat
zsi
(x))
(p(s,))PM + (1 (uss t)) @ (a(s) 20 + (1 - a(s))
Cat
zzi
1- (())
(u(s,t)MM + (1 u(s t))1 o [a(s) + (1 - a(s)) PM
Cat
zsi
1-((()
Zzi ((s, (s)))) + - + +1 p(s, ())))))((()))))))))) + (1 (1- -((s)))
Cat
1- a(t)
2zl ass) -a(t)
1- a(s)
Cat
a(l) 20 + 1- a(t) PM
(15)
where we use the facts that zo ② PM PM PM = PM. and * *.
21 + [MASK].
Correspondingly, this leads to (zt, PM.) = and we have the following simplifi-
cation:
(u(s, () + (1 - ((s 1)) (2,, 20) の (a())=0 + +(1 a(s))Par) a(t)(z): zn) + (1 (z)) PM1)
qZ2,1zt, z0)
Cat
Zzi
[(s,tta((s{ 3 20)
Cat
Zz: a(t) (z11 z0)
Zt Zzi 20 (zt, 20)
Cat
(16)
30

---

Published as a conference paper at ICLR 2025
Here find that, due to the construction of forward marginal in Eq. 13, 2e {[MASK] , 2n} to guarantee the denominator a(t) (2t, 20) + (1-a(t)) Par) to be positive. Such that 2, * [MASK] will imply (z #o) 1 and thus q(2,) =o) becomes Cat (zz: 21) which has probability mass P(= = 2 = 1 and zero elsewhere.
To combine these two cases together, we introduce the indicator function AM(+)) - (pwr,) and let ((s,t) 1=a(s] T-a(f) 0, Eq. 2 can be obtained by plugging in the coefficients to Eq. 14:
(Zzz/Zz, Rt, zo) - Cat (zsi (8(s,t) (1 (((5)))(1-(63)))) AM ((s, ())o)
(17)
G.2.4 LEARNING ORJECTIVE FOR CONDITIONAL MASKED DIFFUSION
Based on the pre-defined masked diffusion process, now we derive the training objective as in Eq- 3. The task we are interested in involves key step of sampling structure tokens Zo based on the amino-acid sequence condition c. We start with discrete-time denoising diffusion objective according to Ho et al. (2020), which is the ELBO of marginal likelihood of z conditioned on (let
4 - max(e, i/T), = 0,1, .))
log ((z()) Ey(z,, (z0) |=0) po00 c)] DKi (((((
T
DKL I 200 (= (z(,,)),
(18)
where each term can be respectively viewed as reconstruction, prior and (discrete-time) diffusion loss. The definition above is general and still valid regardless of continuous or discrete data. Due the special construction of conditional masked diffusion in Eq. 9 and 10, we find the cancellations of the first two terms because: (1) For reconstruction loss, the interpolation dif- fusion process is based on two one-hot vectors 2g and [MASK] which assigns zero probability to all other values. That indicates the infinite small time step to original data, i.e. lim;-00 (t))
= Z1o- or there is no "decoding process" mapping from 1. Thus log Zto c)]
log {(=o] Stg (Ztg, to, c)]] + 0 as to - 0; (2) For the prior loss, both q<>T|20 (0)) and pa (zT (c) are designed to be Cat(2T|pM as the stationary distribution with all probability mass on the [MASK]
token. Thus, DEL (g(23 2r (20) [[6((2)))) DKL (Cat 2r PM (at(2r/pM>) 0.
The ELBO in Eq. 18 is simplified to contain only the multi-step diffusion loss, by parameterization of the posterior distribution:
T Drulg(Zz, 1R,,, zo)l 1z.,, c)),
ELBO
Σ
DKL 200 us uz( ti, c))),
1=2
T
((z = ej 25|2t,, Zp) alzt e,l=0, 20) g(Zm-- Bj Rtc; uo (te, Li, c)))
(19)
1=2 j=1
where ej E {0, 13 VI is the one-hot basis vector with j-th element being non-zero. Note that in
Eq. 16, we show that if Pas, the posterior q(-12,, 20) becomes zn-independent, which imply
31

---

Published as a conference paper at ICLR 2025
DKL O. When 21
PM, according to Eq. 15, we have:
g(Zt, 1 - = 2o)
ELBO Σ
g = 21) 20) log (z = =ee u0 (Zt, tis c)))
i=2 j=1
T Σ
M
(20. eÿ)
(1
((ti-1, e₁) log (zo
(Ze, li e),
j=
T
M
Σ 1(
Σ
1 eÿ) log (z0, ez) (20, eÿ) log (uo(z e),)
i=
j=1
T
[V]
Σ(
(1
(zu, e₁) log ((t:,,,t)) Σ
i=2
T
Σ (1 ((::-1, t:)) B2())1100) [log us uz 4,, c), zo)]
(20)
1=2
Then we push
+ oo and obtain the limit of continuous-time ELBO:
ELBO
lim
((1--,, By(te, (20) [(20, log us
T-X
a(t;-1) - aa(t) B By(2,, (20) ((20, log us ti c)>l
lim
T-
a(li)
[a((t)) alt [[( - ti-1)
lim
Eo(zn, (20) [=00 log (z c))]
T I/T-+0
att:)](ti ti-1)
1=2
1
da(t)
(E(().) a(t) Ōt Eg(2)120) (=0, log up(ze, c))] dt.
(21)
In practice, the integral above can be replaced by the uniform sampling U[0. 1] as Monte Carlo estimation during training. Note that the ELBO,, above is conditioned on pM (ELBO=) otherwise), such that we introduce the indicator function (z(()) and obtain the Eq.
H STRUCTURE GALLERY
This section showcases various structural ensembles generated by our model, ESMDiff (DDPM sampling), across different protein scenarios. The visualizations include comparisons between sam- pled ensembles and reference structure(s) or ground truth ensembles, providing an insight into the model's s ability to capture diverse conformations and structural flexibility different types of pro- teins.
H.1 PROTEIN BPTI
Figure S5 displays the best-matched samples from modeled ensemble (N 100) for each kinetic cluster (in total 5) of BPTI published in Shaw et al. (2010). The superimposed visualization compares the modeled conformations (solid rainbow color) with the reference structure (rendered transparent) after structural alignments.
H.2 CONFORMATION CHANGE PAIRS
Figure S6 presents the sampled ensemble khaki, overlaid on the ground truth structures of conformatiin--caaning pairs from the Apo/Holo dataset, shown in transparent rainbow colors.
32

---

Published as a conference paper at ICLR 2025
RMSD 0.926
RMSD 11.44
RMSD 2,144
RMSD 1.460
RMSD 0.889
Cluster
Cluster 2
Cluster 3
Cluster 4
Cluster 5
Figure S5: The visualization shows the best matched samples for each kinetic cluster of BPTI from the modeled ensemble (N 100) by ESMDifI. The reference structure rendered transparent.
ligp.A| A
Ijej.A Irf4.A
Irf5.A lr4.A
Izal.A| 1995.A
Aju3.A 2ju8.A
2k43.A 2k8rAA
Figure S6: Sampled ensemble (colored in khaki) superimposed with the ground truth conformation changing pairs in Apo/Holo dataset(represeeneed in transparent rainbow colors)
H.3 INTRINSICALLY DISORDERED PROTEIN
The final subsection focuses on intrinsically disordered proteins (IDPs), which are characterized by their flexible, non-fixed structures. Figure $7 demonstrates the sampled ensemble in khaki, superim- posed on the ground truth ensemble of IDPs, rendered in transparent rainbow colors. Additionally, the figure includes comparison of contact maps to the right, further illustrating the model's pacity to reproduce the flexible and dynamic nature of IDPs. Among the examples, we also show failed case for PED00433, where the model struggles to enforee structural diversity at both ter- minal fragments. While the experimental ensemble exhibits significant structural variability, the model-generated ensemble shows reduced diversity, indicating potential direction for further im- provement in modeling terminal regions of the modern structure prediction models.
33

---

Published as a conference paper at ICLR 2025
PED00234
PED00254
PED00247
PED00268
PED00249
PED00284
PED00340
PED00433
Figure S7: Sampled ensemble (colored khaki) superimposed with the ground truth ensemble of intrinsically disordered proteins (IDPs) (represented in transparent rainbow colors), accompanied by a contact map comparison on the right.
34


Reviews:
[{"review_id": "Mn0pG1CXnD", "reviewer": "Reviewer_FWmx", "summary": "In this paper, the authors present a novel Structure Language Modeling (SLM) framework that integrates recent advances in discrete variational autoencoders (dVAEs) to quantise the latent space and language models to learn the conditional distribution on this latent space with masked diffusion models framework. This combined approach is applied to model the conformation space of proteins. The proposed framework offers flexibility in selecting the dVAE and language model components; however, the authors introduce a specific setup, named ESMDiff, which utilizes ESM3 and BERT as its primary components. To demonstrate the effectiveness of this approach, the authors evaluate ESMDiff on several case studies, including the structural dynamics of bovine pancreatic trypsin inhibitor (BPTI), modeling conformational changes in structural proteins, and exploring the conformation space of intrinsically disordered proteins (IDPs).", "questions": "* [major] While the experiments compare a diverse set of baseline models, they lack a comparison with pure language models. Previous work [a] demonstrated that language models can generate molecular and protein structures with atomic coordinates represented as regular text, and other research [b] has shown that this approach is viable for molecular conformation space modeling. Given the context sizes of modern language models, it seems feasible to model protein backbone coordinates directly as text. Including a comparison with pure language models could provide a strong justification for using dVAE in this framework.\n* [minor] In reference to lines 175–186, the training of the dVAE and the language model prior happens in two separate stages. Previous research on learnable priors in VAEs [c, d] showed that the encoder, decoder, and prior can be trained simultaneously. It would be helpful to clarify, either theoretically or through an ablation study, why this work uses a multi-stage training approach rather than a joint training method, as this separation introduces additional complexity to the training process.\n* [minor] The scope of this work is limited to modeling the coordinates of the protein backbone (lines 145–150). However, accurate spatial positioning of side chains is essential for many practical applications, especially when modeling interactions between the protein and other proteins or molecular structures is required [e]. Including a discussion of this limitation in the Limitations section would strengthen the paper by addressing the potential impact of excluding side chains on the model’s practical applications.\n\na. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files, 2023\n\nb. BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning, 2024\n\nc. VAE with a VampPrior, 2017\n\nd. A Prior of a Googol Gaussians: a Tensor Ring Induced Prior for Generative Models, 2019\n\ne. Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction, 2023", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured, with a clear and comprehensive presentation of the theoretical background and framework. Although flexible learnable priors in VAEs have a substantial history of research, the idea of coupling the dVAE and language models as flexible conditional prior is fresh and novel in the area of protein conformation modeling. The selected case studies are relevant and show the potential for wide practical application.", "weaknesses": "While the overall quality of the paper is high, there are a few weaknesses and areas that could benefit from further clarification. Details on these points are provided in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MErrwUtDBg", "reviewer": "Reviewer_yRU8", "summary": "This paper proposes to use protein language models, operating on structure tokens, to model protein conformation ensembles. Several variations on this idea are tried: (1) an encoder-decoder model that translates sequence-tokens to structure-tokens (2) a decoder only model supplied with sequence-token prompts (3) Gibbs sampling with the bidirectional unmasking model ESM3 (4) Gibbs sampling and ancestral sampling with ESMDiff, a version of ESM3 fine-tuned under masked diffusion. These models are evaluated on the BPTI benchmark and conformation pair datasets and compared against several existing methods.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The problem addressed by the authors is important and timely. \n* The approach, although based on accessible and popular ideas, has not been explored in the literature and is likely to be of great interest to the community.\n* The paper is thorough and has detailed appendices describing training, inference, and evaluation.\n* The paper has a diversity of experimental results, including several that appear only in the appendices (?)", "weaknesses": "**Novelty**\n* The paper's approach is relatively straightforward and is almost an off-the-shelf application of ESM3 or any structural-token protein language model. With that said, I don't think this is a big drawback because the focus of the paper is on the novelty of application and empirical benchmarks.\n\n**Quality**\nThe paper's evaluations are, however, relatively limited and could be improved in quality and rigor.\n* The evaluations lean heavily on the BPTI benchmark, and do not make use of MD datasets like ATLAS and MD-CATH. It would be much more interesting to see how well the model can learn from those datasets and generalize to a larger number of new proteins.\n* ESM3, out-of-the-box, can already be used as a sequence->structure generative model with iterative decoding. However, the authors seem to only evaluate ESM3 under Gibbs sampling. Thus a proper comparison with pretrained ESM3 is missing.\n* The results themselves are also relatively mixed, with zero-shot ESM3 outperforming ESMDiff in several instances.\n\n**Clarity**\n* In all tables, the authors have bolded the best method _among the several methods they propose_, not the best method overall. This gives a very misleading impression and is a major issue that must be fixed.\n* At times, the paper has a forced and unnecessary level of formalism. At best, this obscures what the model is actually doing, and at worst leads to misleading or incorrect statements.\n    * The authors spend quite a bit of time building up a probabilistic framework for structure based on a standard variational lower bound. However, the authors universally use the ESM3 tokenizer, which is not trained with maximum-likelihood reconstruction. Indeed, \"we begin by maximizing the ELBO $\\mathcal{L}(\\phi, \\theta)$ with respect to the encoder $\\psi$ and decoder $\\phi$\" as this is soon undercut by \"We start with the pre-trained dVAE established in Hayes et al. (2024) as the structure tokenizer (frozen).\" So are $psi$, $\\phi$, ever trained, or not? Furthermore, the ESM3 decoder is not conditioned on sequence tokens, unlike the formally defined decoder $p_\\phi(x \\mid c, z)$.\n    * The exposition on general discrete diffusion takes up quite a lot of space, but ultimately the authors use masked diffusion, which could be presented much more clearly and succinctly on its own.\n    * Since ESM3 is also trained at all mask levels, it is not immediately clear what is different about the fine-tuning model. As I can currently tell, it is only that the model is explicitly time-conditioned and the unmasking is stochastic instead of a fixed number of tokens unmasked per step. But is this really a significant enough difference to merit a completely different presentation?\n\nTo summarize, I like to paper's core idea and would recommend acceptance based on that promise. However, the paper's presentation obfuscates the simplicity of the method, and the evaluations are incompletely and misleadingly presented. It would be preferable for the authors to de-emphasize methodological novelty but focus on a thorough and convincing empirical study of the ability of structural PLMs to sample conformational ensembles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDHzwIaM2i", "reviewer": "Reviewer_2wMe", "summary": "This paper introduces the Structure Language Modeling (SLM) framework for protein conformation generation. The frameworks consist of an auto-encoder and a conditional language model, which can be BERT-like or GPT-like. This method gets a good speedup than previous methods.", "questions": "In Table 2, the best SLM variant in each cluster is all different. Can authors provide more insights about these results?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  The paper is well motivated.\n-  The framework is a meaningful combination of VAE and LM that take the benefits of both.\n-  The 20x-100x speed up seems to be promising.", "weaknesses": "In results tables, seems all systems (including baselines and SLMs) have different model sizes and pre-training data. It would be better if these details are clearly described in the table to better understand the significance of SLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJWuzzgAj0", "reviewer": "Reviewer_X2Sg", "summary": "This paper aims to generate protein conformation ensembles with Structure Language Modeling (SLM), a method that first tokenizes protein 3D structure through learned autoencoding with a VQVAE, and then generates the 3D structure through language modeling and decoding. Compared to alternative methods such as equivariant diffusion models, the proposed method enjoys a significant efficiency advantage. The proposed method is applied to protein conformation ensemble generation tasks with a diverse set of evaluation metrics.", "questions": "Please see \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method offers an interesting alternative to the task of protein conformation sampling, a task that has so far mostly been tackled with 3D-aware generative models that may suffer from poor computational efficiency. The proposed method naturally leverages pre trained ESM3 model and extends its capablities.\n\n- The experiments validate the efficiency of the proposed method. The performance seems competitive with SOTA structure-based generative models.", "weaknesses": "- There are many metrics being benchmarked and it is hard to evaluate the model performance. It seems that the proposed method is quite competitive with SOTA alternatives. However, this reviewer is not fully convinced about that because the results are so scattered around in different tasks and different evaluation metrics, and the proposed method sometimes works better and sometimes works worse compared to baseline methods. Can the authors explain what metrics are the most practically relevant, or propose consolidated metrics for comparing different models? Can the authors include benchmarks that were originally used in the AlphaFlow/ESMFLow paper (Jing et al, 2024), if they are solving the same task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)