Fast Uncovering of Protein Sequence Diversity from Structure
Authors: luca alessandro silva
Barthelemy Meynard-Piganeau
Carlo Lucibello
Christoph Feinauer
Venue: ICLR 2025 Spotlight (2025)
Keywords: Protein design
inverse folding
generative modelling
transfer learning
Primary area: generative models

Abstract: We present InvMSAFold, an inverse folding method for generating protein sequences optimized for diversity and speed. For a given structure, InvMSAFold generates the parameters of a pairwise probability distribution over the space of sequences, capturing the amino acid covariances observed in Multiple Sequence Alignments (MSA) of homologous proteins. This allows for the efficient generation of highly diverse protein sequences while preserving structural and functional integrity.
We demonstrate that this increased diversity in sampled sequences translates into greater variability in biochemical properties, highlighting the exciting potential of our method for applications such as protein design. The orders of magnitude improvement in sampling speed compared to existing methods unlocks new possibilities for high-throughput in virtual screening.

Published as a conference paper at ICLR 2025
FAST UNCOVERING OF PROTEIN SEQUENCE DIVERSITY FROM STRUCTURE CONFERENCE SUBMISSIONS
Luca Alessandro Silva Department of Computing Sciences Bocconi University
Barthelemy Meynard-Piganeau
Institut de Biologie Paris Seine, Biologie Computationnelle et Quantitative LCQB Sorbonne Université
Milan, MI 20100, Italy
silva. luca Suca@phd.... www.occoni.c....mmmmmm it
Paris, France ba Barthel w hwt..//aardd@pplyytecciii..eeeee
Carlo Lucibello
Christoph Feinauer
Department of Computing Sciences
Department of Computing Sciences
BIDSA
Bocconi University
Bocconi University
Milan, MI 20100, Italy
Milan, MI 20100, Italy
christoph. fei fein it
carlo. wtw.beelloeuiboooooiii.i..oo
ABSTRACT
We present InvMSAFold, an inverse folding method for generating protein se- quences that is optimized for diversity and speed. For given structure, InvM- SAFold generates the parameters of a probability distribution over the space sequences with pairwise interactions, capturing the amino acid covariances ob- served in Multiple Sequence Alignments (MSA) of homologous proteins, This allows for the efficient generation of highly diverse protein sequences while pre- serving structural and functional integrity. We show that this increased diversity in sampled sequences translates into greater variability in biochemical properties, highlighting the exciting potential of our method for applications such as protein design. The orders of magnitude improvement sampling speed compared existing methods unlocks new possibilities for high-throughput virtual screening.
INTRODUCTION
Inverse folding aims to predict amino acid sequences that fold into a given protein structure, and plays fundamental role, for example, in the protein design pipeline of RFDiffusion (Watson et al. (203)). Recent deep learning approaches such as ESM-IF1 (Hsu et al. 2022 or ProteinMPNN (Dauparas) al.) 2022) achieve remarkable accuracy in this task. However, instead of predicting single ground truth sequence, it is often desirable to have method that is able to generate a variety of different sequences with the desired fold, i.e., solving a one-to-may problem, Fig. 2 This diversity could
be leveraged for example by starting from a source sequence Sturnfels et al. (022)); |Byyan et al. (202I) and taking different molecular environments into consideration (Krapp et al. 2023). Such an approach would expand the sequence design space while preserving structural consistency. enabling larger pool of sequences for selection based on additional properties like thermostability. solubility, or toxicity. In drug discovery. for example, it would facilitate the generation of large number of diverse candidates, allowing further selection optimized for properties such as bioavailability. Similarly, in biotechnology and enzyme engineering, it would facilitate the creation of enzymes with tailored properties, such as improved stability and activity under varying conditions. On the computational side instead, even after training, sampling from transformer-baat based architectures such as ESM-IFI et al. (2022) can be very expensive. This can severely
Hsu et al. (2022) or ProteinMPNN limit the widespread use of such models, especially in virtual screening-like settings.

---

Published as a conference paper at ICLR 2025
True sequence the Structure
RMSD with target Structure
ESM-IF1: recovers the original sequence
Homologs
InvMSAfold captures the landscape of sequences with this fold
Sequence Space
Figure 1: InvMSAfolds expands the scope of inverse folding to the retrieval of the entire landscape of homologous proteins with similar folds.
In this work, present efficient method that is able to generate diverse protein sequences given a structure, including sequences far away from the native one (see Fig. I.. Recent architectures for inverse folding based on encoder-decoder architectures, where structure is encoded and sequence decoded. During training, such models typically take into account only the native sequence
of given structure, maximizing its probability given the structure (Hsu et al. 2022 Dauparas et 2022).
In our approach, we use the decoder to generate the complete set of parameters for lightweight model that is sufficiently expressive to describe the sequence di- versity of the multiple sequence alignment (MSA) of the native sequence. This architecture trained end- to-end to capture the broader probability distribution of sequences corresponding to a specific fold. We consider different choices for the lightweight model, and settle on the pairwise family (Figliuzzi et al. 2018), since it has been widely applied to protein sequence data (Coco et al. (2018a), proven to be good generation (Russ et al. 2020) and demonstrated to capture information that enables fitness prediction (Poelwijk et al., (017). A potential issue with pairwise models is that they typi- cally have number of parameters that is quadratic in the sequence length. We solve this issue by considering low-rank approximations, thus reducing the effective number of parameters to the same order of the sequence length. By doing so drastically reduce the number of parameters, boosting training efficiency.
FE-KSORUWHD...
Figure 2: One-to-many nature inverse folding. Top: Structure of IKA0. Bot- tom:
Some homologos sharing the fold.
Our model generates all of these parameters in single forward pass, similar previous research Li jet al. (2023). Once this generation done, resulting pairwise model can be used for generating large number of diverse sequences very efficiently leveraging CPUs dramatically facilitating its use. We show that the models we generate are able to capture the diversity of the protein family better than other models and able to find sequences far away from the natural sequence that are predicted to still fold into the same structure. We also show that this increased diversity translates into a more spread distribution in other properties, enabling selection of promising sequences from a larger pool. Codes to train the models and replicate some of the results be found at the Potts Inverse Folding repository
standard machine,
METHODS
In this Section, describe the components of our architecture, InvMSAFold, and its training procedure. Given a structure-sequence pairing (X, xx), inverse folding methods typically define a probability distribution the space of sequences as p(ar) XX) Most deep learning-base based methods
2

---

Published as a conference paper at ICLR 2025
model this distribution auto-regressively, using ((o|(X) = -I 0.1.....11 00 X), where L is the length, and train by minimizing the loss on the true sequence ox. possibly after adding noise to the coordinates X (Hsu et al. 2022, Dauparas et al. 2022). Sampling amino acids ruto-regressively requires a full forward pass through the neural network for every generated amino acid, making it very expensive to use in virtual screening- like setting, where large number of sequences scanned for properties beyond folding into structure X. Moreover, minimizing the loss on the true sequence ignores the one-to-many property of inverse folding depicted in Figure 2 resulting in distribution peaked around very few sequences (as we will show later) and not capturing other parts of sequence space that might be interesting for the problem at hand.
We address both issues, reduced diversity and slow sampling, by having InvMSAFold output simple and sesy-to-samml--rree model for fast inference and by training our architecture over multiple sequence alignments for any given input structure instead of single sequence.
2.1 THE INVMSAFOLA ARCHITECTURE
InvMSAFold is a neural network whose inputs are the structure backbone coordinates X and whose outputs the parameters of a lightweight sequence model. These parameters, ((X), ), then used to sample amino acid sequences compatible with the given structure. Therefore, our overall model takes the form p( X) ((
We explore different parametric families for the lightweight model. In order to go beyond distributions that treat different positions in the protein as independent and to ensure sufficient expressivity, we focus on pairwise models Cocco et al. (2018)) that correspond to the well-known Potts model (Barrat- Charlaix 2018) from statistical physics. Such models have an experimentally validated ability to capture structure-sequenc relationships Russ et al. (2020). We therefore train the neural network to learn the mapping X - f (J, H), where J is a tensor of size I, x q x I. x q and H is matrix of size I x with I. the sequence length and the number of different amino acids. As is common in pairwise models, we call the quantities His the fields, indexed by position i and amino acid type a, and the quantities j. the couplings, indexed by pair of positions i, and a pair of amino acid types a, b. Fields describe the propensity of amino acids to appear at given positions, while couplings describe the propensity of pairs of amino acids to appear at pairs of positions.
The complete InvlsAFold architecture is composed of two parts, the structure encoder and a decoder which outputs the fields and couplings, see Fig. 3
Pretaiied ESMel- Broader
Grooded
Coec
Kaper
Tha dordamm Decider
beedia cu di..
Lou sppesiiaation Tieeullsaett
Fiilda
W
Libil
ond coaaraaassaaa
Figure 3: Left:: Decoder architecture up to the generation of the low-rank tensor V. Right: The low-rank tensor V is used to generate the couplings J and the fields H.
For the encoder, we use the pre-trained encoder from the ESM-IF1 model of of al. (0022) which follows the GVP-GNN architecture proposed in (Jing et al. 2020). This encoder gives rotationally invariant representation of the input X that has been shown to be effective in capturing the geometric features coming from the protein's 's 3D structure. During training we add Gaussian noise to these representations as commonly done in previous research Hsu et al. (2022). The decoder takes as input batch of encoded structures, padded to common length L. These L x D matrices (D 512 in our experiments) batched into a tensor are passed through 6 Transformer layers with 8 attention heads. The output is first embedded into I. DK dimensional space through a linear layer followed by an
3

---

Published as a conference paper at ICLR 2025
element-wise ReLU activation function, then it is reshaped to a L K D tensor and finally it is projected to a L x K X q tensor V through another linear layer. By selecting K < L, we then create from V low-rank coupling tensor according
K
1
Σ
Vakka VjAk.b-
VK
R=1
Thanks to this low-rank structure, drastically reduce the number of parameters of our pairwise model from L 9² to L x K. The 1/ //VK' factor instead is used to obtain ((1) at initialization, similar to scaled dot product attention in Vaswani et al. (2017). We note that other settings, low-rank decompositions like equation of J have been shown to be similarly effective for most tasks as their full-rank counterparts (Cocco et al. 2013). The fields H are computed by passing through another linear layer, and then contracting the tensor by summing the entries across the latent dimension. Also here after summing we scale by VK to ensure the fields to remain (1).
2.2
INYMSAFOLD-PW
Given the parameters and H we explore two different approaches to define a probability distribution over sequences of amino acids. The first distribution, which we term InvMSAFold-PW a standard pairwise distribution Cocco et al. 2018a) that defines the negative log- eggliiklihood of sequence as
1og (o
(o H - F(o) log(Zuu)
Σ Jianisa
Hi,o EHH
log (7P0),
where Zee is a o-independent normalizing constant and E(or is called energy. This distribution has been explored extensively for proteins Cocco et al.] (2013) it possesses some nice theoret- ical properties (Barrat-Charlai Charlaix (2018). yet has the disadvantage that 2nd intractable, making
likelihood optimization difficult. To overcome this issue, standard alternative is resort pseudo-
loglikelihood (Besag 1977: Ekeberg et al. 2013)
1 H(( L
Σ log TM (o
which is a statistically consistent estimator Baaraat (2018). Moreover, as we will show, pairing Eq. 3 with the low-rank assumption the couplings (Eq. 1b. allows for very efficient computations.
2.2.1 FAST
LIAEDOOLLKEIHHOOOO COMPUTATION
By enforcing the low-rank constraint on the matrix J, the number of parameters is reduced from O(L2) (L). However, calculating the pseudo- likelihood naively by materializing the full coupling matrix J as in Eq. [ leads to a quadratic computational cost. To achieve linear cost in both memory and computational complexity we note that, given the low-rank structure, the coupling term in Eq. becomes
K
L
L
Σ J. i
1
Σ
Σ
Vi.k.
-
V2
Vika
(4)
2
k=1
which can computed in (L) time since it contains a single sum over the positions i. For position p, the pseudo-lkkelihood likelihood is
D (o H.
exp(-E(o))
( H.J) = E peu (o =. H,J) Σ exp(-E((,=
(5)
where the numerator is independent of and therefore the same for all positions. If we can compute the denominators efficiently for every position p. we can then maintain linear cost also for the sum of 3 Let's call gF the configuration that is equal to except for having amino acid in position p. The relevant quantity for the computation of the energy E(GP.c) can be written as
Σ V.S..PPI = Σ Ve.k.. Vp.k.o, + Vpkke
(6)
4

---

Published as a conference paper at ICLR 2025
The first term is common to all positions p, hence can be computed just once. Combining Eq. and Eq. we can compute all the denominator terms in Eq. for for the different positions p efficiently, Finally, for the quadratic regularization on the couplings have
resulting in (L) cost for Eq. 3
))
ΣESaas KJ a.
Σ
1
Σ 2
Vakia Vi
Σ
Σ
Vikka Vi
j.b.
2
k,k'
ki,
which leads again to linear cost. Similar reasoning can be applied to the field-dependent part of the energy.
2.3 INYMSAFOLD-AR
Although, as previously mentioned, the pscudo-log-likeliood Eq. 3 is a statistically consistent estimator, is known in practice could struggle to the second moments the MSAs it is trained on Trinquier et al. (202T). Moreover, for sampling, we have to resort to MCMC algorithms, which can have difficulties in navigating the complex nigh-dimensional landscape given by Eq
Based on the above practical limitations, we consider an alternative efficient suto-regressive vari- of pairwise models Trinquier al. (202T). which we term InvMSAFold-AR, that defines an autoregressive distribution pdr over amino acids having negative log-likelihon
log
.. H,J) Hi,
Since this parametrization decouples into a sequence of univariate distributions, it has the advantage that the normalization factor is tractable allowing for closed-form computation of the likelihood. As a result, we can train with maximum likelihood and also sample much more robustly from the model. Since for InvMSAFold-AR the number of terms in the sum in Eq. depends on the position i, we rescale the couplings as
Ji..5.b
Je..,..
max(i,j)
following what was done in (Ciarella et al. 2023). We found this to be beneficial for training and note that without this scaling the neural network would have to generate couplings of significantly different magnitudes for different sites. This would not be problem if we were to optimize the couplings directly, as previous research Trinquier et al. (202T). but might be problematic the couplings are generated by neural network.
2.4 TRAINING ON HOMOLOGOUS SEQUENCES
In most other works Hsu et al. (202)); Dauparas et al. (2022), inverse folding models are trained predict the ground truth sequences only. In this work, we aim generate distribution that captures the complete sequence space that is compatible with the input structure X. To this end, we structure X to extract an MSA My from
leverage the ground truth sequence corresponding a sequence database (see next section for details). We then use the pairs (X, .M) for training by taking the mean negative (pseudo) log-likelihood of a random subsample sequences in Mx given the parameters 0 ((X) generated by the network. For both HrrMSFFold-R. and InvMSAFold- AR, we add a regularizing Lg term for the fields and the couplings as typical for these models Barrat-Char 2018), Cocco et al. (2018b). For details, see Sec. 4.1
3 DATA AND TRAIN-TEST SPLITS
In order to control the level of homology in our evaluation, we create three test sets, which we call respectively inter-cluster, intra-cluster and MSA test set. We base these on the CATH database Sillitoc al. (2021), which classifies protein domains into superfamilies and then further into clusters based on sequence homology. We use the non-redundant dataset domains at 40% similarity and associate every domain cluster indicated in the CATH database. We then choose 10% of the
5

---

Published as a conference paper at ICLR 2025
sequence clusters uniformly at random and assign them to the inter-cluster test set, excluding these clusters from the training set. Because many superfamilies contain only one sequence cluster, there is significant amount of superfamilies that appear only in the inter-cluster test set and not in the training set, making this a hard test set. We then create the less stringent intra-cluster test set by taking from every domain sequence cluster that is not in the inter-cluster test set and has at least two domains a single random domain. We then use the remaining domains as the training set. Finally, we create MSAs for all sequences in the datasets using the [MMeqs22 software and the Uniprot50 database. We further split the sequences in the MSAs into 90% used for training and 10% for the MSA test set. This last test set is the least stringent one since it is based on domains that are also in the training set. The resulting sizes of the datasets are the following: 22468 for the training set, 22428 for the MSA, 1374 for the intra-cluster, and 2673 for inter-cluster test sets respectively.
4 RESULTS
4.1 MODEL TRAINING
For both training and inference, create embeddings for the structures using the ESM-IFI encoder (Hsu et al. 2022). We then add independent Gaussian noise with standard deviation equal to 5% of the standard deviation of the embeddings across all positions, dimensions, and samples in the training set. The random MSA subsets Mx are independently generated at each training step, where the number of sequences sampled is a model hyperparameter. For InvMSAFold-PWs we train with a single structure in each batch, with a MSA subsample size for Mx of 64, a rank K of 48, a learning rate of 10 and L2 regularization constants of > A 10 for fields and couplings. For LnwMSAFold-ae Both models are trained with Adam W optimizer for a total of 94 epochs. We monitor the negative pseudo-loglikelibne for InvMSAFold-PW and the negative likelihood for InvMSAFold-AR on the train and the different test sets. Training and test curves are reported in Appendix A...I here we just mention that the ordering of the losses the different datasets is consistent with the hardness reasoning behind the split in the last section, yet all curves are significantly better than a null model, signaling that the model is able to generalize in all cases.
we tune the hyperparameters as discussed in Appendix A.2.2 for the details.
4.2 SYNTHETIC DATA GENERATION
Transformer- based architectures such as ESM-IFI (Hsu et al. 2022) or ProteinMPNN (Dauparas et al. 2022) be very expensive use at inference time, as data generation requires forward pass through the decoder for every sample. On the other hand, InvMS A.ldd requires just a single forward pass through the decoder after training; once the parameters J, H of the pairwise model obtained (either of the two parametrizations), it can be run on the CPUs of a standard machine to produce many samples. The sampling speed is drastically faster than ESM-IFF or ProteinMPNN.
As can be seen from Figure the sampling speed on CPU of Aoold-A is orders of magnitudes faster than the one of ESM-IFI on GPU (notice that the y-axis is on log-scale). Moreover, given the lightweight model generated by InvMSAFold-Ar AR,, and the fact that host RAM is typically more abundant than GPU memory, sampling for InvMSAFold-AR can be easily batched, resulting in a virtually constant sampling time across all lengths L. This is crucial in virtual like setting, where proteins in the order of millions are generated and analyzed for properties beyond folding into a desired structure; looking at Figure 4 this would be unfeasible for ESM-IF1. In Figure 4 we report just InvMSAFold AR and not Inv as the latter requires an MCMC sampler, while the former and ESM-IF1 return i.i.d. samples from their respective distribution. To sample from ESM-IFI we utilized a NVIDIA GeForce RTX 4060 Laptop having 8Gb of memory, while for FolddAFo we used single core of 19- 13905H processor.
4.3 COVARIANCE RECONSTRUCTION
The ability of generative model to reproduce covariances between amino acids in MSAs has been shown to be good metric for measuring how well the protein landscape is captured (Figliuzzi et al.) 2018). Models with this ability have been shown to enable efficient sampling of experimentally validated, functional protein sequences (Russ et al. 2020). In order to test this ability, we created

---

Published as a conference paper at ICLR 2025
Comparing samplinn ng times of ESM-IF1 F1 and InvMSAFold-Ar
1a9cN02 206
1000A00,
180cA00, nm 437
Time (seconds)
107
107
10"1
10"
InMMSAFald-Ar
10 25
50
75
100
125
150
175
200
number of samples
Figure 4: Comparing the sampling speeds in seconds (y-axis in logscale) of ESM-IFI (green) and InvMSAFold-Al (blue non-batched, red batched) for la0aA00, Ia8dA02, la0cA00 having increasing lengths L respectively of 63, 206, 437 as increase the number of samples generated (æ-axis).
ESM-IFI
InyMSAFold-PW
InyMSAFold-AR
Inter-Cluster 15.8
4.6
0.49
Intra-Cluster
11.9
11.2
0.67
Table 1: Average KL Divergence across domains reported in Appendix A.3.2 between estimated densities of natural and synthetic samples projected on the first two PC components of the MSA for the Tpee//Intr-Clluttt datasets. For each method and for each backbone, we sampled 2k sequences. Densities are estimated through a Gaussian KDE of kernel size 1.0.
MSAs with synthetic samples from InvMSAFold-PW. InvMSAFold-AR and ESM-IF1, and compared the amino acid covariances in these MSAs with the covariances the MSAs of natural sequences. Given MSA of sampled or natural sequences, we define the covariance between amino acids and b at positions and j as Ciy(a, (a, b) = Jig (a, b) fi(a)); where fiz (a, b) is the frequency of finding amino acids and at these positions in the same sequence in the MSA and f((a) and fy are the overall frequencies of the amino acids and b at these positions. In order to compare two sets of covariances, we calculate their Pearson correlation in previous research (Trinquier et al., 2021). For details on these experiments, we refer to the Appendix A.3.I
From Figure and the attached table, we can see that MnA and InvMSAFold-Pe capture the covariances significantly better than ESM-IF1, especially at longer sequence lengths. In fact, InyMSAFold-Ar shows the best performance overall, with performance robust to L. The worse performance of ESM-IFI in this task is not surprising, it has not been trained for it, yet it signals a significant limitation of the model for many tasks. On the other hand, the stronger performance of InvMSAFold-AN compared InvMSAFold-PPy could be due to the latter being trained with pseudo- poauudo-oolikelhhoods. which do not capture covariances perfectly even when training a pairwise model directly (Ekeberg et al. (2013), due inefficient MCMC sampling. We also tested synthetic MSAs from ProNeinMPNN (Dauparas et al. 2022). detecting an analogous behavior to ESM-IFI. We report the results in Appendix/A.1
The result strengthens our confidence that MSAool is able to model the sequence landscape of an unseen structure. We explore this in more detail in the section.
4.4 SEQUENCE PATTERNS
In this Section, we compare the projections of sampled sequences onto the first two PCA components of the one-hot encoded MSA of natural homologs to get more fine-grained analysis how the models are exploring the space of natural homologs compared to the global measure reported in 5 Results for the NDP Kinase Ixqi (Pédelacq et al. 2005) are shown in Figure 6 The most striking feature is the narrow focus in this space for sequences generated from ESM-IF1, suggesting that the distribution of ESM-IFI is too centered around a single sequence, with very small coverage of the full sequence space. Sequences sampled from InvMSAFFld show broader coverage of
7

---

Published as a conference paper at ICLR 2025
Correlations
InvMSAFold-AR
InvMSAFold-PK
ESM-IF1
intr-Clluste
0.50
0.25
200
400
Correlations
InvMSAFold-AR
InvMSAFold
ESM-IF1
Inter-Cluster
0.50
0.25
0.00
400
200 Length of sequence
Length of sequence
Length of sequence L
ESM-IF1
InyMSAFolddPPW
InvMSAFold-AR
Quartiles (t)
Inter-Cluster
Intra-Clused
Inter-Cluster
Cntra-Cluster
Inter-Cluster
Intra-Cluster
First quartile
0.23
0.21
0.29
0.28
0.37
0.35
Median
0.31
0.31
0.43
0.42
0.53
0.53
Third quartile
0.43
0.45
0.50
0.53
0.60
0.60
Figure 5: Distribution of Pearson's correlation coefficients between the covariances from sampled and natural sequences for domains having least natural sequences. Top: KDE plots representing how the correlations for InvMSAFold-AR (blue), InvMSAFold- sequence length L. The top/bottom rows show results for the Intra/Inter-Cluste test sets respectively. Bottom: Table with quartiles of the above Pearson's 's correlation coefficients. The best model for each quartile is highlighted by boldface numbers.
(red) and ESM-IF1 (green) vary with
Kernel Density Estimates of Components: 1xqiA00
ESM-IF1 KL:
InvMSAfoid:PW)
9.19
ImvMSAfold-AR
0.27
10
MSA ESMIF1
10.
MSA
10.
MSA
InvMSAcld-PW
IR
PCA Companent
6-
PCA Component
PCA Component
4.
12
PCA Component 1
PCA Component
PCA Component
Figure 6: Estimated densities of natural and synthetic samples projected on the first two PC compo- nents of the MSA for IxqiA00, with KL-divergence reported in the title of each subplot.
8

---

Published as a conference paper at ICLR 2025
sequence space but are still concentrated around the single dominant mode. This again could be result of the approximate training given the pseudo likelihood, or of the challenges of sampling from multimodal distributions for MCMC algorithms. While this increased diversity in the samples might already be useful in some settings, it is notable that in this case secondary modes are not captured at all, and that even the dominant mode is still covered only in part. In contrast, sequences sampled from Inv.SAFold-AR cover the sequence space more comprehensively, demonstrating an ability to cover multiple distinct modes. These results are not restricted to this specific NDP Kinase, but are consistent across a variety of out-of-sample proteins we tested, as can be seen from Table []] Other plots are reported in Appendix [8.2] where we report also results for ProteinMPNN, which, although typically samples less narrowly, generally displays a similar performance to ESM-IFI.
4.5 PREDICTED STRUCTURES OF SAMPLED SEQUENCES
Similarly to other works (Li et al. 2023 Dauparas et al. 2022). we test what extent the sequences generated by our models are predicted to fold into the correct structure as we increase the nor- malised hamming distance the native one. From Figure 7 we can that InyMSAFold-AR and FoodSAFol are naturally capable of generating sequences with low sequence similarity to the native sequence, while ESM-IFI often generates sequences very close to the native sequence, which is consistent with what we observed in Subsection 4.4
MSA
MSA
MSA
InvMSAFolc-AR
InvMSAFold-PW
ESM-1F
Density
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Sequence Distance
Sequence Distance
Sequence Distance
Figure 7: Distances to native sequences, averaged over 15 structures from the inter-cluster test set.
We therefore sampled sequences from ESI at a higher temperature (for details of the sampling procedure refer to the Appendix(A.3.2) Following the generation of these sequences, we predicted their structures with AlphaFold 2 (Jumper et al. 2021 Mirdita et al., 2022) with templates switched off. From Figure which shows results for the intra-cluster test set, distances ESM-IFI is the best-performing method. However, its performance drops significantly for larger distances, eventually becoming worse than both InvMSAFold-AR and InvMSAFold-PPy which show a more robust performance at higher distances. For the inter-cluster test set, we observe a similar behavior as the distance increases, with a general worsening of all of them (results reported in Appendix B.I). For the complete list of domains used from the two test sets in this experiment, we refer to Appendix A.3.2
see that for lower
4.6 PROTEIN PROPERTY SAMPLING
Another potential area of application for InyMSAFold are protein design tasks where other structural properties of the designed protein are important. In virtual screening-like setting, one might want to generate a large number of sequences that obey certain structural constraints and then select from this set sequences that have other desirable properties. We test this setting by analyzing the range of (predicted) thermal stabilities and solubility of the sequences generated by the different models. In order to predict thermal stabilities we use Thermoprot (Erickson et al. 2022), which takes into account whether the protein operates within Mesophile or Thermophile species. For solubility, we use predictions from Protein-Sol (Hebditch et al. 2017). Working with aligned sequence also introduces gaps, creating therefore variable effective length of the proteins (in terms of amino acids). We verified that the number of gaps was not discriminative for the two predicted properties. From Figure which shows domain Iny larger than of ESM-IFI. showing that the larger sequence diversity generated by both InvMSAFold PW and InyMSAFold-A translates into wider range of predicted protein properties. Note also that
observe that the support of InvMSAfold is significantly
9

---

Published as a conference paper at ICLR 2025
25
ESMiFI
ESM IF
20
RM50
RMSDD
5-
0.65
0.7
0.75
0.8
0.85
0.65
0.7
0.75
0.8
0.85
Distance
Distance
Figure 8: Comparison of the quality of the generated sequences at increasing hamming distance from native one. Shown an average of 14 structures from the intra-cluster test set. We then refold the sequence with Alphafold and compare the refolded structure with the original one using RMSD.
1ny1A00
Thernophle
1.0
1.0
InvMSAfald-An ESM-IF1
InvM5Afols-PM PW
ESM-F1
0.8
0.8
0.6
0.6
0.4
0.4
Mesophle
0.2
0.2
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
Figure 9: Comparison of the distribution of predicted solubility and thermostabillity of samples from InvlSAFold and ESM-IFI for Iny 1A00. The x-axis reports the solubility predicted by Protein-Sol, while the y-axis the output of Thermoprot, binary classifier (0 is mesophile and thermophile).
while we generated the same number of sequences for a fair comparison, the computational resources very different have shown in Subsection 4.2 These results not specific to this domain, for more examples and for more details on the procedure we refer the reader to Appendix B.3
5 CONCLUSION
InvMSAFold a novel approach to inverse folding that combines modeling the complete sequence space corresponding to structure with computational efficiency. The core idea is frame the problem as a two-stage generation process: an expressive large neural network generates the parameters of a sufficiently expressive family-specific model. During training, the loss is computed data augmented through an MSA, which captures the variability of sequences compatible with the input and prevents the model from narrowly focusing the native sequence. Notably, this idea is not specific to our model formulation and could also be applied to other architectures, such as ESM-IF1. Once trained, for any input structure, the model requires only a single forward pass generate the parameters of the pairwise model, which can then be used to generate diverse sequences orders of magnitude faster on CPUs compared previous architectures. We extensively validate InvMSAFold through numerous out-of-sampii tests, demonstrating its ability to generate sequences that deviate significantly from the native sequence while maintaining structural fidelity and capturing the evolutionary patterns of the MSAs. Additionally, we show that the broader exploration of the sequence domain enables the sampling of a wider range of protein properties of interest. We believe that InvMSAFold, with its combination of diverse sequence exploration and rapid sequence generation, opens up possibilities for virtual screening. for example enabling simultancous optimization of properties like thermostability, altered substrate specificity, and reduced toxicity.
10

---

Published as a conference paper at ICLR 2025
6 ETHICAL CONSIDERATIONS AND CONCERNS
We recognize the importance of biosecurity considerations in protein design and the role of sequence- based screening systems. In principle, approach generates diverse protein sequences that fold into known structures, which raises the question of how dangerous sequences are detected by existing screening methods.
To mitigate this risk, we emphasize that our method relies on homologous sequence data dur- ing training. This inherently means that any generated sequences remain within the evolutionary space of known proteins and would be detectable by homolog detection algorithms. Given that biosecurity screening protocols, such as those employed by DNA synthesis providers Intpps w/genesyythesisseeee srresssconortium org/ increasingly incorporate homology-basee detection, we recommend that safeguards extend beyond simple sequence matching to include homologous sequence analysis for known hazardous proteins.
Moreover, existing tools for structure prediction and homology detection have reached a level of accuracy where they can reliably identify structurally and functionally similar sequences. The maturity of these technologies enables the detection of potentially dangerous sequences even when sequence similarity is low.
REFERENCES
Pierre Barrat- Charlaix. Understanding and improving statistical models of protein sequences. Theses, Sorbonne Université, November 2018. URL httpps:/ttheses hal. science/
tel- 026602266
Julian Besag. Efficiency of pseudolikelihood estimation for simple Gaussian fields. Biometrika, 64(3):61-6618, 12 1977. ISSN 0006-3444, doi: (0.1093/biomeet64....6666 URL https
//dii.org100 1ttp:///..100////mee///.... 34.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick Riley, George M Church, Lucy J Colwell, and Eric D Kelsic. Deep diversification of aav capsid protein by machine learning. Nature Biotechnology. 39(6):9116696 2021.
Simone Ciarella, Jeanne Trinquier, Martin Weigt, and Francesco Zamponi. Machine-learning-ee assisted monte carlo fails at sampling computationally hard problems. Machine Learning: Science and Technology, 4(1 4(1):010501, 2023.
Simona Cocco, Rémi Monasson, and Martin Weigt. Inference of hopfield-potts patterns from covariation in protein families: calculation and statistical error bars. In Journal of Physics: Conference Series. IOP Publishing, 2013.
Simona Cocco, Christoph Feinauer, Matteo Figliuzzi, Rémi Monasson, and Martin Weigt. Inverse statistical physics of protein sequences: a key issues review. Reports on Progress in Physics, 81(3): 032601, 20.
Simona Cocco, Christoph Feinauer, Matteo Figliuzzi, Rémi Monasson, and Martin Weigt. Inverse statistical physics of protein sequences: a key issues review. Reports on Progress in Physics, 81 (3):032601, jan 8b. doi: 10. 01188//16116633/aaee0 1088/1366 063366633369966333
URL htt.:tttt////tt doi org/
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et Robust deep learning- basedd protein sequence design using proteinmpnn. Science, 978(6615):49-66 2022.
Magnus Ekeberg, Cecilia Lövkvist, Yueheng Lan, Martin Weigt, and Erik Aurell, Improved contact prediction in proteins: using pseudolikelihoods infer potts models. Physical Review E, 87(1): 012707, 2013.
Erika Erickson, Japheth E Gado, Luisana Avilán, Felicia Bratti, Richard K Brizendine, Paul A Cox, Raj Gill, Rosie Graham, Dong-Jin Kim, Gerhard König, et al. Sourcing thermotolerant poly (ethylene terephthalate) hydrolase scaffolds from natural diversity. Nature communications, 13(1): 7850, 2022.
11

---

Published as a conference paper at ICLR 2025
Matteo Figliuzzi. Pierre Barrat-Charlaix, and Martin Weigt. How Pairwise Coevolutionary Models Capture the Collective Residue Variability in Proteins? Molecular Biology and Evolution, 35(4): 1018-1027, 01 2018. ISSN 0737-4038. doi: 10.109//ollbe//msx000. URL hetps://dos
org/ 0933 mo hwwobeeemmms00
Max Hebditch, M Alejandro Carballo-Amadie Spyros Charonis, Robin Curtis, and Jim Warwicker. Protein-sol: a web tool for predicting protein solubility from sequence. Bioinformatics, 33(19):
3098-3100, 2017.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.0141, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589) 2021.
Lucien Krapp, Fernado Meireles, Luciano Abriata, and Matteo Dal Peraro. Context-aware geometric deep learning for protein sequence design. bioRxiv, pp. 2023-06, 2023.
Alex J Li, Mindren Lu, Israel Desta, Vikram Sundar, Gevorg Grigoryan, and Amy E Keating. Neural network-derived potts models for structure- protein design using backbone atomic coordinates and tertiary motifs. Protein Science, 32(2))e4554, 2023.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible all. Nature methods, 1966)6679-682, 2022.
Dean- Pédelacq. Geoffrey S Waldo, Stéphamic Cabantous, Elaine C Liong, and Thomas C Terwilliger. Structural and functional features an ndp kinase from the hyperthermophile crenarchaeon pyrobaculum aerophilum. Protein science, 14(10):2562-2575 2005.
FJ Poelwijk. M Socolich, and R Ranganathan. Learning the pattern of epistasis linking genotype and phenotype in a protein. biorxiv: 213835, 2017.
William P Russ, Matteo Figliuzzi, Christian Stocker, Pierre Barrat-Charlaix, Michael Socolich, Peter Kast, Donald Hilvert, Remi Monasson, Simona Cocco, Martin Weigt, al. An evolution-based model for designing chorismate mutase enzymes. Science, 369(6502):440 445, 2020.
Ian Sillitoe, Nicola Bordin, Natalie Dawson, Vaishali P Waman, Paul Ashford, Harry M Scholes, Camilla SM Pang, Laurel Woodridge, Clemens Rauer, Neeladri Sen, et al. Cath: increased structural coverage of functional space. Nucleic acids research, 19661):2266DD233 2021.
Pascal Sturmfels, Roshan Rao, Robert Verkuil, Zeming Lin, Tom Sercu, Adam Lerer, and Alex Rives. Seq2msa: A language model for protein sequence diversification. In Machine Learning in Structural Biology Workshop. NeurIPS 2022, 2022.
Jeanne Trinquier, Guido Uguzzoni, Andrea Pagnani, Francesco Zamponi, and Martin Weigt. Ef- ficient generative modeling of protein sequences using simple autoregressive models. Nature communications, 12(1):1-11, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Joseph I. Watson, David Juergens, Nathaniel R Bennett, Brian L. Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rédiffusion. Nature, 6889 2023.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 COVARTANCE RECONSTRUCTION FOR PROTEINMPNN
In this Appendix, we report the results for another popular method available in the literature, Protein- MPNN (Dauparas al. 2022). The architecture is built upon transformer Vaswani et al.) (2017) encoder-decoder architecture, similar to ESM-IF1 Hsu et al. (2022). but is shallower as consists of only three encoder and decoder layers. Moreover, the training performed only structures of the CATH dataset in contradistinction to ESM-IF1, which augments the training data with millions of structures predicted from Alpha-Fold (Jumper al. 2021).
We begin with the covariance reconstruction capabilities synthetic sequences generated from Protein-MPNN. The sampling procedure is analogous to that reported in Appendix A To have consistency with the other models explored in Section 4.3 we sampled sequences at the training temperature of 1. From Figure I0]
observe that the results similar those observed for ESM-IF1, as the recovered covariances are worse than both InvMSAFold-PPl and InvMSAFold-AR and the performance worsens significantly with sequence length. One difference with ESM-IF1 can be observed in the top-right plot of Figure TO Protein-MPNN at temperature can generate sequences with higher diversity, although still smaller than the one observed in the native MSAs. This could be due to the shallower architecture and the reduced training set of Protein-MPNN.
Inter-Cluster
Intra-Cluster
MSA PPNNein-MPNN
0.6
Correlations
Density
0.4
0.4
3
0.2
0.2
0.0
200
400
200
400
0.00
0.25
0.50
0.75
1.00
Length of sequence
Length of sequence L
Sequence Distances
Protein-MPNN
Quartiles (*)
Inter-Cluster
Intra-Cluster
First quartile
0.23
0.21
Median
0.31
0.31
Third quartile
0.43
0.45
Figure 10: Top: KDE plots representing how the correlations between covariances of sequences from Protein-MPNN and natural homologous vary with sequence length for sequences in the inter-cluster (left) and intra-cluster (center) test sets, and distances to native sequences for samples from Protein-MPNN, averaged over 15 structures from the inter-cluster test set (right). Bottom: Table with quartiles of the above Pearson's 's correlation coefficients.
13

---

Published as a conference paper at ICLR 2025
A.2
TRAINING DETAILS
A.2.1
TRAINING AND TEST CURVES
InVMSAFold-PW
InvSSAFold-AR
22
2.2
Intra Cluster Tect Set
Intra Custer Teel Set
2.1
Inter-Cluster
2.1
Intar-Clustrr Sat
Train Set
Trai Set
MSA Teat Bal
MSA Test Set
2.0
1.9
1.9
1.8
1.7
1.7
1.6
1.6
1.5
40
60
80
20
60
80
Epochs
Epochs
Figure 11: Negative pseudo log-likelihood (npll) for the train and test sets by training epochs. The training loss is rolling average window of 5 epochs, which is why the plotted value is higher than the test losses initially.
As can be seen from Fig. 11 for both models the ordering of the losses on the different datasets is consistent with the reasoning behind the split the last section: The MSA test set has sequences coming from domains which the model has seen during training, and the loss this test set is similar to the training loss. The intru-cluster test set contains domains of which the model has seen domains belonging to the same sequence cluster; the loss this test set is higher than the training loss and seems to saturate during training. For the inter-cluster test dataset, which contains domains that belong to sequence clusters not present in the training set, the loss is significantly higher, even though still much better than random.
A.2.2 HYPERTUNING
To select hyperparameters for InvMSAFold-AR relied the library Optunal The parameters optimized for where Adam's learning rate(Ir), the rank of the approximation for the coupling matrix J(K), the penalties for the couplings and the fields (A,, ),). the batch size used for training (B), the batch MSA size used in the loss ([Mx) \) and the value of dropout. We sampled parameters through the TPESampler, allowing median pruning of bad trials to improve the speed. We gave Optuna 50 trials of 90 epochs each (irrespective of batch size), while as a selecting metric, we used the average of the log- likelihoohod between the inter/intra-cluste test dataset. In the table below we recap the parameter values selected by the hyper-tuning
Hy Hypertuning results
Model
Dropout
B
(d,,AA)
Ir
ArDCA
0.1
8
32
48
(3.2e-6, 5.0e-5)
3.4e-4
Table 2: Parameters selected arDCA by hyperparamter optimization
We applied identical strategy for InvMSAFold yet found that in many applications the tuned model slightly underperformed the original one we were previously using whose hyperparameter values the reported in Section?
14

---

Published as a conference paper at ICLR 2025
A.3 EXPERIMENTS DETAILS
A.3.1 SECOND ORDER RECONSTRUCTION
The experimental procedure for both the inter and intra cluster datasets can be organized in the following steps:
1. We filtered the structures, keeping only those for which the MSA generated by MMseqs2 has at least 2 sequences. This is because small MSAs tend to give very noisy covariances which could bias the benchmarking.
2. For every sequence the filtered dataset, we generate 10k: synthetic samples for three models. To generate the samples from InvMSAFold-PY
use the library bmDCA, running 10 parallel chains of standard Metropolis- -aattings--Hasiing MCMC algorithm and pooling the results, for ESM-IFI we used the built-in sampler while for InvMSAFold-AR we built our sampler.
3. For ESM-IFI, once generated the samples, we re-aligned them using the full MSA to get fair comparison. Note that, while InyMSAFold-PW and InvMSAFold-AR have seen gaps during training, ESM-IFI was trained on the gapless native sequence. It therefore produces un-aligned sequences. We used the PyHMMER library for alignment tasks.
4. Given the samples, we compute the covariance matrices of the generated samples and of the true MSA. We then compute the Pearson correlation between the flattened versions.
A.3.2 PREDICTED STRUCTURES OF SAMPLED SEQUENCES
First, we report the list of the domains used from the intra/inter-cluster test set in this experiment
intra-cluster: 5i0qB02, 4iulB00, 2wfhA00, 2egcA00, 3oz6B02, 3m8bA01, 2pz0B00, 2cnxA00, 4k7zA02, Id12A00, IvjkA00, ludkA00, 3bs9A00.
inter-cluster: lotjA00, A00y 1p9hA00. 2o0bA02, 1b34B00, 1f6mA02, 2hs5A01, 2bh8A02, 2de6A03, lia6A00, 4ge1A01, 3sobBO2, IxqiA00, 4yt9A01.
Synthetic sequences from both InvMSAFold-AR and InyMSAFold-PW produce very diverse se- quences without further intervention. On the other hand, ESM-IF1 produces sequences whose hamming distance from the native sequence significantly lower. Given that we want different models' ability to recover the native structure from sequences having high dissimilarity from the native one, we leverage the built-in temperature parameter of the model to get such sequences from FSM-IFF, increasing it to sample more diverse sequences.
the
Specifically, Iny and InoMSAFo generate sequences having normalized hamming distance between 0.65 and 0.9. We hence split the interval [0.65, 0.9) into 5 bins of equal width of 0.05.
We the following procedure fill each of these bins with least 10 sequences from ESM-IFI:
1. Generate 1000 samples from ESM-IF1.
2. Fill the bins that still are lacking some sequences.
3. If all the bins are full, return the binned sequences, otherwise increase the temperature of ESM-IF1 by 0.1 and go back to step 1.
For InAFol and InvMSAFold-AR we just generate 10k sequences each for each domain, and then fill the bins relative every domain.
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL PLOTS
B.1 PREDICTED STRUCTURES FOR WNTER-CLUSTER TEST SET
AR
ESMJF1
ESMJIF1
25
InVMSAtCIS-PW
InMMSAtols-PW
8-
RMSD
RMSD
15
0.65
0.75
0.8
0.85
0.65
0.75
0.8
0.85
Distance
Distance
Comparison of the quality of the generated sequences at increasing hamming distance from native one. Shown an average of 14 structures from the inter-cluster test set. then refold the sequence with Alphafold and compare the refolded structure with the original one using RMSD.
Figure 12:
16

---

Published as a conference paper at ICLR 2025
B.2
PCA WITH ALSO PROTEINMPNN
Kamel Estimates
ESMIF1 KL 1.40
mpnn KL: 3.70
InVMSATOIS KL. KL:1.83
1.83
InvMSAtoiG KL. 2.10
M5A EPA-T:
MSA
MSA
pSa
mam
hIMSA CAL
PCA Component?
PCA Component
PCL Component
2
O N
V.
PCA Component 1
PCA Component
PCA Companent
PCA Component 1
Kemel Density Estimatis of PCA Components: PDB 166mAg2
N5A
VSA
KSA
USA
Component
PCA Companent
PCA Con Companent
A. bo
PCA Component
PCA Component
PCA Component 1
PCA Component 1
Kemel Density Estimates PCA Companents: POR lis6AD0 KL:
ESMHF1 KL: 78.29
15. InvMSSAlosS-PU 12.27
InvMSAoot--R KL: 2.07
PCA Component
PCA Companent
PCA Component
PCA Compunent
d.
-10-
-1
PCA Component
Component
PCA Component
Component
Kernel Density Estimates PCA Components: PDB 113WAD4
mpnnn KI:223
InvMSAToid.PP
InoMSAfolSAR...
M5A
19A
POA
PSa
PCA Component
PCA Component
N A
Component
2
PCA
d.
4
-15
0.0 2.5
si
-25
00 25
5.
-25
0/3
25
5.b
-15
0/3 25 5.0
PCA Component
PCA Component
PCA Component
PCA Component
Figure 13: Sampled sequences projected onto the first two PCA components of natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leiher (KI.) divergence between the density of the natural data and the density of the sampled data. The values of these KL.s are written in the title each subplots.
17

---

Published as a conference paper at ICLR 2025
Kernel Density Estimates PCA Components: PDB 1m1sA00
ESM-IF1 - 3.18
3.18
mpan - KL: 1.85
Inw/MAArrd--W KL: 6L:6.116 6.16
InvMSAfold-Ai KL- 1.32
MSA
Component
Component
PCA Companent
PCA
N
PCA Component
PCA Component
PCA Component
PCA Component
Kernel Density Estimates PCA InvMSAtald RL: 19.72
Iny1A00
ESM-IF1 KL: 16:35
mpan KL: 5.98
InvMSAfold-AR KL: 0.35
MSA
MSA
10
10
NSA
Component
J. Componentt
PCA Component
PCA Component
J. o
1.
Component
Component
Component
PCA Component
Kernel Density
Compenents:
ESM-IF1 KL. 25.89
-KL 19.01
InNMSArdiG-PW KL 94
InvMSAfold-AR KL: 0.26
12.5
VIA
12.5
VBA
12.5
VIA
12.5
VEA
10.0
10.0
mamn
20.0
10.0
Comporent
7.5-
7.5
7.5
Camparont
7.5
5.0
5.0
5.0
5.0
2.5
2.5
2.5
2.5
0.0
0.0
0.0
0.0
2.5
2.5
25
2.5
-5.0
-5.0
-7.5
-7.5
10
10
PCA Companent
PCA Component
PCA Component
PCA Component
Kernel Density Estimates of PCA Components: PDB 1viuA00
ESM-IFL - KL: 7.84
mpan - KL: 2.59
Inv KL: 11.29
InvMSAfold-AR KL: 0.31
MSA
E. Component
2. 2omponent
PCA Companent
PCA Campanent
1 I
PCA Component
Compenent
PCA Component
PCA Component
Figure 14: Sampled sequences projected onto the first two PCA components natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leiber (KL) divergence between the density of the natural data and the density of the sampled data. The values these KLs written in the title of each subplots.
18

---

Published as a conference paper at ICLR 2025
Kernel Density Estimates of PCA Components: POB 1vjkADD
ESM-IF1 KL: 1.34
mpnn KL: 1.75 fnuM5Meld-7W KL: 1.41
InvMSAfoldAAR 0.98
ESHIP1
ho AR
Companert
Companent:2
PCA Companent2
PCA Companentt
0
PCA Component
PCA Component
PCA Component
PCA Component 1
Kernel Density Estimates PCA Components: PDB 1x9nA0d
ESM-IF1 - KL: 7.13
mpan - KL: 3.85
InwMSArold-PW KL: 4.33
KL-1.22 InvMSAfold-AR -KL: 1.29
10
10
MSA
10
Inv PW
InviSAfu AR
PCA Component'
6-
2 Component
PCA Companent2
PCA Component
N
PCA Component
PCA Component
PCA Component
PCC Component 1
PCA Components PDB 1xplA00
ESMJF1 KL: 18.28
mpnn
KL: 6.23
InvMSArold PW 9.19.19 9.19
InvMSAfold AR KL: 0.27
HSA
4. L. d. Component E
MSA
to
RS4
B
NSA
PCA Companent
PCA Companent2
PCA Companent
0 N
J.
4 L. d.
4.
4.
Component
Component
PCA Component
PCA Component
KemeiDeraity
Components: 2eacA00
ESM-FF- KL: KL:2.01
KL 0.89
WV KL:0.78 0.78
InvM5Wolcc-R KL: 0.32
MSA
MSA
PSA
MSA
mpn
muMS
Component
PCA Companent
PCA Component
2:
PGA
Component
Component
PCA Component
Component
Figure 15: Sampled sequences projected onto the first two PCA components natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leine (KL) divergence between the density of the natural data and the density of the sampled data. The values of these KLL written in the title of each subplots.
19

---

Published as a conference paper at ICLR 2025
Kerne Density Estimates PCA Components POB 2hs5A01
ESM-IF1 - KL: 0.32
mpnn - KL: 0.23
InwMSSreld-WW KL90
InvMSAfold-AR KL- 0.08
PCA Companent
Component
PCA Component
PCA Companent
2
PCA Component
PCA Component
PCA Component
PCA Component 1
Kernel Estimates
Components:
ESM-IF1 KL: 0.47
mpan KL: 1.00
InwMSArold PW KL: 5.09 5.09
InvMSAfold-RR KL: 0.11
M5A
MSA
Component
Component
PCA Component
PCA Component
1.
1.
1.
1.
Component
Component
Component
Kernel Density
Components:
ESM-IF1 KL. 35.55
KL: 27.75
INMMSArGlO-PW
19.03
InvMSAfold-AR- KL: 0.18
VIA
VEA
YIA
VEA
10.0
10.0
mpm
20.0
10.0
Comporent
7.5
Comparent
7.5
7.5
7.5
2.5-
2.5
2.5
2.5
0.0
0.0
0.0
0.0
-2.5
-25
-5.0
+5.0
-10
6
10
10
-10
PCA Companent
PCA Component
PCA Component
PCA Component
Kernel Density Estimates PCA Components: POB 2wfh400
ESM-IF1 KL- 16.31
mpan KL: 9.72
InwMSArold-W KL: 4.64.65 4.65
InvMSAfold-AR KL: 0.21
MSA
PCA Companent?
Component
PCA Component
PCA Campanent
1. d
b. J.
A.
PCA Component
Compenent
PCA Component
PCA Component
Figure 16: Sampled sequences projected onto the first two PCA components natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leiber (KL) divergence between the density of the natural data and the density of the sampled data. The values these KLs written in the title of each subplots.
20

---

Published as a conference paper at ICLR 2025
Kernel Density Estimates of PCA Components: POB 2x64A02
ESM-IF1 KL: 1668
mpnn KL: 0.54 Inv KL: 12
InvMSMold-AR
ISHIP1
InvMSAAsua
PoYMSAMS AR
Comparent 2
PCA Companent 2
Component
PCA Compoment
2
-2.5
0.0
5.0
-2.5
0.0
2.5
5.0
-2.5
0.0
2.5
5.0
-2.5
0.0
2.5
5.0
PCA Component
PCA Component
PCA Component
PCA Component 1
Kereel Density Estimates PCA Components PDB 3b59A00
ESM-IF1 - KL: 0.96
mpnn KL: 0.74
InvMSArold-PW - KL: 1.06
InvMSAfold-AR KL. 0.12
Inv
Inv AR
PCA Component
Component:
PCA Component
PCA Component
PCA Component
PCA Component
PCA Component
PCA Component 1
Kernel Density Estimates PCA Companents: POB 3cx3A01
ESM-IF1 KL: 5.24
mpnn KL: 1.60
InvMSAfald PW KL: 26.16
InvMSAfold AR KL: 0.26
MSA
Component
Comporant
PCA 2omponent
PCA Companant 2
7.5
5,0
9 2
2,54
Component
Component
PCA Component
PCA Component
Density
PCA Componerts: 3c26302
ESM-IF1 KL- 29.16
17.85
In KL: 1.69
InvMSAfeld-AR KL: 0.47
HSA
MSA
KSA
MSA
Component
4 1. Component
PCA Companent
PCA Companent
2 o
d N
5 4.
Component
Component
Component
Component
Figure 17: Sampled sequences projected onto the first two PCA components of natural sequences for
various PDBs. We also used this density estimate compute the Kullbach-Leiber (KL) divergence between the density of the natural data and the density of the sampled data. The values of these KL. written in the title of each subplots.
21

---

Published as a conference paper at ICLR 2025
Kernel Density Estimates PCA Components: POB 3aww/03
ESM-IF1 - 9.59 9.54
mppn KL: 2L:2.95
InVM5Mold-PW - KL: 11.87
InvMSAfold-AR KL 1.63
Component
Component
PCA Component
PCA Companent
2
L.
1.
4.
4
PCA Component
PCA Component
PCA Component
PCA Component 1
Keree Estimates PCA
3sab802
ESM-IF1 KL: 1.01
mpnn KL: 0.70
InMMSArold PW KL: 0.37
InvMSAfold-AR KL: 0.53
M5A
MSA
PCA Component
Component
PCA Component
PCA
J.
1.
Component
Component
Component
PCC Component
Demsity
Components:
ESM-IF1 KL: 1.92
KL: 2.11
(nwMSArdl--PW KL: 1.86 1.36
InvMSAfoRI-AR KL: 1.20
MCA
MSA
MSA
Companare
N.
Compunent
PCA Companent
PCA Companemt
L.
7.
J
L.
1.
1.
Component
Component
PCA Component
PON Component
Density Estimates KL.
Campanents:
ESM-IF1
73.98
InvMSAfoid. PW
InvMSAfald
15.0
VSA
15.0
VEA mpm
15.0
MIA
15.0
V5A
ESWIF]
12.5
12.5-
12.5-
12.5
Component
20.0
PGA Compunant
10.0
Companent
20.0.
Comporact
10.0-
7.5
7.5
7.5
7.5
5.0
5.0
5.0
5.0
2.5
2.5
2.5
0.0
0.0
0.0
0.0
-2.5
-2.5
-2.5
-2.5
55.0
-5.0
-5.0
-5.0
PCA Companent
PCA Component
PCA Component
Companent
Figure 18: Sampled sequences projected onto the first two PCA components natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leiber (KL) divergence between the density of the natural data and the density of the sampled data. The values these KLs written in the title of each subplots.
22

---

Published as a conference paper at ICLR 2025
Kemel Density Estimates of PCA Components PDB 4k7zA02
ESM-IF1 4.3
mpnn - KL: 1.39 InwMSAFold-W KL: 1.43
InvMSAold-AR KL- 0.08
MSA
Component
Component
PCA Compunent
PCA Companent
PCA Component
PCA Component
PCÀ Component
PCA Component
Kernel
AwecC00
ESM-FF1 KL 16.67
monn
KL: 11.14
InvMSAfald KL: 14.18
InvMSAfold AR KL: 0.20
12
HSA
MSA
12
R44
12.5
NSA
10.
1. Companent E
PCA Companent 5
10.0
Componert
PCA Component
+
2,5-
L
-5.0
Component
Component
Component
Component
Density KL. 2.95
Components: 5b7cA02
ESMHF1 KL: 0.91
InyMSAlol--PW
2.57
InvMSWolld-RR KL:
MSA
VSA
MSA
MSA
mpnn
PCA Component
PCA Componentt
PCA Component
PCA
-2.5
7.5
-2.5
0.0
0.0
5,0
1
PCA Component
PCA Component
PCA Component 1
Figure 19: Sampled sequences projected onto the first two PCA components natural sequences for various PDBs. We also used this density estimate to compute the Kullbach-Leehee (KI.) divergence between the density of the natural data and the density of the sampled data. The values these KLs written in the title of each subplots.
23

---

Published as a conference paper at ICLR 2025
B.3
BIVARIATE PLOTS: THERMOSTABILITY VS SOLUBILITY
1b34B00
Thermophile
1.0
1.0
InVMSAfold-AR
InvMSAfold-PW
ESM-IF1
ESM-F1
0.8
0.8
0.6
0.4
0.4
Mesophile
0.2
0.2
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
1f6mA02
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
0.8-
0.8
0.6
0.6
0.4
Mesophile
0.2
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
1ia6A00
Thermophile
1.0
InvMSAfold-AR ESM-IF1
InwMsAflldPWW ESM-IF1
0.8
0.8
0.6
0.6
0.4
0.4
Mesophile
0.2
0.2-
0.0
0.0
20
40
60
80
100
20
40
60
80
100
Solubility
Solubility
Figure 20: Comparison the distribution of predicted solubility an thermostability of samples generated with InvMSAFold and ESM-IF1. These plots show the results for domain 1ia6A00, 116mA02, Ib34B00.
24

---

Published as a conference paper at ICLR 2025
1ny1A00
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
WrMSAfold-PW ESM-F1
0.8
0.8
0.6
Mesophile
0.2
0.2
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
lotjA00
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
0.8
0.8
0.6
0.6
0.4
0.4
Mesophile
0.2
0.0
20
40
80
20
40
60
80
Solubility
Solubility
1viuA00
Thermophile
1.0
1.0
0.6
0.6
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
0.4
0.4
Mesophil
0.2
0.2
0.0
0.0
40
100
60
80
100
Solubility
Solubility
Figure 21: Comparison the distribution of predicted solubility an thermostability of samples generated with InvMSAFold and ESM-IFF. These plots show the results for domain 1ny mny1A00 A00, lotjA00, vviu000.
25

---

Published as a conference paper at ICLR 2025
1m1sA00
Thermophie
1.0
1.0
0.8
0.8
0.6
InvMSAfald ESM-IF1
Mesophile
0.2
0.2
InVMSAfold-PW
ESM-F1
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
1xqiA00
Thermophile
1.0
0.8
0.8
0.6
0.6
InvMSAfold-AR ESM-IF1
InVMSAfold-PW
0.4
0.4
Mesophile
0.2
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
2hs5A01
Thermophile
1.0
1.0
0.6
0.6
InvMSAfold-AR ESM-IF1
InvMSAfold-Poe PW
ESM-IF1
0.4
0.4
Mesophil
0.2
0.2
0.0
0.0
80
100
Solubility
Solubility
Figure 22: Comparison the distribution of predicted solubility an thermostability samples generated with InvMSAFold and ESM-IFF. These plots show the results for domain 1m1sA00, IxqiA00, 2hs5A01.
26

---

Published as a conference paper at ICLR 2025
2m5jA00
Thermophie
1.0
1.0
0.8
0.8
0.6
InvMSAfold ESM-IF1
Mesophile
0.2
0.2
InVMSAfold-PW
ESM-F1
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
2x64A02
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
0.8
0.8
0.6
0.6
0.4
0.4
Mesophile
0.2
0.0
60
80
100
20
40
60
80
100
Solubility
Solubility
3cx3A01
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
InVMSAfold-PW ESM-F11
0.8
0.6
0.6
0.4-
0.4
Mesophie
0.2
0.2
0.0
0.0
40
100
100
Solubility
Solubility
Figure 23: Comparison the distribution of predicted solubility an thermostability of samples generated with InvMSAFold and ESM-IF1. These plots show the results for domain 2m5jA00, 2x64A02, 3cx3A01.
27

---

Published as a conference paper at ICLR 2025
3qwwA03
Thermophie
1.0
1.0
0.8
0.8
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
Mesophile
0.2
0.2
0.0
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
3sobB02
Thermophile
1.0
1.0
InvMSAfold-AR ESM-IF1
0.8-
0.8
0.6
0.6
0.4
0.4
Mesophile
0.2
InVMSAfold-PW
ESM-IF1
0.0
40
60
80
100
40
60
80
100
Solubility
Solubility
4atkB00
Thermophile
1.0
InvMSAfold-AR ESM-IF1
InvMSAfold-PW ESM-IF1
0.8
0.8
0.6
0.6
0.4
0.4
Mesophile
0.2
0.2
0.0
0.0
80
100
40
60
Solubility
Solubility
Figure 24: Comparison the distribution of predicted solubility an thermostability samples generated with InvMSAFold and ESM-IFI. These plots show the results for domain 3qwwA03, 3sobB02, 4atkB00.
28


Reviews:
[{"review_id": "kAaxgCBidR", "reviewer": "Reviewer_Wqzy", "summary": "InvMSAFold is an inverse folding method that is optimized for diversity and speed. The general idea is to use a neural net to predict from an input structure and sequence a pairwise interaction model (a Potts model or Boltzmann machine) that captures the structure-sequence relationship and can be used to efficiently generate sequences that differ largely from the input sequence. To tame the number of parameters (fields and pairwise couplings), InvMSAFold predicts a low-rank approximation of the coupling matrix. The paper proposes two models: 1) InvMSAFold-PW is a full pairwise model that reduces the number of parameters significantly and also allows for efficient learning by using a maximum pseudo-likelihood. A drawback is that sequence generation requires MCMC. 2) InvMSAFold-AR is an autoregressive model whose likelihood is tractable thereby allowing for Maximum-likelihood parameter estimation as well as sampling of sequences in a straight forward fashion. Using various metrics the authors show that InvMSAFold, and in particular InvMSAFold-AR, outperforms current state of the art.", "questions": "* How is sampling of InvMSAFold-PW achieved? Which MCMC algorithm do you use? \n* By using a PCA projection you show that sequences generated by InvFoldMSA have a better coverage of sequence space. But why do you restrict the analysis to the first two principal components? \n* Have you tried AlphaFold3 to validate the sequences generated by InvFoldMSA?\n\n__Typos / grammar__\n\n* Line 117: \"and whos outputs\"\n* Line 212: \"can be reduce to\"\n* Line 248: \"robsutly\"\n* Line 277: \"chose\" - should be present tense\n* Line 302/303: What do you mean by \"consistent with the hardness reasoning behind the split\"\n* Line 475: \"becoming worse that both\"\n* The use of the symbol $\\\\propto$ to indicate quality up to an additive constant is a bit unusual.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* An interesting idea to approach the inverse folding problem (i.e. the problem of generating sequences that fold into a given structure). \n* Proposes a low-rank approximation of the couplings and fields of the lightweight sequence model. \n* Fast generation of sequences that fit a well to a given structure.", "weaknesses": "* The idea of generating a Potts model has already been proposed by Li et al. (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yltIIp6Ngc", "reviewer": "Reviewer_ynEz", "summary": "The paper presents InvMSAFold, an inverse folding model that generates the parameters of a probability distribution over the space of protein sequences with pairwise interwise interactions, allowing for efficient generation of diverse protein sequences while preserving structural and functional integrity. InvMSAFold is a neural network in which the inputs are the structure backbone coordinates X and the outputs are the parameters of a lightweight sequence model. The lightweight sequence model parameters are used to sample amino acid sequences compatible with the input structure. Training is based on the CATH database, which classifies protein domains into superfamilies and further into clusters based on sequence homology. The model is fast and has uses in protein design and virtual screening. Biologically, the model captures amino acid covariances observed in Multiple Sequence Alignments (MSA) of homologous proteins. The model expands the scope of inverse folding to retrieve a landscape of homologous proteins with similar folds (they say the 'entire' landscape, I don't think they have shown this).  I am overall very enthusiastic about this work.", "questions": "I was not clear on the InvMSAFold-AR/-PW. I understand that PW requires MCMC sampling and AR does not but I wonder are there cases/tasks in which one of the two PW/AR models is more appropriate? \n\nWhat would be an example in which you could demonstrate preserved functional integrity that is not directly related to structural integrity in your model's generation of diverse protein sequences?  It seems an important question because when you want to design a protein to do some specific function (bind some small molecule or interact with another protein) you only care about structure to the extent that it acts as a proxy for function. But maybe it doesn't have to be? Do you think your models could get at function outside the restraint of the specific structure that is your input?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The sampling speed of InvMSAFold is a lot faster than ESM-1F or ProteinMPNN, this is important when you want to generate millions of models, as I think could be reasonable for virtual screening/protein design applications. \n\nInvMSAFold seems able to sample more diverse regions of potential protein structure/function space than ESM-1F, again this is important when you are trying to select for particular properties (substrate specificity, thermostability).\n\nThat InvMSAFold is able to capture residue covariances in MSAs may also be useful for better backbone modeling that particular functions could then be engineered into.", "weaknesses": "There is not a specific example taken through to the conclusion that the model preserves \"structural and functional integrity\".  Functional integrity is what you want when you're designing new proteins/doing virtual screening. The authors should consider including such an example or clarifying this statement since that is a major claim of their paper. \n\nI was not clear on the InvMSAFold-AR/-PW. I understand that PW requires MCMC and AR does not but I wonder are there cases/tasks in which a PW vs AR model is more appropriate?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "67ZHwo9rCs", "reviewer": "Reviewer_92iF", "summary": "This paper proposes a neural network called InvMSAFold, which takes the protein structure as input, and outputs the parameters of two statistical models. These models are then used to generate a diverse set of protein sequences corresponding to the input structure. By utilizing these simple statistical models, the proposed pipeline effectively addresses two major challenges faced by other inverse-folding methods, such as ESM-IF: (1) the limited diversity of generated sequences and (2) slow sampling speed.", "questions": "1.\tThe symbols of Eq.(5) is not consistent with that of Eq.(3). It would be better to use consistent symbols.\n2.\tThe proof in section 2.2.1 is incoherent. What is the function of Eq.(5)?\n3.\tIn Eq.(7). It would be better to clarify that Eq.(7) is the L2 regularization term. \n4.\tIn section 3, it would be better to list the number of entries in each dataset.\n5.\tIn section 4.1, what is the necessity of tuning the hyper-parameters of InvMSAFold-AR?\n6.\tIt seems that InvMSAFold-PW performs better than InvMSAFold-AR at larger hamming distance. What is the probable cause?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In their computational experiments, the authors demonstrated that the sequences generated by their models not only fold into the target structure but also exhibit greater diversity and more effectively capture the correlations between residues at different sites. Furthermore, the showed that this sequence diversity extends to other properties, such as predicted solubility and predicted and predicted thermostability. Overall, this paper represents a new methodological advancement.", "weaknesses": "1.\tThe authors only compare their method with ESM-IF1, and do not compare their method with other state-of-the-art inverse folding methods.\n2.\tIn many places such as in section 1, \"ESM-IF\" was wrongly typed as \"ESM-1F\". This may lead readers to perceive the authors as lacking expertise.\n3.\tThe article contains too many grammatical errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81zn2wwNAF", "reviewer": "Reviewer_D2uA", "summary": "In this paper, the authors present an efficient method for designing protein backbones using a neural network that predicts a Potts model. The proposed architecture includes a pre-trained ESM1-IF encoder that encodes the protein backbone, generating rotation-invariant embeddings. These embeddings are then passed through a transformer-based decoder, which produces a low-rank matrix that is ultimately used to compute the fields and couplings. The low-rank approximation is a clever technique that helps mitigate the quadratic scaling cost typically associated with such computations. The neural network was trained using two distinct approaches: (1) a standard pseudo-likelihood loss, and (2) autoregressive sampling (over amino acids) with maximum likelihood training. To avoid training on single sequences, the model was trained on multiple sequence alignments (MSAs), with the mean pseudo-negative log-likelihood calculated over randomly sampled subsets of the MSA. Training and testing data were sourced from the CATH database, following its hierarchical classification to create test sets of varying difficulty, depending on the similarity between the training and test data. The authors demonstrate that their model better reconstructs covariance matrices compared to ESM1-IF, based on Pearson correlations. Moreover, the authors show that projected MSAs using PCA more closely reflect the natural sequence distributions, suggesting that their generated sequences, or predicted MSAs, are more diverse. When refolding designed sequences for test set structures, the InvMSAfold method proves to be more robust than ESM1-IF for sequences that deviate further from the native structure, and comparable to ESM1-IF for sequences that are highly similar to the native.\nIn conclusion, the paper demonstrates how a Potts model can be efficiently constructed, showing that the resulting model generates sequences that are plausible, diverse, refold successfully with AlphaFold2, and possess other promising biochemical attributes.", "questions": "I'd like to raise a few major points:\n* It would strengthen the paper to benchmark against other methods as well. I would suggest for example a simple Potts model without the low-rank approximation and without the pre-trained ESM1-IF encoder, and an additional method such as ProteinMPNN beyond ESM1-IF. This would highlight the contributions of the paper more clearly, as currently, it may seem somewhat reliant on ESM1-IF.\n* I highly recommend adding a plot that shows RMSD versus sequence recovery, as these metrics would provide valuable insights into the model’s performance.\n* In Section 2.2.1, the explanation of Equation 7 and how it maintains linear scaling isn’t entirely obvious, at least to me. I suggest elaborating on this either within the main text or in the supplementary material to clarify the reasoning. It would be helpful to include 1-2 sentences explaining why the method or process is linear and how this linearity is established. This will provide clarity to the reader and strengthen the argument by highlighting the underlying reasoning behind the concept.\n* To make the manuscript even stronger, it would be useful to include (1) an analysis of how the method scales with very large sequences or structures, and (2) a discussion of how the size of the MSA impacts model performance.\n\nA minor point:\n* In Section 2.2, I recommend including the formula that shows the normalization constant, as it is referenced in the text but not explicitly provided.\n\nThere are several typos throughout the manuscript that disrupt the flow. I have listed the ones I noticed while reading, but I recommend a re-read of the manuscript to specifically check for additional typos:\n   - Line 17: The phrase “space of sequences with pairwise interwise interactions, capturing the amino acid…” contains the term “interwise,” which doesn’t seem correct or clear.\n   - Lines 107-108: The word \"Moreover\" is used consecutively, which disrupts the flow.\n   - Line 117: The word \"Whos\" should be corrected to \"Whose.\"\n   - Line 299: \"We monitor the the negative...\"—\"the\" is repeated.\n   - Line 315: \"A can be seen...\" should likely be \"As can be seen...\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The authors present a novel and elegant approach to optimizing Potts model construction, training and sampling. The paper is well-structured, clearly outlining each crucial part of the methodology in a way that is easy to follow. The method is compared and benchmarked against a well-established approach, and performance metrics computed and reported.", "weaknesses": "While the proposed methodology for improving the efficiency of Potts model construction is promising, there are a few areas where the paper could be strengthened. First, Potts models have long been used in fixed-backbone protein design, which makes it difficult to clearly identify the novelty and specific contributions of this work. Additionally, the method relies on components of ESM1-IF and then benchmarks against this model, which may limit the fairness or objectivity of the comparison. Another area for improvement is scalability. The paper does not provide any analysis on how the model handles large structures or long sequences, which could be useful for evaluating its broader applicability. Furthermore, there is no discussion on the significance of using MSAs for training versus single-sequence training, nor is there any exploration of how deep the MSAs need to be if they are indeed important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)