MiniFold: Simple, Fast, and Accurate Protein Structure Prediction
Authors: Jeremy Wohlwend
Mateo Reveiz
Matt McPartlon
Axel Feldmann
Wengong Jin
Regina Barzilay
Venue: Accepted by TMLR ()

Abstract: Protein structure prediction has emerged as a powerful tool for biologists and drug makers. However, the computational cost of state-of-the-art models such as AlphaFold limits their scalability and makes training and fine-tuning prohibitively expensive. Although previous work has achieved considerable inference speedups by replacing the multiple sequence alignment step with protein language models, the overall architecture of structure prediction models, inherited from AlphaFold2, has remained largely unchanged. In this work, we show that protein language model-based structure predictors can be dramatically simplified at little to no loss in accuracy. Our model, MiniFold, consists of a redesigned Evoformer and a lightweight structure module. We also propose two novel GPU kernels, tailored to the proposed architecture. Equipped with the same ESM2 protein language model, MiniFold is competitive with ESMFold on the standard CAMEO and CASP datasets while achieving training and inference speedups of up to 20x, and significant reductions in peak memory. Our results show that MiniFold is an effective solution for large-scale applications and resource-constrained environments.

Published in Transactions on Machine Learning Research (04/2025)
MiniFold:
Simple, Fast, and Accurate Protein Structure Prediction
Jeremy Wohlwend"
juohkwend csoil. mit. edu
MIT
Mateo Reveiz"
mreveiz edu
MIT
Matt McPartlon Universily of Chicago
mmnppatton@uchicngg.o
Axel Feldmann MTT
axelfficsoil mát edu
Wengong Jin
wengong@csail.mian edu
Broad Institute of MIT and Horvard
Regina Barzilay
regina Aanoal mät.ealllllm..a.o.mmmmmmmmmm edu
MIT
Reviewed on OpenReview: https: // openreview. net/ forum? id- 1y9h0To
Abstract
Protein structure prediction has emerged powerful tool for biologists and drug mak- ers. However, the computational cost of seate-of-the-art models such as AlphaFold limits their scalability and makes training and fine-tuning prohibitively expensive. Although pre- vious work has achieved considerable inference speedups by replacing the multiple sequence aligument step with protein language models, the overall architecture of structure predic- tion models, inherited from AlphaFold2, has remained largely unchanged. In this work, we show that protein language model-based structure predictors can be dramatically simplified at little to no loss in accuracy. Our model, MiniFold, consists of redesigned Evoformer and lightweight structure module. We also propose two novel GPU kernels, tailored to the proposed archilecture. Equipped with the same ESM2 protein language model, Mini- Fold is competitive with ESMFold on the standard CAMEO and CASP datasets while achicving training and inforence speedups of up to 20x, and significant reductions in peak memory. Our results show that MiniFold an effective solution for large-scale applica- tions and resource-coonstrainel environments. Our code and trained models are available at httpss//gghhbbbbcc///oonee//nnnn
1 Introduction
With advances in deep learning-based protein structure prediction, models such as Alphafold Jumper et al. 2202)) are now routinely used in biological discovery. These models take as input a protein sequence and predict the 3-D coordinates of every atom in the protein. Yet, the significant computational cost associated with these tools limits their use large scale applications. As an illustration, consider the task of identifying discaso-rolevant antibodics from patients' blood. A sequenced antibody repertoire may contain tens of Joint first authors

---

Published in Transactions on Machine Learning Research (04/2025)
millions of unique clonotypes (i.e unique protein sequences) Briney et al (2019) At speed of just few seconds per sequence, scanning 10 millions sequences would require over a GPU year. In addition to virtual screening, other applications, such as mutational scans, downstream fine-tuning, and the filtering of de novo designs, where ESMFold is broadly utilized [Cmmpeell et al. (2024;; Watson et al. (2023). would all benefit from more efficient models. Importantly, efficient architectures can also accelerate the research life cycle in a space that typically requires extensive computational resources.
Previous works have already recognized this challenge. ESMFold Lin et al. (2023) and OmegaFold Wu et al. (2022) propose to replace the compute-heavy multiple sequence alignment (MSA) stage of Alpha.oold with protein language model (PLM), while conserving most of the original folding trunk design. By bypassing MSA generation, the time required for structure prediction can be reduced from minutes to seconds per sequence. However, for the various use cases outlined above, the cost and throughput of high-quality PLM- basod folding models remain severely limiting, not to mention the largely unchanged cost of model training. We hypothesizod that further improvements these models could be achievod by addressing the bottlenecks inherited from AlphaFold. With the ESMFold architecture as starting point, our goal was to create more efficient model by pinpointing critical performance components and reducing computational bottlenecks.
Despite its considerable number of parameters, the protein language model used in ESMFold only covers a small percentage of the inference time (< 5%)- Instead, the bulk of the compute is spent in the Evoformer blocks (~ 80%). dominated by triangular operations, and to a lesser degree in the structure module (~ 15%), dominated by the invariant point attention (IPA). Our goal, therefore, is to design an efficient protein struc- ture prediction model that addresses these two bottlenecks. Of note, the concurrently published AlphaFold3 and Boltz-1 models retain most of the original Evoformer design Abramson et al. (2024); Wohlwend et al. (0044) making optimization of this architecture highly desirable.
We present MiniFold, an efficient architecture for protein structure prediction which reduces an Evoformer block to a single bidirectional triangular multiplicative operation and feed-forward layers, and replaces the structure module with a lightweight transformer with pairwise bias. In addition, we introduce a hardware- optimized implermentation using newly devised GPU kernels to enhance both the throughput. and memory efficiency of the model. Specifically, our proposed kernel optimizations are applied to the triangular multi- plicative updates and to the food-forward layers in the revised Evoformer (Miniformer). We train MiniFold using the same PLM used in ESMFold on single node of 8x A100 GPU's, and achieve competitive perfor- mance on the standard CAMEO and CASP test sets Kryshtaforych et al. (2021) 2023). When compared against the original architecture, MiniFold achieves 20x speedup and a significant reduction in peak memory utilization. Our work further highlights the potential for PLMs to not only accelerate homology search but also simplify downstream architectures, thereby enabling high-throughput protein structure prediction.
2 Related Works
Protein structure prediction The field of protein structure prediction has seen significant advancements due to recent breakthroughs in deep learning methods. The current state-of-the-ae model AlphaFold (Jupper
et al. 2021 Abramson et al. 2024h consists of two main components: multiple sequence alignment (MSA) and template search, and a folding module responsible for decoding the MSA into a set of coordinates in 3- dimensional space. The derived MSA and templates are used to construct an initial set of sequence-level and pairwise-leved representations. These are then fed to the folding module, more specifically to a pairwise track which learns to predict the pairwise distances between amino acid residues as a distogram. Finally, a structure module uses these pairwise embeddings prediet the 3D coordinates of each residue. RosettaFFold [Bakk al. (2021) and UniFold Li et al. (2022) adopt a similar MSA-based workflow but with different architectures. An important limitation of AlphaFold2, however, is its computational cost. Recently, protein folding models, such as OmegaFold Wu et al. (0022) and ESMFold Lin et al. (2023). accelcrate inference by replacing the
MSA database search with protein language models Rives et al. (2021) Meier et al. (2021); Lin el al. (2023): Elnaggar et al.) (2023); : Chen et al. (2023). The input to these models requires only single sequences, which is idcal for orphan proteins that do not have MSA. In contrast from PLM-based models that keep the same architecture as AlphaFold2, we focus on improving the efficiency of the structure prediction model to further
improve its scalability. Concurrent to this work, the new AlphaFold3 and Boltz-1 models retain most. of
2

---

Published in Transactions on Machine Learning Research (04/2025)
Embedding Module
Miniformer (4B blocks)
Attention maps
ESM2 Protein Lang Model
Pair representation V/.C]
Bi directional Triangular
Muttiplicative Update
Transftonn
Sequence embedidings (r.
Sequence to Pair
Recycling
Pair representation (nric)
Painvise Blased Attentoon
Transftion
Sequance embeddings
Realizer Module blocks)
Our proposed MiniFold architecture. We use the ESM-2 protein language model to extract sequence level embeddings as well pairwise attention maps. These are concatenated and fed to the Miniformer module which updates the pairwise embeddings using efficient triangular multiplicative update operations and feed-forward layers. Finally, structure realizer composed of gated transformer produces the full-atom coordinates from the pairwise and sequence embeddings.
Figure 1:
the original Evoformer, both in the MSA module and the similar Pairformer Abramson al. (2024) Wohlwend et al. (2024), which remains very relevant to work.
Efficiency & scalability Strategies improve the efficiency of neural networks typically fall under one of four categorics: pruning, quantization, GPU kernel optimization, and improvements in algorithmic plexity. Each of these approaches have been extensively studied in the context of the Transformer architec- ture Vaswani et al. (2017]. One such example is the recently proposed Flash-attention Dao et al. (2022) which provides major speedup and memory savings to attention based architecture using an TO-aware im- plementation. Another example is Simple Recurrent Unit Lei (2021), which uses various GPU kernel opti- mization techniques for acceleration. Katharopoulos el al. (2020): Wang al. (2020) propose algorithrnic alterations that result in linear time complexity as function of the input size. Yet, these methods are not obviously applicable to protein folding models, which utilize unique neural layers and transformations. Recognizing the importance of customized solutions to the model architecture ure at hand, our work aims to optimize protein structure prediction models to accelerate their inference speed, and improve their capacity to scale to long protein sequences. While recent work Cheng et al. (2022); Wang et al. (2022) has begun to explore hardware-lovel optimization techniques to accelerate protein folding training. these are limited by the current architecture. Chowdhury al. (2022) propose faster architecture in RGN2 but require refinement through Alphafold2 to reach competitive performance, which negates the potential speod-up. In contrast, our work focuses on both architecture-leeel and custom hardware-level optimizations to improve speed and memory utilization while retaining accuracy.
3 Methods
Our MiniFold architecture composed of three modules: sequence embedding module, an efficient Evo- former which name Miniformer, and lightweight structure module based on gated self self-attention. Figure illustrates the forward pass of model. First, we encode the input protein sequence using the three-billion
3

---

Published in Transactions on Machine Learning Research (04/2025)
Triangular Update
Transition
Pair representation (n/d
Gate (c,
Bi-directional
Gate (o/2, c)
Linear (c, 4c)
Linear
Pair representation (n/d)
Norm
Linear (c1, c2)
Norm
Chunk
Linear (c1, c2)
Sigmoid
Input/ / Output Gate
Bi-directional malmul
Figure Our proposed Miniformer block. The input is a pairwise tensor of dimension (r,r, c). We first compute the input gate displayed in the bottom left panel. In the bi-directional matmul, we first chunk the tensor along the channel dimension to create 4 sub-tensors of shape rr,r,//4), and perform pair of tensor products, once over the vanilla tensors and once over their transpose. The two resulting tensors of shape (r,r, c/4) are then concatenated and fed to the output gate. This is followed by skip connection and a two layer food-forward network with a ReLU activation.
parameter ESM-2 language model Lin et al. (2023), which is pre-trained 36 layer Transformer. The embeddings at the last layer of ESM-2 are fed into a small feed-forward network and tiled into pairwise roprosentations such that every entry (i,j) is the concatenation of the vector representations of residues and j. In addition, we also feed the concatenation of all the network's attention maps, and fine-tune the last 2 layers of the protein language model, both of which accelerate convergence during the early stages of training. This pair representation is then updated through 12 or 48 Miniformer blocks, where we propose the use of a redesigned bi-directional triangular multiplicative layer. Finally, the output of the Miniformer and the sequence embeddings from the protein language model are passed to a structure realizer which produces the 3D coordinates.
contrast to previous work, we apply recycling over the pairwise blocks only. The efficiency of our approach comes from the redesigned Miniformer blocks and the lighter structure realizer. In the following sections, we provide a detailed overview of these design choices and their motivation.
3.1 From Evoformer Miniformer
We propose several modifications to the Evoformer architecture. Our new design is shown in Algorithm I First, we eliminate the sequence track and keep only the pairwise representation and update. While this change does have a substantial impact on speed, dramatically reduces the number of paramoters in the Evoformer. Specifically, the sequence track in ESMFold holds over 600M parameters. When removed, less than 25M parameters remain. Through ablating this change, argue that the representational capacity of the Evoformer is influenced by the depth and complexity of its operations rather than by its parameter count.
Next, we eliminate the Triangular Attention blocks. There are two reasons for this change. First, this operation produces attention maps that encode every node triplet, which results in substantial memory consumption that scales poorly with sequence length. Secondly, early in our experiments, we found that the expressive power of the Evoformer was driven by the triangular multiplicative blocks and not the triangular
4

---

Published in Transactions on Machine Learning Research (04/2025)
Pair-bissed Gated Transformer (8 blocks)
Sequancc umbeddings V,
Norm
Linear
Prinwis B soed Aftention
Transftion
Secusrico bmbeekinng (t.6)
Linear
Gram-
representation
(1//5)
[c, b
Transition
Angles
Figure Our proposed transformer-based realizer. Pairwise embeddings are projected to an (r,r, h) lensor, where b is the number of blocks and h the number of attention heads. We compute 8 blocks of biased transformer layers, and the final sequence embeddings used to predict frame at each residue via Gram-Schmidt and the angles via the Angle Resnet described in AlphaFold2.
altention. This is ideal because although the complexity of the multiplicative update is also cubic sequence length, the space complexity remains quadratic, resulting in a much cheaper operation in practice. Another potential advantage of this change would be to scale up other parts of the model, such as the pairwise embedding dimension, the number of blocks. We leave this exploration for future work.
Finally, propose redesigned triangular multiplicative layer. We merge the incoming and outgoing blocks, fusing the point operations across the two blocks and removing one of the skip connections. We use down- projection of the input dimension prior to the channel-wise matrix multiplicatiin. We also modify the output gating to be a function of the matrix multiplication output instead a function of the layer's input. This final change unifies the input and output gating operations and allows better GPU kernel design, as explained section 2.3. We refer to this layer as bi-directional triangular update, which we combine with feed-forward network in each Miniformer block as shown in Algorithm
3.2 From Structure Module to Realizer
The structure module in AlphaFold2 takes input the pairwise representation from the Evoformer and out- puts the full atom 3D coordinates. We sought to investigate the role of the structure module, hypothesizing that most of the predictive capacity of the model already occurred in the Evoformer. In particular, for the large scale applications mentioned such as virtual screening, it may not be necessary to compute the 3D coordinates. Pairwise distances may sufficient to identify stable folds, and discriminate for binders. Yet, in order substantially alter or omit the structure module, we would first need to ensure that role is primarily realize the 3D coordinates that match the predicted distogram. Motivated by this hypothesis, hereby propose alternatives for the structure module: parameter-free multidimensional scaling realizer (MDS) Mead 1992), and fast transformer realizer.
Parameter-free MDS realizer In order validate our hypothesis we first attempted to directly obtain the coordinates from the predicted distogram, using MDS. To do so, first determine the predicted distance for each pair of residues by choosing the distance bin with the highest probability. Since the maximum distance bin is 25A, we fill the missing entries (i.e., pairs predicted to be farther than 25 by defining graph with nodes as Ca atoms and edges as distances under 25A and run the all-pair shortest path algorithm on this graph. This results in an approximate distance for missing entries. Given the approximated full distance matrix, we then perform classical MDS to generate initial coordinates for the atoms. Because MDS is sensitive noise, the resulting coordinates generally do satisfy the predicted distogram. Therefore, we refine the coordinates through 10 iterations of stress majorization with LBFGS optimizer. We provide pseudo-code for this approach in algorithm In the absence of structure module, also propose

---

Published in Transactions on Machine Learning Research (04/2025)
change to the recycling strategy by directly recycling the argmax of the distogram to the Miniformer. More specifically, convert the maximum likelihood distance bin into one-hot representation which we linearly project and sum into the input to the Miniformer blocks. Only Ca atoms are considered in this setting.*
Tee-beere realizer While the MDS approach serves as a strong indicator of the role of the structure module, the cost of rumning shortest path, MDS and gradient descent is substantial. This is not necessarily an issue in a setting where pairwise distances are sufficient but can be a limitation when the 3D coordinates must be computed. We therefore thought to replace it by learning to perform MDS using a neural network that is similar but more efficient than the existing IPA-based structure module. To achieve this, we propose a transformer with biased attention, similar to what is used in the sequence track in the original Evoformer 2 Starting with the initial pairwise representation and the sequence embeddings s, we first project 2 into a set of attention biases, with one bias matrix per transformer head. This bias is added to the attention scores prior to the softmax. In contrast to the IPA, we do not iteratively update coordinates in each block, but. instead predict the 3D frames of each residue in one-shot after the final layer. To do so, we found it helpful to construct the frame by predicting the coordinates of the N, Ca and C atoms and applying Gram-Schmidt, instead of directly predicting a translation and quaternion at each position. In line with casting the structure module as realizer, the idea of only applying recycling in the Miniformer blocks was kept to further improve training and inference speed. We predict side-chain atoms using 5 torsion angles. All angles are predicted by their cosine and sine contributions as per Alphafold2, using feed-forward layers on top of the sequence representation of the transformer.
3.3 Efficient GPU Inference Kernels
Recent work has shown that hardware-opuinizeed implementations can yield substantial speedups by fusing memory intensive operations into single GPU kernel. Motivated by this success, we identified two promising areas of improvement in the Miniformer architecture: the self-gating operation in the multiplicative update, and the feed-forward layers where the hidden projection quadruples the input dimension. The main idea is to fuse operations to avoid excessive memory transfers. We implement these new kernels using the Triton library Tillet et al. (2019), which provides block-level control over memory transfers while abstracting away the details of issuing tensor core operations.
Self-gating The following gating operation appears twice in each triangular update:
y (Vz+bi) sigmoid Wz + b2)
Because all of these operations are applied entry-wise, and z is loaded twice, speedup and memory savings could be obtained by fusing this operation in single kernel. The key insight is that the hidden dimension of = is small enough (i.e 128) that the GPU SRAM can partition across its large inpul dimension only. As such, we can reduce the number of reads and the mumber of intermediate writes saving both memory and computing time. We provide pseudo-code in Algorithm in the appendix.
Feed-forward The feed-forward layer is ubiquitous in model deep learning architectures. It consists of an initial linear projection, followed by a non-linear activation (in our case a ReLU), and another linear projection. This layer is an essential part of the Transformer architecture Vaswani et al. (2017). as well as the folding blocks described above. This layer typically uses a hidden dimension that is larger than the input and output dimensions. In case, it is 4x the input dimension. Our goal here is to avoid matcrializing the inner malrix, which can causes substantial memory bottleneck:
= (W2ReLU(W12 + b,) + b2)
Here, again we observe that GPU SRAM is sufficiently large to partition z across its large input dimension. Each thread block loads several complete rows of into SRAM and is responsible for producing several
A similar MDS approach was independently developed by Sergey Ovehinnikov and presented talk during CASP14.
6

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset
Model
TMscore
RMSD
IDDT
RMSD-aa
IDDT-aa
Runtime (s/seq)
CAMEO
MiniFold-12
0.825
3.220
0.837
3.946
0.764
0.359
MiniFold-48
0.846
2.781
0.859
3.479
0.786
0.807
ESMFold
0.837
2.888
0.856
3.611
0.784
11.402
OmegaFold
0.815
3.382
0.837
4.111
0.767
61.507
AlphaFold2
0.880
2.100
0.905
2.763
0.828
77.842
RosettaFold1
0.761
4.659
0.798
4.661
0.795
227.368
RosettaFold2
0.883
1.987
0.906
2.666
0.823
14.269
CASP14
MiniFold-12
0.616
7.126
0.663
7.827
0.608
0.531
MiniFold-48
0.641
6.452
0.692
7.154
0.636
1.390
ESMFold
0.656
6.100
0.699
6.850
0.638
12.844
OmegaFold
0.706
5.366
0.737
6.186
0.673
149.303
AlphaFold2
0.831
3.002
0.849
3.698
0.772
105.374
Rolet
0.634
8.174
0.683
8.145
0.682
310.294
RosettaFold2
0.798
3.489
0.826
4.307
0.746
18.176
CASP15
MiniFold-12
0.617
9.424
0.703
10.393
0.650
0.510
MiniFold-48
0.657
8.174
0.736
9.035
0.679
1.312
ESMFoold
0.659
8.172
0.755
9.116
0.688
10.484
OmegaFold
0.631
10.080
0.731
11.271
0.667
128.875
AlphaFold2
0.728
5.754
0.819
6.740
0.748
107.173
RosettaFold1
0,588
11.378
0.683
11.362
0.711
321.987
RosdttaFold2 0.706
6.441
0.798
7.589
0.727
19.786
Table 1: Performance of MiniFold (12 and 48 layers) and baselines on the CAMEO, CASP14 and CASP15 datasets. We report the TMScore, RMSD, IDDT, RMSD All-atoms (RMSD-aa) and IDDT All-atoms (IDDT- aa). Each metric computed by taking the mean across the targets for each dataset, except for the RMSD and RMSD All-atoms metrics where is provided for each model and dataset. Runtimes exclude the generation of multiple sequence alignments (MSA) for MSA-bases models.
the geometric mean. The average runtime in seconds / sequence
complete rows of y. This means that every thread block can loop over the expanded dimension WI without matorializing any intermediates HBM. Instcad we load once and proceed loop over the output dimension of W1 and accumulate the partial matrix multiplication of Wi into the output y. This strategy results in memory usage that is only function of the input and output, but not the inner representation. We provide pseudo-code in Algorithm 5 in the appendix.
We observe that both kernels always preceded by Layer normalization. To further reduce intermediate data transfers, fuse the layer norm into the respective kernels. Unless stated otherwise, our kernel results include the fusing of the layer norm.
Layer-norm
3.4 Training objective
AlphaFold and ESMFold were trained using a combination multiple objective functions including distogram loss, FAPE loss, structural violation loss, confidence loss, etc. Similar to previous work, we construct equally sized bins ranging from 2 to 25 Angstroms, and train the model to predict the pairwise distances between each pair of residues, using a classification objective over the bins described above, with the common cross entropy loss. We initially train using only this objective, and without enabling recyling or the sturcture realizer. After this initial training stage of the Miniformer, we begin a second training stage where we enable both the structure realizer and recycling. The model is supervised using the backbone and side-chain FAPE losses, and the supervised chi losses as previously done in Alphafold2. We also train a pLDDT predictor on top of the structure realizer with the same loss used in AlphaFold2. The model is trained for about 250K steps during each training stage.

---

Published in Transactions on Machine Learning Research (04/2025)
Dataset
MiniFold-12
MiniFold-12 MDS
MiniFold-48
MiniFold-48 MDS
CAMEO
0.88 / 0.14
0.88 / 0.15
0.9 / 0.13
0.9 / 0.14
CASP14
0.73 / 0.22
0.74 / 0.24
0.77 / 0.22
0.78 / 0.25
CASP15
0.8 / 0.23
0.8 / 0.24
0.84 / 0.23
0.84 / 0.23
CAMEO
2.75 / 7.63
2.88 / 22.05
2.34 / 7.2
2.42 / 21.83
CASP14
6.24 / 13.75
11.34 / 23.31
6.19 / 13.6
8.22
21.98
CASP15
6.9 / 32.31
11.05 54.99
6.73 / 33.06
8.11 / 40.74
Table 2: Ca IDDT (top) and Co RMSD (bottom) comparison between the lightweight structure module and the MDS realizer on CAMEO, CASP14 and CASP15 datasets, for MiniFold with 12 and 48 miniFormer blocks. Values correspond to median/sample standard deviation (n-1 normalization).
4 Experiments
We showcase the elficacy of our approach by comparing the structure prediction accuracy, inference speed, and memory consumption
MiniFold against ESMFold Table [ We also provide comparisons against Omegafold, and several MSA-basod models: AlphaFold2, RoscttaFold1, RosottaFold2 Jumper al. (2021); Baek al. (2021); |; Lin et al. (2023); Wu et al. (2022). All models were run with 3 recycling rounds (for total of 4 forward passes), and the same MSA precomputed using the ColabFold service used for all MSA-based models Mirdita et al. (2022). Our detailed experimental setting can be found in the Appendix
and validation curves can be found in appendix
4.1 Prediction Accuracy
4.1.1
Results
Accuracy We report the performance of MiniFold in Table [ and some example predictions in Figure Our results show that MiniFold is competitive with other protein language model- structure prediction models, matching ESSMFold the CAMFO and CASP15 datasets, at fraction of the computational cost. MiniFold is also competitive, though slightly weaker CASP14. Of note, the 12 layer model is surprisingly strong, only underperforming the full-sized model by few IDDT points. MiniFold outperforms Omega Fold on the CAMEO and CASP15 datasets. Of note, Appendix G shows that MiniFold and ESMFold both perform comparably at different depth of sequence homologs, including orphan proteins with little to no homologs.
As limitation, the performance of both ESMFold and MiniFold remains weaker than AlphaFold2 and RoscttaFold2 the CASP datascts. This is an expected result, which can be largely attributed to the language model being less effective than MSA retrieval in extracting evolutionary context Lin et al. (2023 While future work may consider training an MSA-based version of MiniFold that can be more directly compared to these tools, we do not claim that all our observations are directly applicable to MSA based models such as AlphaFold2. Instead, our results highlight that the architecture of pLM based structure prediction models need not necessarily be the same as their MSA counterpart. Since MiniFold, similarly ESMFold, trails the performance of MSA based models, may be beneficial to combine the speod of MiniFoldd with the accuracy of a more computationally expensive method by applying them in succession to achieve good balance in large scale screen.
We also report a comparison between the MDS and transformer-bases structure realizers in Table 2 Sur- prisingly, we find that the non-parametric MDS approach is just as effective as the transformer model in recovering the Ca trace, though a small regression RMSD noticeable. Although using MDS is imprac- tical bocause of its lengthy computation time, this result suggests that the underlying structure has already been discovered by the pairwise track and that distogram supervision alone can be effective in predicting protein structure. It also helps put into perspective the surprising effectiveness of simple transformer
8

---

Published in Transactions on Machine Learning Research (04/2025)
Nirifsid ESNrold
Minifed ESMrald
200
150
Timee (3)
Gipayyies
20-
300
200
400
500
030
2000
1200
200 400
600
800
1000
1208
Lergth
Seq Lengih
Figure Left panel: overall compute time as a function of the sequence length, computed over the 191 largets in the CAMEO lest. set. MiniFold is 10 to 20x faster and shows improved scaling as a function of the input sequence length. Right panel: peak memory usage different sequence lengths.
pared to equivariant architecture such as the IPA. Interestingly, AlphaFold3 reached a similar conclusion, opting for non-cquivariant structure module.
Uncertainty estimation Current protein folding models provide pLDDT scores to quantify the uncer- tainty of predicted structures. We also train pLDDT predictor, which obtains a correlation of 0.76 on CAMEO and 0.9 CASP14. In addition, we demonstrate that the entropy of the predicted distogram is reasonable indicator of uncertainty. As shown in Figure 7 we find that predictions with lower entropy (e.g., higher certainty) tend to have higher LDDT scores. The Pearson correlation between true LDDT and distogram entropy is 0.60 on the CAMEO dataset and 0.9 on the CASP14 dataset. Although not as effective as the PLDDT predictor, the distogram entropy visibly a powerful measure uncertainty.
4.2 MiniFold Efficiency
4.2.1
Experimental Setting
We perform systematic analysis of the throughput and memory usage of our proposed MiniFold architecture. We run MiniFold, ESMFold and OpenFold's implementation of AlphaFold2 the CAMEO test set, and compute the throughput at different sequence lengths. Furthermore, we analyze the step-by-step progression from the Evoformer used in the ESMFold model to our proposed Miniformer:
Full: This is the original Evoformer baseline in the ESMFold implementation, which uses the trian- gular updates from the OpenFold project Ahdritz (002)). We test the full Evoformer has 48 folding blocks and 3 recycling steps.
No sequence: We remove the sequence track, keeping only the pairwise track
No attention: We remove the triangular attention, but keep the triangular update.
Bi-diroctional: We fuse the outgoing and incoming layers and include the down-projcction.
Ours:
We include two optimized GPU kernels in the final Miniformer.
4.2.2 Results
As shown in Figure When measuring speed end to end by running each tool over the CAMEO test set, observe considerable speedup over FSMFold and the reference FpenFold implennetatiion(gnorinn MSA
9

---

Published in Transactions on Machine Learning Research (04/2025)
Evofomer ablation
104
Improvement ratio
Improvement ratio
10a
10"
1024
Protein Size
Protein Size
Figure 5: Ablations for Miniformer memory (left) and time (right) improvement ratio. The full model (cyan bar) corresponds to the original Evoformer implementation in ESMFold. Our Miniformer model (green bar) with all optimization techniques (Removing sequence track and triangular attention, adding kernels, bi-directional) achieves 10-20x improvement in throughput. over Evoformer. The model results significant speedup and memory efficiency depending on the sequence length.
compute time), with speedups ranging from 10x for length 400 protein up 20x for length 1000 protein. We also report controlled ablation on our proposed Miniformer.
Our results show that each of the steps proposed above contribute favorably to the model's improved effi- ciency. We note that the removal of the sequence track has minimal effect on inference speed, but reduces the number of trainable parameters considerably (from 600M down to 23M in ESMFold). The removal of the triangular attention and the use a downward projection in the multiplicative update yield substantial speodups, achieving nearly 5x improvement over the bascline. Our proposed kernels provide an another 3x speedup, leading votal of 10 to 20x improvement in throughput for the full size model, with improved scaling function of protein length.
The bottom panel of Figure 5 measures the peak memory usage during inference over different protein lengths. Here, we see that our proposed kernels result in substantial savings, and that MiniFold improves memory efficiency by nearly 10x for longer proteins. These results have important consequences regarding inference as well as training, allowing larger batch sizes which can result in faster training times.
5 Conclusion
In this work, we propose a highly efficient architecture for protein structure prediction, and a hardware- optimized implementation that results considerable savings in both speed and memory while conserving most of its expressive power and performance. Our results have important implications regarding the use of protein structure prediction models for high throughput applications. On the other hand, we observe that MiniFold, similar to ESMFold, has room for improvement before matching the results of Alphafold2 and the all atom capabilities of TTAFoll All-Atom Krishna et al. (2024), AlphaFold3 Abramson et al. (2024) and Boltz-1 Wohlwend et al. (004)). While our work builds from the success of PLM's, future work could aim to bridge the performance gap by expanding MiniFold to the use of multiple sequence alignments (MSA) as input, and to explore methods to speed up the construction and utilization these MSAs. The methods described here could also be generalized to multi-chain complexes as well as full atom prediction. As such, this work serves as a stepping stone towards the goal of matching AlphaFold3 and Boltz's performance at a fraction of the computational cost.
10

---

Published in Transactions on Machine Learning Research (04/2025)
References
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Scbastian W. Bodenstein, David A. Evans, Chia- Chun Hung, Michael O'Neill, David Reiman, Kathryn Tunyasuvuakkol,, Zachary Wu, Akvilé Zemgulyté, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridgland, Alexey Cherepanov, Miles Congreve, Alexander I. Cowen-Rivers, Andrew Cowie, Michael Figurnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisundaram, Catherine Tong, Sergei Yakneen, Ellen D. Zhong, Michal Zielinski, Augustin Zidck, Victor Bapst, Pushmeet Kohli, Max Jaderberg, Demis Hassabis, and John M. Jumper. Ac- curale structure prediction of biomolecular interactions with AlphaFold 3. pp. 1 3, 2024. ISSN 1476-4687. doi: 10.
URL htttps://wwt nature. wwon//atiiceee/41556666661111111
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy O'Donnell, Daniel Berenberg, lan Fisk, Niccolò Zanichelli, Bo Zhang, et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRrie, pp. 2022-11, 2022.
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnioov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R. Dustin Schaeffer, et al. Accurate prediction protein structures and interactions using a three-track neural network. Science, (677))57) 186, 2021.
Bryan Briney, Anne Inderbiizzin, Collin Joyce, and Dennis R. Burton. Commonality despite exceptional diversity in the bascline human antibody repertoire. Nature, 566(7744):393--)) 397-39777977 2019.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.0499) 2024.
Bo Chen, Xingyi Cheng, Yangli-ao Geng, Shen Li, Xin Zeng, Boyan Wang, Jing Gong, Chiming Liu, Aohan Zeng, Yuxiao Dong, et al. xtrimopglm: Unified 100b-scale pre-trained transformer for deciphering the language of protein. bioRriv, pp. 2023-07, 2023.
Shenggan Cheng, Xuanlei Zhao, Guangyang Lu, Jiarui Fang, Zhongming Yu, Tian Zheng, Ruidong Wu, Xiwen Zhang, Jian Peng, and Yang You. Fastfold: Reducing alphafold training time from 11 days to 67 hours. arXiv preprint ar Xrx:2200 00854, 2022.
Ratul Chowdhury, Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M. Church, Poter K. Sorger, and Mohammed AlQuraishi. Single-sequence protein structure prediction using a language model and deep learning. 40 (11):1617-1623, 2022. ISSN 1546-1696. doi: (0.1088//415557-22--01333 com 2oop/aatttccee//4448---222---010
URL https: //wp:////ww.... nature.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 10-awareness. Advances in Neural Information Processing Systems, 2022.
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general-purpose mod- elling. bioRaie, pp. 2023 01, 2023.
Juergen Haas, Rafal Gumienny, Alessandro Barbato, Flavio Ackermann, Gerardo Tauriello, Martino Bertoni, Gabriel Studer, Anna Smolinski, and Torsten Schwede. Introducing "best single template" models reference baseline for the contimuous automated model evaluation (cameo). Proteins: Structure, Function, and Bioinformatics, 17(12))1378-15 13877 2019.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidck, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin
11

---

Published in Transactions on Machine Learning Research (04/2025)
Steinegger, Michalina Pacholska, Tamas Berghammer, Scbastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. 596(7873):583 589, 2021. doi: 10.1088/s115866-211-088819
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Frangois Fleuret. Transformers are rmns: Fast autorogressive transformers with lincar attention. In International conference on machine learning, pp. 5156-50 5165. PMLR, 2020.
Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S. Morey-Burrows, Ivan Anishchenko, Tan R. Humphreys, Ryan McHugh, Dionne Vafeados, Xinting Li, George A. Sutherland, Andrew Hitchcock, C. Neil Hunter, Alex Kang, Evans Brackenbrough, Asim K. Bera, Minkyung Baek, Frank DiMaio, and David Baker. Generalized biomolecular modeling and design with RoseTTAFold all-atom. 384 xe4(63,2),, https://www.sciie swwencecienc...mm http://c..eee..eooo///.......... wtt.1//////ieec.aaaacoo.c ad12528
2024. doi: 40.1126/science..e
URL
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelís, and John Moult. Critical assess- ment of methods of protein structure prediction (casp) round xiv. Proteins: Structure, Punction, and Bioinformatics, 89(12):1677-161,, 2021.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assess- ment of met of protein structure prediction (casp) round xv. Proleins: Structure, Fanclion, and Bioinformatics, 91(12):1539 1511111)139--19, 2023.
Tao Lei. When attention meets fast, recurrence: Training language models with reduced compute. arXiv preprint ar 0ix:22002. 12459, 2021.
Ziyao Li, Xuyang Liu, Weijie Chen, Fan Shen, Hangrui Bi, Guolin Ke, and Linfeng Zhang. Uni-fold: an open-source platform for developing protein folding models beyond alphafold. bioRriv, pp. 2022-08, 2022.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123 1100:110011111,,,,,,1 2023.
Al Mead. Review of the development of multidimensional scaling methods. Journal of the Royal Statistical Society: Series D (The Statistician), 41(1):27-39, 1992.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protcin function. Advances in Neural Information Processing Systems, 34:29287-29303, 2021.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steineg- ger. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679-683 2022.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C LLarrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15): 22016239118, 2021.
Baris E. Suzck, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H. Wu. UniRef: compre- hensive and non-redundant UniProt reference clusters. 23(10):1282 1288, 2007. ISSN 1367-4811. doi: sattrmmmtttcsbbmm09... btm098.
Philippe Tillet, Hsiang-Tswie Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop Machine Learning and Programming Languages, pp. 10 19, 2019.
Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna, Malarvizhi Radhakr- ishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, Oleg Kovalevskiy, Kathryn Tunyasuvu- nakool, Agata Laydon, Augustin Zidek, Hamish Tomlinson, Dhavanthi Hariharan, Josh A Abrahamson, Tim
12

---

Published in Transactions on Machine Learning Research (04/2025)
Green, John Jumper, Ewan Birney, Martin Steinegger, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic Acids Research, 52(D1):D338 D375, 11 2023. ISSN 0305-1048, doi: 10.1093/nai harggaad/gg...... URL
https::/ddo..rg/// 10. 1093/ www.//00////////e har/g/adi1gg11111
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Guoxia Wang, Xiaomin Fang, Zhihua Wu, Yiqun Liu, Yang Xue, Yingfei Xiang, Dianhai Yu, Fan Wang, and Yanjun Ma. Helixfold: An efficient implementation of alphafold2 using paddlepaddle. arXie preprint arXiv:2207.05.4 2022.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with lincar complexity. Xiv preprint or arXi::2006. 768, 2020.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089-1100 2023.
Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Tally Portnoi, Itamar Chinn, Jacob Silterra, Tommi Jaakkola, and Regina Barzilay. Boltz-l: Democratizing biomolecular interaction modeling. Rioi 2024. doi: 10.1101/2024.1..99
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, Jianzhu Ma, and Jian Peng. High-resolution de novo structure prediction from primary sequence. bioRriv, 2022. doi: early//922//772222202... 07. 217.21..500 21.500999
URL https://ww.biooxivv org/contet//..
Chengxin Zhang, Morgan Shine, Anna Marie Pyle, and Yang Zhang. Us-align: universal structure alignments of proteins, nuclcic acids, and macromolccular complexes. Nature methods, 19(9):1199-1155, 2022.
13

---

Published in Transactions on Machine Learning Research (04/2025)
A
Architecture details
Algorithm outlines the architocture of the miniFormer block, which takes initial input the pairwise representations E R4xLxe from the embedding module and updates it through triangular multiplicative updates and transition layers.
Algorithm Miniformer Block, related to Figure
(langular((2ss):e :
1: Zij -LayorNorm(zz)
2ij E R
2: aiji bijs Chj. dij t sigmoid(Linear(zz))) Linear(zis)
aij. bijs Cij, di Re/4
3: Zi Concat (Ex aik byke En Chi dks)
2i R
4: Zij - LayerNorm(zij)
Fi Ro/2
5: Zij + sigmoid([inear(( Linear(zis)
2ij E R
Transition(1zy))c 128):
1: 4i - LayerNorm(zz))
24j E R
Zi ReLUa(in (zi)))
2ij E R40
Zg Linear(zig)
Zij R
MiniformerBlock/ (2ig), 128):
1: Zij 2ig Triangular(zz)
Aij R
2 Zij -ig Transition(s))
Zij E R
Algorithm outlines the architecture of the transformer-based realizer, which replaces the IPA-based struc- ture module found in AlphaFold 2. The Gram-Schmidt, AngleResnet and ComputeellAtommoordii subroutines are as described in AlphaFold 2 Jumper et al. (2021). The algorithm takes as input the single and pair representations s Rxx, and E RixLxe,
Algorithm Transformer-basedd realizer, related to Figure
TransfornerrRealiiee(((s))) (zy),, N: 8, H 16, c, 1024, c, 128)
1: simitial + LayerNormms!"tit (stritta)
34 E R
2: 8; < Linear(s (sinita))
Σi f- LayerNorm(zg))
2ij E R
4: &Nth Linear(2zy)
6 Nob N
5: For(l E [......]]
6: {si} {8i} + Pair PairBiisadSelAttu((ss)).
7: {si} + {82} Transition((ss:)
# Get backbone frames
8: N,, Ca,, C. Lincar(s:)
9:
Ca,, C.)
# Proceed in AlphaFold2 Structure module
10 of ( (piital .s)
11: : TV. Ta = Compplee All Awl.loooooddnteeeeooooooooo (of,, T.)
Return(T" (T//, xf {st) (..).T.oo. a
14

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 8 outlines the procedure used to realize the distogram coordinates with paramter-frre method. The Shortest Path subroutine uses some variation of the Floyd-Wane Warshall algorithm.
Algorithm Coordinate Realizer
def MDS(logits E NRNNNX61 dmaz 25A,e e E R
1: D,y E (rggaar(logiss;;
b)
2: Da +
1, if (Diy < dmar) & (f
3: Wi
0. otherwise
4: D t ShortestPath(D @ W)
N: if ji
5: Juj t
0, otherwise
6: B -- + * D +
7: A, V + cigen-decomppsitton((]
C, + V
9: for iE [1,2,3
10: C. - LBFGS(D, C.)
11: Return C,
C₂ E Nx3
Kernels
B.1 Algorithms
The following pseudocode outlines the Triton kernels implemented during model inference for the self-gating and feed-forward subroutines. Our kernels show large improvements, even when compared to corch.compio
Algorithm 4 Self-gating kernel. Each parallel kernel thread i is responsible for rows R, and columns C; of the output matrix Y. Computes Y Linear (inear(V,X)) (V,X) Sigmoid(Linear(W:
def SelfGating(X, V, W, Y, B):
: X, + :[
Incurs one read operation for
V, Vl:CC
3: W. W[:CC]]
4: Y1 < DOT(X,
5: Y,2 + DOT(X, Wa)
6: Y+ MUL(SIGMOID(Y)).r7?
Y[R, C.J + Y;
Incurs one write operation forY
15

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 5 Feed--orward kernel. Each parallel kernel thread is responsible for rows R2 and all columns of the output matrix Y. Computes Y = X + Linear Lincar(ReLLU) ULLnearr,,,, V)). W)
def FFN(X, V, W, i, B):
1: X, + :),
Incurs one read operation for
forj N,..,,N//BB
3: Vi < Cj
Here Cjis block of B columns of V
4: W, W[C,,
C3 is also block of B rows of W
5: + RELU(DOT(X, VD)
6: x + W,,
[[ + X;
Incurs one wrile operation forY
B.2 Latency measurements
Size
Torch
Torch-compile
Ours
Size
Torch
Torch-compile
Ours
64.0
0.125465
0.139198
0.217369
64.0
0.047169
0.142224
0.126898
128.0
0.074751
0.136619
0.218885
128.0
0.120064
0.135038
0.124615
256.0
0.263739
0.135980
0.214086
256.0
0.403863
0.373916
0.180016
512.0
1.004088
0.442307
0.218418
512.0
1.543569
1.366344
0.463024
1024.0
3.891327
1.666970
0.868068
1024.0
6.119958
5.369287
1.726875
2048.0
18.806410
6.639621
3.469150
2048.0
32.731358
23.447216
6.805521
Table 3: Self-gating kernel latency (ms)
Table 4: MLP kernel latency (ms)
C
Predictions
Figure 6: Predicted structures for MiniFold (blue), ESMFold (red) and the ground truth PDB structure
(green). PDB ID from left to right: 7c5j, 7efs, 7rez, 7dki.
16

---

Published in Transactions on Machine Learning Research (04/2025)
D Uncertainty prediction
R=0 841 p=0.781
0.842
Protein stoc
0.601
Thue LODT Cr
66-
04
35
85
11
p1
Uistogram entropy
R=0 958
R=0.958
R=0 892
p=0
p=0.9
G
LDOT
06 37 p6 DET
-30 Dbitggrem entropy
Figure 7: The trained pLDDT predictor (left) and the entropy over the predicted distogram (right) correlate with the true IDDT Ca as evaluated the CAMEO dataset. CAMEO on top block and CASP14 on bottom block. The R value was computed using the scipy.stats. linrerres function and the spearman correlation p was computed with stats.spearmanr with default parameters in python.
E Experimental Setting
Training and validation set Similarly to ESMFold, we constructed training set from the AlphaFold Protein Structure database. Varadi et al. (2023). We first cluster Uniref50 sequences Suzek et al. (2007) at 30% sequence similarity, and then select the structures with an average pLDDT score above 0.7. This results in a high quality, diverse dataset of roughly 10 million structures. Interestingly, we found that training on this distillation set alone was sufficient to achieve high accuracy. We randomly sample 10000 structure as our fixed validation set for model selection and hyper-parameter optimization.
Benchmarks We evaluate MiniFold using CAMEO targets Haas et al. (019) released between April 1, 2022 and June 31, 2022, for total of 191 structures. Similar to ESMFold, also evaluate on 51 structures from the CASP14 targets, chosen to avoid overlaps with the ESMFold training set. We also use 47 targets from CASP15 with structures made recently publicly available. The specific protein ID's used for each dataset can be found in appendix
Metrics We report the local distance difference test (IDDT) and root mean square deviation (RMSD) for each dataset and model. We also report both full-atom metrics. A higher metric correspond to better performance for IDDT, and lower is better for RMSD. When computing metrics we align the reference sequence to the residues available in the structure using the USalign tool Zhang et al. (022). We also ignore ambiguous residues (X, U, B, Z) in the evaluation, as well as any missing atom in the target structure.
and training details We train MiniFold using 12 and 48 Miniformer blocks. We train the models on a single node using 8x A100 GPU's and batch size of per GPU for an effective batch size of 128 similar to ESMFold. We use the Adam optimizer with learning rate of 3c-5 for the tuned PLM layers, le-4 for the structure module parameters and 1e-3 for all others for all parameters. We use
17

---

Published in Transactions on Machine Learning Research (04/2025)
pairwise embedding dimension of 128 in the Miniformer, and sequence embedding dimension of 1024 with 16 self-attention heads each of 64 dimensions in the structure realizer. Training is performed in two stages: first by limiting to the embedder and miniFormer blocks and then by enabling recycling and the structure realizer.
F
Validation Curve
0.800
Menhal--12 Mixivalde ca
Minttud-3 12
MISPUUU-18 48
08
0.800
0.-
0.25
0.750
0.725
50000
100030
150000
200003
250000
50000
100000
150003
200000
250000
Steps
Figure 8: Validation curve as a function of training steps. Left panel: full scale, Right pancl: zoomed-in.
G
MSA Depth
We show the performance of ESMFold and MiniFold as function of MSA depth, showing that MiniFold performs comparably well on orphan proteins with shallow to no sequence homologs. Both models show degradation when less evolutionary signal is available as was previously reported by Lin et al. (2023). This figure was constructed by comparing the performance of the model against the depth of the MSA produced by the colabfold MSA server on these sequences.
Meed Fetereance MSA Dephin CAPRO
Medid Netfrrmanees MSA Dest CAPPLA
Figure LDDT-Co as a function of MSA depth (i.e known sequence homologs).
18

---

Published in Transactions on Machine Learning Research (04/2025)
H Targets
The following list contains the target id and corresponding chain (if applicable) used for evaluation.
H.1 CAMEO
7W74-A 7PC6-A, 7L8N-A, 7VW0-B, 7E4S-A, 7QDV-A, 7K2Z-A, 7F6J-C, TEIC-A, 7EA4-A, 7WKQ-A,
7PW1-A, 7EL8-A, 7X8V-A, 7U5Y-A, 7NTN-A, 709F-B, 7BGS-B, 7RCZ-A, 7WWR-A 7TJ1-D, 7VOH-A,
7K09-A, 7LGR-A, 7LVE-A, 7STT-A, TQDW-A, 7ESO-A, 7EKZ-A, 7RXE-A, 7PUO-A, 7QU2-A, 7EIB-A,
7E52-A, 7ESK-B, 7VU7-A 7VNO-A, 7ETR-A, 7TOC-A, 7TLF-I, TQBG-E, 7WEW-A, 7ED1-A, 7ACY-B,
7RCW-A, 7VMC-B, 7BI4-A, 7EFS-D, 7E4J-, 7WIN-B, 7TLF-G, 7TA5-A, 7PMO-G, TEAD-A, A,W6-A
7Q51-A, 7OPY-F, 7S8T-J, 7MYV-B, 7S94-, 7TKV-A 7TA9AA 7LQN-A, 7M5W-A, 7KPJ-F, 7RGV-A
7EGT-B, 7M06-B, 7EOV-C, 7E3T-A, 7POH-A, 7040-A, 7U5F-D, 7BEW-A, 7NDE-A, 7P31-B, 7THI-A
7QBZ-A, 7BLL-A, 6YOD-A, 7VU5-A 7LH6-A 7KDX-B, 7FIW-B, 7TBU-A, 7S2R-B, 7RW4A, 7LK4D, 7MKK-B, 7PHW-E, 7U4H-A, 7 A 7LVF-A, 7PRD-A, 7A67-A, 7LI0-A, 7QSU-A, A
7WJ9-A, 7TIY-C, 7WIF-B, 7PUJ-A, 7R74B, 7EHG-E, 7WCK-A 7D66-C, 7VEE-A, 7MCC-A, 7EBQ-A, 7LSV-B, 7THW-A 7SGN-C, 7LQM-D, 7ETR-C, 7F6L-B, 7A67-B, 7V9P-A, 7N40-C, 7LXK-A, AQBP-A, 7V4S-A, 7F8A-A, 7JIZA, 7PUG-A, 7PZJ-A, A 7WNW-B, 7DKI-A, 7JT9-, 7KOBA, A, 7Q05-E, 7E0L-A, 7MHU-A, 7B7T-A 7TB5-A, 7WRK-A 7VWT-A 7LXS-A 7VSP-C, 7BBZ-A, 7EBT-B, 7ON9-A, 7AAL-A, 7VSE-B, 7B9P-A, 7WRP-A 7F3A-A, 70S0-A, 7EAU-A, 7ERV-A, 7W26-A, 7SKC-A, 6ZPP-A, 7TXP-A, 7NUV-A, 7DUP-A, 7V9H-A, 7EYM-A, 7MWR-A, 7STZ-C, 7WWX-A, 7ED6-A, 7V6I- A, 7EWU-B, 7BCB-B, 7E5J-A, 7TCR-C, 7REJ-A, 7ERP-B, 7PAB-A, 7R6P-A, 7EZG-A, 7E8J-A, 7TV9-C, 7OM4-B, 7NMI-B, 7E3Q-A, 7T03-A, 7FFA-A, 7VJS-A 7EYL-A, 7R09-A, 7TMU-C, 7B0K-A, 7A 7DT1-A, 7N40-B, 7RY7-A, 7EJG-C
H.2 CASP14
T1024, T1025, T1026, T1027, T1028, T1029, T1030, T1031, T1032, T1033, T1034, T1035, T1036s1, T1037, T1038, T1039, T1040, T1041, T1042, T1043, T1044, T1045s1, T1045s2, T1046s1, T1046s2, T1047s1, T1047s2, T1049, T1050, T1053, T1054, T1055, T1056, T1057, T1058, T1064, T1065s1, T1065s2, T1067, T1070, T1073, T1074, T1076, T1078, T1079, T1080, T1082, T1089, T1090, T1091, T1099
H.3 CASP15
T1104, T1106s1, T1106s2, T1112, T1113, T1114s1, T1114s2, T1114s3, T1119, T1120, T1121, T1122, T1123, T1124, T1125, T1132, T1133, T1134s1, T1134s2, T1137s1, T1137s2, T1137s3, T1137s4, T1137s5, T1137s6, T1137s7, T1137s8, T113769, T1145, T1147, T1151s2, T1152, T1154, T1158, T1159, T1170, T1173, T1174, T1176, T1178, T1179, T1185s1, T1185s2, T1185s4, T1187, T1188, T1194
19


Reviews:
[{"review_id": "Qvo7APXsRk", "reviewer": "Reviewer_3CjT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nMiniFold is substantially faster at inference time than ESMFold, the most widely used single-sequence structure prediction method. For high-throughput applications like filtering and assessing computational protein designs, these speed increases are very practically useful. The performance is comparable to ESMFold, and the tradeoff in quality vs MSA-reliant methods like AlphaFold is well discussed in the literature and it is clear that for many applications, this tradeoff is acceptable given the significantly faster inference time. So I do not consider the weaker performance compared to AlphaFold as a “weakness” of the method; rather it is a more nuanced tradeoff that practitioners should take into account. And I believe the authors have done a nice job acknowledging this tradeoff and the limitations of their method, which is much appreciated.\n\nWeaknesses\n\nMy two main questions / weaknesses relate to the choice of Triton kernels and the availability of the method.\n1. Can the authors expand on why they chose to use Triton? How do the advantages and disadvantages of the framework compare to the more commonly used CUDA framework, specifically with respect to structure prediction methods? Are there any concerns about usability or interoperability because of the Triton kernels? Are there plans to develop and release a CUDA-based version of MiniFold, or are there compelling reasons not to do so?\n2. Because MiniFold is a structure prediction tool, the claims in the paper must be supported by some usable code. Using a service like https://anonymous.4open.science/, can the authors make at least some portion of the code available for review, and allow for verifying the usability of the model?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors introduce MiniFold, an optimized protein structure prediction method that uses protein language models, novel GPU kernels, and ablations to construct an optimized prediction method, without requiring MSAs. They redesign the Evoformer module and adapt the structure module into a simpler, faster alternative, achieving up to 20x speed ups in training and inference. MiniFold is competitive with ESMFold, AlphaFold, and other methods on the CAMEO and CASP benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Thorough responses to the two above questions would secure my recommendation for acceptance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2p0mml5Get", "reviewer": "Reviewer_iLp6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Efficiency and Resource Savings:\nThe paper introduces optimizations by simplifying the Evoformer architecture: removing the sequence channel and triangular attention, and retaining only the bi-directional triangular multiplicative update. These improvements are particularly valuable for large-scale protein structure prediction and virtual screening tasks.  \n\n2. Practical MSA Replacement:\nBy leveraging a pre-trained protein language model (ESM-2) to extract sequence embeddings, MiniFold avoids the computationally expensive generation of MSA. This is especially useful for orphan proteins or scenarios where rich evolutionary data is unavailable, making the method more broadly applicable.  \n\nWeaknesses:\n\n1. Accuracy Trade-offs:\nAlthough MiniFold performs well on many datasets, its accuracy, particularly on the CASP14 dataset and in full-atom structure prediction, is inferior to that of more mature methods such as AlphaFold2 or RosettaFold2. This suggests that the architectural simplifications might compromise the ability to capture fine-grained structural details in challenging cases.\n\n2. Dependence on Pre-trained PLMs:\nWhile replacing MSA with a pre-trained language model saves significant time, it may also limit performance in cases where deep evolutionary information (typically captured via MSA) plays a critical role in accurate structure prediction. This reliance might result in diminished performance for proteins where evolutionary context is paramount.\n\n3. Lack of Orphan Protein Evaluation:\nAnother potential weakness is that while the method is naturally suitable for orphan proteins, since it relies on a pre-trained language model rather than MSAs, the paper does not provide experiments specifically evaluating its performance on orphan proteins. This omission means that its effectiveness in such scenarios remains unverified on targeted benchmarks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MiniFold, an efficient approach to protein structure prediction that simplifies the complex Evoformer architecture from models like AlphaFold2. By eliminating the sequence channel and triangular attention in favor of a bi-directional triangular multiplicative update, and by leveraging a pre-trained protein language model (ESM-2) to bypass the computational cost of generating multiple sequence alignments (MSA)—the authors achieve significant improvements in both speed and memory efficiency. Additionally, custom GPU kernel optimizations using Triton further enhance the model’s performance, making MiniFold a promising candidate for high-throughput and resource-constrained applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to weaknesses.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ed4TekrE3H", "reviewer": "Reviewer_AYru", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Stengths**\n1. The authors propose a simplified architecture (MiniFold) that reduces computational cost while maintaining competitive accuracy. The reported speedups (10x-20x) and memory savings are impressive and make MiniFold a strong candidate for large-scale applications like virtual screening, mutational scans, and de novo protein design.\n2. MiniFold is benchmarked against state-of-the-art models on widely recognized datasets (CAMEO, CASP14, CASP15). The results are well-documented, showing that MiniFold performs comparably to ESMFold while being significantly faster.\n\n**Weaknesses**\n1. While MiniFold achieves significant efficiency gains, its accuracy (e.g., lDDT scores) lags behind AlphaFold2 and RosettaFold2 on CASP datasets. This limitation is acknowledged but could be emphasized more clearly in the discussion section, especially for applications requiring high precision.\n2. The authors mention that MiniFold could be extended to MSA-based inputs to improve accuracy, but no concrete experiments or evidence supporting this claim are presented.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MiniFold, a simplified and computationally efficient architecture for protein structure prediction, aiming to address the scalability and resource limitations of existing models such as AlphaFold2 and ESMFold. By redesigning the Evoformer and structure module, and introducing optimized GPU kernels, the authors claim significant improvements in efficiency (up to 20x speedup) with minimal loss in accuracy. The architecture is evaluated on standard datasets (CAMEO, CASP14, CASP15), demonstrating competitive performance with existing methods while achieving drastic reductions in runtime and memory usage.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1. Address Accuracy-Performance Trade-off**\nProvide more analysis or discussion on the scenarios where MiniFold’s reduced accuracy might limit its use, and suggest potential solutions (e.g., hybrid models that combine MSAs and PLMs).  \n\n**2. Evaluate on More Datasets**\nTesting MiniFold on additional benchmarks or datasets, particularly those with multi-chain complexes or orphan proteins, would provide stronger evidence of its generalizability.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and this action editor all appreciate this contribution and the authors effort to give the research community access to more efficient protein structure prediction. It can be expected that this paper will have a substantial impact and help spur further progress in accelerating computation in deep learning for point cloud data.
Decision: Accept (poster)