Triangle Multiplication is All You Need for Biomolecular Structure Representations
Authors: Jeffrey Ouyang-Zhang
Pranav Murugan
Daniel Jesus Diaz
Gianluca Scarpellini
Richard Strong Bowen
Nate Gruver
Adam Klivans
Philipp Kraehenbuehl
Aleksandra Faust
Maruan Al-Shedivat
Venue: ICLR 2026 Poster (2026)
Keywords: structure prediction
cofolding
triangle multiplication
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Simple biomolecular structure prediction architecture

Abstract: AlphaFold has transformed protein structure prediction, but emerging applications such as virtual ligand screening, proteome-wide folding, and de novo binder design demand predictions at a massive scale, where runtime and memory costs become prohibitive.
A major bottleneck lies in the Pairformer backbone of AlphaFold3-style models, which relies on computationally expensive triangular primitives—especially triangle attention—for pairwise reasoning.
We introduce Pairmixer, a streamlined alternative that eliminates triangle attention while preserving higher-order geometric reasoning capabilities that are critical for structure prediction.
Pairmixer substantially improves computational efficiency, matching state-of-the-art structure predictors across folding and docking benchmarks, delivering up to 4x faster inference on long sequences while reducing training cost by 34%.
Its efficiency alleviates the computational burden of downstream applications such as modeling large protein complexes, high-throughput ligand and binder screening, and hallucination-based design.
Within BoltzDesign, for example, Pairmixer delivers over 2x faster sampling and scales to sequences 30% longer than the memory limits of Pairformer.
Code is available at https://github.com/genesistherapeutics/pairmixer.

Published as a conference paper at ICLR 2026
TRIANGLE MULTIPLICATION IS ALL YOU NEED FOR BIOMOLECULAR STRUCTURE REPRESENTATIONS
Jeffrey Ouyang-Zhang Pranav Murugan', Daniel J. Diaz², Gianluca Scarpellini', Richard Strong Bowen', Nate Gruver', Adam Klivans?, Philipp Krähenbühh!, Aleksandra Faust Maruan A
Genesis Research
UT Austin
ABSTRACT
AlphaFold has transformed protein structure prediction. but emerging applications such virtual ligand screening, proteome-wide folding, and de novo binder de- sign demand predictions at a massive scale, where runtime and memory costs become prohibitive. A major bottleneck lies in the Pairformer backbone of Alphafoold--tylle models, which relies on computationally expensive triangular primitives-especialll triangle attention-for pairwise reasoning. We introduce Pairmixer, a streamlined alternative that eliminates triangle attention while pre- serving higher-order geometric reasoning capabilities that are critical for structure prediction. Pairmixer substantially improves computational efficiency, match- ing state-of-the-arr structure predictors across folding and docking benchmarks, delivering up to 4x faster inference on long sequences while reducing train- ing cost by 34%. Its efficiency alleviates the computational burden down- stream applications such modeling large protein complexes, high-throughput ligand and binder screening, and hallucination-base sign, for example, Pairmixer delivers over 2 faster sampling and scales to quences ~30% longer than the memory limits of Pairformer. Code is available at https///ghhhbb..oommnnessseeettt
design. Within BoltzDe-
INTRODUCTION
AlphaFold (Senior et al., 2020; Jumper et al., 2021) has transformed protein structure prediction and become an indispensable tool across the biological sciences. Yet emerging applications increasingly demand massive scale. Virtual screening of protein-ligande interactions, modeling of large protein complexes, prdeeome-wide folding, and iterative de novo binder design already require millions (and soon billions) of inference calls. At this scale, runtime and memory efficiency are critical bottlenecks: for example, Boltz-1 1 (Wohlwend et al., 2024) requires over 15 minutes to process single 2048- token sequence on an A100 GPU (see Section 5.3). The dominant computational cost comes from pairwise token representations and triangular primitives, which scale cubically with sequence length L. While triangle multiplication is implemented efficiently via matrix multiplications, triangle attention requires L attention operations, introducing substantial memory and runtime overhead.
We introduce Pairmixer, a streamlined alternative to the Pairformer backbone of AlphaFold3 (Abram- son et al., 2024). By retaining triangle multiplication and feed-forward networks while eliminating triangle and sequence attentions, Pairmixer preserves the ability to reason over higher-order geometric interactions that critical for structure prediction while alleviating Pairformer's heavy computa- tional burden. Despite this simplification, Pairmixer performs comparably on RCSB and CASPI5 test sets against Sate-of-the-an predictors such as AlphaFold, Chai-1, and Boltz-1, while providing 4x faster inference on long sequences. Pairmixer consistently matches the performance of Pairformer backbone across protein- protein-ligand, siiibody-antigee protein- nucleic acid and RNA structures while training in 34% % fewer GPU-days across multiple model sizes (see Figure 1).
WWrr done during an internship at Genesis Research
{ (johhang, danny. di az, kli ivans, philkr @cs wtw.aa.e........... ettexas.edu (pranav, giansca rpe, iccardd rd, ngruver, sandra, maruan] @genest stherapauticss as

---

Published as a conference paper at ICLR 2026
0.75
Mean LDDT
0.72
10t
1023
100
150
200
250
300
350
400
Training FLOPs
Training Time (GPU-days)
Paimbker (Ours)
Pairformer (Boltz-1)
Transfomerr
Figure 1: Pairmixer is an efficient architecture for biomolecular structure prediction. Across multiple model sizes, Pairmixer matches the performance of the leading Pairformer architecture while delivering greater training efficiency.
By reducing both runtime and memory requirements, Pairmixer expands the scope of feasible downstream applications structure prediction. enables modeling of larger protein complexes beyond the limits of triangle attention, supports high-throughput screening of ligands and binders, and accelerates hallucination-basee
design pipelines (Pacesa et al., 2025). Within the BoltzDesignl (Cho al., 2025) framework, Pairmixer provides over 2x faster sampling and scales sequences beyond 700 amino acids, where BoltzDesign otherwise fails due to memory overflow. Our analysis highlights the role of the pair representation in learning precise distances between residues and suggests that triangle multiplication learns to capture sparse long-range interactions among residue triplets.
RELATED WORK
Biomolecular Structure Prediction. Protein structure prediction has progressed rapidly in recent years, with early efforts primarily focused modeling monomeric proteins (Senior et al., 2020; Jumper et al., 2021; Baek et al., 2021; Yang et al., 2020; Ahdritz et al., 2024). As these approaches matured, structure predictors expanded to handle multimeric assemblies (Evans al., 2021; Baek et al., 2023) and other modalities such as nucleic acids (Baek et al., 2024). Today, state-of-the-ar predictors can fold complexes that span a wide range of biomolecular types (Abramson al., 2024; IntFold et al., 2025; Boitreaud et al., 2024; Wohlwend et al., 2024; ByteDance et al., 2025).
Biomolecular structure predictors rely on specialized backbones that capture complex geometric relationships among molecular entities. Early approaches such as trRosetta (Yang et al., 2020) and AlphaFoldi (Senior al., 2020) leveraged convolutional neural networks to extract pairwise residue features from multiple sequence alignments (MSAs) and predict inter-residue distances. AlphaFold2 introduced the transformer- based Evoformer to jointly model MSA and pair representations (Jumper et al., 2021), while AlphaFold3 refined it with the Pairformer, which decouples MSA and pair processing (Abramson et al., 2024). The Pairformer has since become the dc-facto backbone architecture for biomolecular structure prediction (IntFold al., 2025; Boitreaud et al., 2024; Wohlwend et al., 2024; ByteDance et al., 2025). However, despite its strong performance, the Pairformer remains complex and computationally demanding.
Several alternative architectures have been proposed to simplify structure prediction backbones. Mini- Fold (Wohlwend et al., 2025) streamlines Alphafold2's Evoformer using lightweight Miniformer based on triangle multiplications. SimpleFold (Wang et al., 2025) replaces the Evoformer with a sequence-only transformer that omits pair representations. Our work also simplifies backbone design, but unlike prior efforts focused on monomeric folding, Pairmixer' is developed for AlphaFold cofolding models, enabling structure prediction across broader biomolecular modalities.
Downstream Applications of Structure Prediction. The success of biomolecular structure predic- tion has enabled a growing number of downstream applications, many of which leverage predicted structures at unprecedented scales. Large-scale resources such as the AlphaFold Database (Varadi et al., 2022) and OpenFold (Ahdritz et al., 2024) have generated massive synthetic protein structure datasets using AlphaFold2, powering advances in structure search ( (Van Kempen al., 2024), pro- tein language modeling (Heinzinger et al., 2024; Ouyang-Zhang et al., 2024; Hayes et al., 2025), and diffusion-barn based structure generation (Geffner al., 2025; Lin et al., 2024; Daras et al., 2025).
2

---

Published as a conference paper at ICLR 2026
MSKQNSDH
MSA Module
TATAGGCA
Input Embedder
Diffusion Module
Pair
Backbone
[Mg+&]
Single
Sequences
Structure
Figure 2: Overview of Biomolecular Structure Prediction. Given list of sequences, our model predicts the 3D folded structure of all sequences within a single complex. Input sequences are first embedded into a single representation for each residue and a pair representation to capture the relationship between pairs of residues. The MSA Module and Backbone (e.g., Pairformer) extracts deep pairwise features capturing inter-residue interactions, which are then passed to the diffusion module to generate the 3D structure. (Additional inputs such as MSAs, conformers, and templates are omitted for clarity.)
Structure predictors now drive virtual screening pipelines that evaluate millions of candidate drugs based on predicted protein-ligand interactions (Wong et al., 2022; Shamir & London, 2025; Scardino et al., 2023), and large-scale folding studies that map the human interactome (Ille et al., 2025; Zhang al., 2025). Hallucination-baan basodddd--dddd---eo generation methods such as BindCraft (Pacesa et al., 2025) further use predictors in iterative optimization loops requiring millions of model evaluations. As these applications expand scope and scale, inference speed becomes critical bottleneck. We introduce structure predictor that matches state-of-the-an accuracy while operating at fraction of the runtime, enabling faster and broader deployment of downstream workflows.
Attention-frre Architectures. While transformers lead modern architectures, attention-free variants aim to improve scalability. FNet (Lee-Thee Thorph et al., 2021) and related models (Poli et al., 2023; Zhai al., 2021) replace attention with Fourier convolutional mixing for sub-quadratic efficiency, while MLP-Mixer (Tolstikhin et al., 2021) achieves competitive performance using token- and channel-wise multi-layer perceptrons (MLPs). Puirmixer removes attention entirely from the backbone and mixes tokens through matrix multiplication.
Architectures based on triangle multiplication have been explored in several prior works. Genie2 (Lin et al., 2024) performs de-novo structure generation by iteratively updating a pair representation through triangle multiplications, while MSA Pairformer (Akiyama al., 2025) applies similar opera- tions to extract features from multiple sequence alignments. IgFold (Ruffolo et al., 2023) incorporates triangle operations within GNN layers. Pairmixer likewise learns rich protein representations through triangle multiplication, but in the context of biomolecular structure prediction.
3 PRELIMINARIES
Let x (x(1),..... x [k) denote a collection of K biomolecular sequences. Each sequence ((k)
(z.(4), (k) (k)
2())) consists of tokens z! E T corresponding to amino acid, nucleic acid, or small molecule heavy atoms. (k) denotes the number of tokens in biomolecule (*),, The goal
of biomolecular structure prediction is to map the sequences to direc-dimensional structure a = (1) a(k),, where each biomolecular structure (k) = (a
(k)
) consists of
atomic coordinates a (M) R33 and (k) denotes the number of atoms in biomolecule k. See Figure for overview.
The Input Embedder concatenates the sequences x = {x(1) ,.."),........ (k)} and embeds into a "single" length L = Ek= (b) sequence representation ginit = LxCC of dimension C., Modern structure predictors (Jumper et al., 2021) additionally initialize "pair" representation x zinit IR LxLxx.
Zi 8; + 8, PE(i,))
where PE( is positional encoding that incorporates both intra- and C. is the pair embedding dimension. Intuitively, 2, R captures the relational context between tokens s; and sj and enables the model to reason about longer-rang, couplings. Since pairwise reasoning is critical for structure prediction, adopt the same input embedding scheme.
distances and
3

---

Published as a conference paper at ICLR 2026
Z
Triangle Multiplication
Triangle Attention
FFN
Sequence Attention w/Pair Bias
S
FFN
(a) Pairformer architecture. The de facto biomolecular structure prediction backhone.
Z
Triangle Multiplication
FFN
S
(b) Pairmixer architecture. An efficient yet effective biomolecular structure prediction backbone.
Figure Schematic comparison of the Pairformer and Pairmixer backbones. Pairmixer simplifies the Pairformer architecture by removing redundancies. This results in faster training and inference, expanding the scale of downstream applications.
The MSA Module encodes evolutionary information that is crucial for structure prediction (Benner Yanofsky et al., 1964; Ovchinnikov et al., 2017; 2014; Morcos et al., 2011; Weigt et al., 2009). For each amino acid or nucleic acid sequence (*()), we perform a homology search to construct multiple sequence alignment (MSA) of related sequences that likely adopt the same (AAP))MMM
& Gerloff, 1991;
fold. Formally, (z)()( E (Tu U (GAP)) M(3) (K)) contains (k) aligned sequences of length
(k), This alignment establishes positional correspondence across homologous sequences, enabling detection of conserved sites and co-evolutionary couplings. The resulting MSAs are then paired, concatenated, and embedded into miint Mx where M is the number of filtered homologous sequences and C.m is the MSA embedding dimension.
The MSA module takes (nitt .init) as input, extracts structuralll-relevan evolutionary patterns from m init and encodes pairwise interactions into mas to guide folding. Since processing all M sequences the MSA computationally expensive, AlphaFold3 introduced a shallow 4-layer MSA module after which the MSA is discarded while the evolutionary-ant to be refined. Our model derives tma from an MSA module but introduces a more efficient feature extractor to refine its evolutionary signals.
pair representation msa continues
The Pairformer backbone serves as the primary feature extractor for AlphaFold3 (Abramson al., 2024), producing structrually-aware representations that encode geometric constraints between residues (see Figure 3a). It takes (iit ms)) as input and employs several specialized modules that iteratively update the sequence and pair representations produce (s backbone backbone). See Figure for more detailed treatment of the entire architecture.
The Pairformer contains two specialized modules for processing the pair representation; triangle attention and triangle multiplication. These modules treat the pair representation z E R Lxxxx as edge features of a cully-connected graph of nodes and reason over triplets of residues (nodes) to learn geometric constraints.
Triangle attention computes attention (with pair bias) along every row (and column) of the pair representation. Formally, the update to row is
TriAtt(z); softmax (ootmax(((Qzz)(WM)) (Woz)) + Wnz Wvz:
where (We WK. Wv) are standard attention projection matrices, and Wn projects the pair repre- sentation into an attention bias term TriAtt(=); effectively performs attention over all residues while conditioning residue
single head and removed scaling for brevity
4

---

Published as a conference paper at ICLR 2026
Algorithm Puirmixer Backbone
Require: Input pair representation 2 LXXX
Require: Number of backbone layers N
Ensure: Updated pair representation NN
1: _
2: for = to N do
3:
2: TriMullncomingg zi)
4: # 21 TriMulOutgoingg
5: 21-1
(2N(
6: end for
7:
return ZN
Triangle mulriplication performs matrix multiplications to integrate features across different rows (and columns) of the pair representation. Formally, the update to edge Zij is
TriMul(z)y = Σ (aaz = O (Wizjk)
where Wa, W, are linear projection layers. For each edge Zij. triangle multiplication computes how every node interacts with query nodes and through edges Zík and Zjk-
Both operations scale cubically with sequence length, making the processing of long sequences computationally expensive. Triangle multiplication more efficient, as it can be implemented with matrix multiplications (e.g., torch. ei einsum) whereas triangle attention incurs the higher cost of full attention computations. components and show that triangle multiplication yields representations powerful those from triangle attention, but at substantially lower computational cost, supporting range of downstream applications.
this work, we streamline the cofolding backbone to its essential
While the Pairformer trained with an auxiliary distogram loss that ensures z backhone accurately represents all pairwise token distances, it does not yet specify an atomic 3-D structure.
The Diffusion Module samples the atomic coordinates conditioned on (s backbone backboue)) It uses transformers derive atomic representations from the token-level sequence and pair representations, and subsequently denoises all-atom coordinates based on these representations. We leverage the diffusion module as-is to realizes 3-D structures conditioned on single and pair representations derived from our efficient backbone.
METHOD
We introduce Pairmixer, attention-free feature extractor for biomolecular structure prediction and design (see Figure 3). Pairmixer exclusively updates the pair representation msa leaving the single-sequence representation giiit unchanged. Through triangle multiplication, Pairmixer efficiently mixes features within the pair representation, facilitating reasoning over residue triplets and their geometric constraints. Combined with ceed-forward networks (FFN) that process all residue pairs, this architecture provides an effective and expressive backbone for biomolecular structure prediction.
The full algorithmic specification of Pairmixer is available in Algorithm 1. In developing Pairmixer, we identified and removed two unnecessary modules from the Pairformer: sequence updates and triangle attention.
Removing Sequence Updates. In AlphaFold2's Evoformer backbone, sequence updates were essential components that processed the MSA to capture evolutionary features. However, the MSA Module in cofolding models now preprocesses the MSA and encodes this evolutionary information directly into the pair representation x mma eliminating the need for sequence updates to provide evolutionary information. Since the pair updates proved more expressive. bypass sequence processing entirely and pass the initial sequence representation directly to the diffusion module (i.c., bhackboue iiiit).
5

---

Published as a conference paper at ICLR 2026
Small
Medium
Large
Mean LDDT
0.72
0.64
0.56
0.48
50
100
150
50
100
150
200
100
200
300
400
Training Time (GPU-days)
Pairmixer (Ours)
Pairformer (Boltz-1)
Transformer
Figure 4: Performance curves on RCSB test set across model sizes. We compare three backbone architectures across three model sizes over training. Pairmixer matches or surpasses the Pairformer baseline while training more efficiently.
Removing Triangle Attention. Triangle attention reasons over residue triplets by applying attention to each row of the pair representation 2,, using the full z as pairwise bias (see Figure 11b). How- ever, this approach is computationally expensive, requiring L separate attention operations over L tokens per layer. Triangle multiplication offers equivalent capability for capturing geometrically consistent pair representations via triplet reasoning mechanism, but with significantly lower compu- tational cost. Since both methods have independently demonstrated strong performance in structure prediction (Jumper et al., 2021), adopt the more efficient triangle multiplication approach.
RESULTS
5.1 IMPLEMENTATION DETAILS
We implement Pairmixer on top of Boltz-1, an AlphaFold3 descendant. More specifically, we replace the Pairformer backbone with Pairmixer and remove triangle attention from the MSA Module. Note that we do not alter the diffusion module's transformer architecture. We also introduce a transformer baseline that preserves the sequence update while removing the pair update in the backbone. To ensure this baseline as strong as possible, we modify the architecture allow features flow effectively from the MSA module into the diffusion module (see Section A.2).
Following Boltz-1 training schedule (Wohlwend et al., 2024), we train on 384/3456 token/atom crops for the first 53k iterations using the PDB and OpenFold distillation dataset. We then finetune for 15k iterations on the PDB dataset with a larger crop size of 512/4608. To evaluate the generality of approach, train models of multiple sizes. Our large configuration matches Boltz-1, with 48 Pairformer layers and 24 diffusion transformer layers. In addition, we develop small and medium variants with 12/24 Pairformer layers and 6/24 diffusion transformer layers, respectively. During inference, we default to 10 recycling steps and 200 sampling steps for all models. In our main evaluation, we sample 5 poses and report the metrics on the top pose (oracle evaluation). Full hyperparameter details are in Table 10.
5.2 COMPARISONS ON COFOLDING PERFORMANCE ACROSS MODEL SIZES
We evaluate our efficient Pairmixer architecture against two baselines, Pairformer (Abramson et al., 2024) and a sequence-only Transformer. All models are evaluated on the RCSB test set introduced in Boltz-1 (Wohlwend et al., 2024), which contains 533 structures with at most 40% sequence identity to the training set, maximum small-molecule similarity of 80%, and resolution better than 4.5Á All models are evaluated at 15, 30, 45, 60, and 68 epochs, totalling 53k iterations and the large model is additionally evaluated during the second phase of 15k iterations. We additionally extend training for small and medium Pairmixer and Transformer models until the total training time matches the Pairformer. We report the final mean LDDT, averaged across all residues.
Our Pairmixer consistently outperforms or matches the Pairformer across all model sizes (see Fig- 4). At the large scale, Pairmixer reaches Pairformr-level accuracy (mean LDDT of 0.78) while requiring only 66% of the training time. The trend holds at smaller scales: Pairmixer surpasses Pairformer the medium scale and matches it at the small scale under equal training budgets. Fur-

---

Published as a conference paper at ICLR 2026
50
50
50
Time
30
30
30
20
20
20
10.
256
512
1024
2048
01
20
512
1024
2040
4096
8192
36
Number of Tokens
Number of Recycles
MSA Depth
Number of Blocks
Pairmixer
Boltz-1 (Pairformer)
Transformer
Figure 5: Inference speed analysis. We measure runtime across architectures and input sizes. While the Transformer the fastest overall, Pairmixer achieves substantially lower inference times than Pairformer, particularly on longer sequences.
thermore, under the same training time, Pairmixer exceeds the sequence-only Transformer baseline across all scales. These results suggest that sequence-only Transformer is inadequate for extracting structural features, while the triangle multiplications and feed-forward networks in Pairmixer are sufficient to capture rich structural representations. Full tabular results are provided in Table and Table 5, and detailed FLOPS analysis is provided in Section B.
5.3 INFERENCE TIME COMPARISONS
Many downstream applications require running the structure predictor thousands to millions of complexes, making inference efficiency critical. Figure 4, we benchmark Puirmixer against the Pairformer and a sequence-only transformer under a default setting of 512 tokens, 4608 atoms, MSA depth of 4096, 10 recycles, 48 blocks, and 200 sampling steps.
On this setup, Boltz-1 I requires 34 seconds to generate single sample on GH200 GPU, while Puirmixer completes in 21 seconds, yielding 1.6x speedup. This advantage holds consistently across different recycle counts, MSA depths, and backbone sizes. The scaling benefits striking for longer sequences: at 1024 tokens, Pairmixer is 2x faster, and at 2048 tokens, it delivers a xx speedup, reducing runtime from 1000 seconds 250 seconds. These results establish Pairmixer as a scalable and efficient architecture, making large-scale cofolding more practical.
more
5.4 COMPARISONS TO PRIOR WORKS
Figure 6 compares Pairmixer to other cofolding models on the RCSB test set, evaluating protein folding, protein-protein interactions (DockQ), and protein-ligand interactions (IDDT-PLI and ligand RMSD < 2). See Section 5.2 for description of the test dataset. We generate five poses per complex and report both the performance of the best pose (oracle) and the average across poses. Results
PDB Test Performance
1.0
0.8-
0.6
655
0.4-
Mean LDDT
DeckQ > 0.23
Mean LDOT-PLI
L-AMSD < 2A (n-238)
(n-538)
(n-242)
(n-238)
AF3 oracle
Chai-1 oracle
Boltz-1 oracle Boltz-1 top-1
Pairmixer (Ours) oracle
AF3 top-1
Chai-1 top-1
Pairmixer (Ours) top-1
Figure 6: System-level comparison on the RCSB test set. We evaluate against AlphaFold3, Chai-1, and Boltz-1 on protein and small-molecule structure prediction. Pairmixer performs competitively with these -f of-the-art approaches. Error bars denote bootstrapped 95% confidence intervals. "Since we do not train a confidence model, results are reported using the first prediction.
7

---

Published as a conference paper at ICLR 2026
Table Performance diverse biomolecular structure prediction benchmarks. Number of test
samples for each dataset is indicated in parentheses.
(a) PoseBusters: Protein- Ligand Complex (298)
(b) Antibody-Antigen Complex (70)
Method
RMSD<2
RMSD
IDDTPu
Method
DOCKQ>0.22
Pairformer (Boltz-I)
0.68
0.46
0.74
Pairformer (Boltz-1)
0.23
Paimixer (Ours)
0.67
0.45
0.73
Pairmixer (Ours)
0.23
Transformer
0.08
(c) Protein-Nucleic Acid Complex (172)
(d) RNA Structure (27)
Method
ICS
IPS
Method
IDDT
Pairformer (Boltz-1)
0,50
0.65
Pairformer (Boltz-1)
0.58
Pairmixer (Ours)
0.51
0.66
Pairmixer (Ours)
0.59
Transformer
0.48
0.64
Transformer
0.61
for existing methods are taken from the literature. Pairmixer matches Boltz-1 in mean IDDT and protein-ligand IDDT, slightly improves ligand RMSD < (0.55 vs. 0.54), but lags on DockQ > 0.23 (0.63 vs. 0.64). These results indicate that even at the largest scale, triangle multiplication and pair FFNs in Pairmixer are sufficient for cofolding across diverse interaction types. We show similar results the CASP15 test set in Section C.I.
5.5 COMPARISONS ON DIVERSE STRUCTURE PREDICTION TASKS
Table shows evaluation results across a variety of biomolecular structure prediction benchmarks, including protein-ligand complexes (PoseBusters), antibody-antigen complexes, protein-nucleic acid complexes, and RNA structures. Experimental details are provided in Section C.3. Pairmixer performs par with Pairformer, while standard Transformers generally lag behind. The exception is RNA structures, where the Transformer baseline slightly outperforms both Pairformer and Pairmixer, likely due the limited availability of RNA structural training data. Notably, Pairmixer achieves comparable performance to Pairformer despite not using sequence attention. These results highlight Pairmixer's generality and robustness in modeling diverse biomolecular interactions.
5.6 COMPARISONS ON BINDER DESIGN (BINDFAST)
Hallucination- based protein design methods have shown that structure predictors can act as differen- tiable scoring functions for sequence optimization. However, they are memory-intensive and slow, requiring hundreds of runs to generate single sequence. We introduce BindFast, which replaces BoltzDesign's (Cho et al., 2025) Pairformer backbone with Pairmixer, reducing runtime and memory usage. On 80GB A 100 GPU, BoltzDesign encountered OOM errors on targets with over 500 residues, while BindFast handled targets up 650 residues (+30%) and ran over 2x speedups (see Table 2). Qualitative comparisons in Figure 13 show comparable designs, suggesting BindFast enables faster in-silico iteration and design of larger, biologically relevant binders. Details are in Section C.4.
Table 2: Runtime comparison of generating proteins with Pairmixer and Pairformer in the BoltzDesign framework. For biologically relevant targets of various sequence lengths, we generate three 110-residue binders using 160 iterations in all settings and report the average rumning time.
Complex
Target
Pairformer
Pairmixer
Target
PDB_Chain
Speedup
Length
Length
Time (sec)
Time (sec)
GIP peptide
2QHK_ B
140
30
680
337
2.01x
Ubiquitin
IUBQ_A
186
76
1113
532
2.09x
TP53
4MZL_ A
303
193
3198
1390
2.30x
hSDH
IP5J_A
429
319
7289
2920
2.50x
hMAO
IGOS
607
497
17134
6601
2.60x
bsDNA Polymerase
3TAN_A
702
592
0OM
9184
00
hTLR3
IZIW
739
629
OOM
10568
0c
Prostate Antigen (PSA)
A
805
695
00M
0OM
8

---

Published as a conference paper at ICLR 2026
IDDT
RMSD
DockQ
IDDT-PLI
1.0
0.75
40
Pairmixer
Patrmbxar
Pairmizer
Pairmixer
0.3
93.7%
Win 73.3%
0.0
71.6%
0.25
0,50 0.75
20
40
0.0
0.5
0.0
0.5
10
Transformer
Transformer
Transformer
Transformer
Figure 7: Head-to-head comparison between Pairmixer and the Transformer backbone. The win rate shows how often the Pairmixer architecture achieves a better score than the Transformer architecture. Pairmixer outperforms the Transformer on the distance-based IDDT metric 93.7% of the cases. highlighting that its advantage lies in capturing pairwise interactions.
ANALYSIS
Predicting biomolecular structure requires reasoning over the entire sequence to capture diverse interactions among residues. We analyze how the architectural design of modern structure predictors facilitates such reasoning and how the simplified Pairmixer architecture achieves this.
Pair representations. central challenge biomolecular structure prediction is determining the strength of the interactions between all residue pairs. This is difficult because folding involves nonlocal tertiary interactions in which residues distant in sequence often interact physically in three- dimensional space. Modern structure predictors address this challenge with a pair representation. Our results indicate that the pair representation enables the model to capture fine-grain spatial relationships between all residue pairs.
We compare the performance of Pairmixer, which incorporates pair representations, against our sequence-only Transformer baseline in Figure 7. On the IDDT metric computed from pairwise distances, Pairmixer achieves higher scores in 93.7% of test complexes. In contrast, on the RMSD metric, which requires global structural aligmment, the improvement is smaller (74.7%). These findings show that pair representations provide greater benefits for local, pairwise accuracy over sequence attention, suggesting their effectiveness in capturing residue-residue interactions.
Triangle multiplication. Modern structure predictors employ triangle attention and triangle multiplication within the pair representation to capture geometric relationships among residue triplets. While triangle attention allows the model to reason sparsely over interacting residues, triangle multiplication densely aggregates features across the entire sequence. However, our analysis shows that triangle multiplication also efficiently captures sparse geometric relationships among residue triplets by adjusting the magnitudes in the pair representations.
LDDT
DockQ
LDDT-PLI
RMSD
1.0
0.8
0.0
Dropout Rate
Dropout
Dropout Rata
Dropout
Low-Norm Dropout
Random Dropout
Figure 8: Pairmixer Performance under different sparse triangle multiplication variants. The model is trained with standard triangle multiplication and evaluated under various dropout conditions. While performance degrades rapidly under random dropout, it remains stable when low-norm entries the triangle multiplication are zeroed out.

---

Published as a conference paper at ICLR 2026
LDDT
DockQ
LDDT-PLI
RMSD
N: Cogpost
0.6
0.4
0.2
Block Size
Block Size
Block Size
Block Size
Figure 9: Pairmixer Performance under blockwise dropout. The model is trained with standard triangle multiplication and evaluated under local blockwise triangle multiplication. Performance quickly degrades even for local metrics like IDDT.
We explicitly sparsify triangle multiplication by introducing dropout during inference. Formally,
TriMulWithDropuutee
Σ (W (Wozja) (zix) M (zju)
new dropout musks
where M(zj)) E {0,1} determines whether particular interaction is active.
In random dropout with dropout rate y [0, 1], the masks are sampled independently as (zza,, M(zjk) Bernoulli(1 - 77). We experiment with a low-norm dropout scheme, drop- ping any interaction (1,5) whose pair representation lies in the 7 E [0, fraction of smallest
[1, i E (t]==al
magnitudes. Formally, M(za)) = 00 otherwise
Under both dropout schemes,
each term (Waza) @ (Wozjk) ) is retained only if both corresponding masks M (z;) and M (zji) are active, resulting higher effective dropout rate.
Figure 8 shows the performance of the model where both dropout schemes are applied to every layer with y 0.10, 0.25, 0.50, 0.75. We observe that performance starts degrade rapidly once the random dropout rate exceeds 25%, indicating that the model is not robust to random removal of interactions. However, the performance is very similar under the low-norm dropout of 75%. This suggests that, like attention, triangle multiplication identifies and processes small subset of interactions that are essential for accurate folding of biomolecular complexes.
To probe which interactions the model relies on in its sparse computation, evaluate it using a local dropout scheme. For block size B, we retain only local interactions: M(2x)) =
[1, if ii k| < B
The results in Figure 9 show that performance already begins to degrade B = 512, with a substantial drop at B = 256 tokens. This suggests that triangle multiplication processes sparse, long-range interactions.
0, otherwise
CONCLUSION
We introduce Pairmixer, a simplified, efficient feature extractor for biomolecular structure prediction. Models using Pairmixer train 1.5x faster and sample up 4x faster than those with Pairformer, enabling large-scale, compute-intensiee applications of structure prediction. The key idea is to explicitly materialize a 2-D pair representation, updated via triangle multiplications that capture interactions among residue triplets. We hypothesize that transforming 1-D sequences into 3-D structures most effective when mediated through this intermediate pair representation, which naturally encodes distance information. Triangle multiplication provides a simple and efficient mechanism to do so.
LEDMONLE
This work in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML) and UT-Austin Center for Generative AL. The authors thank Yue Zhao, Sergey Ovchinnikov, Chengyue Gong and Luca Naef for their thoughtful feedback this manuscript.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):495-000, 2024. 1, 2,
Gustaf Ahdritz, Nazim Bouatta, Christina Floristean, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy J O' O'Donnell, Daniel Berenberg, Ian Fisk, Niccolò Zanichelli, et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. Nature methods, 188:11514-5524 2024. 2
Yo Akiyama, Zhidian Zhang, Milot Mirdita, Martin Steinegger, and Sergey Ovchinnikov. Scaling down protein language modeling with msa pairformer. bioRxiv, pp. 2025-08. 2025.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang. Asim K Bera, et al. De novo protein design by deep network hallucination. Nature, 600(7889):577-552 2021. 20
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(5557:877--766 2021.
2
Minkyung Baek, Ivan Anishchenko, Ian R Humphreys, Qian Cong. David Baker, and Frank DiMaio. Efficient and accurate prediction of protein structure using rosettafold2. BioRxiv, pp. 2023-05,
2023. 2
Minkyung Baek, Ryan McHugh, Ivan Anishchenko, Hanlun Jiang, David Baker, and Frank DiMaio. Accurate prediction of protein-nucleic acid complexes using rosettafoldna. Nature methods, 21(1): 117-121, 2024.
Steven A Benner and Dietlinde Gerloff. Patterns of divergence in homologous proteins as indicators of secondary and tertiary structure: a prediction of the structure of the catalytic domain of protein kinases. Advances in Enzyme Regulation, 31:121-181, 1991. 4
Jacques Boitreaud, Jack Dent, Matthew McPartlon, Joshua Meier, Vinicius Reis, Alex Rogozhnikov, and Kevin Wu. Chai-1: Decoding the molecular interactions of life. BioRxiv, 2024. 2
Patrick Bryant and Arne Elofsson. Evobind: in silico directed evolution of peptide binders with alphafold. bioRxiv, pp. 2022-07, 2022. 20
ByteDance, Xinshi Chen, Yuxuan Zhang, Chan Lu, Wenzhi Ma, Jiaqi Guan, Chengyue Gong, Jincai Yang, Hanyu Zhang, Ke Zhang. Shenghao Wu, Kuangqi Zhou, Yanping Yang, Zhenyu Liu, Lan Wang. Bo Shi, Shaochen Shi, and Wenzhi Xiao. Protenix advancing structure prediction through a comprehensive alphafold3 reproduction. bioRxiv, 2025. doi: 10.1 2025.01.08.631967. URL https:/ //tp::////wwww..// www.biioxxii. wttp://////cooeee/eee///////// 11/2025. 01. 08. 631967.2
Yehlin Cho, Martin Pacesa, Zhidian Zhang, Bruno E Correia, and Sergey Ovchinnikov. Boltzdesignl: Inverting all-atom structure prediction model for generalized biomolecular binder design. bioRxiv, pp. 2025-04, 2025. 2, 8, 20
Giannis Daras, Jeffrey Ouyang-Zhang, Krithika Ravishankar, William Daspit, Costis Daskalakis, Qiang Liu, Adam Klivans, and Daniel J Diaz. Ambient proteins: Training diffusion models low quality structures. bioRxiv, pp. 2025-07, 2025. 2
Richard Evans, Michael O'Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Zidek, Russ Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with hyphafold-multinan biorxiv, pp. 2021-10, 2021. 2
Christopher Frank, Ali Khoshouei, Lara Fu,3, Dominik Schiwietz, Dominik Putz, Lara Weber, Zhixuan Zhao, Motoyuki Hattori, Shihao Feng, Yosta de Stigter, et al. Scalable protein design using optimization in relaxed sequence space. Science, 386677)49946 2024. 20
11

---

Published as a conference paper at ICLR 2026
Tomas Geffner, Kieran Didi, Zuobai Zhang, Danny Reidenbach, Zhonglin Cao, Jason Yim, Mario Geiger, Christian Dallago, Emine Kucukbenli, Arash Vahdat, et al. Proteina: Scaling flow-base protein structure generative models. arXiv preprint arXtv:2503.007100 2025.
Casper A Goverde, Benedict Wolf, Hamed Khakzad, Stéphane Rosset, and Bruno Correia. De novo protein design by inversion of the alphafold structure prediction network. Protein Science, 32 (6):e4653, 2023. 20
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. Science, 38766336 2025.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure. NAR Genomics and Bioinformatics, 6(4)) 11 2024. ISSN 2631-9268. doi: 10.10933/aargab/lgee5555 URL https::///// /doi/diooooo org / 10 1093 http///aaggabgggg//////
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint arXiv: 2019. 18
Alexander Ille, Christopher Markosian, Stephen K Burley, Renata Pasqualini, and Wadih Arap. Human protein interactome structure prediction at scale with boltz-2. bioRxiv, pp. 2025-07, 2025.
IntFold, Leon Qiao, Wayne Bai, He Yan, Gary Liu, Nova Xi, Xiang Zhang, and Siqi Sun. Intfold: A controllable foundation model for general and specialized biomolecular structure prediction. arXiv preprint arXiv:2507.02000 2025. 2
Michael A Jendrusch, Alessio LJ Yang, Elisabetta Cacace, Jacob Bobonis, Carlos GP Voogdt, Sarah Kaspar, Kristian Schweimer, Cecilia Perez-Borrajero, Karine Lapouge, Jacob Scheurich, al. Alphadesign: A de novo protein design framework based on alphafold. Molecular Systems Biology,
pp. 1-24, 2025. 20
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-88)) 2021. 1, 2, 3, 6, 16
Alexander Kozachinskiy, Felipe Urrutia, Hector Jimenez, Tomasz Steifer, Germán Pizarro, Matías Fuentes, Francisco Meza, Cristian B. Calderon, and Cristóbal Rojas. Strassen attention: Unlocking compositional abilities in transformers based new lower bound method. arXiv, 2025. doi: 10.48550/ARXI..250..00
URL https: / arxiv org/ abs 2201 112..19211 19215. 22
James Lee-Thorp, Joshua Ainslie, lya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint 1rXiv:2105.03300 2021. 3, 21
Yeqing Lin, Minji Lee, Zhao Zhang, and Mohammed AlQuraishi. Out of many, one: Designing and scaffolding proteins the scale of the structural universe with genie 2. arXiv preprint arXiv:2405. 1548, 2024. 2,
Faruck Morcos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora S Marks, Chris Sander, Riccardo Zecchina, José N Onuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, 108(49):E1293EEI) E1301,, 2011.
John Moult, Krzysztof Fidelis, Andriy Kryshtafovych, Torsten Schwede, and Anna Tramontano. Critical assessment of methods of protein structure prediction (casp)-round x. Proteins: Structure, Function, and Bioinformaticx, 82:1-6, 2014. 24
Jeffrey Ouyang- -Zaan.. Chengyue Gong, Yue Zhao, Philipp Krähenbühl, Adam R Klivans, and Daniel J Diaz. Distilling structural representations into protein sequence models. bioRxiv, pp. 2024-11, 2024. 2
12

---

Published as a conference paper at ICLR 2026
Sergey Ovchinnikov, Hetumandan Kamisetty, and David Baker. Robust and accurate prediction of interactions across protein interfaces using evolutionary information. elife, c02030, 2014. 4
Sergey Ovchinnikov, Hahnbeom Park, Neha Varghese, Po-Ssu Huang, Georgios A Pavlopoulos, David E Kim, Hetumandan Kamisetty, Nikos C Kyrpides, and David Baker. Protein structure determination using metagenome sequence data. Science, 355(6322):294 299-299, 2017. 4
Martin Pacesa, Lennart Nickel, Christian Schellhaas, Joseph Schmidt, Ekaterina Pyatova, Lucas Kissling, Patrick Barendse, Jagrity Choudhury, Srajan Kapoor, Ana Alcaraz- Serna, al. One-shot design of functional protein binders with bindcraft. Nature, pp. 1-10, 2025. 2, 20
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv, 2023. doi: 10.48550/ARXIV22302.0000
URL atttps://arxii
org/abs/ 2302. 10866. 3
Aurko Roy, Timothy Chou, Sai Surya Duvvuri, Sijia Chen, Jiecao Yu, Xiaodong Wang. Manzil Zaheer, and Rohan Anil. Fast and simplex: 2-simplicial attention in triton. arXiv, 2025. doi: 10. 10.48550/ARXIV.5507.00 025504 URL https:
02754. 22
Jeffrey A Ruffolo, Lee-Shin Chu, Sai Pooja Mahajan, and Jeffrey J Gray. Fast, accurate antibody struc- ture prediction from deep learning on massive set of natural antibodies. Nature communications, 14(1):2389, 2023. 3
Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. 37th Conference on Neural Information Processing Systems, 2023. doi: 10.48550/ ARXIV.2306.02896. URL https: /arxiv org/ abs 2306 02896. 22
Valeria Scardino, Juan I Di Filippo, and Claudio N Cavasotto. How good are alphafold models for docking-basse based virtual screening? Iscience, 26(1), 2023. 3
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre. Tim Green, Chongli Qin, Augustin Židek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):76-710, 2020. 1, 2
Yoav Shamir and Nir London. State-of-the-art covalent virtual screening with alphafold3. bioRxiv, pp. 2025-03, 2025.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021. 3
Michel Van Kempen, Stephanic S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature biotechnology. 42(2))243-244 2024.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50KD1):D43--D44 2022. 2
Yuyang Wang, Jiarui Lu, Navdeep Jaitly, Josh Susskind, and Miguel Angel Bautista. Simplefold: Folding proteins is simpler than you think. arXiv preprint arXiv:2509. 18480, 2025. 2
Martin Weigt, Robert A White, Hendrik Szurmant, James A Hoch, and Terence Hwa. Identification of direct residue contacts protein-protein interaction by message passing. Proceedings of the National Academy of Sciences, 106( 106(1):67-72, 2009. 4
Basile IM Wicky, Lukas F Milles. Alexis Courbet, Robert J Ragotte, Justas Dauparas, E Kinfu, Tipps, Ryan D Kibler, Minkyung Back, Frank DiMaio, et al. Hallucinating symmetric protein assemblies. Science, 378(6615):56-4) 2022. 20
13

---

Published as a conference paper at ICLR 2026
Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Noah Getz, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Liam Atkinson, Tally Portnoi, Itamar Chinn, Jacob Silterra, Tommi Jaakkola, and Regina Barzilay. Boltz-1: 1: Democratizing biomolecular interaction modeling. bioRxiv, 2024. doi: 10.1 10.1101/2024.1 11. 19.62446... 1, 2, 6, 18
Jeremy Wohlwend, Mateo Reveiz, Matt McPartion, Axel Feldmann, Wengong Jin, and Regina Barzilay. Minifold:
Simple, fast, and accurate protein structure prediction. Transactions on Machine Learning Research, 2025. 2
Felix Wong, Aarti Krishnan, Erica J Zheng, Hannes Stärk, Abigail L Manson, Ashlee M Earl, Tommi Jaakkola, and James Collins. Benchmarking alphafold-enabled molecular docking predictions for antibiotic discovery. Molecular systems biology, 18(9):e 1081, 2022.
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3): 196-11503, 2020. 2
Charles Yanofsky, Virginia Horn, and Deanna Thorpe. Protein structure relationships revealed by mutational analysis. Science, 146(3651):1593-15))
1964.
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. arXiv, 2021. doi: 10448550ARXIV22105...44 14100.
URL /tttpp://arxiiii org/ abs /alss//15552 14103.
Jing Zhang, Ian R Humphreys, Jimin Pei, Jinuk Kim, Chulwon Choi, Rongqing Yuan, Jesse Durham, Siqi Liu, Hee-Jung Choi, Minkyung Back, et al. Predicting protein-protein interactions in the human proteome. Science, pp. eadt1630, 2025. 3
14

---

Published as a conference paper at ICLR 2026
A
ARCHITECTURAL BASELINES
The full cofolding pipeline for all methods can be found at Figure 10.
MSKQNSDH
MSA
Module
TATAGGCA
Input Embedder
Diffusion Module
Pair
Pairformer
[Mg+2]
Sequences
Single
Structure
(a) Pairformer-based predictor
MSKQNSDH
MSA Module
Pairmixer
TATAGGCA
Input Embedder
Diffusion Module
Pair
[Mg+2]
Sequences
Single
Structure
(b) cerrmixer-based predictor
MSKQNSDH
MSA Module
TATAGGCA
Input Embedder
Diffusion Module
Pair
[Mg+2]
Transformer
Single
Sequences
Structure
(c) Transformer-basee predictor
Figure 10: Overview of biomolecular structure predictors. We study the effect of varying backbone architectures while keeping other modules fixed, except in the Transformer model, where we adjust the connections between the MSA module outputs and the Diffusion module inputs.
A. PAIRFORMER BASELINE
Here describe the Pairformer architecture of Figure 11 in detail.
Attention Primitive. The Pairformer extends the standard attention mechanism by incorporating pairwise bias term derived from the pair representation z. Formmaly, this update
Attn WithPairBias(aa =) softmax ((W
WB2 Wvæ,
where E E LxC,2 is a sequence representation, z E RLXLXC, is a pair representation, (WQ, W,, Wv) are standard attention projection matrices, and WB projects the pair represen- tation into attention bias term
The Sequence Update first performs attention with pair bias (see Figure 11a) and then applies feed-forward network. At layer we compute the update
$ 81 Attn AttnWithPairBinss 21)
81+1 = 8 + FFN(
siiggle head and removed scaling for brevity
15

---

Published as a conference paper at ICLR 2026
x L
2
Triangle Multiplication
Triangle Attention
FFN
Sequence Attention w/Pair Blas
S
FFN
Z
Z
OKT
+
M
of OKTT + ++)
(a) Seq. Attention w/Pair Bias
(h) Triangle Attention
(c) Triangle Multiplication
Figure 11: Pairformer Architecture and Module Details. The main architecture (top) outlines the general Pairformer layer. The detailed module architectures (bottom) illustrate the key components: (a) Sequence Attention with Pair Bias, (b) Triangle Attention, and (c) Triangle Multiplication modules.
The Pair Update mixes the tokens in pair representation _L using triangle attention and triangle multiplication, then applies feedforward network.
The Triangle Attention operates on each row of the pair representation 2; E <-xxx as an independent sequence, applying sequence attention with pair bias to each row separately' (see Figure 11b). Formally, the update for row is defined
TriAttn(2) 2); = Attn WithhPairBias(2..2.
The Triangle Multiplication integrates features across different rows of the pair representation (see Figure lc)). Formally, the update for feature Zıj defined as
TriMul())y
W (.. O (Wozya)
where W., W, are linear projection layers.
Both pair operations were introduced to reason over triplets of residues, intuitively enabling the model learn to follow geometric constraints in 3-D space (Jumper et al., 2021).
A.2 TRANSFORMER BASELINE
Our transformer baseline removes the pair update from the Pairformer and keeps only the sequence update. We also modify the MSA module to make it more effective with the transformer baseline. Instead of outputting only msa it produces additional sequence representation msa obtained by indexing the first row of the processed MSA representation. This & maa is fed into the transformer, while mm serves as the pair bias. Additionally, the diffusion module expects both sequence and pair representations. Because the pair features are otherwise less processed in this baseline, we update them with the outer sum of the sequence representation. Formally,
backbene ij
msa
W backbene W backbone
where W. E IR XC, is projection layer. This is illustrated in Figure 10c.
In practice another layer of triangle attention is performed on the columns. IIn practice another layer of triangle multiplication is performed the columns.
16

---

Published as a conference paper at ICLR 2026
Module Operation
FLOPs
Backbone MSA Module
Pair Update
Triangle Attention
Matrix Multiply
L
Projection
20 C
Triangle Multiplication
EinSum
4 4
Projection
24 C'
Pair FFN
24 L'
Sequence Update
Sequence Attention (with Pair Bias)
Matrix Multiply
ALAC
Projection
10 LC
Sequence FFN
24
Diffusion Transformer
Attention (Pair Bias) Matrix Multiply
A
Attention (Pair Bias) Projection
10 LC
Soquence FFN
16 LC
Full Modules
MSA Module
RDm (12L 6
Pairformer
R Dr (12L"C, + 68L + +
Structure Module
M (4L' 26LC.)
Table 3: Breakdown of FLOPs in AlphaFold3 architectural components. Variables: I, = max max_tokens, C2 token_z, C. token_s, C. 2 token_2, recycles, Dr = pairformer_depth Dm = msa_depth, Da = diffusion_depth, M multiplicity.
B FLOPS CALCULATIONS
Our biomolecular structure predictor uses multi-resolution transformer that denoises atom coordi- nates at both the token and heavy-atom levels (see Figure 2). this design, a backbone refines token representations, which are then processed by conditional diffusion transformer. The backbone runs once per sequence, while the diffusion transformer can generate arbitrarily many samples.
In Table 3, present the mathematical FLOP calculations for each component, and in Table 10 report the total training and inference FLOPs for all model architectures.
Boltz-1 Hyperparameters The Boltz-1 I architecture defined by several key components and hyper- parameters that influence its performance. We identify the following set of critical hyperparameters:
Input: The input is defined by the number of input tokens (L), the single token dimension (C.), and the pair token dimension (Cz).
Feature extractor: The feature extractor consists of Pairformer and MSA blocks that process single and pair representations; its configuration is determined by the number of Pairformer blocks Dp, MSA blocks Dm-
Diffusion model: The diffusion model is a transformer architecture made up Multi-Head Attention (MHA) transformer layers. Its configuration is determined by the number of diffusion blocks (Da) and the widths of its layers C, = 2C..
Feature extractors. The feature extractors is a concatenation of Dm, MSA blocks and D, pairformer blocks. Each pairformer block primarily consists two parallel update paths: the pair representation path and the single representation path (see Figure 11). Each path is further processed by FFN. The pair representation path includes two triangular self-attention updates and two triangular multiplication updates (applied row-wise and column-wise). These analogous to axial attention
17

---

Published as a conference paper at ICLR 2026
mechanisms (Ho al., 2019) operating L x L pair matrix, where each attention pass involves computations along one length-L L dimension for each of the L rows or columns.
Each pair of triangular attention pass incurs computational cost of Cg,L9C,) FLOPs. The triangle multiplication einsum operations require a quadratic FLOPs term per input token (total FLOPs of CT) Following the triangle updates, feed-forward network (FFN) is applied each pair representation entry. The single representation path also contributes to the computational load, but its cost quadratic in L.
Each MSA block is lighter than the full pairformer blocks and consists of a pair of triangular attention layers and pair of triangular operations, followed by FFN network for pair representation FFN (C2), but without single representation FFN and attention with pair bias. It also includes an additional OuterProductMean and pair-weighted averaging on the MSA, which we omit from our FLOPs calculations.
Diffusion Model. Each diffusion module block resembles a standard transformer block with a standard self-attention mechanism and a conditioning block. As with the trunk block analysis, we ignore bias terms, gating, and layer normalization for simplicity. Atom Attention Encoder and Atom Attention Decoder that sequence-lood loccll attention (Wohlwend et al., 2024) and their computational cost is negligible. The conditioned transition block of the diffusion model is dominated by dense matrix multiplications that scale quadratically with the hidden size Ca. The bulk of the compute arises from the SwiGLU feed-forward pathway, which contributes both pair of linear projections (4 C.) and the associated activation matmul (2 (2). In addition, cross cross-path transformations are introduced via the a - b and b + projections (each 2 C2), followed by an output projection (2). Finally, the gating mechanisms for both the and b streams contribute another 2 C² apiece. The total FLOPs per structure block can therefore be approximated as the sum of the attention, MatMuls, and feed forward components (see Table 3).
also ignore the cost of atoms, since those modules adopt
C ADDITIONAL RESULTS
C.I SYSTEM-LEVEL COMPARISONS ON THE CASP15 DATASET
We report results on the CASP15 dataset in Table 12. These numbers differ slightly from Table because further filter proteins to ensure all methods evaluated on the same set.
CASP15 Performance
0.8
0.47
0.6-
3.24
0:21
0.4-
G26124
0.2
Mean LDDT
DockQ 0.23
Mean LDOT-PLI (n=12)
L-RM5D 24
(n=66)
(n=13)
(n-12)
AF3 oracle
Chai-1 oracle
Boltz-1 oracle
Pairmixer (Ours) oracle
AF3 top-1
Chai-1 top-1
Boltz-1 top-1
Pairmixer (Ours) top-1
Figure 12: System-level comparison on the CASPI5 test set. We evaluate against AlphaFold3, Chai- and Boltz-1 on protein and small-molecule structure prediction. Pairmixer performs competitively with these state-of-the-- art approaches. Error bars denote bootstrapped 95% confidence intervals. Since we do not train a confidence model, results are reported using the first prediction.
18

---

Published as a conference paper at ICLR 2026
C.2 FULL BIOMOLECULAR STRUCTURE PREDICTION RESULTS FOR RCSB AND CASP15
Table 4 and Table 5 report the full set of evaluation metrics across all architectures, along with the number of complexes evaluated by each metric. We retrain Boltz-1 for Pairformer baselines and additionally include comparisons against the public checkpoint.
Table 4: Model Performance on the Boltz RCSB test The metric computed on the best- performing protein out of five samples (oracle).
Archilecture
Epoch
GPU-Days
IDDT (ned39)
DOCKQ
DOCKQ 0.49 (Qx342)
IDDTuu (S=250)
RMSD (n=250)
RMSD <2 (Q=220)
Smnall
Lrussoomer
86
0.68
0.35
0.47
0.43
Pairforter
125
Pairatixer (Curs)
68
98
0.73
0.59
0.44
0.51
0.33
0.45
Medium
Transfomeer
68
128
0.67
0.50
0.36
0.47
0.33
0.46
Pairformr (Boltz-))
194
0.75
0.47
0.53
0.36
0:19
Pairniser (Ours)
Large
Transfurmer
173
0.69
0.51
0.37
0.48
0.33
0:46
Pairformer (Bolte- ()
68
200
0.76
0.61
0.49
0.54
0.41
0.52
Puirmixer (Ours)
192
0.75
0.61
0:46
0.55
0.38
0.51
Phase
Traesformmr
232
0.38
0.48
Pairforee (Rolts-))
0.78
0.50
0.57
Pairmisur (Ouns)
20
269
0.78
0.63
0.49
0.57
0.45
0.55
Boltz-1 public nsodel Puirfomer (Belz-1)
0.79
0.64
0.51
0.58
0.46
0.57
Table 5: Model Performance on CASP15 test set. protein of five samples (oracle).
The metric is computed on the best-performing
Architecture
Epoch
GPU-Days
DOCKQ 23
DOCKQ (n-14)
IDDTE (e-12)
RMSD 21 (n-12)
RMSD 22
(n-66)
[er 14)
(a-12)
Small
Transfurmer
0.35
0.22
0:06
Painformer (Dolte: 1)
68
125
0.39
0.46
0.24
0.36
0.10
0.21
Paimixer (Ouns)
68
0.37
0.39
0.21
0.35
0.06
0.16
Medium
Treasfomer
68
0.39
0.04
Painformer (Bolte-)) 1)
0.38
0.66
0.35
0.14
Paimiixer (Qurs)
68
146
0.39
0.49
0.39
0.38
0.12
0.34
Large
Traasfomer
68
173
0.56
0.29
0.16
0.26
0.06
0.10
Paiffommer (Boltz
68
290
0.41
0.68
0.43
0.12
(Qurs)
0.50
Large
Transfomeer
20
232
0.37
0.34
0.17
0.26
0.11
0.11
Painformer (Bolte- l)
20
421
0:42
0.64
0.43
0.36
0.10
0.28
Paimiser (Ours)
20
269
0.41
0.52
0.14
Bolte-1 public model
Painformer (Bulte:
0.68
C.3 DETAILS FOR DIVERSE BIOMOLECULAR STRUCTURE PREDICTION
We evaluate Pairmixer across several benchmark datasets listed in Table 1. Below, we describe the dataset preparation and evaluation protocols used for these benchmarks.
Protein-ligand complexes. We evaluate performance pigted complexes using the Pose- Buster benchmark. The original dataset contains 428 complexes. Applying a training-date cutoff of September 30, 2021 reduces this to 373, and after removing redundant -otteinnligand complexes, the final benchmark includes 298 structures. Evaluation uses standard Mineiin-lgaand metrics, including RMSD 2A, RMSD < IÅ, and pigtein-ligand IDDT. We compare Pairmixer model finetuned for longer to the publicly available Boltz-1 checkpoint with Pairformer in Table 1a. Under this setup, Pairmixer performs comparably to Pairformer, with at most a 1 1% drop in performance, while the architecture is significantly simpler and more efficient, requiring attention in the backbone.
Antibody-antigen complexes. We evaluate on the AlphaFold3 paper. First, we extract the relevant chains from the publicly released AlphaFold3
benchmark introduced in the
19

---

Published as a conference paper at ICLR 2026
(a) Pairformer-based predictions
(b) Pairmixer-based predictions
Figure 13: Qualitative visualizations of de-novo binders. Target is shown in green and binder is shown in blue. PDB code: IP5J
files and remove unresolved residues. Of the 71 total complexes, 70 pass our data pipeline. The protein-protein interface evaluated using the DockQ > 0.23 metric. We compare large Pairformer, Pairmixer, and Transformer models trained for the same number of iterations in Table 1b. We find that Pairmixer matches the performance of Pairformer (0.23), while the sequence-only Transformer performs substantially worse (0.08).
Protein-nucleic acid and RNA-only complexes. We evaluate our models on the protein-nucleic acid dataset from the AlphaFold-3 paper. Of the 199 structures, 172 pass our data pipeline. We also consider a subset of 27 RNA-only structures. For RNA-only complexes, we assess folding quality using IDDT, while for puulein-nucleic acid complexes Contact Similarity (ICS) and Interface Patch Similarity (IPS). We compare large Pairformer, Pairmixer, and Transformer models trained for the same number of iterations in Table 1d and Table 1c. On protein-nucleic acid complexes, Pairmixer performs comparably to Pairformer, with the Transformer lagging behind. For RNA-only structures, Pairmixer again matches Pairformer, while the Transformer performs better, likely due to limited RNA structural training data. Notably, Pairmixer achieves performance comparable Pairformer despite removing sequence attention from the trunk.
evaluate interface accuracy using Interface
C.4 DETAILS OF PAIRMIXER APPLIED TO PROTEIN DESIGN
BindCraft (Pacesa et al., 2025), BoltzDesign (Cho et al., 2025), and hallucination- based protein design methods (Frank et al., 2024; Wicky et al., 2022; Jendrusch et al., 2025; Goverde et al., 2023; Bryant Elofsson, 2022; Anishchenko et al., 2021) have recently demonstrated that structure predictors can be repurposed as differentiable scoring functions for sequence optimization. The input sequence treated as a set of learnable parameters and is updated by backpropagating through a structure predictor, thereby jointly refining sequence and structure toward favorable interactions with the target protein or small molecule. While powerful, these methods have practical limitations: memory demands are high and sequence generation is slow, requiring hundreds of runs of the structure predictor per design. This inefficiency makes the approach prohibitively expensive, particularly for larger systems.
To address these challenges, we introduce BindFast, a scalable and efficient framework for binder design which replaces BoltzDesign's Pairformer backbone with Pairmixer. BindFast substantially reduces the runtime and memory footprint of binder generation and aim to accelerate the discovery of higl-quality binders, particularly for large targets.
In Table 2, we benchmark the runtime performance of BindFast against BoltzDesign for generating 110-residue binders across a range of target proteins with biotechnological relevance, using an A GPU with 80 GB memory. BoltzDesign failed with oft-of-memory (OOM) errors on targets larger than 500 residues, whereas BindFastt extended this limit to 650 residues, 30% improvement in target size. For protein targets where both models executed without memory overflow, BindFast achieves speedups of 2x to 2.6x at total sequence lengths ranging from 140 to 607, respectively. Qualitative comparisons in Figure 13 further indicate that BindFast produces designs comparable to those of BoltzDesign, underscoring its potential for faster in-silico iteration and enabling the design of binders against larger, more biologically relevant targets.
20

---

Published as a conference paper at ICLR 2026
Table Pairformer Ablation. We remove each module in the Pairformer one time.
Ablation
GPU days
IDDT
DOCKQ 20.23
IDDTpLI
RMSD
82
0.74
0.57
0.52
0.50
No Seq Update
0.73
0.57
0.54
0.49
No Tri Att
66
0.70
0.55
0.50
0,48
No Tri Mul
71
0.70
0.53
0.49
0.46
D ADDITIONAL ANALYSIS
D.I PAIRFORMER ABLATIONS
We performed ablation experiments on a small 12-layer Pairformer model to isolate the contributions of triangle multiplication, triangle attention, and sequence updates in Table 6. The results show that, under short training schedule of 60 epochs (3M samples), both triangle multiplication and triangle attention are essential for performance, while sequence updates have minimal impact. Notably, Puirmixer recovers performance with additional training.
D.2 ADDITIONAL ABLATIONS
Triangle Multiplication Feed-Forwart Network. We aim understand how the performance is affected by the triangle multiplication and pair forw-fo networks, the two core ingredients of the Pairmixer architecture. In Table 7a and Table 7b, we vary the hidden dimensions of these components to evaluate model's sensitivity. For the FFN, we change the hidden dimension that the model expands to. For triangle multiplication, we instead project the features into higher- lower- dimensional spaces before the multiplication and then project them back to the input dimension. We find that decreasing the FFN hidden dimension does not change performance much, while doubling the FFN dimension increases the mean IDDT from 0.71 to 0.74. We see similar trend with triangle multiplication dimensions doubling the hidden dimension improves the mean IDDT from 0.71 to 0.73, while reducing the dimensionality does not change IDDT.
Other mixing methods. Triangle multiplication mixes features within the z E LXLXD pair representation. In Table 7c, we replace this operation with alternative, simpler mixing functions. First, we ablate the outgoing triangle multiplications, retaining only the incoming variant. Second, we introduce an FFT mixer that applies the discrete Fourier transform along rows and columns, following FNet (Lee-Thorp Thorp et al., 2021). Finally, we test pooling mixer that averages representations across each row (and column) and adds the result back to positions along the corresponding axis.
We find that these simplified approaches are insufficient and underperform compared to vanilla triangle multiplication. For instance, the FFT mixer likely fails because it mixes features solely based on sequence position, ignoring discontinuities introduced by multiple chains.
Table 7: Pairmixer ablations experiments. Default settings are marked in grey, See Section D.2 for details. Dp: number of pairmixer layers. Dy: number of diffusion transformer layers.
(a)
FFN Hidden Dimension
(b) Triangle Mul Dimension
dim
IDDT
DOCKQ>0..t as IDDTuu
RMSD
dim
DOCKQ so IDDTru
RMSD
256
0.71
0.38
0.50
0.34
64
0.71
0.41
0.50
0.34
512
0.71
0.42
0.50
0.33
128
0.71
0.42
0.50
0.33
1024
0.74
0.40
0.53
0.35
256
0.73
0.42
0.52
0.37
(c) Mixing Method
(d)
Diffusion Transformer Depth
mixer
IDD
DOCKO IDDTu
RMSD
Dp
D2
IDT
DOCKO IDDTuu
RMSD
FFT
0.66
0.34
0.45
0.27
12
12
0.73
0.43
0.52
0.34
AvgPool
0.69
0.35
0.31
24
24
0.75
0.45
0.54
0.40
TriMul-rows
0.70
0.35
0.49
0.32
TriMul-both
0.71
0.42
0.50
0.33
21

---

Published as a conference paper at ICLR 2026
Diffusion Module. The diffusion module takes the latent representations as input and decodes the 3-dimensional protein structure using 24-layer transformer. Table 7, we evaluate how sensitive the Pairformer and Pairmixer architectures are to the size of the diffusion module.
D.3 TRIANGLE MULTIPLICATION APPLIED THE MATCH3 TASK
To separate the effect of quadratic pair representations from the cubic cost of triangle operations, we evaluate these components on the Match3 task. benchmark for learning 3-way interactions (Sanford et al., 2023; Kozachinskiy et al., 2025). Match3 gives the model sequence x E [M] and asks whether any triple of distinct elements sums to zero mod M. We use = 16, M 64, a hidden dimension of 8, comparable parameter counts (900-1 100), and standard embedding, projection, and max-pooling layers, training on balanced datasets.
We compare three architectures that separate representational and computational factors: standard self- attention (linear representations, quadratic compute), third-order self-attention (linear representations, cubic compute (Roy al., 2025)), and triangle multiplication (quadratic pair representations, cubic compute). This setup isolates whether performance gains stem from the richer pair representation or from cubic-order computation.
Across data regimes, all architectures struggle under extreme data scarcity. However, as data and depth increase, standard self-attention consistently lags behind both cubic-compute methods (see Figure 14). Notably, triangle multiplication on quadratic pair representations outperforms both variants of Transformer-style attention in shallow settings, demonstrating a particular advantage in capturing nonlocal 3- 3-token interactions even when computational budgets are matched.
Mode: Depth: Layer
Model Depen: Layers
2nd
Trienglc muliplication
Accomnyy
0.70
0,70
0.00
0.58
0.50
0.80
Number
Number
Figure 14: Comparison between architecture variants on Match3. We report classification accuracy on Match3 task function of training data size and model depth.
22

---

Published as a conference paper at ICLR 2026
D.4 DISSECTING PAIRMIXER PERFORMANCE ON THE RCSB TEST SET
To gain deeper insight into these results, we analyze Pairmixer, Pairformer, and Transformer perfor- mance across the RCSB test through two complementary perspectives.
We first examine the correlation between Pairformer and Pairmixer performances. In Figure 15, each point represents single structure, with coordinates indicating the respective model's performance. The strong correlation between the two models, with minimal outliers, suggests the two architectures share similar failure and success modes. Furthermore, Pairmixer outperforms Pairformer on IDDT in 44.9% of the test cases, demonstrating near- eaar-equivalent predictive capability despite the architectural simplification.
Next, we investigate whether Pairmixer's competitive performance is confined to favorable conditions, specifically, short sequences or proteins with abundant homologous sequences. We partition the RCSB test set by sequence length and MSA depth, then evaluate all three models across these stratified subsets in Figure 16. As expected, all models achieve higher accuracy on shorter proteins and those with richer MSA information, while accuracy degrades for longer sequences and sparser alignments. Critically, these performance trends remain consistent across architectures, with Pairmixer maintaining parity with Pairformer across all difficulty regimes.
IDDT
RMSD
DockQ
IDDT-PLI
1.00
1.0
0.75
40
Palmmier
Pairmixer
Falmixer
Pairmixer
0.50
0.5
0.5
0.25
Win Rate:
44.9%
Win Rate: 45.8%
0.0
Win Rate: 51.2%
0.0
Win Rate: 40.0%
0.25
0.50
0.75
1.00
20
0.0
0.5
0.0
0.5
1.
Boltz-1
Boltz-1
Boltz-1
Boltz-1
Figure 15: Head-to-head comparison between Pairmixer and the Pairformer backbone. The win rate shows how often the Pairmixer architecture achieves a better score than the Transformer architecture, 89% of cases, two models' IDDT scores differ by less than 5 points.
LDDT
pocko
RMSDx2
0.5
Weon LOOT
Mean
8.65
2003
1305
2003 1200 15000
2003
5330
12300 1500
2008 2835 Depth
1204
12033
300
(33) J00K 1206 12403
Pa miiur (Dung
Paitommr (Bazz31
Tansforer
(a) Performance stratified by MSA depth. LDOT
LOOT
Teat DOCKO
RMSD-2
Moun LOOT
Number Raniduss 2013 250
2330
2830
452
Number Realdues
B5 2200
402 123 2230
Number Residues
Pririeer Y0urs
Paiffomer
Tareforner
(b) Performance stratified sequence length.
Figure 16: Performance across different data difficulty metrics. Pairmixer maintains comparable performance to Pairformer across all difficulty levels.
23

---

Published as a conference paper at ICLR 2026
D.5 DISTOGRAM PERFORMANCE
A potential confounding factor in evaluating Pairmixer is that the diffusion module may correct errors in lower-quality backbone outputs, producing high-quality structures independently of the backbone. To isolate the backbone's contribution, we evaluate distogram predictions on the RCSB test set. The distogram head predicts 64-bin discretized distance matrix directly from backbone features buckbone, and accuracy is measured both across the full system and between chains in Table 8a. Following standard contact prediction evaluation (Moult al., 2014), we also report precision and L/5 in Table 8b. Across all metrics, Pairmixer performs comparably to Pairformer, suggesting that the Pairmixer with triangle multiplication and feed-forward networks produce equally expressive backbone features.
Table 8: Distogram Prediction Performance.
(a) Global and Inter-Chain Accuracy
Method
Global
Inter-Chain
Top-1 1 Acc
Top-5 Acc
Top-1 Acc
Top-5 Acc
Pairformer (Boltz-1)
0.74
0.89
0.67
0.73
Pairmixer (Ours)
0.73
0.88
0.67
0.73
Transformer
0.72
0.86
0.67
0.72
(b) Contact Prediction
Method
Short
Medium
Long
P@L
P@L/5
P@L
P@L/5
P@L
P@L/5
Pairformer (Boltz-1)
0.72
0.75
0.72
0.76
0.73
0.81
Pairmixer (Ours)
0.72
0.75
0.72
0.76
0.73
0.80
Transformer
0.69
0.72
0.69
0.74
0.70
0.79
D.6 SENSITIVITY ANALYSIS TO NUMBER OF RECYCLING STEPS
To ensure Pairmixer's performance not solely due to greater benefits from recycling, we evaluate Pairformer, Pairmixer, and Transformer with 0, 1, 3, and 10 recycling steps in Table 9. All models use the large setting (48 layers) and are trained for the same number of iterations. Pairformer and Pairmixer achieve similar results across different numbers of recycles, suggesting that Pairmixer's comparable performance not simply result additional recycling.
Table 9: Impact of Recycling Steps. Performance increases but quickly saturates for all architectures.
Architecture
Recycles
IDDT
DOCKQ
IDDTpr
RMSD <2
Pairformer (Boltz-1 1)
0
0.75
0.59
0.55
0.53
Pairformer (Boltz-1 1)
1
0.77
0,59
0.58
0.56
Pairformer (Boltz-1)
0.78
0.62
0.57
0.55
Pairformer (Boltz-1) 1)
10
0.78
0.64
0.57
0.54
Pairmixer (Ours)
0
0.74
0.59
0.54
0.52
Pairmixer (Ours)
1
0.76
0.61
0.56
0.56
Pairmixer (Ours)
3
0.77
0.61
0.56
0.54
Pairmixer (Ours)
10
0.78
0.63
0.57
0.55
Transformer
0
0.62
0.40
0.48
0.47
Transformer
1
0.67
0.49
0.51
0.48
Transformer
3
0.70
0.52
0.52
0.49
Transformer
10
0.70
0.53
0.51
0.48
24

---

Published as a conference paper at ICLR 2026
E MODEL HYPERPARAMETER,
Table 10 includes a thorough list of the hyperparameters used for our experiments. This table additionally includes the training FLOPs for all model architectures and sizes.
Table 10: Model Hyperparameters. Dashes (-) indicate same value as the previous column. The large model is trained with smaller crops and mixed data, then with larger crops and PDB-only data.
Hyperparameter
Small
Medium
Large Stage
Large Stage
Model Architecture
Number of Backbone Layers
12
24
48
48
Number of MSA Layers
4
Token representation dim (C.)
384
Pair representation dim (C.)
128
Backhone dropout
0.25
MSA Module dropout
0.15
Number Diffusion Layers
24
24
24
Atom representation dim
128
Atom pair representation dim
16
Training
Optimizer
Adam
Maximum learning
1.8 10
Diffusion multiplicity
16
Recycling
0,1,2,3
Epochs
68
68
68
20
Training Samples
6.8M
6.8M
6.8M
2M
Data Processing
Data source
PDB + OpenFold
PDB
Maximum tokens
384
384
384
512
Maximum atoms
3,456
3,456
3,456
4,608
Maximum MSA sequences
2,048
Samples per epoch
100,000
Total Batch size
128
Inference
Number of sampling steps
200
Maximum MSA Sequences
4096
Recycling
10
Diffusion samples
5
Training Infrastructure
GPU Type
H200
Number GPUs
32
32
32
64
Total Training FLOPs
Boltz-1 1 (Pairformer)
8.306c+19
1.467c+20
2,707c+20
1.572c+20
Pairmixer
4.817e+19
8.557e+19
1.572e+20
8.716e+19
Transformer
5.784e+18
7,888e+18
8.941e+18
4,205e+18
Inference FLOPs
Boltzl (Pairformer)
9,100e+15
1.595e+16
2.964e+16
Pairmixer
4.474e+15
7.849e+15
1.460e+16
Transformer
4.137c+14
4.975c+14
6.652c+14
25


Reviews:
[{"review_id": "QSbjp62ACj", "reviewer": "Reviewer_qqDT", "summary": "This paper introduces a new operator, Triangle Multiplication, as a core mechanism for relational reasoning and geometric representation learning. The method is presented as a lightweight yet expressive alternative to traditional self-attention, aiming to capture higher-order interactions among triplets of entities efficiently. The authors apply this operation to tasks, showing that it can achieve competitive or improved performance compared to transformer-style baselines.\n\nThe conceptual idea is creative and well-motivated; however, the presentation lacks sufficient technical clarity, the experimental scope is somewhat limited, and the empirical analysis does not fully demonstrate the operator’s claimed generality.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tInteresting conceptual direction: The idea of moving from pairwise attention to triangle-based relational modeling is novel and aligns with emerging research on geometric and higher-order attention.\n2.\tSimplicity of the operator:  The formulation is elegant and could potentially be a computationally efficient substitute for attention in specific contexts.\n3.\tPotential for extension: The proposed mechanism could inspire further work in 3D molecular or graph-structured domains, where triplet relations are natural.\n4.\tReadable overall motivation:  The high-level rationale and related work are generally well-written.", "weaknesses": "- Limited Comparative Breadth\n\nThe experiments benchmark against a few baselines but omit several directly relevant contemporary models, including:\n\t1. Higher-order attention variants (e.g., Tensor Attention, Relational Transformer)\n\t2. Geometric and 3D reasoning frameworks (e.g., SE(3)-Transformer, EGNN)\n\t3. Diffusion-based relational models and equivariant graph networks.\n\nWithout these comparisons, it is difficult to judge whether Triangle Multiplication provides a fundamentally better abstraction or merely a reparameterization of higher-order attention.\n\n- Lack of Theoretical Clarity\n\nThe mathematical definition of “Triangle Multiplication” is presented at a high level but lacks rigorous derivation or clear connection to known tensor operations:\n\t1. The operator’s expressive power (what functions it can approximate) is not discussed.\n\t2. No complexity analysis is provided—readers cannot tell if it scales better than self-attention for large N.\n\n- Weak Empirical Validation\n\nWhile the experiments show some improvement, they remain qualitative and dataset-limited:\n\t1. The selected tasks are small-scale and do not reflect real-world complexity.\n\t2. No ablation studies are provided to isolate the contribution of the triangle operator vs. other components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZjLt1O7zIZ", "reviewer": "Reviewer_r3YY", "summary": "The paper proposes Pairmixer, a modified AlphaFold3-style backbone that (a) deletes triangle attention and sequence updates from the Pairformer backbone, (b) keeps only triangle multiplication + pairwise FFNs to update the pair representation, and (c) leaves the single-sequence representation unchanged and feeds it directly to the downstream diffusion module. The experiments are shown with maintaining folding / docking accuracy while significantly improving speed (up to 4× faster inference on 2048-token sequences and 34% less training compute), and unlock downstream design workflows that previously ran out of memory.", "questions": "**Missing sequence update ablation:**\nThe paper removes sequence updates but doesn’t isolate their effect. It’s unclear how much of the performance change comes from dropping triangle attention versus removing sequence updates.\n\n**Lack of discussion on limitations vs AlphaFold3:**\nCan authors discuss where the simplified model may fail compared to full AlphaFold3 — for example, in modeling highly flexible regions, RNA–protein complexes, or subtle side-chain rearrangements requiring long-range attention?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Reviewer appreciates the following contributions:\n\n- **impactful and practical**: the paper addresses a real bottleneck in AlphaFold-style models by removing triangle attention for faster, more scalable inference and training. Furthermore, it enables long-sequence and large-complex modeling that was previously infeasible due to memory or computational limits.\n\n- **Empirical validation**: Demonstrates near-identical accuracy to AlphaFold3-class baselines across folding, docking, and binder design tasks, with up to 4× speedup and lower memory use. Experiments include comparisons on Boltz-1, Transformer, and ablations, FLOPs analysis, and realistic large-scale design benchmarks.\n\n- **Simple but be efficient**: the paper shows that triangle multiplication alone suffices for capturing higher-order geometric consistency, providing a clearer understanding of what inductive biases matter. In particular, the final architecture will be the form of *Pairmixer = triangle multiplication + FFN over z, recycle it N times*. \n\n- **presentation**: the paper is well-written and easy to follow.", "weaknesses": "- **Method Novelty**: Novelty somewhat incremental: Prior works (e.g., Genie2, MiniFold) already suggested triangle multiplication is key; this paper mainly extends the idea to AF3 scale rather than introducing it conceptually. This requires further analysis to highlight key differences between Pairmixer versus prior works, for e.g., what should we do to adapt for the protein structure design task?\n\n- **Limited generalization tests:**\nWhile the paper benchmarks extensively on protein–protein and protein–ligand systems, all evaluations remain within domains similar to the training distribution of Boltz-1 (PDB-scale protein complexes). The work does not assess generalization to more diverse biomolecular systems, such as RNA–protein assemblies, RNA-only structures, metalloproteins, or highly flexible/transient complexes. These categories often require different geometric reasoning and long-range constraints, where triangle attention might still provide advantages. Without such tests, it’s quite unclear whether the proposed architecture truly generalizes beyond well-structured protein complexes, or if its performance degrades on systems with **unconventional topologies or more dynamic conformational behavior**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IvCyt32JN", "reviewer": "Reviewer_GgDp", "summary": "This paper addresses the computational efficiency issue in AlphaFold and argues that the triangle attention operation contributes little to the final predictive performance while introducing significant computational overhead. Based on this observation, the authors propose the PairMixer block, a simplified variant of the Pairformer block used in AlphaFold3, in which operations with minimal performance benefits are removed. Experimental results on the RCSB test set demonstrate that PairMixer achieves comparable performance to the original Pairformer while offering up to a 4x improvement in inference speed.", "questions": "No further questions. Please see the “Weaknesses” section for details.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper focuses on an important research problem, i.e., to accelerate AlphaFold and make it more lightweight, which is critital for down-stream applications like virtual screening;\n2. This paper is well-written and clearly-structured. The figures effectively support the understanding of the proposed method, and the authors provide sufficient background and preliminaries to contextualize their work.", "weaknesses": "1. The contribution of this paper appears limited. The proposed method can be viewed primarily as an engineering optimization of the original Pairformer, without introducing substantial new insights. Without deeper analysis or justification of the design choices, the current contribution may not meet the novelty threshold typically expected for a venue such as ICLR. Furthermore, the finding that the triangle attention module contributes minimally to performance is not particularly surprising; this has been informally noted by several researchers through ablation studies, even though such observations have not been formally published.\n\n2. The experimental evaluation of PairMixer is insufficient. The results are reported only on the RCSB dataset (533 structures), which limits the generalizability of the conclusions. It is recommended that the authors adopt a broader evaluation protocol, such as the one used in Boltz-2, to strengthen the empirical validation of their method.\n\n3. The use of the phrase “all you need” in the title, while common in machine learning literature, is not appropriate in this context. The PairMixer/Pairformer serves only as the model trunk within a structure prediction framework, whereas the diffusion module plays an equally important role. Therefore, the current title may overstate the scope and completeness of the proposed contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpvXc0GfSk", "reviewer": "Reviewer_J8sg", "summary": "This work applies triangle multiplication in the large scale of current biomolecular structure prediction. Such application scenario has not been investigated with models using triangle multiplication. The authors show that, for the Pairformer backbone, when maintaining triangle multiplication, and omitting triangle attention, the training and inference efficiency can be improved.", "questions": "Whether there is any non-trivial part for deleting triangle attention and other modules? This could be a potential methodological contribution if there is any.\n\nSuggestion: If my evaluation remains consistent after I read the authors' response, I would suggest that the author considers submitting this work to an application-oriented venue, e.g., Nat. Comm., Nat. Comp. Sci., Sci. Adv. The value of this work is mostly about the application results but without meaningful methodological insights. After giving stronger application evaluations, packing as a tool box or executable empirical platform, this work is more suitable to those top application-oriented venues. The methodological novelty, in my opinion, is enough for them. \n\nI am also open to change my mind if the author can prove the methodological value that I failed to see.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written, clearly depicting the methodology;\n2. The training cost and inference speed are both improved;\n3. The large-scale biomolecular structure prediction scenario is of great practical importance;\n4. The experiments are sufficient.", "weaknesses": "1. This work lacks of methodological contribution. The authors omit other modules and maintaining only the triangle multiplication. This modification is considered very trivial and not that novel. The importance of triangle multiplication has already been investigated by several proceeders, as the authors themselves claimed in the paper. The author is the first one deleting triangle attention for PairFormer.  The main contribution is testing if we can achieve better performance-efficiency trade-off using only triangle multiplication. \n2. The efficiency improvement is not that satisfying. To be fair, it is adequate when the methodological contribution is enough. But since the contribution is minor, I would expect giant efficiency leap to complement the limited contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)