Generating Highly Designable Proteins with Geometric Algebra Flow Matching
Authors: Simon Wagner
Leif Seute
Vsevolod Viliuga
Nicolas Wolf
Frauke Gräter
Jan Stuehmer
Venue: NeurIPS 2024 poster (2024)
Keywords: Proteins; Flow Matching; Geometric Algebra; Generative models; Equivariant models; Protein design; AlphaFold; Local frames
Primary area: generative_models
TL;DR: Geometric Algebra based architecture for protein structure that achieves high designability and structural diversity as flow matching model for protein generation.

Abstract: We introduce a generative model for protein backbone design utilizing geometric products and higher order message passing. In particular, we propose Clifford Frame Attention (CFA), an extension of the invariant point attention (IPA) architecture from AlphaFold2, in which the backbone residue frames and geometric features are represented in the projective geometric algebra. This enables to construct geometrically expressive messages between residues, including higher order terms, using the bilinear operations of the algebra. We evaluate our architecture by incorporating it into the framework of FrameFlow, a state-of-the-art flow matching model for protein backbone generation. The proposed model achieves high designability, diversity and novelty, while also sampling protein backbones that follow the statistical distribution of secondary structure elements found in naturally occurring proteins, a property so far only insufficiently achieved by many state-of-the-art generative models.

Generating Highly Designable Proteins with Geometric Algebra Flow Matching
Simon Wagner" 1,2 Leif Seute" 1,2,3 Vsevolod Viliuga 1,3,4 Jan Stühmer
Nicolas Wo.f
Frauke Gräter
Heidelberg Institute for Theoretical Studies, Heidelberg, Germany IWR, Heidelberg University, Heidelberg. Germany
3 Max Planck Institute for Polymer Research, Mainz, Germany SciLLifeLab and DBB at Stockholm University, Stockholm, Sweden SIRR, Karlsruhe Institute of Technology, Karlsruhe, Germany
Abstract
We introduce a generative model for protein backbone design utilizing geometric products and higher order message passing. In particular, we propose Clifford Frame Attention (CFA), an extension of the invariant point attention (IPA) archi- tecture from AiphaFold2, in which the backbone residue frames and geometric features are represented in the projective geometric algebra. This enables to con- struct geometrically expressive messages between residues, including higher order terms, using the bilinear operations of the algebra. We evaluate our architecture by incorporating it into the framework of FrameFlow, model for protein backbone generation. The proposed model achieves high des- ignability, diversity and novelty, while also sampling protein backbones that follow the statistical distribution of secondary structure elements found in naturally occur- ring proteins, property far only insufficiently achieved by many state-of-the--rt generative models.
state-of-in of-te-of-thh-art flow matching
Introduction
Recent years have shown tremendous progress applying deep learning computational chemistry, where applications of learning-based approaches have enabled unprecedented progress across a broad range of problems, such as molecular property prediction [55]
28
50
protein-ligand
docking [22]. protein structure prediction 322 B 38). and de novo protein design
663
68
10
67
37
In case of protein design, state-of-the-art methods typically represent the structure of a protein of N residues as an element of SE(3)~ i.e. as a collection of N frames, each of which describes the position and orientation of individual protein residue. Among the most successful methods are those based on diffusion models [53 54
37 1 and flow matching [39 19 56 67], which make use of architectures that incorporate the invariant point attention (IPA) of AlphaFold2 [2). By modeling protein through the frames of its backbone, the task of protein structure generation is reformulated as to model the distribution of a set of frames, an inherently geometric problem.
63 68
Advances in geometric deep learning have lead to architectures that are equivariant towards rotations and translations [14 21 15), which can be regarded as geometric inductive bias that enhances performance and data efficiency. Most generative models for protein design achieve equivariance by using geometric features that are expressed in the canonical local coordinate frames representing the protein backbone. The coordinates of these features are thus invariant with respect to global rotations and translations, which allows apply general layers and non-linee non-linearities.
Equua contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

AC
A
c
C
Co
N
B
Rotation
Translation
Figure 1: (A) Protein backbone residue with three backbone atoms represented by coordinate frame. (B) In PGA, frame can be represented via the geometric product of four planes, Two of the planes parameterize the frame's 's rotation around their line of intersection, while the other two encode the frame's translation along the separation vector between them. (C) An exemplary protein backbone structure containing an a-helix and a f--heet. Lines (red), planes (violet) and Euclidean frames (blue) can all be embedded as elements of PGA, facilitating geometric inductive bias for learning representations of the abstract geometry of the protein.
Another widely used approach to construct equivariant architectures is to embed internal features in symmetry group representations and restrict neural network operations equivariant functions
6 6 36 52]. Explicit equivariance in these models has usually been limited to the orthogonal group (3)), which contains rotations and reflections, while most models are instead invariant towards translations. Only recently, Brehmer et al. [13] proposed an architecture based on the framework of Ruhe et al. [48] that enables explicit equivariance towards both translations and rotations by utilizing the projective geometric algebra (PGA) 29 25).
[55 28 50
Inspired by their work, demonstrate that apart from its provides powerful framework for representing the geometry of protein backbones when incorporated into the local frame formulation of protein design architectures. While utilize PGA explicitly represent the frames of the protein backbone as elements of the algebra, PGA additionally provides strong inductive bias its elements represent abstract geometric objects like points, lines and planes, which are well suited to capture the geometry of secondary structure elements, such as a-helices and B-sheets (Figure Moreover, bilinear operations of the algebra enable to compute many geometric relations between those objects, such as distances, angles, projections and incidences.
E(3) eeqiivariant formalism, PGA
Main contributions: We introduce Clifford frame attention (CFA), an extension of the invariant point attention (IPA) architecture of AlphaFold2 [32] by expressing geometric features elements of the projective geometric algebra and using its bilinear operations to construct geometrically expressive, higher order messages. We incorporate CFA into existing flow matching framework for protein backbone generation, FrameFlow (67], and demonstrate in our experiments that the proposed method achieves state-of-the-an performance in the combination of designability, diversity and novelty of generated protein samples. While, especially for small proteins, other models with high designability often over-represent a-helices, the proposed method captures the broad distribution of secondary structures of naturally occurring proteins, which we believe is crucial for designing proteins with vast functionalities. Notably, since IPA widely used throughout the field, CFA may be readily included in many other protein-relatee machine learning models.
1.1 Related Work
Geometric (Clifford) algebra in neural networks Neural networks that use the Clifford algebra were first proposed by Pearson and Bisset [46), an extension of the multi layer perceptron (MLP) by Clifford algebras, which was later studied further by Buchholz and Sommer [16]. More recently, Ruhe al. [49] propose Geometrie Clifford Algebra Networks, using geometric (Clifford) algebras based their Clifford Neural Layers [2], and extend this framework in 48 to E(3) ((3)-cquivariant

---

representations. Brehmer et al. 13 propose to use projective geometric algebra, which enables (E(() -equivarrar feature representations, and the representation of frames as elements of the algebra.
Generative models for protein design Watson et al. 63 propose RFdifffusion, a generative model for protein backbone design that utilizes the pre-trained protein structure prediction network RoseTTAAold [2]. Yim et al. [68] propose FifmeDiff, which defines diffusion model over a set
of frames, SB(3) N and extend this method within the flow matching framework [67]. Bose et al.
10 propose FoldFlow and its variants, diffusion and flow matching models over SE(3) Lin and
AIQuraishi (37 propose an equivariant encoder and decoder architecture. Wu et al.
65 train
transformer model to predict angles between adjacent residues. Mao et al. [42) propose vector field networks. which is also an extension of IPA, and employ them in Frame The main difference
to our approach is that VFN uses virtual atoms as geometric features and vector field operators to model interactions, whereas CFA use multivectors and the geometric bilinears of PGA respectively.
2
Background
This section provides an introduction to the mathematical frameworks that we refer to in this paper, flow matching and Geometric algebra, and a brief introduction to protein design.
2.1
Geometric Algebra
A Clifford algebra over a real vector space, typically referred to as geometric algebra, a power- ful mathematical framework for describing geometric objects including points, lines, planes and operations on these objects in an algebraically concise way 26].
In more technical terms, given a vector space V and a quadratic form q V + FF from the vector space to the underlying field F, we construct geometric algebra the unitary, associative, non-commutative algebra with the property v² = q(v) for every v E V 49 v2 = vv denotes the geometric product, the bilinear operation of the algebra, of the vector v with itself. In the geometric context, q may be thought of as the metric of the vector space, meaning that for a vector v, q(v) is its squared norm. Elements of the algebra are called multivectors. They can be constructed by forming geometric products between basis vectors of V and linearly combining them.
In this paper we are mainly interested in the projective geometric algebra (PGA) 22 25 which we denote as G3,0,1- It is the geometric algebra over V = R with vector basis
Co C1 C2 C3 and q(c1) = (((2) q(cs) 1, g(co) 0.
(1)
The full algebra has 16 basis elements, which can be grouped in grades according to the number of vector basis elements they are constructed from. Following Gunn [29], we can interpret elements of different grades as geometric objects such as planes, lines and points in 3D space, as listed in Table I and visualized in Figure A.3 Working in a four dimensional space has the advantage that we are able to represent lines and planes that do not necessarily include the origin, which is crucial for the description of translations as shown below.
Central to the proposed method is the fact that PGA allows the representation of elements of the Euclidean group F(3) as elements of the algebra. Given plane p E G3,0.1, an arbitrary geometric object X E G3.0.1 can be reflected across the plane via the sandwich product,
X' -XpX..
where X is the grade involution that flips the sign of elements with odd grade. A vector in PGA may thus be interpreted both as a plane and as a reflection operator. One can extend this idea using the Curtan-Dieudonte theorem, which states that any E(3) transformation can be represented by repeated reflections. Two consecutive reflections through intersecting planes result in a rotation around the line of intersection and two reflections through two parallel planes correspond to translation along the separation vector of the planes. The associated elements of PGA are obtained by taking the geometric product of the respective planes. We thus represent each residue frame of a protein by a multivector, a so called motor, corresponding to a rotation followed by translation as shown in Figure Similar to eq. motor M E G3.0,1 can be applied to an arbitrary element X E G3,0,1 according
X' MXM
We provide more detailed introduction to geometric algebra in Appendix A.I.I
3

---

2.2 Flow Matching
Flow matching 33 as generalization of diffusion models [53] 54 offers framework for learning continuous normalizing flows (CNFs) (0), Ø: [0.1] x S - S that transform a general prior distribution Po, defined on the domain Ω2 target distribution p1 by evaluating the probability path
Pt
[d
at / = 1, where denotes the pushforward. The flow di can be expressed in terms of a vector field ey by solving the ordinary differential equation (ODE)
d (1()) (x)())))) dt
do(x)=
If the vector field is known, integrating the flow ODE. Lipman et al. [39] showed that oy can be learned by regressing on a vector field (z) (x)) that is conditioned
generate samples from p1 by sampling from the prior and
data sample 21 Pi, ie. by minimizing the loss function
L = (0, po propee [I1o((xs)) ue (xae(2))))
Here, (a xr is the conditional probability path induced by ue( which is commonly chosen as linear interpolation between the prior sample zo and the target 2ı by setting the conditional flow to
Ve( x1) = (1 - t) + tæ0
from which (x) (xr)) can be obtained by forming the time derivative. As shown by 39 sampling from Pe (1aa21)) in eq. [ can then be realized by transforming sample from the prior Po to Fe = (xul(a))) 11)) This formulation in terms of conditional distributions allows to learn dynamic optimal transport (OT) plans from prior to target using minibatch OT as shown by Tong et al. [56).
The framework of CNFs and flow matching be generalized for sampling from distributions on Riemannian manifolds M, such as SE(3) as described by Chen and Lipman [19]. Then, the flow Φ: [o, 1] M M is diffeomorphism, and vector field learned as a smooth map ue: [0, 1] x M + TM to the tangent space T T.M. A natural choice for the map (z(oo)))) on Riemannian manifold is the connecting path with minimal length, the geodesic.
2.3 Flow Matching for Protein Structures
For sampling from the distribution of protein backbones, we represent each protein residue by a frame T (r, x)) SF((3 that corresponds to translation
RR3 and rotation SO(3), thus the domain for the flow matching process is M = SE(3) as described in FrameDiff (68). The frame for a given residue is defined through the backbone atoms [N, Ca. CJ, where the translation is chosen as the displacement of the Ca atom relative to the origin, while the rotation r is constructed using a Gram-Schmidt process 332 the vectors [C' C., Ca].
Similar to 67 we define the conditional flow (To as the geodesic between To and Ti,
4/ (TolTi) expr, (flog,, (Ti)).
(8)
We calculate the time derivative of the flow described in FrameFlow 667 and regress it as in eq. 6 As prior Po, we choose a 3.N -imeensiiaa normal distribution with unit variance N I) for translations and, for rotations, adopt the heuristic trick of using IGSO(3) during training and
U(S((3)) for inference from Yim et al. (67]. Additionally, we use minibatch optimal transport [56
with the equivariant cost function from 34
2.4 Designability of Proteins and Importance of their Structural Composition
Until recently, computational tools for protein design have relied on physics-based energy func- tions B5] 35| 64), restricting protein design campaigns pre-defined topologies, geometries, and secondary structure elements. In contrast, deep generative models learn the probability distribution of protein structures from the training set, enabling less constrained sampling of the structure space.
Ideally, the main goal of any generative model should be the creation of structurally diverse and novel proteins, thereby maximizing the access unseen structures and possibly functions. Since

---

the functionality of proteins is grounded in their structure, it is crucial to extensively cover the broad range of secondary structure elements and topologies found in natural proteins. Typically, the properties of generated proteins are assessed by the scores of diversity and novelly, indicating how similar generated backbones are to each other, and known native proteins, respectively. While explicit analyses of secondary structure elements of generated proteins are often overlooked, [37]6 65]
suggest that diffusion models commonly generate redundant protein structures composed mostly of a-helices, which raises the question of how much functionality can be hypothetically encoded in these proteins.
A key quality check for generated proteins is their biophysical consistency in the sense that their sequence, indeed, folds into the intended structure [44. In the case of backbone generation, this is typically evaluated by predicting a sequence from the generated structure with the inverse folding
model ProteinMPNN [33. The obtained sequence re-folded with ESMFold (38 and if the resulting
structure aligns well with the original backbone, the latter is called designable (57 [3 Thus, the
designability metric measures consistency between well-established sequence and structure prediction
models, which can be seen as proxy for biophysical validity and practical realizability 47
3 Geometric Algebra Flow Matching for Protein Backbone Generation
We propose geometric algehra-based neural network architecture to predict the vector fields in flow matching process on SE(3) and call this approach Geometric Algebra Flow Matching (GAFL). The proposed neural network architecture is an extension of the one introduced in FrameDiff [68), where we replace its central component, the invariant point attention (IPA) block from AlphaFold2 [32 by Clifford Frame Attention (CFA), which we explain in more detail in the next paragraph.
Input features are the noised frames TG, pairwise spatial distances, positional encodings absolute and relative sequence positions, and the flow matching time As FrameDiff, we use self-conditioning. The network relies on a series of six blocks, in which the frames are updated consecutively to predict the denoised backbone structure and from this the respective conditional vector fields. Each block uses CFA to perform message passing between protein residues, in particular processing geometric information as given by geometric node features and the current set of frames. The SE(3) invariant output of CFA is fed through an MLP and a transformer (60) and then used to predict frame updates. For detailed explanation of the architecture, we refer Appendix A.2.I
Clifford Frame Attention
The original IPA mechanism uses geometric node features the form of 3D points for the calculation of attention scores and as queries, keys and values, as described in Appendix [A.2.2 The features are expressed in local coordinate frames, which allows the use of arbitrary layers without breaking equivariance. Messages between nodes are constructed as a linear combination of attention values, weighted by attention scores. While other generative models for protein design incorporate the original version of IPA directly [68] 67 [0], we propose to enhance its geometric expressivity by performing the following modifications.
We replace the point-valued attention values by multivectors, which can encode points, lines, planes and Fuclidean frames shown in Figure]] [ At the same time we decrease the number of channels to retain approximately the same number of parameters.
PGA features:
Geometric messages: Instead of linearly combining geometric features in the message passing step, we construct messages utilizing the geometric bilinear layer that was introduced as part of the Geometric Algebra Transformer in Brehmer et al. 113, effectively calculating geometric product and join, another bilinear operation of PGA (see Definition A.10). between the node features V, and V (see Algorithm 4. These two operations able to compute many geometric relations as detailed in Appendix A. and (25) The messages m.
h.p from node to node can then be written as
h.p mij
= GeometricBilinear (T.' (I, T,, V!
99)
where T,, Ty are the frame transformations of the corresponding node, which ensure that the bilinear operations performed in the same reference frame. When aggregating the messages make use
5

---

of the bilinearity of the products to exchange the sum and the bilinear layer to obtain:
Lah h.p
GeometricBBlinear
T
Σ au T, (T) T. v."),
(10)
The operation is thus performed on a node level and scales linearly with the number of nodes.
Higher order message passing: We also incorporate higher order messages into the proposed architecture, that is, messages that depend on more than two nodes and are thus capable of describing relationships beyond pairwise interaction. As in [6], we use bilinearity of geometric product and join to construct higher order messages by multiplying aggregated two-body messages m and m', e.g.
(I h
Σ mi
m'k
Σ jk
mig =
Σ f
m
(11)
can be seen as sum of three-body messages mijk- In practice, we use learnable linear combination of versions of eq. 1 projected onto different multivector grades using the geometric product and the join, as described in algorithm 5
(3)
Since embed both residue frames and geometric node features in G3,0,1+ is a natural idea to combine them on node level such that they can interact via the geometric bilinears during message passing. To this end we compute relative frame transformations for all pairs and aggregate them with the attention weights,
Frames as features:
T
Σ a. TT
(12)
We concatenate these with the remaining geometric features before the construction of geometric messages and also pass them directly to the backbone update block using residual connection. The full CFA algorithm is provided in Appendix A.2.3
Although PGA would in principle allow construct fully equivariant architecture (13 without using local frames, we decide to keep the local frame formulation of IPA, since it allows to use more general layers and non-linearities. Moreover, the common problem of ambiguous local frame choices that other architectures suffer from [62] is not apparent in our case since backbone residues provide canonical, geometrically meaningful choice for the local frames. We discuss the equivariance of GAFL in Appendix A.2.4
4 Experiments
We train GAFL on a subset of the Protein Data Bank (PDB) dataset 9 comprised of monomeric protein structures with up to 512 residues and perform extensive ablations on the smaller, curated SCOPe dataset [27] [8] filtered by proteins with length of up to 128 residues (SCOPe-12) 128) as in [67 37). A representative selection of designable protein backbones generated by GAFL trained on the PDB dataset is illustrated in Figure 2
4.1 Checkpoint Selection
With the aim of finding a good balance between designability and secondary structure content in mind, we introduce checkpointing criterion that takes secondary strueture into account. We first train the model for Nuain epochs. Then, from epoch Nusia to Nuai + NNsekeet we calculate the relative occurrence of a-helices and 3-strands T. and r3 of 100 generated proteins after each epoch. We keep the top i checkpoints in terms of secondary structure content deviation, which we define as d, = Iro (ref) + Irs- r we choose the checkpoint with the highest designability and filter by a threshold d, < dmax- The
(rel) with the training set as reference. Among those k checkpoints,
hyperparameters Ntrain. Neclect dmax and & for the different training runs are listed in Appendix A.3.8
Source code and trained model weights available

---

60
100
N 200
300
Figure 2: Representative examples of designable protein backbones generated with GAFL (white) and the output of the refolding pipeline (colored), comprising ProteinMPNN and ESMFold.
4.2 Metrics
To assess the performance of given model, we follow a well-cstablished pipeline of self-consisteney evaluation [57 63]. For each generated backbone, we design candidate sequences with Protein- MPNN 233 which are subsequently refolded with ESMfold (38 and define scRMSD the smallest RMSD between our generated backbone and the 8 refolded, aligned candidates. As [63 68), we define designability as the fraction of generated samples with scRMSD < 2.0 Å. We also report diversity and novelty of the designable backbones average TM-similarid [70] scores within the set of generated backbones and with respect to the PDB correspondingly (see Appendix [A.3.] and [67]).
To evaluate how well the secondary structure distribution of the training set is captured, calculate the average helix and strand content all designable backbones using the DSSP algorithm [33
4.3 Baselines
At the task of generating backbones of up to 300 residues, we compare GAFL trained on the PDB to the diffusion models REdiffusion (63 and FrameDiff [68] and to the flow matching models FoldFlow [10] and FrameFlow (67 69 At the time of submitting the paper, FrameFlow had not been trained on the PDB yet. however, we include it in our results as contemporary work. Further, perform ablation study for generating smaller backbones, where compare GAFL models to the originally published FrameFlow 667 model, which was trained on the SCOPe dataset with backbones of up 128 residues. While all of the baselines above incorporate the original IPA [32 architecture, we also compare GAFL with VEN [42], in which an alternative modification IPA is proposed. For all models considered, generate backbones using published model weights and the respective default inference settings (Appendix A.3.2).
4.4 Results
We train GAFL for 15 days two NVIDIA A100-80GB GPUs on the dataset used in FrameDiff, which comprises monomeric structures from the Protein Data Bank (PDB), filtered by a maximum length of 512 and maximum coil content of 50%, resulting in total of around 25, 50000 backbones (68). As in FoldFlow, we evaluate GAFL and the respective baselines on the task of generating backbones of lengths (100, 150, 200, 250, 300). For GAFL,, we 200 inference timesteps (Figure A.9p. In Table [ we report the metrics described in Section (.2 for each model together with the time needed to generate a single backbone of length 100 on an NVIDIA A100 GPU without batching.
GAFL has state-of-the-art performance
We find that GAFL can reliably generate designable, diverse and novel backbones while capturing the statistical distribution secondary structure elements of natural proteins. all metrics considered, GAFL outperforms both variants of FoldFlow and is better or as good as FrameDiff. GAFL also outperforms FrameFlow, which was trained the same dataset, terms of designability and, at the same time, achieves better secondary structure content. GAFL's designability is only matched by RFdiffusion, which not directly comparable since relies on pre-trained model weights from the folding model RoseTTAFold 2 and has around three times more parameters. For diversity, novelty and helix content, GAFL performs better than RFdiffusion. We also observe that GAFL and FrameFlow can generate backbones around three times faster than the other evaluated models.
7

---

Helix
Strand
Mean Helix of PDB
Mean Strand of PDB
Designability
Secondary Structure
Secondary Structure
1.0
Secondary Structure
1.0
0.6-
Designability
0.8-
-08
Desiggaat Designabiiityy
0.61
-066
0.6
0.6
Designabll
0.6-
00
0.4-
0.4
0.4
0.4
0.4
0.4
60
100
150 Length
200
250
300
60
100
150
200
250
300
100
150
200
250
300
Length
Length
FrameDiff
FoloFlow, OT
C
N - 450
Secondary Strucure
Secondary Structure
0.8-
-.88
Designabiltty
0.8
0.0
Desigrabiitt
TM 0.45
0.6-
10.6
0.6
0.6
04-
-04
0.4-
0.4
0.0
0.0
0.0-
0.0
00
100
150
200
250
300
00
100
150
200
300
Length
Length
N 500
Strand Percentage
WD 0.25
WD-0.3
0.35
Norm Trequenns Sy
TM hit 0.29
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0:75 1:00
0.00 0.25 0.50 0.75
1.00 Helix Percentage
Helix Percentage
Helix Percentage
Figure 3: (A) Performance of evaluated models in terms of designability and secondary structure content as a function of backbone length. 200 backbones were generated for each model at each length {60, 80, 100, 150, 200, 250, 300). (B) Comparison of the secondary structure distributions of backbones generated by GAFL and RFdiffusion from (A) to the PDB dataset filtered by the respective protein lengths along with the Wasserstein distance (WD) between the distributions. (C) Examples of designable backbones generated by GAFL for lengths 450 and 500. We also report TM scores of the backbones the closest hit in the PDB database computed with FoldSeek.
Table Performance of GAFL and baseline models for the generation of 200 protein backbones for
each length in {100, 150, 200, 250, 300). We report the metrics from Section 4.2 including standard errors obtained by bootstrapping, and the time needed to generate a backbone of length 100. The best values and values within the respective margin of error bold.
Method
Designability (1)
Diversity (.)
Novelty (4)
Helix Content
Strand Content
Time [s]
PDB Dataset (300)
0.39 (0.00)
0.23 (0.00)
FrameDiff
0.54 (0.02)
0.45 (0.00)
0.71 (0.00)
0,53 (0.01)
0.20 (0.00)
24.3
ColdFlow-SFM
0.69 (0.01)
0.44 (0.00)
0.77 (0.00)
0.91 (0.00)
0.01 (0.00)
24.3
FoldFlow-OT
0,82 (0.01)
0.44 (0.00)
0.79 (0.00)
0.88 (0.00)
0.00 (0.00)
24.3
FrameFlow
0.85 (0.01)
0.35 (0.00)
0.70 (0.00)
0.56 (0.01)
0.20 (0.00)
6.6
REdiffusion"
0.89 (0.01)
0.37 (0.00)
(0.00)
0.58 (0.02)
0.24 (0.02)
21.0
GAFL. (ours)
0.88 (0.01)
0.36 (0.00)
0.71 (0.00)
0.53 (0.01)
0.25 (0.01)
8.8
"Pretrained weights from folding model trained on dataset larger than PDB.
GAFL generates proteins with diverse secondary structures at varions lengths
Although protein design campaigns span a wide range of protein sizes [ 17 43 8 63). the primary goal remains encode maximum functionality into the smallest protein possible, driven by the growing costs of synthesis as protein size increases. Thus, we further assess designability and secondary structure content of generated backbones as function of their length (Figure (3A). For all lengths considered, backbones generated by GAFL, FrameFlow and RFdiffusion are highly designable while FrameDiff and FoldFlow struggle with the generation of long proteins. The length dependence of designability and secondary structure content for GAFL and FrameFlow is qualitatively similar; however, GAFL achieves overall better results (Table Figure A.8). Crucially, we find that for generating proteins with less than 150 residues, GAFL is well suited is capable of generating

---

highly designable backbones with similar amount of 8-strands as naturally occurring proteins, while REdiffusion over-represents a-helices. This also reflected by the Wasserstein distances between the secondary structure distribution of naturally occurring backbones and those generated by GAFL and RFdiffusion, respectively: Purely a-helical proteins are over-represented by RFdiffusion, leading to higher Wasserstein distance of 0.35 compared to 0.25 for GAFL (Figure 3)).
GAFL generate large proteins
To compare GAFL, with VFN 42 we evaluate it at generating five backbones for each length in {100, 105,..., 500}, some of which are portrayed in Figure 3C. We find that, with value of 0.74, GAFL outperforms not only VFN (0.44) but also FrameFlow (0.64) and REdiffusion (0.71) in designability. However, GAFL and FrameFlow over-represent helices for large proteins (Table A.3)
4.5 Ablation of CFA
In order to investigate the effect of the proposed architectural changes, we conduct an ablation study on the semi-manuale curated SCOPe dataset [8), which clusters proteins by their sequence and structural similarities ensuring that its entries are evolutionary and structurally non-redundant. We further filter SCOPe by the length of up 128 residues, which results in 3938 proteins.
We compare GAFL with FrameFlow and models for which we leave out all proposed architectural changes or only higher order message passing, respectively, while scaling the width of the lay-
0.9
ers such that the number of parameters remains the same. All models are trained with three dif- ferent random seeds for 6500 epochs on one
Designability
0.8-
0.7
NVIDIA A100-80GB GPU, which takes around days, and evaluated by generating backbones for lengths between 60 and 128 as in [67.
0.6-
FrameFlow (IPA)
We find that GAFL's designability of 90.5% is 8 percentage points higher than that of the pub- lished FramcFlow model [67] trained SCOPe (81.2%) as depicted in Table 3 If retrained with GAFL's training procedure, FrameFlow's architecture with original IPA achieves des- iigabiliity 88.2%. Using PGA-valued features and higher order message passing as proposed in CFA increases the designability by 1.4 and 2.3 percentage points, respectively. While the training procedure has a larger effect on the des- ignability, the improvement due to CFA can be considered to be significant since it persists across different random seeds. This can also be observed in the distribution of designabilities of 30 check- points sampled during the training procedure (Figure A. described in Section 4.1
GAFL (CFA)
0.4 0.5
0.5 0.6
0.6 0.7
Helix Percent
Figure 4: Helix content and designabilities of 90 model checkpoints sampled during three train- ing runs the PDB dataset for GAFL and re- trained FrameFlow, respectively. For each check- point, sample 40 backbones per length in {100, 150...., ..00......0)..
For the much larger PDB dataset, we conduct small ablation study, in which we compare the performance of GAFL with CFA and retrained FrameFlow with original IPA across three differ- ent training runs, respectively (Table 2). Since GAFL consistently achieves higher designabil- ities, we can validate that the trend observed on SCOPe also holds true for training the PDB.
Table 2:
Ablation of GAFL and FrameFlow on the PDB dataset. We report the results of three training runs and the published FrameFlow model.
Method
Designability (?)
Diversity (L)
GAFL
87.8 87.0
0.35 0-36
84.3
0.34
FrameFlow
84.4 78.1
0.35 0:34
Furthermore, we observe a correlation between rvererepres a- achelices and achieving high designability (Figure 4) for 90 checkpoints sam- pled for each model during checkpoint selection as described Section 4.T Crucially, GAFL checkpoints consistently show better designability in the low helix-contens regime, which attribute to using CFA instead of IPA.
FrameFlow"
84.6
0.35
*Published model weights

---

Table Ablation of GAFL models with different elements of the proposed CFA architecture, trained on the SCOPe-128 dataset. For each model, we report the performance of three training runs with different random seeds, evaluated by generating backbones for each length {60, 61,.... 128).
PGA Higher Ord. Msg.
Checkp. Selection
Designability [%] (1)
Diversity (4)
90.5
90.6
0.30 0.38 0.36
89.6
89.6 90.3
0.37 0.40
89.0
0.37
88.2
0.38 0.38
0.39
FrameFlow-SCOPe
81.2
0.37
4.6
Discussion
Our results suggest that GAFL one of the current state-of-the-art models for unconditional protein structure generation. GAFL outperforms the widely used, pre-trained model RFdiffusion in terms of diversity, novelty and inference time and is on par at designability. Remarkably, GAFL can achieve this performance without requiring pre-trained model weights while other con-pre-ttrined models often lack designability or show a mode-collapse towards generating helical structures.
Especially for generating small, highly designable backbones with distinct secondary structures, GAFL performs well, in particular better than RFdiffusion. Since in most protein design campaigns the goal is to incorporate the desired functionality into the smallest protein possible while exploring large structural space, we consider this advantage of GAFL be highly relevant for future developments and real-world applications of generative models for proteins.
Our ablation studies provide evidence that achieving high designability without over-representing a-helices can be attributed to the replacement of IPA by the proposed CFA architecture. Since IPA is used in many current state-of-the-ar architectures for backbone structure, CFA has the potential to enable improvements in many protein-related tasks.
Limitations While the proposed method GAFL achieves satte-of-the-ae performance in protein backbone generation, there is still room for improvement. We note that achieving high designability without compromising the diversity of secondary structures remains grand challenge. GAFL does not perfectly capture the secondary structure distribution of natural proteins, especially for large proteins. Further, unconditional protein generation can regarded as suitable benchmarking task for protein design models, however, most applications require conditional sampling. GAFL can be readily incorporated into existing frameworks for conditioning, c.g. on motif's [6] or symmetry 633
5 Conclusion
We introduced Geometric Algebra Flow Matching (GAFL), flow matching model for protein design based on FrameFlow [67]. GAFL relies on the proposed Clifford Frame Attention (CFA), an extension of the invariant point attention block from AlphaFold2 [32] by representing residue frames and geometric features of protein in the projective geometric algebra. This enables the usage of the bilinear operations in the algebra to construct geometrically expressive messages between residues. The experiments demonstrate that the resulting model is state-of ofetthe-rr in the combination of the established metrics designability, diversity and novelty and performs notably well at generating designable small backbones with distinct secondary structures, containing 3-strands particular. Given the promising results of the extension of invariant point attention with geometric algebra presented in this work, we look forward to exploring benefits for other protein-related tasks.
This study received funding from the Klaus Tschira Stiftung gGmbH (HITS Lab). We acknowledge the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2021-29 for awarding this project access to the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre. The authors acknowledge support by the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG.
Acknowledgments
10

---

References
[1] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15011 2022.
[2] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang. Qian Cong, Lisa N Kinch, R Dustin Schaeffer, al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871 8-7-776 2021.
[3] Minkyung Back, Ivan Anishchenko, Ian Humphreys, Qian Cong, David Baker, and Frank DiMaio. Efficient and accurate prediction of protein structure using rosettafold2. bioRxiv, pages 2023-05, 2023.
[4] David Baker and George Church. Protein design meets biosecurity. Science, 383(6681) 349-349, 2024.
[5] David Baker, Susana Vázquez Torres, Melisa Benard Valle, Stephen P. Mackessy, Stefanie Men- zies, Nicholas R. Casewell, Shirin Ahmadi, Nick J. Burlet, Edin Muratspahić, Isaac Sappington, Max Overath, Esperanza Rivera de Torre, Jann Ledergerber, Andreas Hougaard Laustsen, Kim Boddum, Asim K. Bera, Alex Kang, Evans Brackenbrough, Iara Cardoso, Edouard Crittenden, Rebecca Edge. Justin Decarreau, Robert J. Ragotte, Arvind Pillai, Mohamad H. Abedi, Hannah Han, Stacey R. Gerben, Analisa Murray, Rebecca Skotheim, Lynda Stuart, Lance Stewart, Thomas Fryer, and Timothy Patrick Jenkins. De novo designed proteins neutralize lethal snake venom toxins. Research Square, 2024.
[6] llyes Batatia, David P Kovaes, Gregor Simm, Christoph Ortner, and Gábor Csányi Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances Neural Information Processing Systems, 35:1 12:11423-1110 1136,, 2022.
[7] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess Smidt, and Boris Kozinsky. (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):
2453, 2022.
[8] Nathaniel R Bennett, Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Po Peng, Justas Dauparas, Minkyung Baek, Lance Stewart, et al. Improving de novo protein binder design with deep learning. Nature Communications, 14(1):2625, 2023.
[9] Helen M Berman, John Westbrook, Zukang Feng. Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1):235-242, 2000.
[10] Joey Bose, Tara Schound-Sadeghl Guillaume Huguet, Kilian FATRAS, Jarrid Rector-Brooke Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael M. Bronstein, and Alexan- der Tong. SEE()) flow matching for protein backbone generation. In The Twelfth International Conference on Learning Representations, 2024.
[11] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing. In International Conference Learning Representations, 2022.
[12] Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for PDE modeling. In The Eleventh International Conference Learning Representations, 2023.
[13] Johann Brehmer, Pim de Haan, Sönke Behrends, and Taco Cohen, Geometric algebra trans- former. In Advances in Neural Information Processing Systems, volume 37, 2023.
[14] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18-42, 2017. 2017.
11

---

[15] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[16]
Sven Buchholz and Gerald Sommer. On clifford neurons and clifford multi- perceptrons. Neural Networks, 21()):925 2008.
[17] Longxing Cao. Inna Goreshnik, Brian Coventry, James Brett Case, Lauren Miller, Lisa Kozodoy, Rita E Chen, Lauren Carter, Alexandra Walls, Young-Jun Park, et De novo design of picomolar sars-cov-2 miniprotein inhibitors. Science, 37066515):4264433 2020.
[18] John-Marc Chandonia, Lindsey Guan, Shiangyi Lin, Changhua Yu, Naomi Fox, and Steven Brenner. Scope: improvements to the structural classification of proteins-extended database to facilitate variant interpretation and machine learning. Nucleic acids research, 50(D)): D553- 2022.
[19] Ricky T. Q. Chen and Yaron Lipman. Flow matching on general geometries. In The Twelfth International Conference Learning Representations, 2024.
[20] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[21] Taco Cohen. Equivariant convolutional networks. PhD thesis, Taco Cohen, 2021.
[22] Gabriele Corso, Bowen Jing, Regina Barzilay, Tommi Jaakkola, et al. Diffdock: Diffusion steps, twists, and turns for molecular docking. In International Conference on Learning Representations (ICLR 2023), 2023.
[23] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. L. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning-basse based protein sequence design using ProteinMPNN. Science, 378(6615):49950 56, 2022.
[24] Chris Doran and Anthony Lasenby. Geometric algebra for physicists. Cambridge University Press, 2003.
[25] Leo Dorst and Steven De Keninck. A guided tour the plane-based geometric algebra pga. 2020. URL httpttttt////beetttttt :ewp/GA4CS........ htmm
[26] Leo Dorst, Daniel Fontijne, and Stephen Mann. Geometric Algebra for Computer Science: An Object-Oriented Approach to Geometry. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1st edition, 2007. ISBN 0123694655.
[27] Naomi K Fox. Steven E Brenner, and John-Marc Chandonia. Scope: Structural classification of proteins-extenned,, integrating scop and astral data and classification of new structures. Nucleic acids research, 423DI))D304-D
2014.
[28] Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.
[29] Charles G. Gunn. Projective geometric algebra: A new framework for doing euclidean geometry. arXiv preprint 1901 005873, 2020.
[30] Charles R. Harris, Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard- Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362 September 2020.
Kevin Sheppard, Tyler
[31] David Hestenes and Garret Sobezyk. Clifford algebra to geometric calculus: a unified language for mathematics and physics, volume 5. Springer Science & Business Media, 2012.
12

---

[32] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger. Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with AlphaFold. Nature, 596(7873):583-588)) August 2021.
[33] Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):2577-266 1983.
[34] Leon Klein, Andreas Krämer, and Frank Noe. Equivariant flow matching. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 59886-59910. Curran Associates, Inc., 2023.
[35] Andrew Leaver-Fay, Michael Tyka, Steven Lewis, Oliver F Lange, James Thompson, Ron Jacak, Kristian W Kaufman, P Douglas Renfrew, Colin A Smith, Will Sheffler, et al. Rosetta3: an object-oriented software suite for the simulation and design of macromolecules. In Methods in enzymology, volume 487, pages 545-574. Elsevier, 2011.
[36] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. The Eleventh International Conference on Learning Representations, 2023.
[37] Yeqing Lin and Mohammed AlQuraishi. Generating novel. designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. arXiv preprint arX1v:2301.12485. 2023.
[38] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction atomic-level protein structure with language model. Science, 379(6637):1123-111 2023.
[39] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations, 2023.
[40] Pertti Lounesto. Clifford algebras and spinors. In Clifford algebras and their applications in mathematical physics, pages 25-37. Springer, 2001.
[41] Douglas Lundholm and Lars Svensson. Clifford algebra, geometric algebra, and applications. arXiv preprint arXiv:0907.5356 2009.
[42] Weian Mao, Muzhi Zhu, Hao Chen, and Chunhua Shen. Modeling protein structure using geometric vector field networks. bioRxiv, pages 2023-05. 2023.
[43] Audrey Olshefsky, Christian Richardson, Suzie Pun, and Neil P King. Engineering self- assembling protein nanoparticles for therapeutic delivery. Bioconjugate chemistry, 33(11): 2-34,, 2022.
[44] Sergey Ovchinnikov and Po-Ssu Huang. Structure-based protein design with deep learning. Current opinion in chemical biology, 65:136-144, 2021.
[45] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perf-peeformance deep learning library. In Advances in Neural Information Processing Systems. 2019.
[46] J.K. Pearson and D.I. Bisset. Neural networks in the clifford domain. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) volume 3, pages 1465-1469 vol.3, 1994.
13

---

[47] James P Roney and Sergey Ovchinnikov. State-of-the-ar estimation of protein model accuracy using alphafold. Physical Review Letters, 129(23):238101. 2022.
[48] David Ruhe, Johannes Brandstetter, and Patrick Forré. Clifford group equivariant neural networks. In Teirth- Conference on Neural Information Processing Systems, 2023.
[49] David Ruhe, Jayesh K Gupta, Steven De Keninck, Max Welling, and Johannes Brandstetter. Geometric clifford algebra networks, In International Conference on Machine Learning, pages 29306-29337. PMLR, 2023.
[50] Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pages 9323-9332. PMLR, 18-24 Jul 2021.
[51] Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederie Rousseau, and Luis Serrano. The foldx web server: an online force field. Nucleic acids research, 33(suppl_2); W382-W388, 2005.
[52] Guillem Simeon and Gianni De Fabritiis. Tensornet: Cartesian tensor representations for efficient learning of molecular potentials. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vol- ume 36, pages 37334-37353, Curran Associates, Inc., 2023.
[53] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256 2265. PMLR, 2015.
[54] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[55] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equii equivariant neural networks for 3d point clouds. arXiv preprint arXiv:/802.002 19, 2018.
[56] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00482. 2024.
[57] Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif motif-scaffolding problem. arXiv preprint 2rXhv:2206.04113 2022.
[58] Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology, 42(2):243-246, 2024.
[59] Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 2020.
14

---

[62] Xiyuan Wang and Muhan Zhang. Graph neural network with local frame for molecular potential energy surface. Learning on Graphs Conference, pages 19-1. PMLR, 2022.
[63] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, pages 1-3, 2023.
[64] Benjamin Webb and Andrej Sali. Comparative protein structure modeling using modeller. Current protocols in bioinformatics, 54(1):5-6, 2016.
[65] Kevin E Wu, Kevin K Yang, Rianne van den Berg, Sarah Alamdari, James Y Zou, Alex Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature Communications, 15(1):1059, 2024.
[66] Onry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
[67] Jason Yim, Andrew Campbell, Andrew YK Foong. Michael Gastegger, José Jiménez-Luna Sarah Lewis, Victor Garcia Satorras, Bastiaan Veeling, Regina Barzilay, Tommi Jaakkola, et al. Fast protein backbone generation with (3) flow matching. arXiv preprint arXiv:2310.05297, 2023.
[68] Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application protein backbone generation. arXiv preprint arXiv:2302.02277, 2023.
[69] Jason Yim, Andrew Campbell, Emile Mathieu, Andrew Y. K. Foong, Michael Gastegger, Jose Jimenez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S. Veeling, Frank Noe, Regina Barzilay, and Tommi Jaakkola. Improved motif-scaffolding with SE(3) flow matching. Trunsactions on Machine Learning Research, 2024. ISSN 2835-8856.
[70] Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004.
15

---

A
Appendix
A.1
Background
A.1.1
Background on Geometric Algebra
In this section give short introduction to Clifford algebra, loosely following the presentation of and Doran and Lasenby [24).
Dorst et al. 26 Dorst and De Keninck [25
General construction of Clifford algebras
Clifford algebra be constructed from vector space V by extending it with additional bilinear operation called the geometric product. We write this algebra as G(V). Elements of the algebra A G(V) called mulfivectors and are written in bold. In cases where want to highlight that an algebra element coincides with vector in V, we write it with (bold) lower case letter. Other algebra elements are generally denoted by upper case letters. The geometric product has to fulfill the following properties [24]
Definition A.1: Properties of the geometric product
Associativity: (AB) C ABC)
A,B,CeG(V)
2. Distributivity: A (B C) AB AC
3. Vectors square to scalars:
a² E R
We denote the geometric product as juxtaposition of elements in order distinguish it from other products liker inner and outer product.
Algebra basis Given basis of the underlying vector space V. fe where n is the dimension
of the vector space, we can use the geometric product to construct basis for the algebra.
(1, e en ej ele3, ..,..... e ei...en]
In general, geometric algebra general multivector can be written as a linear combination these basis vectors. The basis elements can be further categorized into called grades according the number of basis vectors they contain, i.e. 1 will be of grade 0, e; of grade 1, e,ey of grade and so on. With this definition it is possible to define the projection of arbitrary multivector onto specific grade 26
dimensional vector space will have 2n basis vectors. A
Definition A.2: Grade projection
Let GBR) (V) be the subspace of G(V) spanned by all multivectors of grade k. We then define the grade projection operator
() -GVVV-VGV (k)
(13)
which selects the k-th grade of given multivector.
Multivectors that contain only elements of grade receive specific names according their grade. Elements of grade are vectors, elements of grade are bivectors, elements of grade 3 are trivectors and so
The metric So far we have only specified that the square of vector should yield scalar. In order to uniquely define a geometric algebra it is important to specify the exact values of these scalars, which will define a metric on the algebra. The choice of metric is crucial for the properties of the algebra and the geometrie interpretation its elements. For an algebra over an n-diimensional vector space may choose scalars, one for each basis vector. It is common to work with a metric that assigns to all non-zero values. This convention leaves (-1, 0, 1) as possible choices for the scalars. One defines G... the geometric algebra with basis vectors squaring to 1, basis
16

---

vectors squaring to -1 and basis vectors squaring to O. Having fixed metric for the vectors of the algebra we can go on discuss the construction of norm for general multivectors. The definition of the norm in geometric algebra uses the concept of reversion which is defined as follows:
Definition A.3: Reversion
the geometric product of Let a1, ag.... an E (V), n E N be vectors and A II these vectors, then the reverse of A is defined as
1411
(14)
i.c. we reverse the order of the vector elements. For general multivector, which may consist of sum of elements of the form of 14 we apply the reverse operation each summand individually.
A norm general mutltivectors can then be defined as:
Definition A.4: Norm
Let A E CVV be a multivector, then the norm of A is defined as
||| V (AA)o-
(15)
The Euclidean geometric algebra G3
One of the most prominent examples for a geometric algebra is the Euclidean geometric algebra G3, which is the geometric algebra over R33 with the standard Euclidean metric, vector basis fei, e3} fulfilling oî c2 ca 1. Apart from the geometric product we can define two additional operations that will be useful for further analysis.
Definition A.5: Inner and outer product
Given two arbitrary vectors b Gl" symmetric and antisymmetric part
separate their geometric product into
1 ab 2 (ab ba)
1
2 (ab ba).
(16)
One typically refers to (ab + ba) =: a as the inner product and (ab ba) =: a as the outer product of the two vectors. The geometric product of and b can then be written as
ab aa.bban b+aAb.
(17)
This definition only holds for vectors. For extension to arbitrary multivectors 24.. To further investigate the properties of these products we can look at the square of the sum of two vectors (a + b)². Expanding the product and rearranging the terms yields
1 (a b)² ba (ab + ba) =
(18)
2
Since all terms the right side are scalars due to the third property in Definition A. conclude that the inner product two vectors a scalar itself. In fact, 18 corresponds to the well known polarization identity from linear algebra, and since we defined the norm of vectors to be the standard Euclidean norm, the inner product from Definition A.3 is the standard Euclidean inner product.
can
Using 17 we can also introduce the notion an orthogonal basis. Just as linear algebra a vector basis ( is called orthogonal if e, e, 0 for i # j. In the following discussion we will
17

---

2
2
2
1
eg ege1
e2e3
e2
y
y
y
x
e2
e1 e1e263
Figure A.1: Visualization of the different geometric primitives in G3. Vectors directed line segments, bivectors are oriented areas and trivectors are oriented volumes. Orientations are indicated by arrows that have the same sense of rotation as the corresponding basis vectors when linked together end to tip.
always assume an orthogonal basis. As a consequence of 17 for + basis elements anticommute e,e, e, and the geometric product is equal to the outer product e,e; = e, Cy.
Geometric interpretation of the algebra elements As the name geometric algebra suggests, is possible to assign geometric meaning to the algebra elements with grade greater than O. Grade one elements vectors keep their usual geometric interpretation directed line segments. To find an interpretation for higher grade elements, one can observe that the outer product of two vectors A A b defines a homogeneous subspace in the sense that for every vector the span of and the following equation holds
x E = span (a. - x A
(19)
as shown in 26). Additionally A has a magnitude according to Definition A.4 A.44andd and also an orientation due to the antisymmetry of the outer product. For example e₁ e2 has opposite orientation compared to e2 A e1 = -e A 02 as indicated by the relative minus sign. 2-blades, that is multivectors which can be written purely as the outer product of vectors, can thus be interpreted oriented areas, which lie within the subspace spanned by their generating vectors and area equal to the norm of the blade. Analogously 3-blades, multivectors equal the outer product of three vectors, correspond to oriented volumes. Notably like vectors, 2-blades and 3- 3-blades neither have position in space nor do they have a specified shape. The only fixed properties are the magnitude of their area/volume as well as their orientation. The geometric interpretation of algebra elements is visualized in A.]
closely related by relation called duality. Graphically speaking, a plane in 3D space for example can be represented both as the plane itself or by the normal vector that is orthogonal it. In algebraic terms this translates to c.g. es representing the normal vector of the plane described by the bivector eje2. One says that these two elements are dual to each other. the same way, scalars are dual trivectors since trivectors, like scalars, have only one degree of freedom in three dimensions and are thus also sometimes called pseudoscalars. Formally duality in Gy can be defined as follows:
Grades of the same dimensionality
Definition A.6: Duality
Let A E G3 and = e1ege3. Then the dual of A is given by
IA
(20)
Finally, it is also insightful to calculate the magnitude of the area represented by 2-blade. To this end we notice that, using the Einstein sum convention, the outer product of two vectors a = a,e, b = aje, can be written as
a A b- aAb= Ae, = a,a,e,e, = E,ja,,, = I (a b)
(21)
where Eygk is the Levi-Civita symbol and x denotes the usual vector cross product. This shows that in G3 the outer product yields the 2-blade which is dual to the resulting vector of the cross product
18

---

Figure A.2: Visualization of the reflection of a vector at plane.
with the same magnitude. In fact, the outer product be generalization of the cross product to arbitrary dimensions. The magnitude can be calculated from 15
yielding
a A bl (la x b/ b/ laa x bl
|a|l
b sin(a)),
(22)
where a is the angle enclosed by a and b. This exactly the area of the parallelogram spanned by the two vectors. Similar calculations can be performed for the case of 3-blades to show that their magnitude is equal the volume of the parallelepiped spanned by their three generating vectors. These findings further support the idea of interpreting bivectors and trivectors area and volume elements respectively.
Orthogonal transformations Another important class of operations which geometric algebra can describe in an efficient way are orthogonal transformations, that transformations T' G3 + G3 which preserve the inner product between vectors. In group theoretic terms, these transformations form the group (3).. In order construct such transformations we first look at how reflections handled in Gg.
Definition A.7: Reflection Let E G be a vector and n E G3 be the unit normal vector of a plane. The reflection of at that plane is then given by
nan.
(23)
To why this indeed corresponds reflection, we decompose into parallel and orthogonal part with respect to the normal vector as shown in A.2 Algebraically this can be written as
a n²a n(na) n(n a nA n(n An)
(24)
(25)
The first summand corresponds the usual projection formula from linear algebra and can thus be identified as the parallel part. Since the whole expression equals a, the second summand must be the orthogonal part. The reflected vector can now be obtained by reversing the sign of a
a' -aj +ai = n(a n) + nnnn A a) n(a n) n(a n) -nn(a ma nan.
(26)
(27)
(28)
19

---

From 26 to 27 we used the antisymmetry of the outer product. This construction of reflections shows that elements Euclidean geometric algebra can simultaneously interpreted as geometric objects as well as transformation operators, a concept which also translates to other geometric algebras.
To construct general orthogonal transformations we can make use of the following theorem.
Theorem A.1: Cartan-Dieudonné theorem
Let (V, q) be nondegenerate space dimension n, then any orthogonal transformation can be written composition of at most reflections.
For proof see [1.. In the context of G3. this implies that any orthogonal transformation of vector, which in addition to reflections comprises rotations around lines through the origin as well as compositions of both operations, can be written
a' = vaV, VE G3, |V||= =1.
(29)
The definition can be extended to higher grade blades, by transforming each of its vectors individually, The transformation of a general blade can then be written as
h A +
A a. 2m1
(=va.V - (+1)AV -
A a;
V.
(30)
The property that the outer product of the transformed vectors is equal to the transformed outer product makes any orthogonal transformation so called outermorphism. As a consequence, orthogonal transformations preserve the grade of blades, which geometrically translates to the fact that vectors are transformed to vectors, bivectors bivectors and so on, which is what we would expect from geometric transformation (a vector will not become volume when rotated). A formal proof of this property can be found in (26].
The multivector V in 29 is called a versor. Versors together with the geometric product form group on their own, the called Pin(3) group. This group is said be the double cover of (3) (see 440) 30 is both a representation of 0(3) and Pin( One can generalize this definition to arbitrary geometric algebras in the following way:
Definition A.8: The Pin(n,m,l) and Spin(n,m,I) groups
Let V = (u G.,,. I ||v|| =1 be the set of versors. Then V together with the geometric product forms the group Pin m, 1).
Let V = (u E G m. ||v|| = 1} be the set of even versors, i.e. versors that only involve even grades. Then V together with the geometric product forms the group Spin(n, m, l).
The projective geometric algebra G3.0.1
The Fuclidean geometric algebra G3 allows for a powerful description of geometric objects in 3D space. However, it lacks the ability to describe absolute positions. For example, planes parametrized by 2-blades are restricted to go through the origin, and vectors only describe relative displacement, not points in space. These problems are solved by the projective geometric algebra (PGA), which is the geometric algebra G3,0,1 with three Euclidean basis vectors squaring to 1, o e = c 1 and one null vector squaring to 0, og 0. We thus use an algebra based on a four dimensional vector space to describe 3D space, similarly what one does with homogeneous coordinates. the following we will denote PGA vectors which only contain a Fuclidean part by ă.
There are multiple ways interpret the different algebraic elements geometrically. Here, we will focus on the flane-based approach as described in (25), since this yields the most useful description of Euclidean motions orthogonal transformations, Elements of grade 1 interpreted not as points but as planes, i.e. vector of the form
π + deo
(31)
20

---

Figure A.3: Visualization of the different geometric primitives in G3,0.1. Vectors represent planes, 2-blades represent lines resulting from the intersecting of its generating planes and 3-blades represent points the intersection of three planes.
Table A.1: Basis elements of the projective geometric algebra along with their geometric interpretation
Grade
Basis Elements
Geometric Interpretation
0
{1}
scalar
1
(eo, e1, e2, 03)
planes
2
(0001: e000; Cpe2; Coc3, e1 C1 0203)
lines, vanishing lines points, vanishing points
3 [c00102, tpe C3, Co C10203] {egere2e3}
4
pseudoscalar
corresponds to plane with (Euclidean) normal vector ń and distance N from the origin. In turn, the Euclidean vectors e e2, ea correspond to basis planes through the origin, whereas ep interpreted as the plane at infinity. Notably any multiple of 3 of the form a( ñ + dev), a E IR represents the same plane.
Similarly to the case of G3. we can make use of 19 to obtain a geometric interpretation for blades of higher grade. For 2-blades of the form a b, the subspace spanned by and corresponds to all planes which contain the line of intersection between the two original planes. It is thus sensible to take as representation of this line. Similarly, a 3-blade X A b A e corresponds to the subspace spanned by all planes which contain the point of intersection of b, and and thus 3-blades represent points PGA. The logic behind this construction visualized in AA.3]
PGA also contains additional classes of geometric objects which e.g. result from taking the outer product parallel planes. For a further discussion of these, we refer 25
Meet and Join We have seen that the outer product in PGA can be used to calculate the intersection of geometric objects. In this context opposite operation called the join which, the name suggests, e.g. maps two points to the line which passes through both of them. In order to properly define the join we first extend the concept of duality from Gy to G3,0,1-
thus also known as so called meet. There is also an
Definition A.9: Hodge duality
Let X G be blade. Then its hodge dual +X is defined via
X+X
X
I.
(32)
where X. is the Euclidean part of X without eg. For general multivectors the hodge dual is performed bladewise.
As explained in 25,, this definition is slightly different compared to Definition A.6 since in PGA en does not have multiplicative inverse. However, the overall idea to map between the different
In some literature the roles of meet and join are actually reversed, i.e. the outer product takes the role of the join. This seeming contradiction can be explained by the fact that we follow the approach of interpreting vectors planes and not points.
21

---

subspaces of equal dimensionality is still the same. In practice the hodge dual maps basis element simply the element which contains all the basis vectors not present the initial multivector (in some situations with an additional minus sign), e.g. *0pe3 = e, (confer [5] for more details).
Using the hodge dual one can define the join as follows.
Definition A.10: Join
Let A, B E G3,0,1, then the join between these multivectors is defined as
AvB -1 (+B +A),
(33)
where * - is the inverse hodge dual, which differs from the usual hodge dual by sign for some elements.
As mentioned earlier, the join can be viewed as being dual to the meet, linking geometric objects together instead of finding their incidence. In that way, the join of two points is the line connecting both of them and the join of three points results in a plane containing all three points.
In PGA norm for multivectors can be defined analogously to Definition A.4 However, since PGA contains the null vector eo, only half of the components of a general mulfivector contribute to value of the norm. It is thus useful define second norm which depends all of the components which contain eg.
Definition A.11: Infinity norm
Let A E C3,u, then its infinity norm is given by
|| ||+A||-
(34)
Euclidean transformations in another plane p
Similarly to the case G3, one can write the reflection of a plane m
m' pmmp, |pl| = 1.
(35)
Reflection of higher grade blades, e.g. lines and points, can again be achieved by reflecting each vector in the blade individually and making use of the fact that the reflection is an outermorphism
& a
-app)) = (-1)*
\ a; p.
(36)
1=1
The crucial difference in comparison to G3 is that reflecting plane no longer restricted to go through the origin. This allows one to do consecutive reflections not only in intersecting planes, but also parallel planes. Two important special cases are reflections two intersecting planes and the reflecions in two parallel planes as shwon in A.4 The first case corresponds to rotation around the line of intersection by an angle which is twice the angle enclosed by the planes. The second case results in translation along the distance vector by amount equal to twice the distance between the planes (see 25
Compositions of these two types of transformations make up the special Euclidean group SE(3). which can be seen as the group of rigid body motions. This means that every Euclidean transformation which does not involve reflections can be embedded as element of the even subalgebra G {A G3.0.1 (A)x 0, & odd). stated in the following theorem:
22

---

Figure A.4: Visualization of the different geometric transformations in G3,0,
Theorem A.2: Euclidean motors
Let I'' SE(3) be an element of the special Euclidean group. Then there exists multivector called motor in the even subalgebra M G3.o.1 with M 1 such that
X MXM,
(37)
with an arbitrary multivector X E G3.0, is representation of on G3,0.1
We emphasize that the expression in 37 can applied to any geometric object which is representable in the algebra, Le. it does not matter if X is a point, vector, plane; the correct equation to transform it always takes the above form. This remarkable property can also be found for other important operations in PGA. In fact, have already mentioned the meet operation which allows to calculate intersections arbitrary geometric objects A, B E G3,0,1 via the universal formula B, well the join which can likewise be applied to pairs of arbitrary objects.
Metric Relations We have already seen how to transform objects, find incidences between them, and join them together. Furthermore the basic operations of PGA allow to calculate a host different metric relations between its elements. Below we provide non-complete list of examples.
Theorem A.3: Metric relations
22 Let P1, P2 E G be planes, L E 1 be line and Pı, P2 E G3.o, points, which all normalized, i.e.
[3]
| = | = |L| [P = P2 11
then we can calculate the following relations:
Distance between points P1, P2
(38)
Distance between point P, and line L
J V
(39)
Distance between point P1 and plane
[ A pillao
(40)
Angle between planes PI, P2
sin -1 -1 A P2ll)
(41)
Angle between plane Pı and line L
sin -1 (Il (p, II)
(42)
23

---

For an extensive list of operations see 23
A.2
Methodology
A.2.1
GAFL architecuer
21
Edge Update
2141
Node update
81
Transformer
MLP
Clifford Frame Attention
G
G
T
Backbone update
Tits
Figure A.5: High level overview over the GAFL architecture. The architecture was adapted from FrameDiif/FrameFlovv [68 67]. We replaced invariant point attention (IPA) with Clifford frame attention (CFA) and added geometric node features G.
We adapted the GAFL architecture from FrameDif//FrameFloo [68
67 and replaced invariant point
attention (IPA) with Clifford frame attention (CFA) (see Algorithm
3 Furthermore we introduced
geometric node features G; that used in the prediction of geometric attention values and backbone frame updates. An overview over the architecture is shown in Figure/..3 The backbone update block presented in Algorithm 6
For details on the edge update we refer
A.2.2 Invariant Point Attention
In the following we provide short overview over the IPA architecture.
Invariant point attention uses local frames to construct the messages between nodes, where the coordinate frames are given by the frames of the individual residues. The full procedure is presented in A visualized in A.6 The part of IPA that processes geometric information (red nodes in A.6 can be summarized as follows:
1. Learn a certain number of local vector valued queries, keys and values q k, v.
2. In the calculation of attention scores, transform the local queries and keys into the global frame via 7; qi and calculate the squared distance between them. Importantly they should be thought of as points in 3D space rather than vectors, meaning that they change under
25j
Σ
Linear
Linear
Linear
h.P
Norm
Linear
q.
Actention
.h.p
Linear
T.
Figure A.6: High level overview of invariant point attention [32 Blue nodes represent layers which process scalar information, while red nodes represent layers which process geometric information.

---

12) Algorithm Invariant point attention (IPA)
32
1:
procedure [sA((s:) (Zig) (Ti). Nhed 8, 128, N
query points
8, N paint values
2:
q. kh.
v LinearNoBias (s,)
q
k
v E RR, h E {I,..., Necad
3:
q
hpp hp
LinearNoBias (s;)
q
hp
R P [1, [I...., Nueery points
4:
vlp LincarNoBias (s:)
D V
hp
E R
, f1..... point values
5:
LinearNoBiee (Zig)
6:
WC
9. eaes poin
2
7:
W7
8:
4 softmax,
WI Ve J. k2. +bg
1 "ogk"kk
>/we
Σn T,
-T; R2 1))
2
9:
o Σ, Zij
10:
o E, a.j
0 T, E3 aij T3 o v/
11:
5 Linear concata.p ( oft The 6 1)
12:
return {
14: end procedure
Algorithm 2 Original backbone update [32]
1: procedure BACKBONFUPDATH((() (T:))
2: bi, di, f = Linear(s:)
3: (a;, by. Ci d:) = (1,b,, d V 1 bg + d.
4: R. QuaternionToMatrinn
b,, Cz d:)
5:
Т, - (R,,, ())
6: return T; o T;
7: end procedure
translations, which vectors would not, and also making the L norm of their separation somewhat natural map to construct invariant attention scores.
3. In the message passing step. transform vector valued features from the neighboring frame the node frame i via T T3 q and aggregate the vector valued information all neighbors.
4. Finally, concatenate the output vectors along with their norm with the remaining scalar output features and put them through a final linear layer.
The update of the backbone frames, which is used in conjunction with IPA in 32 67 is accomplished by learning an SE(3) transformation per frame which is concatenated with the current. frame representation. To this end, the network predicts quaternion for the rotation and translation vector from the scalar node features s; as shown algorithm
A.2.3 Clifford frame attention
An overview over the CFA architecture is given in Figure A.7 The proposed changes to the original IPA and FrameDiff architecture are shown in algorithms and highlighted in blue. They can be summarized as follows:
1. We replace polut-valued attention values with multivectors Vhv and also introduce geometric node features G; E G3.0.1-
2. We compute node features Tee containing aggregated relative transformations between frames T; and T3
25

---

26j
Linear
Geometric MP
G
Linear
V
Linear
Bilinear
Linear
9i
fs
k
Attention
ManyBody
di
Linear
qi
Norm
T.
G
Figure A.7: Overview of Clifford frame attention. Blue nodes represent layers which process scalar information, red nodes represent layers which process point valued information and orange nodes represent layers which process features the PGA. The central innovation is the novel construction of geometric messages, summarized in the grey box. To retain readability we omit the calculation of relative frame transformations the chart.
T = I.
(43)
These features are concatenated with the remaining geometric features and also passed directly to the backbone update block via residual connection.
3. Messages formed by applying geometric bilinear layers from 13 to node features of each node pair, as described in algorithm 4 The EquiLinear layer refers to the most general equivariant linear layer in PGA, also introduced in written as
13 which for input features X,, be
EquiLinear Xx.)
Σ
Σ Wkam m))x +
knnm Go (X
(44)
k=
Although we do not need this property for enforcing equivariance, we use it for its parameter efficiency and geometric inductive bias.
4. We construct implicit higher body messages inspired by MACE [6]. Bilinearity of the geometric product and the join means that repeated products of aggregated two-body messages correspond sum of N-body messages shown in eq. TT In algorithm 5
use the geometric product and the join to construct three-body messages from aggregated two-body messages, stored in node features A.
5. In addition to the Euclidean norm we also compute the infinity norm (see Definition A.11 of multivector features after message passing.
6. In the Backbone update step (Algorithm 6). concatenate scalar and geometric features along the feature dimension and pass them through an ML.P to predict multivectors R. and S that parameterize rotation and translation respectively. These are then multiplied with the current frames to compute frame updates. The main difference to the backbone update as the inclusion of geometric features and the use of the MLP. The prediction of rotor and translator instead of rotation matrix and translation vector is just a matter of representation.
used in [32 67
A.2.4 Equivariance of GAFL
As in the original IPA formulation, SE(3) equivariance of the GAFL architecture based on express- ing geometric features in canonically induced local frames, described in section 3.
More specifically, going through the architecture step by step, the attention scores are calculated using the L2 norm of the difference of point features, which is E(3) invariant. Equivariance of the
26

---

Algorithm 3 Clifford frame attention
1:
procedure ) (G.). (Zigh. (Ti) Nheat = 8, 128, N
query points
8, N point values
4)
2:
q. k. v
LinearNoBias (s;)
q kg v E E h {I,..., Nheud
3:
q
LinearNoBias (s:)
hp k E R$ P [I,..., N quury points q
4:
VP
Linear (8.)
V R16
p (1,.... point values
5:
V
EquiLinear (concat, (V?, G
6:
LinearNoBias (zi))
7:
WC
garer poins
8:
WL
[] o q. Tÿ 0 k2 1 (')
9:
a softmax,
UL
Vc
" kt. + +b
2
Σ,
of E, al
11:
of E, a vi
T Ej a
13:
hp EquiL Equirinear (conctt
V T
14:
O hp GeometricBillinaar T: Σ Aij
T V p
Tú, Vith)
O hp GeomeetricMannyyooyyyooooooo
15:
S Linear cconatt..p (oft of O
1 0 T
17:
G. EquiLinear O (P)
18:
return 8; G., T
19: end procedure
Algorithm 4 GeometricBilinear [3
1: procedure GEOMETRICBILINEARRA B)
2: GL EquiLinear (A)
3: GR = EquiLinear (B)
4:
G GcometriicroduueticC GR)
5:
Jz EquiLincar (A)
6:
JR EquiLincar (B)
7: J (J,
8: return EquiLinear (concal (G,J))
9:
end procedure
message aggregation step is guaranteed by the expression
T o dhP
in line 11 of algorithm where Th are SE(3) invariant point values and the frames (T.) transform according to T, T Talobat T; such that the whole expression remains invariant:
T
T
globat Tytobus T3 o T =T. T,; o T...
In GAFL, we do not modify the calculation attention scores from IPA, which means that their invariance remains ensured. During message passing, use the same construction above (see line 11 of Algorithm B. J but use a different representation of SE(3), namely multivector features instead of point features. The choice of representation, however, does not influence the invariance of the whole expression. Also the relative frame transformations T-TT, we compute are invariant. All subsequent layers including the GeometricBilinear layer and the exclusively on invariant node features, hence overall equivariance is retained throughout those layers.
layer operate
27

---

Algorithm 5 GeomeriicMnnyBodyProdde
procedure GFOMETTCCMAAYYHOORRRROCCCCA Wrijke W Waegk)
2:
X EquiLinear(A
A contains aggregated messages
3: Y EquiLinear(A)
Whish (Xn)s (Yn)x) )) YY,,
4: O n ΣnjR
Whisk (Xn)s(Xn)s)
5: return O
6: end procedure
7: b n is a feature dimension, W, W are learnable, (X); is the projection onto the i-th grade of X
Algorithm 6 Backbone update
procedure BACKBONEUPDATE((S)) {G,), (TT'), (T,}) 2: by, Cj de, f MI.P (conccat (8,, G., )
3: R. EmbedRotor (b,, c d;)
4: S; Eemeednranslatorrrrrrrr
5: return T,S.S.
6: end procedure
Finally, in the backbone update step, predict invariant frame update, just like in IPA, which when concatenated with the original frame transforms equivariantly:
T,T Tupdate + T global T; Tupdate.
Permutation equivariance is also maintained by the GAFL architecture, since we use message passing on the fully connected graph. In the setting at hand, however, we break this permutation equivariance intentionally by introducing positional encodings for the nodes, as done in models that rely on original IPA such as RFdiffusion.
A.3 Experiments
A.3.1 Definition of novelty and diversity
For each sampled length, we compute the pairwise Template Modeling (TM) score between designable backhones as measure for similarity between folds (70 and report diversity as the TM score averaged over all lengths,
72g
Diversity
TM(X; x,)
N
1=1 ni m
with the total number N of sampled lengths, the number of designable samples ny at given length, and the x; and x, being the i-th and j-th samples, respectively.
The resulting diversity score reports on how similar are sampled backbones to each other. We compare the structures of designable samples to the natural proteins found in the PDB database using FoldSeek [58) in T'Malign mode, and define novelty as the average of the highest T'M score calculated over all designable samples:
Novelty max TM(X,, x,)
with n the number of all designable samples and x, being the sample from the PDB database with the highest TM score the sample x;
Since diversity and novelty are defined as averaged similarity scores, small values are desirable for the generation of structurally distinct and de novo proteins.
28

---

Table A.2: Performance of different models when generating 10 backbones for each length in
{60, 61..... 128). For strand and helix content, we choose the PDB dataset filtered for lengths 60 to 128 as reference. FrameFlow denotes the model trained on the PDB published in
Method
Designability (t)
Diversity (4)
Novelty (1)
Helix Content
Strand Content
PDB (128)
0.36 (0.00)
0.22 (0.00)
SCOP)
0.33 (0.00)
0.26 (0.00)
Genie-SCOPe
0.72 (0.02)
0.38 (0.00)
0.67 (0.00)
0.66 (0.01)
0.07 (0.01)
Sross-
0.79 (0.02)
0.37 (0.00)
0.68 (0.00)
0.66 (0.01)
0.09 (0.01)
FoldFlow-OT
0.99 (0.00)
0.49 (0.01)
0.83 (0.01)
0.87 (0.00)
0.00 (0.00)
FrameDiff
0.81 (0.02)
0.44 (0.00)
0.72 (0.00)
0.39 (0.01)
0.30 (0.01)
FrameFlow
0.95 (0.01)
0.38 (0.00)
0.75 (0.00)
0.53 (0.01)
0.22 (0.01)
RFdiffusion
0.98 (0.01)
0.43 (0.01)
0.8 (0.00)
0.78 (0.00)
0.07 (0.00)
GAFL
0.96 (0.01)
0.38 (0.00)
0.78 (0.00)
0.49 (0.01)
0.27 (0.01)
A.3.2 Baselines used for comparison with GAFL
We generate backbones using RFdiffusion relying on publicly available weights with default settings (GitHub RFdiff). For Genie, we use the model with the published weights trained either on the SCOPe or SwissProt datasets containing proteins of up to 128 or 256 residues long(GitHun Genie), and generate backbones with sampling for 1000 time steps, as it has demon- strated the best performance. FrfmeDifff was used with the newest published model weights with sampling for 500 timesteps (GitHub FrameDiff). FrameFloww was used with the published weights trained on the same dataset as GAFL (GitHub FrameFlow). The originally model, denoted as FrameFlow" can be found (GitHub FrameFlow (legacy)). We note that the GitHub repository supporting FrameFlow contains an implementation of minibatch OT (TO), which we use for retraining FrameFlow. For FoldFlow we use the optimal transport model (foldflow-ot.pll with inference annealing suggested by [10 and sampling for 500 timesteps (GitHub FoldFlow).
A.3.3 Results for small proteins
In order to evaluate GAFL's performance for small proteins in particular, we evaluate a range of models in the setting from the original FrameFlow (67 paper, where 10 backbones per length in {60, (60,61,... 128} are generated. Apart form the models in Table 1 we also evaluate Genie (37], which was trained on backbones smaller than 300 residues and were therefore excluded from the evaluation in [
We find that GAFL and FrameFlow are the only models capable of generating highly designable backbones with diverse secondary structures for the considered length range, where GAFL outper- forms FrameFlow in designability and secondary structure content. While FoldFlow and RFdiffusion, like GAFL, achieve designabilities over 90%, they generate backbones with 0% and 7% 3-strand content, respectively, indicating mode-collapse towards generating a-helical structures. GAFL also outperforms both REdiffusion and FoldFlow in terms of diversity and novelty. All other models considered have significantly lower designability. For Genie in particular, we observe good novelty but it has the lowest designability among the models considered and also under-represents beta strands.
A.3.4 Results for long proteins
We also evaluate the performance of GAFL for generating proteins up to length 500, which enables to compare GAFL against vector field networks (VFN) [4), which the time of writing have no pub- lished model weights. We thus evaluate GAFL and other baselines trained on the PDB with the same inference settings as in [42], sampling protein backbones for each length in [100, 105,... 10,,...,5000 500). We report the results in Table A.3 For VFN, we can only report on designability since [4] have
different definitions for diversity and novelty respectively and do not calculate the secondary structure content of generated structures.
29

---

Table A.3: Comparison of GAFL with VFN, FrameDiff, and RFdiffusion for the generation
protein backbones at each length {100, 105,... 500}. Values for VFN from [2].
Method
Designability (1)
Diversity ())
Novelty ())
Helix Content
Strand Content
FrameDiff
0.28 (0.02)
0.44 (0.01)
0.70 (0.01)
0.57
0.17
VFN
0.44
REdiffusion
0.71 (0.01)
0.36 (0.00)
0.68 (0.00)
0.52
0.28
FrameFlow"
0.64 (0.02)
0.33 (0.00)
0.70 (0.01)
0.64
0.15
GAFL (Ours)
0.74 (0.01)
0.35 (0.00)
(0.00)
0.68
0.14
*Published model weights
We find that GAFL generates the highest fraction of designable backbones, while also yielding the highest diversity. At the same time RFdiffusion has the best novelty and generates structures with a secondary structure content which is closest to the PDB. The trend that GAFL outperforms RFdiffusion with respect to secondary structure content on small proteins but performs worse on longer proteins also already apparent from Figure
A.3.5 Ablation for CFA on the PDB
As described in Section 4.5 we retrained FrameFlow in the same setup as GAFL with default hyperaarameters from the repository in A.3.2 For both FrameFlow and GAFL, we performed three training runs with the hyperparameters reported Appendix A.3.8 For all runs we selected checkpoints using our checkpointing criterion, described in Section 4.1 In Table we report the performance of the best checkpoints per run.
As extension of Figure we directly compare the designability of GAFL and the published Frame- Flow model by length Figure A. We find that GAFL has higher designability for lengths smaller than 200 and greater than 250. Since we only generate 200 backbones for each length, the margins of error of the individual designabilities overlap, except for length 300. The total designability of GAFL, averaged over all 1000 backbones, however, is significantly better than that of FrameFlow (Table I.
GAFL
FrameFlow
1.0
Designability
0.8
0.6
60
100
150
200
250
300
Length
Figure A.8: Designability of GAFL and the published FrameFlow model as a function of length with standard emrrs obtained by bootstrapping the set of generated samples. 200 backbones are generated for each length, as in Figure
A.3.6 Timestep analysis GAFL
We perform timestep analysis, evaluating the best GAFL checkpoint with a different number of timesteps taken during inference, order to judge impact on the designability metric. To this end we generate 100 backbones for each length in {100, 1,00).,, for different numbers of timesteps. We see that the performance beyond 200 timesteps stays almost constant, suggesting that 200 timesteps represents good tradcoff between performance and inference speed.
30

---

GAFL Designability
0.90
0.85
Designability
0.80
0.75
0.70
0.65
0.60
0.55
20
50
100
200
300
Timesteps
Figure A.9: Designability as a function of timesteps used during inference. 100 backbones for each length in (100, 150. 300} were sampled per each data point. Vertical lines denote the standard deviation.
A.3.7 Inference Efficiency
We report the time needed for sampling protein backbones length 100 on NVIDIA A100 GPU with published model weights and default settings as described in A.3.2
Table A.4: Time needed for sampling protein backbones of length 100 on NVIDIA A100 GPU.
Method
Timesteps
Time per Structure (s)
RFdiffusion
50
21.0
FrameDiff
500
24.3
FoldFlow
500
24.2
FrameFlow
200
6.6
GAFL
200
8.8
A.3.8 Training on PDB
On the PDB dataset, train both GAFL and FrameFlow for 5200 epochs, where one epoch is defined as one iteration over all 4776 clusters, which we define as in FrameDiff' by 30% sequence similarity. The learning rate is increased in 50 warmup steps to 0.0002 and then kept constant for 3500 epochs. From there we use a cosine-annealing schedule to decrease the learning rate to 0.0001 epoch 5000. From epoch N train 5000 to Nurain + NNettct 5200 we employ our checkpointing criterion described Section 4.1 evaluating secondary structures and storing checkpoints every second epoch. We keep the 30 best checkpoints and filter them for checkpoints with secondary structure content deviation of less than dmux < 0.2.
evaluate all filtered checkpoints sampling 40 backbones for In order to select the best checkpoint
each length in { [100, 150, 200, 250, 300} and choosing the checkpoint with the highest designability. We then run new and independent inference runs for all experiments conduct on the selected checkpoint.
A.3.9 Training on SCOPe (Ablation)
On SCOPe, we train both GAFL and FrameFlow for 6500 epochs. We use a constant learning rate of 0.0001 for the whole procedure. From epoch Nuain = 4000 to Nuain + N 6500 we = best
use our ceckpointing criterion, storing checkpoints every 25 epochs. We keep checkpoints also filtering them for checkpoints with secondary structure content deviation of less than d'mxx < 0.2. We evaluate each filtered checkpoint by sampling 10 backbones for each length in (60,6l..... .188) and visualize the resulting designability distribution in Figure A.10
31

---

1.0
GAFL
90
0.8
FrameFlow
Designability (%)
0.6
Loss
84
0.4
82
0.2
80
0.0
78
2
8
Step
le5
GAFL
FrameFlow
Figure A.10: Left: Total train loss averaged over flow matching times 0.5 to 0.75 on SCOPe for GAFL and FrameFlow. Right: Designabilities of 30 checkpoints sampled during three training runs on SCOPe for GAFL and retrained FrameFlow, respectively and filtered for secondary structure content deviation of less than dmax < 0.2. For each checkpoint, sample 10 backbones per length in (60.61....,122 128)..
A.3.10 Model hyperparameters
report the most important hyperparameters the CFA/IPA modules used in GAFL, FrameFlow respectively in Table A.5 For an extensive list of hyperparameters we refer the respective config files on GitHub.
Parameter
GAFL
FrameFlow
Node embedding size Edge embedding size Number of attention heads
240
256
120
128
8
Number of query/key points
8
8
Number of geometric value channels
Number of IPA/CFA blocks
6
6
Table A.5: Comparison of the most important hyperparameters used for the CFA/IPA modules in GAFL and FrameFlow
The model hyperparameters GAFL are choosen such that its total number of trainable parameters roughly equals that of FrameFlow as detailed in Table A.66
Model component
GAFL
FrameFlow
Embedding
135 K
150 K
CFA/IPA
9.2 M
8.4 M
Seq transformer
4.5 M
5.1 M
Node update
1.0 M
1.2 M
Edge update
1.7 M
1.9 M
BB update
169
10 K
Total
16.7 M
16.7 M
Table A.6: Comparison between the number of parameters GAFL and FrameFlow, broken down into contributions from different components.
32

---

A.3.11
Memory consumption
Training on SCOPe with the model hyperparameters listed in A0..10 training hyperparameters given A.3.9 results in GPU memory consumption of 48.2 for FrameFlow and 59.5 GB for
GAFL.
A.4
Miscellaneous
A.4.1 Societal Impact
Societal impact is considered mostly positive, as de novo protein design holds the promise to develop. for example, drugs against discases, personalized therapies against cancer or also new nanomaterials, which out-wesghts potential risks, while of course security concerns remain, see e.g. Baker and Church []]
A.4.2 Implementation and Software Libraries
Our implementation based on the implementation of FrameFlow our code together with the camera ready version of this manuscript. The implementation is in the Python [59] programming language and uses the PyTorch framework of FrameFlow: Numpy [30], Hydra [66], and SciPy [6]]
69
We will publish
45 and further dependencies
'The implementation of FromeFlow is published under the MIT license at https: https://github.com Licrosofttprooeii--raaaa
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction made the following claims, which accurately reflect the papers contributions and scope:
(a) Represent frames as elements of the projective geometric algebra
(b) Geometrically more expressive bilinear products of the geometric algebra
(c) Higher order message passing
(d) High designability and diversity of sampled protein backbones
(e) The proposed method closely captures the distribution of protein secondary structures, while other models with high designability often over-represent alpha helices
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the proposed method were discussed in dedicated para- graph at the end of Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
34

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not introduce any novel theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Full explanation of the experiments is provided in Section and Ap- pendix A.3.2 The source code of the implementation will be published together with the camera ready version the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
35

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The SCOPe dataset used for training the model is publicly available. The source code of the implementation of the method proposed in this work will be published together with the camera ready version of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ public/guides/Den
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.ccc/pulic//guu
Cooe/Sbmeeeeeeeee wwwiibbmmsssiinni ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We the standard data set split of the SCOPe data set. For the baselines, use the standard parameters defined the respective publicly available official implementa- tions. The source code of the implementation of the method proposed in this work will be published together with the camera ready version of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
36

---

Justification: We report errors of the mean where applicable. Also we performed three additional complete training runs of the proposed method and the baseline FrameFlow with different seeds for verification. The results of this verification are provided in Table and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided information on the hardware used for the experiments at the beginning of Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpehghhhttee/eepppppoo///a///a
Answer: [Yes]
Justification: No human subjects or data with privacy concerns was used. Training was only performed on publicly available standard academic datasets. Societal impact considered mostly positive, as de novo protein design holds the promise to develop drugs against diseases and personalized therapies against cancer, which out-weights potential risks, while of course security concerns remain, see e.g. Baker and Church(2024). The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low.
Guidelines:
37

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impact briefly discussed in Appendix A.4.1 while refer the reader to e.g Baker and Church(2024) for recent discussions of potential risks. The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
38

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our work is based on the implementation of FrameFlow, which was published under the MIT license which allows to create derivative works. The license terms will be followed when puiblishing own implementations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets have been introduced.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:No crowd sourcing research with human subjects performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
39

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No insitutional review board approval required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
References NeurIPS Paper Checklist
David Baker and George Church. Protein design meets biosecurity. Science, 383(6681):349 2024.
40


Reviews:
[{"review_id": "gP42EbdPjl", "reviewer": "Reviewer_5KkW", "summary": "This work focuses on the protein backbone generation task and improves FrameFlow, a flow-based protein backbone generation framework, based on geometric algebra. More specifically, this work proposes to represent residue frames with the elements of projective geometric algebra, which allows for the usage of higher-order message passing based on bilinear geometric products. The results in terms of designability, diversity, and the distribution of secondary structure distribution are all good compared with various baselines.", "questions": "See weaknesses.\n\nBesides, it seems GAFL is trained on the smaller dataset (SCOPe-128). I wonder its performance if it is trained on the same dataset (the larger one) used in FrameDiff.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed model architecture is effective according to the experimental results. It is challenging to achieve both high designability and proper distribution of secondary structures, and this work has well addressed the problem.\n- The presentation of the paper is good. The algorithms between the original version of Framediff and the proposed version show the changes clearly.", "weaknesses": "- The novelty of this work is limited. It seems that the application of geometric-algebra-based models on the protein backbone design task. The author might show the novelty of the architecture compared with the existing ones, though they may still not be used in this task.\n- Lack of theoretical analysis or understanding of the proposed architecture. It would be more solid if the author could theoretically show the superiority of the proposed architecture compared with the original IPA.\n- Lack of experiments. There are many other settings that are more useful in practical scenarios beyond unconditional backbone generation, such as motif scaffolding. The authors can refer to [1] for more details. \n- It would be better if the author could provide a more extensive background of geometric algebra considering the audience of this paper may hold a background in machine learning for protein engineering. This would also make this paper more self-containing.\n\n\nReferences:\n\n[1] Huguet, G., Vuckovic, J., Fatras, K., Thibodeau-Laufer, E., Lemos, P., Islam, R., Liu, C.H., Rector-Brooks, J., Akhound-Sadegh, T., Bronstein, M. and Tong, A., 2024. Sequence-Augmented SE (3)-Flow Matching For Conditional Protein Backbone Generation. arXiv preprint arXiv:2405.20313.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and helpful review. Below we will discuss the comments line by line.\n\n**1. Novelty of the architecture**\n\nWe consider the proposed model architecture as novel from a theoretical point of view since, to the best of our knowledge, it comprises the first method that uses multivectors from Geometric Algebra as feature representation in a local frame framework. \nBesides that, it is also, to the best of our knowledge, the first time an architecture that uses Projective Geometric Algebra is applied in the context of protein design.\n\n*Differences to other architectures with Geometric Algebra*\n\nAll previous architectures that relied on Geometric Algebra (e.g. [5,6]) have employed Geometric Algebra layers for their equivariance properties and were formulated as E(3) equivariant graph neural networks.\nIn our case, equivariance is already ensured by working in the canonical local frames induced by the protein backbone, as in the original IPA. This enables to use more expressive layers that do not need to be E(3) equivariant.\nWe use Geometric Algebra layers not for obtaining equivariance, but for the geometric inductive bias that multivector representation, join, and geometric product offer in contrast to e.g. MLPs on coordinate vectors.\n\nOn top of that, we construct higher order messages that represent not only pairwise relationships but also the relationship of three or more nodes. This approach was introduced in MACE [4] but has never been applied together with Geometric Algebra or with local frames. \n\n*Differences to other local frame architectures*\n\nLocal frames in principle enable the use of arbitrary SE(3) representations.\nMost often, vector-, or more generally tensor-representations are used, for instance in IPA and [3].\nThe GAFL architecture, on the other hand, leverages the multivector representation of Geometric Algebra, which increases expressivity as we explain below and sets it apart from other local frame architectures.\nThe performance increase we see over the original IPA formulation suggests that using a multivector representation might also be beneficial for other architectures that rely on local frames.\n\n**2. Theoretical analysis of the architecture**\n\nOne can explicitly show that GAFL contains the original IPA based architecture as a special case, meaning that every function parametrized by IPA can also be parametrized by GAFL, which makes GAFL more expressive than the original IPA. \n\nTo see this one can note that the original point representation of IPA is a subrepresentation of the multivector representation used in GAFL, where points can be embedded as trivectors (see e.g. chapters 2.5 and 6 of [2]). The parametrization of IPA is recovered by choosing weights such that the model only operates on this subrepresentation and ignoring any additionally added layers by setting them to the identity. \n\nThe main theoretical motivations, which make this generalization of IPA actually favorable are:\n\n* The use of the geometric product and join as interaction between residues in the message passing step allows to capture more information on the geometric relation between them than the simple linear aggregation of point features which is done in IPA. As examples these bilinear products are able to compute quantities like distances, angles, areas, and volumes between geometric objects represented in the algebra (see [2]).\n\n* Since PGA enables the representation of frames, residue frames can be used as node/edge features as we do by calculating $\\mathbf{T}_i^{-1}\\mathbf{T}_j$.\n\n**3. More background on Geometric Algebra**\n\nWe agree that more background on Geometric Algebra would help to better follow the theoretical reasoning behind the proposed architecture.\nWe will thus add a self-contained introduction to Geometric Algebra to the appendix, following parts of the introductory presentations in [1,2], which we find well-suited for understanding the Geometric Algebra related theory part in more depth.\n\n**4. Lack of experiments / practical scenarios**\n\nWe consider the task of unconditional backbone generation as well suited for benchmarking the proposed novel architecture, and for demonstrating the benefits of incorporating the proposed Geometric Algebra layers for representing the geometry of protein backbones.\nWhile we consider this as the main contribution of the paper (see general remarks), we believe that the GAFL architecture can also be integrated into motif scaffolding or conformational ensemble prediction as these tasks also rely on capturing the backbone geometry.\n\nFurther we want to note that [1] was published after the paper submission deadline, however, GAFL can also be incorporated in the method presented there, which uses original IPA.\n\nFollowing the suggestions of the reviewers, we performed additional experiments during the rebuttal period, including training on the PDB dataset, a more extensive ablation (Table A.3) and a number-of-timesteps analysis (Figure A.4).\n\n**5. Training on the dataset used in FrameDiff**\n\nWe trained GAFL on the FrameDiff dataset during the rebuttal period and obtained very competitive results for sampling backbones with lengths of up to 300, as is done in FrameDiff and FoldFlow.\nDue to the new results, we consider GAFL as state-of-the-art for protein backbone generation trained solely on the PDB or the SCOPe-128 dataset.\nFor further details, we refer to Table A.1, Figures A.1 - A.3.\n\n**References:**\n\n[1] Doran, C., Lasenby, A., Geometric Algebra for physicists (2003)\n\n[2] Dorst, L., A Guided Tour to the Plane-Based Geometric Algebra PGA Leo Dorst (2020)\n\n[3] Weitao Du et al., Se(3)equivariant graph neural networks with complete local frames (2022).\n\n[4] Batatia I. et al., MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2022)\n\n[5] Ruhe D. et al., Clifford Group Equivariant Neural Networks (2023)\n\n[6] Brehmer J. et al., Geometric Algebra Transformer (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1C51eWAaMw", "reviewer": "Reviewer_ZJDu", "summary": "This paper extends frame-based protein backbone generation with projective geometric algebra. It allows higher-order geometric tensors in frame modeling as seen in EGNNs. Based on FrameFlow, it demonstrates great designability with relatively small increases in computational consumption.", "questions": "It’s coherent to see GAFL has better designability and less diversity/novelty since EGNNs are well-suited to model the structure’s geometric nature. I would like to see how this architecture performs on protein function prediction or structure prediction.", "limitations": "Limitations are discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Although not the first work to introduce geometric algebra to proteins or molecules, it is the first to leverage its advantages in protein backbone generation, achieving competitive results.\n- Introducing concepts from EGNNs like MACE is innovative and effective.\n- The inclusion of DSSP distribution comparisons between different methods is informative.", "weaknesses": "- It is unclear whether this Geometric Algebra Extension maintains SE(3) invariance/equivariance and permutation invariance.\n- In Tables 1 and 2,  what is the length stride used for generation?\n- It would be interesting to see GAFL's performance when trained on the PDB dataset and its comparison with RFDiffusion.\n- GAFL appears to be more complex than the original IPA. Information on its training consumption and runtime is missing. Additionally, it is important to know if GAFL is sensitive to the timesteps used for generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time invested in reading the paper and for their constructive and helpful review. Below we will discuss the comments line by line.\n\n**1. SE(3) and permutation equivariance of GAFL**\n\nAs in the original IPA formulation, SE(3) equivariance of the GAFL architecture is based on expressing geometric features in canonically induced local frames, as described from lines 211 to 216 in the original submission.\n\nMore specifically, going through the architecture step by step, the attention scores are calculated using the $L_2$ norm of the difference of point features, which is E(3) invariant.\nEquivariance of the message aggregation step is guaranteed by the expression $T_i^{-1} \\circ T_j \\circ \\vec{v}_j^{hp}$ in line 11 of algorithm 1, where $\\\\vec{v}_j^{hp}$ are SE(3) invariant point values and the frames $\\\\{T_i\\\\}$ transform according to $T_i \\\\rightarrow T^{global} \\\\circ T_i$ such that the whole expression remains invariant:\n\n$T_i^{-1} \\\\circ {T^{global}}^{-1} \\\\circ T^{global}\\\\circ T_j \\\\circ \\\\vec{v}^{hp}_j = T_i^{-1} \\\\circ T_j \\\\circ \\\\vec{v}^{hp}_j.$\n\nIn GAFL, we do not modify the calculation of attention scores from IPA, which means that their invariance remains ensured.\nDuring message passing, we use the same construction as above (see line 11 of algorithm 2), but use a different representation of SE(3), namely multivector features instead of point features.\nThe choice of representation, however, does not influence the invariance of the whole expression.\nAlso the relative frame transformations $\\\\mathbf{T}_i^{-1} \\\\mathbf{T}_j$  we compute are invariant. All subsequent layers including the *GeometricBilinear* layer and the *ManyBodyContraction* layer operate exclusively on invariant node features, hence overall equivariance is retained throughout those layers.\nFinally, in the backbone update step, we predict an invariant frame update, just like in IPA, which when concatenated with the original frame transforms equivariantly:\n\n$\\\\mathbf{T}_i\\\\mathbf{T}^{update} \\\\rightarrow \\\\mathbf{T}^{global}\\\\mathbf{T}_i\\\\mathbf{T}^{update}.$\n\nPermutation equivariance is also maintained by the GAFL architecture, since we use message passing on the fully connected graph.\nIn the setting at hand, however, we break this permutation equivariance intentionally by introducing positional encodings for the nodes, as done in models that rely on original IPA such as RFdiffusion.\n\nWe thank the reviewer for this suggestion and will add this clarification about equivariance to the camera-ready version of the paper.\n\n**2. Length stride in tables 1 and 2**\n\nFor both tables, we sampled ten backbones for each length in [60,61,...,127,128].\nFor the new results from training GAFL on the PDB, we report the results for sampling 200 backbones for each length in [100,150,200,250,300] (1,000 backbones in total), as is done in FrameDiff, FoldFlow and RFDiffusion in Table A.1.\n\n**3. Training on PDB**\n\nDuring the rebuttal period, we trained GAFL on the PDB dataset and obtained very competitive results, which we report in Table A.1 and Figures A.1 - A.3.\nAs we argue in the overarching comment, due to the new results, we consider GAFL as state-of-the art for generating protein backbones trained solely on the PDB or SCOPe-128.\nFor further details, please refer to the overarching comment.\n\n**4. Training consumption and runtime**\n\nThe introduction of bilinear products which operate on the 16 dimensional algebra indeed leads to an increase in runtime of GAFL compared to FrameFlow. We already report inference time in Table 6 in the supplementary of the original submission, which is about 33\\% higher than the one of FrameFlow but still approximately a factor of five below the other baselines if evaluated with 100 timesteps and a factor of three if evaluated with 200 timesteps as in Table A.1. For identical training hyperparameters, the training time per epoch of GAFL and FrameFlow on one A100 80 GB are:\n\n| Frameflow | GAFL   |\n|-----------|--------|\n| 22 min    | 34 min |\n\nSince Clifford networks are such a novel type of architecture, there are still no optimized GPU kernels available, as mentioned in [1]. Thus future development in this field might as well increase the runtime performance of GAFL.\n\n**5. Timestep analysis**\n\nDuring the rebuttal period, we conducted an analysis on the influence of the number of inference timesteps on designability of the sampled backbones for both the GAFL model trained on SCOPe-128 and for the GAFL model trained on PDB.\nIn Figure A.4, the designability of five sampled backbones for each length in [60,61,...,127,128] is shown for a different number of inference timesteps.\nWe observe that both models already have competitive designabilities for 50 timesteps and that the designability does not improve much when going beyond 200 timesteps.\nThis speaks for the efficiency of our Flow Matching formulation, in which we aim to learn rectified flows that transform noise to data on approximately straight paths as explained in section 2.2 of the paper.\n\nFor other metrics such as diversity and novelty, we did not observe a strong dependency on the amount of timesteps.\n\n**6. Application to protein function and structure prediction**\n\nWe regard the main contribution of the present work to be the introduction of the novel architecture (and of a state-of-the-art model for protein backbone generation).\nIn extensive ablations (Table A.3, original too), we have shown that using the GAFL architecture instead of original IPA is beneficial for protein backbone generation.\nAs this task relies on capturing the geometry of protein backbones, we consider our work as demonstration of the potential of using GAFL for tasks beyond protein generation as well, such as protein folding or conformational ensemble prediction.\n\n[1] Ruhe D. et al. Clifford Group Equivariant Neural Networks (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwCDFdH25X", "reviewer": "Reviewer_yVxp", "summary": "This paper introduces a new architecture based on Geometric Algebra Transformer (GAT) for protein backbone design. By using adapting the protein design IPA architecture with GAT, they train a SE(3) flow matching method to generate protein backbone. The contributions of the papers are essentially on the architecture. They are able to generate designable and diverse (in terms of secondary structure) proteins. They evaluated their proposed architecture on the SCOPe dataset and against standard baseline.", "questions": "See weakness section. \n\nWhile the paper proposes a very interesting idea, I believe some additional experiments, comparisons and ablations are needed for it to pass the acceptance bar at NeurIPS. I am willing to improve my score if the reviewers are able to answer the limits I see in the current manuscript.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "First of all, the paper is clearly written, clearly presented and easy to read. I really like the idea to revisit the IPA architecture with more recent transformer architectures that were developed for SE(3) data.", "weaknesses": "The weaknesses I see are ont the performed experiments and ablation study. \n\ni) I think the authors should have run experiments on the PDB dataset because it makes the comparison between their method and other methods trained on the PDB datasets incomparable (Table 2). As some models were trained on longer proteins, their performance might not be optimal compared to GAFL.\n\nii) I would like to see the model trained on longer proteins. Currently, the largest protein size is 128 which is very small. Some of the competitors were trained on proteins of much larger size. For instance, FrameDiff and FoldFlow were trained on size up to 384. I wonder if the proposed architecture would still have been performing on longer proteins especially on the generated secondary structure. Maybe it would be harder for it and that would be a valuable input about the limits of the current method. If the authors retrain their method on the PDB dataset, I would encourage them to increase the protein length up to 384.\n\niii) I would like to see in the ablation study (Table 3) the use of different body order. The body order was set up at '3' and I would like to see the performance of when it is at least '2'.\n\niv) Some important papers should be discussed. Recently [Mao et al.,] introduced a new architecture to improve IPA with frame-diff. The comparison of GAFL with this method would strengthen the proposed manuscript.\n\n\n[Mao et al] De novo Protein Design Using Geometric Vector Field Networks, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time invested in reading the paper and for their informative and constructive feedback. Below we will discuss the comments line by line.\n\n**1. Training on longer proteins**\n\nDuring the rebuttal period, we trained GAFL on longer proteins (on the FrameDiff dataset of proteins up to length 512) and obtained very competitive results for sampling backbones of length up to 300, as is done in FrameDiff, FoldFlow and RFdiffusion.\nAs we argue in the overarching comment, with the new results, we consider GAFL as state-of-the-art for protein backbone generation of various lengths trained solely on the PDB or SCOPe-128 dataset.\nFor further details, we refer to Table A.1, Figures A.1 - A.3 and the part in the overarching comment on PDB training.\n\n**2. Incomparability to methods trained on the PDB datasets**\n\nHaving trained GAFL on the PDB dataset from FrameDiff makes it more comparable with FrameDiff, FoldFlow and RFdiffusion, which have been trained on (largely) the same dataset.\nIn Table A.2, we report the values of both GAFL trained on the PDB and on SCOPe-128 using the same lengths and metrics as in Table 2 of the original submission.\nGAFL trained on the PDB with longer proteins outperforms GAFL trained on the SCOPe-128 dataset on designability, diversity and novelty and has competitive secondary structure distribution.\nThus we find that, for GAFL, training on larger proteins has no harming effect on the performance for smaller proteins.\n\n**3. Ablation study for different body order**\n\nThe second line in the ablation study in the original submission (Table 3) already corresponds to a body order of 2:\n\nMessages after the application of the *GeometricBilinear* layer (see equation 12) are of body order 2.\nHigher body order messages are implicitly constructed by repeatedly taking products between such 2 body messages as explained in equation 14, which we implemented in GAFL for body order 3 through algorithm 5.\nIn the second line of the ablation study in Table 3 without higher order messages, we essentially remove this layer from the network, resulting in a message passing step which uses body order 2.\nWe will clarify this in the final version of the paper and thank the reviewer for the remark.\n\n\n**4. Comparison with vector field networks**\n\nWe thank the reviewer for bringing the paper of [Mao et al. 2024] to our attention. This paper discusses interesting modifications to IPA, which are in some sense complementary to ours and in principle allow for a combined model that uses both methods as we discuss below.\n\n*Geometric feature representation*\n\nThe vector field networks (VFN) in [Mao et al. 2024] use a set of virtual atoms as geometric features, i.e. a set of point features $\\\\{\\\\vec{q}_k \\\\in \\\\mathbb{R}^3\\\\}$. GAFL, on the other hand, uses multivector valued features $\\\\{\\\\mathbf{V}_k \\\\in \\\\mathbb{G}^{3,0,1}\\\\}$ in the projective Geometric Algebra (PGA). These contain points as subrepresentation but are also able to represent lines, planes and most importantly frames i.e. elements of the special Euclidean group SE(3).\n\n*Residue interactions*\n\nThe central part of VFN is how it models interactions between residue frames. To this end VFN introduces the \"vector field operator\", which models residue interactions in a common local reference frame and thus also allows to utilize nonlinearities such as radial basis functions.\n\nGAFL uses the canonical bilinear operations of PGA namely the geometric product and the join to model interactions between residue frames. In contrast to VFN, which only considers 2-body interactions, GAFL also models higher body order interactions (specifically n=3 in the published version) similar to the construction in MACE [1].\n\n*Scope of modifications*\n\nThe modifications of GAFL focus on learning more expressive geometric node features, which are able to parametrize the target space of residue frames. In the attention mechanism of IPA we thus only modify the calculation of attention values $\\\\mathbf{V}_i$ and their subsequent manipulations.\n\nIn VFN additionally the calculation of attention scores $a_{ij}$ is modified, while GAFL uses the original attention mechanism of IPA.\n\n*Conclusion*\n\nBoth models try to enhance the geometric expressivity of IPA, relying on different representations and interaction layers respectively. Both modifications could be built into one model by adding the improved calculation of attention scores via vector field operators to the GAFL architecture. We think that this would be an interesting line of research and look forward to exploring it in the future.\n\nWe will add above discussion to the camera-ready version of the manuscript.\n\n**GAFL outperforms VFN for unconditional backbone generation**\n\nWe also note that GAFL trained on the PDB outperforms VFN on the metric reported in [Mao et al. 2024].\nAs done there, we sample 5 protein backbones for each length in [100,105,...,495,500] for 500 timesteps and re-fold 8 sequences predicted by ProteinMPNN.\nWe obtain a designability score of 72 for GAFL compared to 56 reported in [2] (see overarching comment).\nUnfortunately, VFN's model weights are not published and a different metric for diversity and novelty is reported, thus we can only compare with the designability value reported in the paper.\n\n[1] Batatia I. et al., MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2022)\n\n[2] Mao et al., De novo protein design using geometric vector field networks (2024)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BdBmpNDuby", "reviewer": "Reviewer_zVgj", "summary": "This paper tackles de-novo protein design, focusing on structure generation, where the goal is to generate novel protein backbones. The paper builds on top of FrameFlow, a well-established and popular method that uses flow matching, a residue frame-based protein backbone representation, and equivariant neural networks (invariant point attention and transformers). The proposed method, however, represents the oriented residue frames as elements of a projective geometric algebra. Different orientations and translations of the residue frames are essentially represented through reflections along a set of planes. The advantage of the approach is that it allows to use more complex operations on this algebra and design higher order messages in the message passing of the equivariant network. This can improve expressivity and performance, as the paper shows. The experiments are carried out on standard, relatively small scale, protein generation benchmarks, where the method demonstrates overall slightly favourable performance compared to baselines.", "questions": "The authors do not in detail write down their basic training hyperparameters, like learning rates, batch sizes, etc. I assume these are all taken exactly from FrameFlow? It would be great if the authors could clarify this.", "limitations": "Limitations and potential negative societal impact have been briefly, but appropriately discussed. I have no concerns.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clarity:** While the math underlying the paper's main contribution is not easy, the high level intuitions are explained well and the paper is overall written clearly and presented well.\n\n**Originality:** The paper essentially extends the Invariant Point Attention (IPA) layers used in previous works with ideas from geometric projective geometry. Such geometric projective geometry methods have been used before, but the integration into IPA and application for protein design seems to be new and original.\n\n**Quality:** The paper is of overall good quality. The paper is well written, the experiments and baseline comparisons carried out as well as the ablations and analyses seem to be appropriate.\n\n**Significance:** I think bringing novel ideas regarding new architectures and representations into the protein design field is timely. However, the results are only very incrementally better than the baselines and the scalability of these architectures is an open question (more below).", "weaknesses": "Methodologically, the paper seems interesting and novel, but the results are somewhat weak. On the important designability metric, the proposed geometric algebra flow matching method is only incrementally better than the most relevant FrameFlow baseline. This becomes clear especially in Table 5 in the Appendix, where the authors re-trained FrameFlow. On diversity and novelty, the method is not better than any baselines, sometimes actually slightly worse. Hence, the claimed performance boosts seem to be barely significant.\n\nMoreover, the trained models are small and the generated protein backbones short. Whether the proposed approach is scalable remains an open question.\n\nThe authors said they kept the number of parameters of the model similar to the FrameFlow baseline, but more details here would be helpful, i.e. which components of the model have how many parameters. Furthermore, it would also be nice to report other properties of the proposed model. Does the proposed architecture converge quicker or slower? How does it affect memory consumption (which affects scalability)?\n\n**Conclusions:** For these reasons, it is overall unclear how practically impactful the proposed method can be. These issues negatively affect the paper's significance. That said, despite these somewhat weak results, I think this is overall a well-written and interesting paper with insightful analyzes, and it proposes novel and original ideas. Hence, I am still leaning towards suggesting acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their detailed and helpful review. Below we will discuss the comments line by line.\n\n**1. Scalability to larger proteins and datasets**\n\nAs suggested by the reviewers, we trained GAFL on the PDB dataset of FrameDiff, which contains proteins of length up to 512 and is approximately ten times larger than SCOPe-128.\nWe obtained very competitive results for sampling backbones of length up to 300, as is done in FrameDiff, FoldFlow and RF Diffusion.\nThese new results demonstrate the scalability of GAFL to both large proteins and large datasets.\nFor further details, please refer to Table A.1, Figures A.1 - A.3 and the part in the overarching comment on PDB training, where we argue that we consider GAFL a new state-of-the-art for protein backbone generation.\n\n**2. Performance on Diversity/Novelty**\n\nRegarding the performance of GAFL on diversity and novelty we would like to refer to the results achieved on the PDB dataset which we report in Table A.1.\nThese results demonstrate that GAFL outperforms the other baselines on PDB with respect to diversity, and is on par with FrameDiff with respect to novelty.\n\n**3. Incrementally better results**\n\nWhile the designability improvement of GAFL over the published FrameFlow model is around ten percentage points (90.5\\% vs 81\\%), we indeed observed in our ablation that by using the original IPA architecture within our training setup (which includes the GAFL checkpoint selection algorithm), we can obtain a performance of up to 88\\% designability for FrameFlow.\nWe thus observe that the absolute improvement in percentage points due to the checkpoint selection algorithm is higher than that of the architecture.\nHowever, regarding that 88.2\\% comprises already a high ratio of designable backbones, we consider an improvement from 88.2\\% to 90.5\\% a more significant step than an improvement by the same number of percentage points in lower regions of designablility.\nAfter all, the fraction of non-designable proteins was significantly reduced from 11.8\\% to 9.5% by the GAFL model architecture.\n\nDuring the rebuttal period, we performed more extensive ablations to compare the architectures and to demonstrate that the difference in performance is significant.\nIn Table A.3, we report the performance of three runs with different random seeds for different ablations of GAFL as median, min and max across these three runs.\nWe observe that the GAFL architecture achieves consistently higher designabilities than the FrameFlow architecture: In this ablation study (Table A.3), row 2 corresponds to GAFL without data filtering, and rows 4 and 5 correspond to FrameFlow with and without the GAFL checkpoint selection criterion respectively.\n\nWe also note that GAFL's good performance for longer proteins (see overarching comment, Table A.1, Figure A.1) sets it apart from other models trained on the PDB.\nTo the best of our knowledge, similar results have not been achieved by any other architecture that does not rely on pretrained weights from folding models.\n\n\n**4. Number of parameters of GAFL**\n\nAlthough we applied fundamental changes to the model architecture by using multivector representations of the Geometric Algebra, we were able to retain a similar amount of parameters as the FrameFlow architecture, by reducing the node embedding size from 256 to 240 and the edge embedding size from 128 to 120.\n\nThese changes result in the following number of parameters, which we will add to the appendix of the final version of the paper:\n\n|| GAFL | FrameFlow |\n|--|--|--|\n|Embedding| 135 K | 150 K |\n|IPA| 9.2 M | 8.4 M |\n|Seq transformer| 4.5 M | 5.1 M |\n|Node update| 1.0 M | 1.2 M |\n|Edge update| 1.7 M | 1.9 M |\n|BB update| 169 K | 10 K |\n|Total| 16.7 M | 16.7 M |\n\nThe complete hyperparameter config files of the model will be published along with the source code of the implementation with the final version of the paper.\n\n**5. Other properties of the model**\n\nIt is challenging to judge the convergence properties of generative models in general since often the train loss has no absolute meaning and effects like mode collapse are difficult to notice from the training loss alone.\nThis is also the case for GAFL.\nStill, we observe consistently better training loss values for GAFL in comparison to the FrameFlow architecture trained in our setup, which we provide as reference in Figure B.1. \n\nWe observe that the training of GAFL is more stable in the sense that the distribution of designabilities for checkpoints selected with our checkpoint criterium is more narrow than for FrameFlow (see Figure B.2).\n\nDespite that both FrameFlow and GAFL have the same number of parameters, the need to store tensors of 16 dimensional multivectors increases the memory consumption of GAFL. More specifically, training on SCOPe with the same training hyperparameters results in the following GPU memory consumption:\n\n| Frameflow | GAFL    |\n|-----------|---------|\n| 48.2 GB   | 59.5 GB |\n\n**6. Training hyperparameters**\n\nWe used the same hyperparameters as FrameFlow to enable a direct comparison between GAFL and FrameFlow and because we consider the hyperparameters as somewhat established. We will include the hyperparameters explicitly in a table in the supplementary in the camera-ready version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers cordially for reviewing our paper and appreciate their helpful comments.
We are happy to read that the reviewers find our approach novel (ZJDu,zVgj) and effective (ZJDu,5KkW), and consider the topic as important (zVgj,yVxp) and challenging (5KkW) and the paper to be clearly written and well presented (zVgj,yVxp,5KkW).
Following the constructive remarks of all reviewers, we conducted several new experiments during the rebuttal period and obtained new results, which, we think, greatly improve the impact of our submission.

We retrained GAFL on the PDB dataset, which is roughly ten times larger than the SCOPe-128 dataset considered in the original submission and comprises proteins with lengths of up to 512.
For all protein lengths considered, GAFL outperforms the other baselines that do not rely on pretrained folding models on the task of unconditional generation of highly designable and structurally diverse backbones.

In the following, we will elaborate on that and address other points of general relevance.

**General remarks on the scope of the paper**

We consider our main contributions to be the introduction of

a) an enhanced architecture for representing protein structure

b) a checkpoint selection algorithm that prevents the mode collapse towards alpha helices of generative models with very high designability and

c) a new state-of-the-art model for generation of highly designable and yet structurally diverse protein backbones that
    is trained on the PDB or SCOPe-128.

**Training on the PDB dataset**

Several reviewers suggested to train GAFL on the Protein Data Bank (PDB) dataset, which comprises longer proteins and is approximately ten times bigger than the SCOPe-128 dataset used in the original submission.
We used the same PDB dataset used for training FrameDiff, which is composed of proteins with lengths of up to 512 clustered by 30\% sequence similarity.
In the following, we report preliminary results from training GAFL on this dataset for three days on two A100 GPUs during the rebuttal period with the same hyperparameters as in the original submission.

As in FoldFlow and FrameDiff, we evaluate GAFL for proteins of length up to 300 on the metrics stated in the original submission and report the results in Table A.1.
GAFL outperforms (or is on par with) all baselines with respect to diversity, novelty, helix and strand content.
On designability, GAFL is only surpassed by FoldFlow-OT, which performs poorly on all other metrics considered (e.g. it samples backbones with strand content below 0.5\%) and by RFdiffusion, which relies on weights from the folding model RoseTTaFold (trained for 1792 V100-GPU days [1]) and is trained for 21 A100-GPU days [2] on a dataset that is significantly larger than the PDB [2, SI 1.4].
Moreover, since GAFL uses only 200 timesteps for the backbones sampled in Table A.1, its inference time is the lowest among all models considered; we also implemented a batched inference mode, which reduces the inference time to 3.1 s per sampled backbone of length 100 with 200 timesteps on an A100 GPU.
From the above we conclude that GAFL is a *current state-of-the* art model for unconditional sampling of highly designable protein backbones with robust secondary structure distribution among models trained solely on the PDB or the SCOPe-128 dataset.

We also investigate the performance by protein length and note that the mode collapse of RFdiffusion towards predominantly helical structures (that we already observed in our experiments in the original submission) persists for proteins below length 150, which is not the case for GAFL trained on the PDB.
Since in most protein design campaigns the goal is to incorporate the desired functionality into the smallest protein possible (the cost of synthetizing and testing proteins experimentally increases with their size), we consider this advantage of GAFL over RFdiffusion to be *highly impactful for future developments and real-world applications* of generative models for protein backbones.

Finally, we evaluate GAFL's performance also for proteins of length up to 500.
In the table below, we report designability scores of backbones of lengths [100,105,...,500] in percent for different numbers of timesteps and ProteinMPNN-sequences and compare with the recent model VFN [4] and the values reported for FrameDiff in [4].

|Designabilities averaged over lengths [100,105,...,500]||||
|-|-|-|-|
|Num Timesteps |500| 500|200|
|Num pMPNN Seq.| 100  |8| 100|
|*FrameDiff*| 38 | 28 | - |
|*VFN*| 56| 44 | - |
|*GAFL*|**72**|**53**|62|

GAFL outperforms both baselines with less timesteps required, while still achieving diversity of 0.35 and strand and helix content of 0.18 and 0.59 for all timesteps and number of sequences considered.
We do not report metrics apart from designability for the baselines as the model weights are not published and different metrics are reported in [4].

**Outlook**

We demonstrate that incorporating the proposed Geometric Algebra layers is beneficial for the task of unconditional protein backbone generation, for which representing the protein backbone geometry is crucial.
We consider this task as a fundamental benchmarking task for further development of application-related conditional generative methods like motif scaffolding.

Further, since the proposed method extends the widely used original IPA architecture, it may be integrated easily into other state of the art architectures and methods for general protein backbone related tasks such as folding or conformational ensemble prediction [1,3,4].

[1] Baek, M. et al. Accurate prediction of protein structures and interactions using a three-track neural network (2021)

[2] Watson, L. et al. De novo design of protein structure and function with RFdiffusion (2023)

[3] Jing, B. et al. AlphaFold Meets Flow Matching for Generating Protein Ensembles (2024)

[4] Mao et al., De novo protein design using geometric vector field networks (2024)

Meta-review: This paper introduces a novel architecture called the Geometric Algebra Transformer (GAT) for protein backbone design. This work leverages geometric algebra in an interesting way. While previous works have explored geometric algebra in the context of proteins or molecules, this work is the first to focus on protein backbone generation, achieving results that are competitive with state-of-the-art methods. The GAT architecture achieves equivariance by combining higher-order message passing with the expression of geometric features in canonically induced local frames. This method allows for more expressive layers that do not require E(3) equivariance, as the geometric algebra layers provide inductive biases through multivector representation, join, and geometric product, which is advantageous w.r.t. traditional MLPs on coordinate vectors, inspired by the original IPA method.

The paper is well-written, with comprehensive experiments, baseline comparisons, ablations, and analyses that effectively support the proposed approach. The rebuttal answers were convincing. I would ask the authors to include the new experiments in the final version of the paper. I found the work a compelling case for considering using geometric algebra beyond E(3) equivariance (which is guaranteed by the canonical frame).
Decision: Accept (poster)