Generating Highly Designable Proteins with Geometric Algebra Flow Matching

Authors: simon  wagner
leif  seute
vsevolod  viliuga
nicolas  wolf
frauke  gräter
jan  stühmer
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 20:01:59

Abstract: We introduce a generative model for protein backbone design utilizing geometric products and higher order message passing. In particular, we propose Clifford Frame Attention (CFA), an extension of the invariant point attention (IPA) architecture from AlphaFold2, in which the backbone residue frames and geometric features are represented in the projective geometric algebra. This enables to construct geometrically expressive messages between residues, including higher order terms, using the bilinear operations of the algebra. We evaluate our architecture by incorporating it into the framework of FrameFlow, a state-of-the-art flow matching model for protein backbone generation. The proposed model achieves high designability, diversity and novelty, while also sampling protein backbones that follow the statistical distribution of secondary structure elements found in naturally occurring proteins, a property so far only insufficiently achieved by many state-of-the-art generative models.

Main paper: 
 introduction: 
 Recent years have shown tremendous progress in applying deep learning to computational chemistry, where applications of learning-based approaches have enabled unprecedented progress across a broad range of problems, such as molecular property prediction [55, 28, 50, 11] , protein-ligand docking [22] , protein structure prediction [32, 3, 38] , and de novo protein design [63, 68, 10, 67, 37] . In case of protein design, state-of-the-art methods typically represent the structure of a protein of N residues as an element of SE(3) N , i.e. as a collection of N frames, each of which describes the position and orientation of an individual protein residue. Among the most successful methods are those based on diffusion models [53, 54, 63, 68, 37, 1] and flow matching [39, 19, 56, 67] , which make use of architectures that incorporate the invariant point attention (IPA) of AlphaFold2 [32] . By modeling a protein through the frames of its backbone, the task of protein structure generation is reformulated as to model the distribution of a set of frames, an inherently geometric problem.Advances in geometric deep learning have lead to architectures that are equivariant towards rotations and translations [14, 21, 15] , which can be regarded as a geometric inductive bias that enhances performance and data efficiency. Most generative models for protein design achieve equivariance by using geometric features that are expressed in the canonical local coordinate frames representing the protein backbone. The coordinates of these features are thus invariant with respect to global rotations and translations, which allows to apply general layers and non-linearities. (B) In PGA, a frame can be represented via the geometric product of four planes. Two of the planes parameterize the frame's rotation around their line of intersection, while the other two encode the frame's translation along the separation vector between them. (C) An exemplary protein backbone structure containing an α-helix and a β-sheet. Lines (red), planes (violet) and Euclidean frames (blue) can all be embedded as elements of PGA, facilitating a geometric inductive bias for learning representations of the abstract geometry of the protein.Another widely used approach to construct equivariant architectures is to embed internal features in symmetry group representations and restrict neural network operations to equivariant functions [55, 28, 50, 11, 7, 6, 36, 52] . Explicit equivariance in these models has usually been limited to the orthogonal group O(3), which contains rotations and reflections, while most models are instead invariant towards translations. Only recently, Brehmer et al. [13] proposed an architecture based on the framework of Ruhe et al. [48] that enables explicit equivariance towards both translations and rotations by utilizing the projective geometric algebra (PGA) [29, 25] .Inspired by their work, we demonstrate that apart from its use as E(3)-equivariant formalism, PGA provides a powerful framework for representing the geometry of protein backbones when incorporated into the local frame formulation of protein design architectures. While we utilize PGA to explicitly represent the frames of the protein backbone as elements of the algebra, PGA additionally provides a strong inductive bias as its elements can represent abstract geometric objects like points, lines and planes, which are well suited to capture the geometry of secondary structure elements, such as α-helices and β-sheets (Figure 1 ). Moreover, bilinear operations of the algebra enable to compute many geometric relations between those objects, such as distances, angles, projections and incidences.Main contributions: We introduce Clifford frame attention (CFA), an extension of the invariant point attention (IPA) architecture of AlphaFold2 [32] by expressing geometric features as elements of the projective geometric algebra and using its bilinear operations to construct geometrically expressive, higher order messages. We incorporate CFA into an existing flow matching framework for protein backbone generation, FrameFlow [67] , and demonstrate in our experiments that the proposed method achieves state-of-the-art performance in the combination of designability, diversity and novelty of generated protein samples. While, especially for small proteins, other models with high designability often over-represent α-helices, the proposed method captures the broad distribution of secondary structures of naturally occurring proteins, which we believe is crucial for designing proteins with vast functionalities. Notably, since IPA is widely used throughout the field, CFA may be readily included in many other protein-related machine learning models.Geometric (Clifford) algebra in neural networks Neural networks that use the Clifford algebra were first proposed by Pearson and Bisset [46] , an extension of the multi layer perceptron (MLP) by Clifford algebras, which was later studied further by Buchholz and Sommer [16] . More recently, Ruhe et al. [49] propose Geometric Clifford Algebra Networks, using geometric (Clifford) algebras based on their Clifford Neural Layers [12] , and extend this framework in [48] to E(3)-equivariant representations. Brehmer et al. [13] propose to use projective geometric algebra, which enables SE(3)-equivariant feature representations, and the representation of frames as elements of the algebra.Generative models for protein design Watson et al. [63] propose RFdifffusion, a generative model for protein backbone design that utilizes the pre-trained protein structure prediction network RoseTTAFold [2] . Yim et al. [68] propose FrameDiff, which defines a diffusion model over a set of frames, SE(3) N , and extend this method within the flow matching framework [67] . Bose et al. [10] propose FoldFlow and its variants, diffusion and flow matching models over SE(3) N . Lin and AlQuraishi [37] propose an equivariant encoder and decoder architecture. Wu et al. [65] train a transformer model to predict angles between adjacent residues. Mao et al. [42] propose vector field networks, which is also an extension of IPA, and employ them in FrameDiff. The main difference to our approach is that VFN uses virtual atoms as geometric features and vector field operators to model interactions, whereas CFA use multivectors and the geometric bilinears of PGA respectively.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: In the abstract and introduction we made the following claims, which accurately reflect the papers contributions and scope: (a) Represent frames as elements of the projective geometric algebra (b) Geometrically more expressive bilinear products of the geometric algebra (c) Higher order message passing (d) High designability and diversity of sampled protein backbones (e) The proposed method closely captures the distribution of protein secondary structures, while other models with high designability often over-represent alpha helices Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 This section provides an introduction to the mathematical frameworks that we refer to in this paper, flow matching and Geometric algebra, and a brief introduction to protein design.A Clifford algebra over a real vector space, typically referred to as geometric algebra, is a powerful mathematical framework for describing geometric objects including points, lines, planes and operations on these objects in an algebraically concise way [31, 26] .In more technical terms, given a vector space V and a quadratic form q : V -→ F from the vector space to the underlying field F, we can construct a geometric algebra as the unitary, associative, non-commutative algebra with the property v 2 = q(v) for every v ∈ V [49] . v 2 = vv denotes the geometric product, the bilinear operation of the algebra, of the vector v with itself. In the geometric context, q may be thought of as the metric of the vector space, meaning that for a vector v, q(v) is its squared norm. Elements of the algebra are called multivectors. They can be constructed by forming geometric products between basis vectors of V and linearly combining them.In this paper we are mainly interested in the projective geometric algebra (PGA) [29, 25] , which we denote as G 3,0,1 . It is the geometric algebra over V = R 4 with vector basis e 0 , e 1 , e 2 , e 3 and q(e 1 ) = q(e 2 ) = q(e 3 ) = 1, q(e 0 ) = 0 .(1) The full algebra has 16 basis elements, which can be grouped in grades according to the number of vector basis elements they are constructed from. Following Gunn [29] , we can interpret elements of different grades as geometric objects such as planes, lines and points in 3D space, as listed in Table A .1 and visualized in Figure A.3. Working in a four dimensional space has the advantage that we are able to represent lines and planes that do not necessarily include the origin, which is crucial for the description of translations as shown below.Central to the proposed method is the fact that PGA allows the representation of elements of the Euclidean group E(3) as elements of the algebra. Given a plane p ∈ G 3,0,1 , an arbitrary geometric object X ∈ G 3,0,1 can be reflected across the plane via the sandwich product,EQUATION)where X is the grade involution that flips the sign of elements with odd grade. A vector in PGA may thus be interpreted both as a plane and as a reflection operator. One can extend this idea using the Cartan-Dieudonné theorem, which states that any E(3) transformation can be represented by repeated reflections. Two consecutive reflections through intersecting planes result in a rotation around the line of intersection and two reflections through two parallel planes correspond to a translation along the separation vector of the planes. The associated elements of PGA are obtained by taking the geometric product of the respective planes. We thus represent each residue frame of a protein by a multivector, a so called motor, corresponding to a rotation followed by a translation as shown in Figure 1 . Similar to eq. 2, a motor M ∈ G 3,0,1 can be applied to an arbitrary element X ∈ G 3,0,1 according toX ′ = MXM -1 .(3) We provide a more detailed introduction to geometric algebra in Appendix A.1.1Flow matching [39] , as a generalization of diffusion models [53, 54] , offers a framework for learning continuous normalizing flows (CNFs) [20] , ϕ : [0, 1] × Ω → Ω that transform a general prior distribution p 0 , defined on the domain Ω, to a target distribution p 1 by evaluating the probability pathEQUATIONat t = 1, where * denotes the pushforward. The flow ϕ t can be expressed in terms of a vector field v t by solving the ordinary differential equation (ODE)EQUATIONIf the vector field v t is known, one can generate samples from p 1 by sampling from the prior and integrating the flow ODE. Lipman et al. [39] showed that v t can be learned by regressing on a vector field u t (x|x 1 ) that is conditioned on a data sample x 1 ∼ p 1 , i.e. by minimizing the loss functionEQUATIONHere, p t (x|x 1 ) is the conditional probability path induced by u t (x t |x 1 ) , which is commonly chosen as linear interpolation between the prior sample x 0 and the target x 1 by setting the conditional flow toEQUATIONfrom which u t (x|x 1 ) can be obtained by forming the time derivative. As shown by [39, 19] , sampling from p t (x t |x 1 ) in eq. 6 can then be realized by transforming a sample x 0 from the prior p 0 to x t ≡ ψ t (x 0 |x 1 ). This formulation in terms of conditional distributions allows to learn dynamic optimal transport (OT) plans from prior to target using minibatch OT as shown by Tong et al. [56] .The framework of CNFs and flow matching can be generalized for sampling from distributions on Riemannian manifolds M, such as SE(3) N , as described by Chen and Lipman [19] . Then, the flow ϕ : [0, 1] × M → M is a diffeomorphism, and a vector field is learned as a smooth map u t : [0, 1] × M → T M to the tangent space T M. A natural choice for the map ψ t (x 0 |x 1 ) on a Riemannian manifold is the connecting path with minimal length, the geodesic.For sampling from the distribution of protein backbones, we represent each protein residue by a frame T = (r, x) ∈ SE(3) that corresponds to a translation x ∈ R 3 and rotation r ∈ SO(3), thus the domain for the flow matching process is M ≡ SE(3) N , as described in FrameDiff [68] . The frame for a given residue is defined through the backbone atoms [N, C α , C], where the translation x is chosen as the displacement of the C α atom relative to the origin, while the rotation r is constructed using a Gram-Schmidt process [32] on the vectors[C -C α , N -C α ].Similar to [67] , we define the conditional flow ψ t (T 0 |T 1 ) as the geodesic between T 0 and T 1 ,EQUATIONWe calculate the time derivative of the flow as described in FrameFlow [67] and regress on it as in eq. 6. As prior p 0 , we choose a 3N -dimensional normal distribution with unit variance N (0, I) for translations and, for rotations, adopt the heuristic trick of using IGSO(3) during training and U(SO(3)) for inference from Yim et al. [67] . Additionally, we use minibatch optimal transport [56] with the equivariant cost function from [34, 10] .Until recently, computational tools for protein design have relied on physics-based energy functions [51, 35, 64] , restricting protein design campaigns to pre-defined topologies, geometries, and secondary structure elements. In contrast, deep generative models learn the probability distribution of protein structures from the training set, enabling less constrained sampling of the structure space.Ideally, the main goal of any generative model should be the creation of structurally diverse and novel proteins, thereby maximizing the access to unseen structures and possibly functions. Since the functionality of proteins is grounded in their structure, it is crucial to extensively cover the broad range of secondary structure elements and topologies found in natural proteins. Typically, the properties of generated proteins are assessed by the scores of diversity and novelty, indicating how similar generated backbones are to each other, and to known native proteins, respectively. While explicit analyses of secondary structure elements of generated proteins are often overlooked, [37, 65] suggest that diffusion models commonly generate redundant protein structures composed mostly of α-helices, which raises the question of how much functionality can be hypothetically encoded in these proteins.A key quality check for generated proteins is their biophysical consistency in the sense that their sequence, indeed, folds into the intended structure [44] . In the case of backbone generation, this is typically evaluated by predicting a sequence from the generated structure with the inverse folding model ProteinMPNN [23] . The obtained sequence is re-folded with ESMFold [38] and if the resulting structure aligns well with the original backbone, the latter is called designable [57, 63] . Thus, the designability metric measures consistency between well-established sequence and structure prediction models, which can be seen as a proxy for biophysical validity and practical realizability [47] .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of the proposed method were discussed in a dedicated paragraph at the end of Section 4 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
geometric algebra flow matching for protein backbone generation: 
 We propose a geometric algebra-based neural network architecture to predict the vector fields in a flow matching process on SE(3) N and call this approach Geometric Algebra Flow Matching (GAFL). The proposed neural network architecture is an extension of the one introduced in FrameDiff [68] , where we replace its central component, the invariant point attention (IPA) block from AlphaFold2 [32] by Clifford Frame Attention (CFA), which we explain in more detail in the next paragraph.Input features are the noised frames T t , pairwise spatial distances, positional encodings of absolute and relative sequence positions, and the flow matching time t. As FrameDiff, we use self-conditioning.The network relies on a series of six blocks, in which the frames are updated consecutively to predict the denoised backbone structure and from this the respective conditional vector fields. Each block uses CFA to perform message passing between protein residues, in particular processing geometric information as given by geometric node features and the current set of frames. The SE(3) invariant output of CFA is fed through an MLP and a transformer [60] and then used to predict frame updates. For a detailed explanation of the architecture, we refer to Appendix A.2.1.The original IPA mechanism uses geometric node features in the form of 3D points for the calculation of attention scores and as queries, keys and values, as described in Appendix A. [68, 67, 10] , we propose to enhance its geometric expressivity by performing the following modifications.PGA features: We replace the point-valued attention values by multivectors, which can encode points, lines, planes and Euclidean frames as shown in Figure 1 . At the same time we decrease the number of channels to retain approximately the same number of parameters.Geometric messages: Instead of linearly combining geometric features in the message passing step, we construct messages utilizing the geometric bilinear layer that was introduced as part of the Geometric Algebra Transformer in Brehmer et al. [13] , effectively calculating geometric product and join, another bilinear operation of PGA (see Definition A.10), between the node features V i and V j (see Algorithm 4). These two operations are able to compute many geometric relations as detailed in Appendix A.1.1 and [25] . The messages m h,p ij from node i to node j can then be written asEQUATIONwhere T i , T j are the frame transformations of the corresponding node, which ensure that the bilinear operations are performed in the same reference frame. When aggregating the messages we make use of the bilinearity of the products to exchange the sum and the bilinear layer to obtain:EQUATIONThe operation is thus performed on a node level and scales linearly with the number of nodes.Higher order message passing: We also incorporate higher order messages into the proposed architecture, that is, messages that depend on more than two nodes and are thus capable of describing relationships beyond pairwise interaction. As in [6] , we use bilinearity of geometric product and join to construct higher order messages by multiplying aggregated two-body messages m and m ′ , e.g.EQUATIONcan be seen as sum of three-body messages mijk . In practice, we use a learnable linear combination of versions of eq. 11 projected onto different multivector grades using the geometric product and the join, as described in algorithm 5.Frames as features: Since we embed both residue frames and geometric node features in G 3,0,1 , it is a natural idea to combine them on a node level such that they can interact via the geometric bilinears during message passing. To this end we compute relative frame transformations for all pairs and aggregate them with the attention weights,EQUATIONWe concatenate these with the remaining geometric features before the construction of geometric messages and also pass them directly to the backbone update block using a residual connection. The full CFA algorithm is provided in Appendix A.2.3.Although PGA would in principle allow to construct a fully equivariant architecture ( [13] ) without using local frames, we decide to keep the local frame formulation of IPA, since it allows to use more general layers and non-linearities. Moreover, the common problem of ambiguous local frame choices that other architectures suffer from [62] is not apparent in our case since backbone residues provide a canonical, geometrically meaningful choice for the local frames. We discuss the equivariance of GAFL in Appendix A.2.4.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not introduce any novel theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We train GAFL2 on a subset of the Protein Data Bank (PDB) dataset [9] comprised of monomeric protein structures with up to 512 residues and perform extensive ablations on the smaller, curated SCOPe dataset [27, 18] filtered by proteins with length of up to 128 residues (SCOPe-128) as in [67, 37] . A representative selection of designable protein backbones generated by GAFL trained on the PDB dataset is illustrated in Figure 2 .With the aim of finding a good balance between designability and secondary structure content in mind, we introduce a checkpointing criterion that takes secondary structure into account. We first train the model for N train epochs. Then, from epoch N train to N train + N select , we calculate the relative occurrence of α-helices and β-strands r α and r β of 100 generated proteins after each epoch. We keep the top k checkpoints in terms of secondary structure content deviation, which we define as EQUATIONTo assess the performance of a given model, we follow a well-established pipeline of self-consistency evaluation [57, 63] . For each generated backbone, we design 8 candidate sequences with Protein-MPNN [23] , which are subsequently refolded with ESMfold [38] , and define scRMSD as the smallest RMSD between our generated backbone and the 8 refolded, aligned candidates. As in [63, 68] , we define designability as the fraction of generated samples with scRMSD < 2.0 Å . We also report diversity and novelty of the designable backbones as average TM-similarity [70] scores within the set of generated backbones and with respect to the PDB correspondingly (see Appendix A.3.1 and [67] ). To evaluate how well the secondary structure distribution of the training set is captured, we calculate the average helix and strand content of all designable backbones using the DSSP algorithm [33] .At the task of generating backbones of up to 300 residues, we compare GAFL trained on the PDB to the diffusion models RFdiffusion [63] and FrameDiff [68] and to the flow matching models FoldFlow [10] and FrameFlow [67, 69] . At the time of submitting the paper, FrameFlow had not been trained on the PDB yet, however, we include it in our results as contemporary work. Further, we perform an ablation study for generating smaller backbones, where we compare GAFL models to the originally published FrameFlow [67] model, which was trained on the SCOPe dataset with backbones of up to 128 residues. While all of the baselines above incorporate the original IPA [32] architecture, we also compare GAFL with VFN [42] , in which an alternative modification of IPA is proposed. For all models considered, we generate backbones using published model weights and the respective default inference settings (Appendix A.3.2).We train GAFL for 15 days on two NVIDIA A100-80GB GPUs on the dataset used in FrameDiff, which comprises monomeric structures from the Protein Data Bank (PDB), filtered by a maximum length of 512 and a maximum coil content of 50%, resulting in a total of around 25,000 backbones [68] . As in FoldFlow, we evaluate GAFL and the respective baselines on the task of generating backbones of lengths {100, 150, 200, 250, 300}. For GAFL, we use 200 inference timesteps (Figure A.9). In Table 1 , we report the metrics described in Section 4.2 for each model together with the time needed to generate a single backbone of length 100 on an NVIDIA A100 GPU without batching.We find that GAFL can reliably generate designable, diverse and novel backbones while capturing the statistical distribution of secondary structure elements of natural proteins. In all metrics considered, GAFL outperforms both variants of FoldFlow and is better or as good as FrameDiff. GAFL also outperforms FrameFlow, which was trained on the same dataset, in terms of designability and, at the same time, achieves better secondary structure content. GAFL's designability is only matched by RFdiffusion, which is not directly comparable since it relies on pre-trained model weights from the folding model RoseTTAFold [2] and has around three times more parameters. For diversity, novelty and helix content, GAFL performs better than RFdiffusion. We also observe that GAFL and FrameFlow can generate backbones around three times faster than the other evaluated models. Although protein design campaigns span a wide range of protein sizes [5, 17, 43, 8, 63] , the primary goal remains to encode maximum functionality into the smallest protein possible, driven by the growing costs of synthesis as protein size increases. Thus, we further assess designability and secondary structure content of generated backbones as a function of their length (Figure 3A ). For all lengths considered, backbones generated by GAFL, FrameFlow and RFdiffusion are highly designable while FrameDiff and FoldFlow struggle with the generation of long proteins. The length dependence of designability and secondary structure content for GAFL and FrameFlow is qualitatively similar; however, GAFL achieves overall better results (Table 1 , Figure A.8). Crucially, we find that for generating proteins with less than 150 residues, GAFL is well suited as it is capable of generating highly designable backbones with a similar amount of β-strands as naturally occurring proteins, while RFdiffusion over-represents α-helices. This is also reflected by the Wasserstein distances between the secondary structure distribution of naturally occurring backbones and those generated by GAFL and RFdiffusion, respectively: Purely α-helical proteins are over-represented by RFdiffusion, leading to a higher Wasserstein distance of 0.35 compared to 0.25 for GAFL (Figure 3B ).To compare GAFL with VFN [42] , we evaluate it at generating five backbones for each length in {100, 105, . . . , 500}, some of which are portrayed in Figure 3C . We find that, with a value of 0.74, GAFL outperforms not only VFN (0.44) but also FrameFlow (0.64) and RFdiffusion (0.71) in designability. However, GAFL and FrameFlow over-represent helices for large proteins (Table A .3).In order to investigate the effect of the proposed architectural changes, we conduct an ablation study on the semi-manually curated SCOPe dataset [27, 18] , which clusters proteins by their sequence and structural similarities ensuring that its entries are evolutionary and structurally non-redundant. We further filter SCOPe by the length of up to 128 residues, which results in 3938 proteins. We compare GAFL with FrameFlow and models for which we leave out all proposed architectural changes or only higher order message passing, respectively, while scaling the width of the layers such that the number of parameters remains the same. All models are trained with three different random seeds for 6500 epochs on one NVIDIA A100-80GB GPU, which takes around 6 days, and evaluated by generating backbones for lengths between 60 and 128 as in [67] .We find that GAFL's designability of 90.5% is 8 percentage points higher than that of the published FrameFlow model [67] trained on SCOPe (81.2%) as depicted in Table 3 . If retrained with GAFL's training procedure, FrameFlow's architecture with original IPA achieves a designability of 88.2%. Using PGA-valued features and higher order message passing as proposed in CFA increases the designability by 1.4 and 2.3 percentage points, respectively. While the training procedure has a larger effect on the designability, the improvement due to CFA can be considered to be significant since it persists across different random seeds. This can also be observed in the distribution of designabilities of 30 checkpoints sampled during the training procedure ( For the much larger PDB dataset, we conduct a small ablation study, in which we compare the performance of GAFL with CFA and retrained FrameFlow with original IPA across three different training runs, respectively (Table 2 ). Since GAFL consistently achieves higher designabilities, we can validate that the trend observed on SCOPe also holds true for training on the PDB. Furthermore, we observe a correlation between over-representing α-helices and achieving high designability (Figure 4 ) for 90 checkpoints sampled for each model during checkpoint selection as described in Section 4.1. Crucially, GAFL checkpoints consistently show better designability in the low helix-content regime, which we can attribute to using CFA instead of IPA. FrameFlow-SCOPe 81.2 0.37Our results suggest that GAFL is one of the current state-of-the-art models for unconditional protein structure generation. GAFL outperforms the widely used, pre-trained model RFdiffusion in terms of diversity, novelty and inference time and is on par at designability. Remarkably, GAFL can achieve this performance without requiring pre-trained model weights while other non-pre-trained models often lack designability or show a mode-collapse towards generating helical structures.Especially for generating small, highly designable backbones with distinct secondary structures, GAFL performs well, in particular better than RFdiffusion. Since in most protein design campaigns the goal is to incorporate the desired functionality into the smallest protein possible while exploring a large structural space, we consider this advantage of GAFL to be highly relevant for future developments and real-world applications of generative models for proteins.Our ablation studies provide evidence that achieving high designability without over-representing α-helices can be attributed to the replacement of IPA by the proposed CFA architecture. Since IPA is used in many current state-of-the-art architectures for backbone structure, CFA has the potential to enable improvements in many protein-related tasks.Limitations While the proposed method GAFL achieves state-of-the-art performance in protein backbone generation, there is still room for improvement. We note that achieving high designability without compromising the diversity of secondary structures remains a grand challenge. GAFL does not perfectly capture the secondary structure distribution of natural proteins, especially for large proteins. Further, unconditional protein generation can be regarded as suitable benchmarking task for protein design models, however, most applications require conditional sampling. GAFL can be readily incorporated into existing frameworks for conditioning, e.g. on motifs [69] or symmetry [63] .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Full explanation of the experiments is provided in Section 4 and Appendix A.3.2. The source code of the implementation will be published together with the camera ready version of the paper. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 We introduced Geometric Algebra Flow Matching (GAFL), a flow matching model for protein design based on FrameFlow [67] . GAFL relies on the proposed Clifford Frame Attention (CFA), an extension of the invariant point attention block from AlphaFold2 [32] , by representing residue frames and geometric features of a protein in the projective geometric algebra. This enables the usage of the bilinear operations in the algebra to construct geometrically expressive messages between residues. The experiments demonstrate that the resulting model is state-of-the-art in the combination of the established metrics designability, diversity and novelty and performs notably well at generating designable small backbones with distinct secondary structures, containing β-strands in particular.Given the promising results of the extension of invariant point attention with geometric algebra presented in this work, we look forward to exploring its benefits for other protein-related tasks.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: The SCOPe dataset used for training the model is publicly available. The source code of the implementation of the method proposed in this work will be published together with the camera ready version of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We use the standard data set split of the SCOPe data set. For the baselines, we use the standard parameters defined in the respective publicly available official implementations. The source code of the implementation of the method proposed in this work will be published together with the camera ready version of the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report errors of the mean where applicable. Also we performed three additional complete training runs of the proposed method and the baseline FrameFlow with different seeds for verification. The results of this verification are provided in Table 3 and 2 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provided information on the hardware used for the experiments at the beginning of Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 BackgroundIn this section we give a short introduction to Clifford algebra, loosely following the presentation of Dorst et al. [26] , Dorst and De Keninck [25] and Doran and Lasenby [24] .A Clifford algebra can be constructed from a vector space V by extending it with an additional bilinear operation called the geometric product. We write this algebra as G(V ). Elements of the algebra A ∈ G(V ) are called multivectors and are written in bold. In cases where we want to highlight that an algebra element coincides with a vector in V , we write it with a (bold) lower case letter. Other algebra elements are generally denoted by upper case letters. The geometric product has to fulfill the following properties [24] :Definition A.1: Properties of the geometric product 1. Associativity: (AB)C = A(BC) A, B, C ∈ G(V ) 2. Distributivity: A(B + C) = AB + AC 3. Vectors square to scalars: aa = a 2 ∈ RWe denote the geometric product as juxtaposition of elements in order to distinguish it from other products liker inner and outer product.Algebra basis Given a basis of the underlying vector space V , {e i } n i=0 , where n is the dimension of the vector space, we can use the geometric product to construct a basis for the algebra.{1, e 1 , . . . , e n , e 1 e 2 , e 1 e 3 , . . . , e 1 . . . e n } In general, a geometric algebra over an n dimensional vector space will have 2 n basis vectors. A general multivector can be written as a linear combination of these basis vectors. The basis elements can be further categorized into so called grades according to the number of basis vectors they contain, i.e. 1 will be of grade 0, e i of grade 1, e i e j of grade 2 and so on. With this definition it is possible to define the projection of an arbitrary multivector onto a specific grade [26] .Let G [k] (V ) be the subspace of G(V ) spanned by all multivectors of grade k. We then define the grade projection operatorEQUATIONwhich selects the k-th grade of a given multivector.Multivectors that contain only elements of one grade receive specific names according to their grade. Elements of grade 1 are vectors, elements of grade 2 are bivectors, elements of grade 3 are trivectors and so on.The metric So far we have only specified that the square of a vector should yield a scalar. In order to uniquely define a geometric algebra it is important to specify the exact values of these scalars, which will define a metric on the algebra. The choice of metric is crucial for the properties of the algebra and the geometric interpretation of its elements. For an algebra over an n-dimensional vector space we may choose n scalars, one for each basis vector. It is common to work with a metric that assigns 1 or -1 to all non-zero values. This convention leaves {-1, 0, 1} as possible choices for the scalars. One defines G n,m,l as the geometric algebra with n basis vectors squaring to 1, m basis vectors squaring to -1 and l basis vectors squaring to 0. Having fixed a metric for the vectors of the algebra we can go on to discuss the construction of a norm for general multivectors. The definition of the norm in geometric algebra uses the concept of reversion which is defined as follows:Definition A.3Let a 1 , a 2 , . . . a n ∈ G [1] (V ), n ∈ N be vectors and A = n i=1 a i the geometric product of these vectors, then the reverse of A is defined asEQUATIONi.e. we reverse the order of the vector elements. For a general multivector, which may consist of a sum of elements of the form of 14, we apply the reverse operation to each summand individually.A norm on general mutltivectors can then be defined as:Definition A.4: Norm Let A ∈ G(V )be a multivector, then the norm of A is defined asEQUATION)The Euclidean geometric algebra G 3One of the most prominent examples for a geometric algebra is the Euclidean geometric algebra G 3 , which is the geometric algebra over R 3 with the standard Euclidean metric, i.e. a vector basis {e 1 , e 2 , e 3 } fulfilling e 2 1 = e 2 2 = e 2 3 = 1. Apart from the geometric product we can define two additional operations that will be useful for further analysis. One typically refers to 1 2 (ab + ba) =: a • b as the inner product and 1 2 (ab -ba) =: a ∧ b as the outer product of the two vectors. The geometric product of a and b can then be written asEQUATIONThis definition only holds for vectors. For an extension to arbitrary multivectors see [24] . To further investigate the properties of these products we can look at the square of the sum of two vectors (a + b) 2 . Expanding the product and rearranging the terms yieldsEQUATION)Since all terms on the right side are scalars due to the third property in Definition A.1, we can conclude that the inner product of two vectors is a scalar itself. In fact, 18 corresponds to the well known polarization identity from linear algebra, and since we defined the norm of vectors to be the standard Euclidean norm, the inner product from Definition A.5 is the standard Euclidean inner product.Using 17 we can also introduce the notion of an orthogonal basis. Just as in linear algebra a vector basis {e i } i is called orthogonal if e i • e j = 0 for i ̸ = j. In the following discussion we will always assume an orthogonal basis. As a consequence of 17 for i ̸ = j basis elements anticommute e i e j = -e j e i and the geometric product is equal to the outer product e i e j = e i ∧ e j .Geometric interpretation of the algebra elements As the name geometric algebra suggests, it is possible to assign geometric meaning to the algebra elements with grade greater than 0. Grade one elements or vectors keep their usual geometric interpretation as directed line segments. To find an interpretation for higher grade elements, one can observe that the outer product of two vectors A = a ∧ b defines a homogeneous subspace in the sense that for every vector in the span of a and b the following equation holdsEQUATIONas shown in [26] . Additionally A has a magnitude according to Definition A.4 and also an orientation due to the antisymmetry of the outer product. For example e 1 ∧ e 2 has opposite orientation compared to e 2 ∧ e 1 = -e 1 ∧ e 2 as indicated by the relative minus sign. 2-blades, that is multivectors which can be written purely as the outer product of two vectors, can thus be interpreted as oriented areas, which lie within the subspace spanned by their generating vectors and area equal to the norm of the blade. Analogously 3-blades, multivectors equal to the outer product of three vectors, correspond to oriented volumes. Notably like vectors, 2-blades and 3-blades neither have a position in space nor do they have a specified shape. The only fixed properties are the magnitude of their area/volume as well as their orientation. The geometric interpretation of algebra elements is visualized in A.1.Grades of the same dimensionality are closely related by a relation called duality. Graphically speaking, a plane in 3D space for example can be represented both as the plane itself or by the normal vector that is orthogonal to it. In algebraic terms this translates to e.g. e 3 representing the normal vector of the plane described by the bivector e 1 e 2 . One says that these two elements are dual to each other. In the same way, scalars are dual to trivectors since trivectors, like scalars, have only one degree of freedom in three dimensions and are thus also sometimes called pseudoscalars. Formally duality in G 3 can be defined as follows:Definition A.6: Duality Let A ∈ G 3 and I = e 1 e 2 e 3 . Then the dual of A is given byEQUATIONFinally, it is also insightful to calculate the magnitude of the area represented by a 2-blade. To this end we notice that, using the Einstein sum convention, the outer product of two vectors a = a i e i , b = a j e j can be written asEQUATIONwhere ϵ ijk is the Levi-Civita symbol and × denotes the usual vector cross product. This shows that in G 3 the outer product yields the 2-blade which is dual to the resulting vector of the cross product with the same magnitude. In fact, the outer product can be seen as generalization of the cross product to arbitrary dimensions. The magnitude can be calculated from 15, yieldingEQUATION)where α is the angle enclosed by a and b. This is exactly the area of the parallelogram spanned by the two vectors. Similar calculations can be performed for the case of 3-blades to show that their magnitude is equal to the volume of the parallelepiped spanned by their three generating vectors. These findings further support the idea of interpreting bivectors and trivectors as area and volume elements respectively.Orthogonal transformations Another important class of operations which geometric algebra can describe in an efficient way are orthogonal transformations, that is transformations T : G 3 → G 3 which preserve the inner product between vectors. In group theoretic terms, these transformations form the group O(3). In order to construct such transformations we first look at how reflections are handled in G 3 .Definition A.7: Reflection Let a ∈ G [1]3 be a vector and n ∈ G[1]3 be the unit normal vector of a plane. The reflection of a at that plane is then given byEQUATIONTo see why this indeed corresponds to a reflection, we decompose a into a parallel and orthogonal part with respect to the normal vector n as shown in A.2. Algebraically this can be written asEQUATION)The first summand corresponds to the usual projection formula from linear algebra and can thus be identified as the parallel part. Since the whole expression equals a, the second summand must be the orthogonal part. The reflected vector can now be obtained by reversing the sign of a ∥ .EQUATION)From 26 to 27, we used the antisymmetry of the outer product. This construction of reflections shows that elements of Euclidean geometric algebra can simultaneously be interpreted as geometric objects as well as transformation operators, a concept which also translates to other geometric algebras.To construct general orthogonal transformations we can make use of the following theorem.Theorem A.1: Cartan-Dieudonné theorem Let (V, q) be a nondegenerate space of dimension n, then any orthogonal transformation can be written as a composition of at most n reflections.For a proof see [41] . In the context of G 3 , this implies that any orthogonal transformation of a vector, which in addition to reflections comprises rotations around lines through the origin as well as compositions of both operations, can be written asEQUATIONThe definition can be extended to higher grade blades, by transforming each of its vectors individually.The transformation of a general blade can then be written asEQUATION)The property that the outer product of the transformed vectors is equal to the transformed outer product makes any orthogonal transformation a so called outermorphism. As a consequence, orthogonal transformations preserve the grade of blades, which geometrically translates to the fact that vectors are transformed to vectors, bivectors to bivectors and so on, which is what we would expect from a geometric transformation (a vector will not become a volume when rotated). A formal proof of this property can be found in [26] .The multivector V in 29 is called a versor. Versors together with the geometric product form a group on their own, the so called Pin(3) group. This group is said to be the double cover of O(3) (see [40] ). 30 is both a representation of O(3) and Pin(3). One can generalize this definition to arbitrary geometric algebras in the following way:Definition A.8: The Pin(n,m,l) and Spin(n,m,l) groups Let V = {v ∈ G n,m,l | ∥v∥ = 1} be the set of versors . Then V together with the geometric product forms the group Pin(n, m, l). Let V = {v ∈ G + n,m,l | ∥v∥ = 1} be the set of even versors, i.e. versors that only involve even grades. Then V together with the geometric product forms the group Spin(n, m, l).The projective geometric algebra G 3,0,1 The Euclidean geometric algebra G 3 allows for a powerful description of geometric objects in 3D space. However, it lacks the ability to describe absolute positions. For example, planes parametrized by 2-blades are restricted to go through the origin, and vectors only describe a relative displacement, not points in space. These problems are solved by the projective geometric algebra (PGA), which is the geometric algebra G 3,0,1 with three Euclidean basis vectors squaring to 1, e 2 1 = e 2 2 = e 2 3 = 1 and one null vector squaring to 0, e 2 0 = 0. We thus use an algebra based on a four dimensional vector space to describe 3D space, similarly to what one does with homogeneous coordinates. In the following we will denote PGA vectors which only contain a Euclidean part by ⃗ a.There are multiple ways to interpret the different algebraic elements geometrically. Here, we will focus on the plane-based approach as described in [25] , since this yields the most useful description of Euclidean motions as orthogonal transformations. Elements of grade 1 are interpreted not as points but as planes, i.e. a vector of the form {e 0 e 1 e 2 e 3 } pseudoscalar corresponds to a plane with (Euclidean) normal vector ⃗ n and a distance δ from the origin. In turn, the Euclidean vectors e 1 , e 2 , e 3 correspond to basis planes through the origin, whereas e 0 is interpreted as the plane at infinity. Notably any multiple of 31 of the form α(⃗ n + δe 0 ), α ∈ R represents the same plane.EQUATIONSimilarly to the case of G 3 , we can make use of 19 to obtain a geometric interpretation for blades of higher grade. For 2-blades of the form L = a ∧ b, the subspace spanned by a and b corresponds to all planes which contain the line of intersection between the two original planes. It is thus sensible to take L as representation of this line. Similarly, a 3-blade X = a ∧ b ∧ c corresponds to the subspace spanned by all planes which contain the point of intersection of a, b, and c, and thus 3-blades represent points in PGA. The logic behind this construction is visualized in A.3.PGA also contains additional classes of geometric objects which e.g. result from taking the outer product of parallel planes. For a further discussion of these, we refer to [25] .We have seen that the outer product in PGA can be used to calculate the intersection of geometric objects. In this context it is thus also known as the so called meet. There is also an opposite operation called the join3 , which, as the name suggests, e.g. maps two points to the line which passes through both of them. In order to properly define the join we first extend the concept of duality from G 3 to G 3,0,1 .Definition A.9: Hodge duality Let X ∈ G 3,0,1 be a blade. Then its hodge dual ⋆X is defined viaEQUATIONwhere X e is the Euclidean part of X without e 0 . For general multivectors the hodge dual is performed bladewise.As explained in [25] , this definition is slightly different compared to Definition A.6 since in PGA e 0 does not have a multiplicative inverse. However, the overall idea to map between the different subspaces of equal dimensionality is still the same. In practice the hodge dual maps a basis element simply to the element which contains all the basis vectors not present in the initial multivector (in some situations with an additional minus sign), e.g. ⋆e 0 e 3 = e 1 e 2 (confer [25] for more details).Using the hodge dual one can define the join as follows.Definition A.10: Join Let A, B ∈ G 3,0,1 , then the join between these multivectors is defined asEQUATIONwhere ⋆ -1 is the inverse hodge dual, which differs from the usual hodge dual by a sign for some elements.As mentioned earlier, the join can be viewed as being dual to the meet, linking geometric objects together instead of finding their incidence. In that way, the join of two points is the line connecting both of them and the join of three points results in a plane containing all three points.In PGA a norm for multivectors can be defined analogously to Definition A.4. However, since PGA contains the null vector e 0 , only half of the components of a general multivector contribute to the value of the norm. It is thus useful to define a second norm which depends on all of the components which contain e 0 .Let A ∈ G 3,0,1 , then its infinity norm is given byEQUATIONEuclidean transformations Similarly to the case of G 3 , one can write the reflection of a plane m in another plane p asEQUATIONReflection of higher grade blades, e.g. lines and points, can again be achieved by reflecting each vector in the blade individually and making use of the fact that the reflection is an outermorphismEQUATIONThe crucial difference in comparison to G 3 is that the reflecting plane is no longer restricted to go through the origin. This allows one to do consecutive reflections not only in intersecting planes, but also in parallel planes. Two important special cases are the reflections in two intersecting planes and the reflecions in two parallel planes as shwon in A.4. The first case corresponds to a rotation around the line of intersection by an angle which is twice the angle enclosed by the planes. The second case results in a translation along the distance vector by an amount equal to twice the distance between the planes (see [25] ).Compositions of these two types of transformations make up the special Euclidean group SE(3), which can be seen as the group of rigid body motions. This means that every Euclidean transformation which does not involve reflections can be embedded as an element of the even subalgebra G + 3,0,1 = {A ∈ G 3,0,1 | ⟨A⟩ k = 0, k odd}, as stated in the following theorem: 3) be an element of the special Euclidean group. Then there exists a multivector called motor in the even subalgebra M ∈ G + 3,0,1 with ∥M∥ = 1 such thatEQUATIONwith an arbitrary multivector X ∈ G 3,0,1 , is a representation of T on G 3,0,1 .We emphasize that the expression in 37 can be applied to any geometric object which is representable in the algebra, i.e. it does not matter if X is a point, a vector, or a plane; the correct equation to transform it always takes the above form. This remarkable property can also be found for other important operations in PGA. In fact, we have already mentioned the meet operation which allows to calculate intersections of arbitrary geometric objects A, B ∈ G 3,0,1 via the universal formula A ∧ B, as well as the join which can likewise be applied to pairs of arbitrary objects.We have already seen how to transform objects, find incidences between them, and join them together. Furthermore the basic operations of PGA allow to calculate a host of different metric relations between its elements. Below we provide a non-complete list of examples.Let p 1 , p 2 ∈ G [1] 3,0,1 be planes, L ∈ G [2]3,0,1 be a line and P 1 , P 2 ∈ G [3] 3,0,1 be points, which are all normalized, i.e.∥p 1 ∥ = ∥p 2 ∥ = ∥L∥ = ∥P 1 ∥ = ∥P 2 ∥ = 1,then we can calculate the following relations:• Distance between points P 1 , P 2EQUATION• Distance between point P 1 and line LEQUATION• Distance between point P 1 and plane p 1EQUATION• Angle between planes p 1 , p 2EQUATION• Angle between plane p 1 and line LEQUATIONFor an extensive list of operations see [25] .A.2.1 GAFL architecturez l s l G l T l z l+1 s l+1 G l+1 T l+1Transformer MLPBackbone update Edge Update We adapted the GAFL architecture from FrameDiff/FrameFlow [68, 67] and replaced invariant point attention (IPA) with Clifford frame attention (CFA) (see Algorithm 3). Furthermore we introduced geometric node features G i that are used in the prediction of geometric attention values and backbone frame updates. An overview over the architecture is shown in Figure A.5. The backbone update block is presented in Algorithm 6. For details on the edge update we refer to [68] .In the following we provide a short overview over the IPA architecture.Invariant point attention uses local frames to construct the messages between nodes, where the coordinate frames are given by the frames of the individual residues. The full procedure is presented in algorithm 1 and visualized in A.6. The part of IPA that processes geometric information (red nodes in A.6) can be summarized as follows:1. Learn a certain number of local vector valued queries, keys and values ⃗ q i , ⃗ k i , ⃗ v i . 2. In the calculation of attention scores, transform the local queries and keys into the global frame via T i • ⃗ q i and calculate the squared distance between them. Importantly they should be thought of as points in 3D space rather than vectors, meaning that they change unders i z ij T i Linear Linear q h i k h i ⃗ q h,p i ⃗ k h,p i Attention a h ij Linear Linear v h i ⃗ v h,p i j j Norm Linear s ′ i Figure A.6: High level overview of invariant point attention [32] . Blue nodes represent layers which process scalar information, while red nodes represent layers which process geometric information.Algorithm 1 Invariant point attention (IPA) [32] 1: procedure IPA({s i } , {z ij } , {T i } , N head = 8, c = 128, N query points = 8, N point values = 12)2:q h i , k h i , v h i = LinearNoBias (s i ) ▷ q h i , k h i , v h i ∈ R c , h ∈ {1, . . . , N head } 3: ⃗ q hp i , ⃗ k hp i = LinearNoBias (s i ) ▷ ⃗ q hp i , ⃗ k hp i , ∈ R 3 , p ∈ {1, . . . , N query points } 4: ⃗ v hp i = LinearNoBias (s i ) ▷ ⃗ v hp i ∈ R 3 , p ∈ {1, . . . , N point values } 5: b h ij = LinearNoBias (z ij ) 6:w C = 2 9Nquery points , 7:EQUATION:õh i = j a h ij z ij 10: o h i = j a h ij v h j 11: ⃗ o hp i = T -1 i • j a h ij T j • ⃗ v hp j 12: si = Linear concat h,p õh i , o h i , ⃗ o hp i , ⃗ o hp i 13:return {s i } 14: end procedure Algorithm 2 Original backbone update [32] 1: procedure BACKBONEUPDATE({s i }, {T i })2: b i , c i , d i , ⃗ t i = Linear(s i ) 3: (a i , b i , c i , d i ) = (1, b i , c i , d i )/ 1 + b 2 i + c 2 i + d 2 i 4: R i = QuaternionToMatrix(a i , b i , c i , d i ) 5: Ti = (R i , ⃗ t i ) 6:return T i • Ti 7: end procedure translations, which vectors would not, and also making the L 2 -norm of their separation a somewhat natural map to construct invariant attention scores.3. In the message passing step, transform vector valued features from the neighboring frame j to the node frame i via T -1 i • T j • ⃗ q i and aggregate the vector valued information over all neighbors.4. Finally, concatenate the output vectors along with their norm with the remaining scalar output features and put them through a final linear layer.The update of the backbone frames, which is used in conjunction with IPA e.g. in [32, 67] , is accomplished by learning an SE(3) transformation per frame which is concatenated with the current frame representation. To this end, the network predicts a quaternion for the rotation and a translation vector from the scalar node features s i as shown in algorithm 2An overview over the CFA architecture is given in Figure A.7. The proposed changes to the original IPA and FrameDiff architecture are shown in algorithms 3 and 6, highlighted in blue. They can be summarized as follows:1. We replace point-valued attention values with multivectors V hp i and also introduce geometric node features G i ∈ G 3,0,1 .2. We compute node features T rel containing aggregated relative transformations between frames T i and T j as si zij Ti Linear Linear q h i k h i ⃗ q h,p i ⃗ k h,p i Attention a h ij Linear Gi Linear v h i V h,pEQUATION)These features are concatenated with the remaining geometric features and also passed directly to the backbone update block via a residual connection. 3. Messages are formed by applying geometric bilinear layers from [13] to node features of each node pair, as described in algorithm 4. The EquiLinear layer refers to the most general equivariant linear layer in PGA, also introduced in [13] , which for input features X n can be written asEQUATION)Although we do not need this property for enforcing equivariance, we use it for its parameter efficiency and geometric inductive bias. 4. We construct implicit higher body messages inspired by MACE [6] . Bilinearity of the geometric product and the join means that repeated products of aggregated two-body messages correspond to a sum of N -body messages as shown in eq. 11. In algorithm 5, we use the geometric product and the join to construct three-body messages from aggregated two-body messages, stored in node features A. 5. In addition to the Euclidean norm we also compute the infinity norm (see Definition A.11) of multivector features after message passing. 6. In the Backbone update step (Algorithm 6), we concatenate scalar and geometric features along the feature dimension and pass them through an MLP to predict multivectors R and S that parameterize a rotation and translation respectively. These are then multiplied with the current frames to compute frame updates. The main difference to the backbone update as used in [32, 67] is the inclusion of geometric features and the use of the MLP. The prediction of rotor and translator instead of a rotation matrix and translation vector is just a matter of representation.As in the original IPA formulation, SE(3) equivariance of the GAFL architecture is based on expressing geometric features in canonically induced local frames, as described in section 3.More specifically, going through the architecture step by step, the attention scores are calculated using the L 2 norm of the difference of point features, which is E(3) invariant. Equivariance of the q h i , k h i , v h i = LinearNoBias (s i ) ▷ q h i , k h i , v h i ∈ R c , h ∈ {1, . . . , N head } 3: ⃗ q hp i , ⃗ k hp i = LinearNoBias (s i ) ▷ ⃗ q hp i , ⃗ k hp i , ∈ R 3 , p ∈ {1, . . . , N query points } 4: V p i = Linear (s i ) ▷ V p i ∈ R 16 , p ∈ {1, . . . , N point values } 5: V hp i = EquiLinear (concat p (V p i , G p i )) 6: b h ij = LinearNoBias (z ij ) 7:w C = 2 9Nquery points , 8:EQUATION:õh i = j a h ij z ij 11: o h i = j a h ij v h j 12: T rel h i = j a h ij T -1 i T j 13: Ṽhp i = EquiLinear concat V hp i , T rel h i 14: O hp i = GeometricBilinear T -1 i j a ij T j Ṽhp j T -1 j T i , V hp i 15: O hp i = GeometricManyBodyProduct(O hp i ) 16: si = Linear concat h,p õh i , o h i , O hp i , O hp i , O hp i ∞ , T rel h i 17: Gi = EquiLinear(O hp i ) 18:return si , Gi , T rel i 19: end procedure Algorithm 4 GeometricBilinear [13] 1: procedure GEOMETRICBILINEAR(A, B)2:G L = EquiLinear (A)3: G R = EquiLinear (B) 4: G = GeometricProduct(G L , G R ) 5:J L = EquiLinear (A) 6:J R = EquiLinear (B)7: J = Join(J L , J R ) 8:return EquiLinear (concat (G, J)) 9: end procedure message aggregation step is guaranteed by the expression T -1 i • T j • ⃗ v hpT -1 i • T -1 global • T global • T j • ⃗ v hp j = T -1 i • T j • ⃗ v hp j .In GAFL, we do not modify the calculation of attention scores from IPA, which means that their invariance remains ensured. During message passing, we use the same construction as above (see line 11 of Algorithm 3), but use a different representation of SE(3), namely multivector features instead of point features. The choice of representation, however, does not influence the invariance of the whole expression. Also the relative frame transformations T -1 i T j we compute are invariant. All subsequent layers including the GeometricBilinear layer and the ManyBodyProduct layer operate exclusively on invariant node features, hence overall equivariance is retained throughout those layers. O n = ijk W nijk ⟨X n ⟩ j ⟨Y n ⟩ k i + Wnijk ⟨X n ⟩ j ∨ ⟨Y n ⟩ k i + Y n 5:return O 6: end procedure 7: ▷ n is a feature dimension, W , W are learnable, ⟨X⟩ i is the projection onto the i-th grade of X Algorithm 6 Backbone update1: procedure BACKBONEUPDATE({s i }, {G i }, {T rel i }, {T i }) 2: b i , c i , d i , ⃗ t i = MLP(concat s i , G i , T rel i ) 3: R i = EmbedRotor(b i , c i , d i ) 4:S i = EmbedTranslator( ti )5:return T i R i S i 6: end procedure Finally, in the backbone update step, we predict an invariant frame update, just like in IPA, which when concatenated with the original frame transforms equivariantly:T i T update → T global T i T update .Permutation equivariance is also maintained by the GAFL architecture, since we use message passing on the fully connected graph. In the setting at hand, however, we break this permutation equivariance intentionally by introducing positional encodings for the nodes, as done in models that rely on original IPA such as RFdiffusion.For each sampled length, we compute the pairwise Template Modeling (TM) score between designable backbones as a measure for similarity between folds [70] and report diversity as the TM score averaged over all lengths,Diversity = 1 N N l=1 1 n l (n l -1) n l i=1 n l j=1 j̸ =i TM(x i , x j )with the total number N of sampled lengths, the number of designable samples n l at a given length, and the x i and x j being the i-th and j-th samples, respectively.The resulting diversity score reports on how similar are sampled backbones to each other. We compare the structures of designable samples to the natural proteins found in the PDB database using FoldSeek [58] in TMalign mode, and define novelty as the average of the highest TM score calculated over all designable samples:Novelty = 1 n n i=1 max j TM(x i , x j )with n the number of all designable samples and x j being the sample from the PDB database with the highest TM score to the sample x i Since diversity and novelty are defined as averaged similarity scores, small values are desirable for the generation of structurally distinct and de novo proteins. We generate backbones using RFdiffusion relying on publicly available weights with default settings (noise_scale_ca: 1) (GitHub RFdiff). For Genie, we use the model with the published weights trained either on the SCOPe or SwissProt datasets containing proteins of up to 128 or 256 residues long(GitHub Genie), and generate backbones with sampling for 1000 time steps, as it has demonstrated the best performance. FrameDiff was used with the newest published model weights (best_weights.pth) with sampling for 500 timesteps (GitHub FrameDiff). FrameFlow was used with the published weights trained on the same dataset as GAFL (GitHub FrameFlow). The originally model, denoted as FrameFlow * can be found at (GitHub FrameFlow (legacy)). We note that the GitHub repository supporting FrameFlow contains an implementation of minibatch OT [10] , which we use for retraining FrameFlow. For FoldFlow we use the optimal transport model (foldflow-ot.pth) with inference annealing as suggested by [10] and sampling for 500 timesteps (GitHub FoldFlow).In order to evaluate GAFL's performance for small proteins in particular, we evaluate a range of models in the setting from the original FrameFlow [67] paper, where 10 backbones per length in {60, 61, . . . , 128} are generated. Apart form the models in Table 1 , we also evaluate Genie [37] , which was trained on backbones smaller than 300 residues and were therefore excluded from the evaluation in Table 1 .We find that GAFL and FrameFlow are the only models capable of generating highly designable backbones with diverse secondary structures for the considered length range, where GAFL outperforms FrameFlow in designability and secondary structure content. While FoldFlow and RFdiffusion, like GAFL, achieve designabilities over 90%, they generate backbones with 0% and 7% β-strand content, respectively, indicating a mode-collapse towards generating α-helical structures. GAFL also outperforms both RFdiffusion and FoldFlow in terms of diversity and novelty. All other models considered have significantly lower designability. For Genie in particular, we observe good novelty but it has the lowest designability among the models considered and also under-represents beta strands.We also evaluate the performance of GAFL for generating proteins up to length 500, which enables to compare GAFL against vector field networks (VFN) [42] , which at the time of writing have no published model weights. We thus evaluate GAFL and other baselines trained on the PDB with the same inference settings as in [42] , sampling 5 protein backbones for each length in {100, 105, . . . , 500}.We report the results in Table A . 3 . For VFN, we can only report on designability since [42] have different definitions for diversity and novelty respectively and do not calculate the secondary structure content of generated structures. We find that GAFL generates the highest fraction of designable backbones, while also yielding the highest diversity. At the same time RFdiffusion has the best novelty and generates structures with a secondary structure content which is closest to the PDB. The trend that GAFL outperforms RFdiffusion with respect to secondary structure content on small proteins but performs worse on longer proteins is also already apparent from Figure 3 .As described in Section 4.5, we retrained FrameFlow in the same setup as GAFL with default hyperparameters from the repository in A.3.2. For both FrameFlow and GAFL, we performed three training runs with the hyperparameters reported in Appendix A.3.8. For all runs we selected checkpoints using our checkpointing criterion, described in Section 4.1. In Table 2 , we report the performance of the best checkpoints per run.As extension of Figure 3 , we directly compare the designability of GAFL and the published Frame-Flow model by length in Figure A.8. We find that GAFL has higher designability for lengths smaller than 200 and greater than 250. Since we only generate 200 backbones for each length, the margins of error of the individual designabilities overlap, except for length 300. The total designability of GAFL, averaged over all 1000 backbones, however, is significantly better than that of FrameFlow (Table 1 ). We perform a timestep analysis, evaluating the best GAFL checkpoint with a different number of timesteps taken during inference, in order to judge its impact on the designability metric. To this end we generate 100 backbones for each length in {100, 150, . . . , 300}, for different numbers of timesteps. We see that the performance beyond 200 timesteps stays almost constant, suggesting that 200 timesteps represents a good tradeoff between performance and inference speed. We report the time needed for sampling protein backbones of length 100 on an NVIDIA A100 GPU with published model weights and default settings as described in A.3.2. We then run new and independent inference runs for all experiments we conduct on the selected checkpoint.On SCOPe, we train both GAFL and FrameFlow for 6500 epochs. We use a constant learning rate of 0.0001 for the whole procedure. From epoch N train = 4000 to N train + N select = 6500 we use our ceckpointing criterion, storing checkpoints every 25 epochs. We keep the k = 10 best checkpoints also filtering them for checkpoints with a secondary structure content deviation of less than d max < 0.2. We evaluate each filtered checkpoint by sampling 10 backbones for each length in {60,61,. . . ,128} and visualize the resulting designability distribution in We report the most important hyperparameters of the CFA/IPA modules used in GAFL and FrameFlow respectively in Table A Training on SCOPe with the model hyperparameters listed in A.3.10 and training hyperparameters given in A.3.9, results in a GPU memory consumption of 48.2 GB for FrameFlow and 59.5 GB for GAFL.Societal impact is considered mostly positive, as de novo protein design holds the promise to develop, for example, drugs against diseases, personalized therapies against cancer or also new nanomaterials, which out-weights potential risks, while of course security concerns remain, see e.g. Baker and Church [4] .Our implementation is based on the implementation of FrameFlow [67, 69] 4 . We will publish our code together with the camera ready version of this manuscript. The implementation is in the Python [59] programming language and uses the PyTorch framework [45] and further dependencies of FrameFlow: Numpy [30] , Hydra [66] , and SciPy [61] .Justification: No human subjects or data with privacy concerns was used. Training was only performed on publicly available standard academic datasets. Societal impact is considered mostly positive, as de novo protein design holds the promise to develop drugs against diseases and personalized therapies against cancer, which out-weights potential risks, while of course security concerns remain, see e.g. Baker and Church(2024) . The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Societal impact is briefly discussed in Appendix A.4.1, while we refer the reader to e.g Baker and Church(2024) for recent discussions of potential risks. The presented method does not enable the design of proteins with a certain function, so risk of misuse is considerably low. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Source code and trained model weights are available at https://github.com/hits-mli/gaflIn some literature the roles of meet and join are actually reversed, i.e. the outer product takes the role of the join. This seeming contradiction can be explained by the fact that we follow the approach of interpreting vectors as planes and not as points.The implementation of FrameFlow is published under the MIT license at https://github.com/ microsoft/protein-frame-flow 


Decision: This paper introduces a novel architecture called the Geometric Algebra Transformer (GAT) for protein backbone design. This work leverages geometric algebra in an interesting way. While previous works have explored geometric algebra in the context of proteins or molecules, this work is the first to focus on protein backbone generation, achieving results that are competitive with state-of-the-art methods. The GAT architecture achieves equivariance by combining higher-order message passing with the expression of geometric features in canonically induced local frames. This method allows for more expressive layers that do not require E(3) equivariance, as the geometric algebra layers provide inductive biases through multivector representation, join, and geometric product, which is advantageous w.r.t. traditional MLPs on coordinate vectors, inspired by the original IPA method.

The paper is well-written, with comprehensive experiments, baseline comparisons, ablations, and analyses that effectively support the proposed approach. The rebuttal answers were convincing. I would ask the authors to include the new experiments in the final version of the paper. I found the work a compelling case for considering using geometric algebra beyond E(3) equivariance (which is guaranteed by the canonical frame).

Reviews: review: paper_summary: paper_summary: This work focuses on the protein backbone generation task and improves FrameFlow, a flow-based protein backbone generation framework, based on geometric algebra. More specifically, this work proposes to represent residue frames with the elements of projective geometric algebra, which allows for the usage of higher-order message passing based on bilinear geometric products. The results in terms of designability, diversity, and the distribution of secondary structure distribution are all good compared with various baselines.
strength_weakness: strength_weakness: - The novelty of this work is limited. It seems that the application of geometric-algebra-based models on the protein backbone design task. The author might show the novelty of the architecture compared with the existing ones, though they may still not be used in this task.
- Lack of theoretical analysis or understanding of the proposed architecture. It would be more solid if the author could theoretically show the superiority of the proposed architecture compared with the original IPA.
- Lack of experiments. There are many other settings that are more useful in practical scenarios beyond unconditional backbone generation, such as motif scaffolding. The authors can refer to [1] for more details. 
- It would be better if the author could provide a more extensive background of geometric algebra considering the audience of this paper may hold a background in machine learning for protein engineering. This would also make this paper more self-containing.


References:

[1] Huguet, G., Vuckovic, J., Fatras, K., Thibodeau-Laufer, E., Lemos, P., Islam, R., Liu, C.H., Rector-Brooks, J., Akhound-Sadegh, T., Bronstein, M. and Tong, A., 2024. Sequence-Augmented SE (3)-Flow Matching For Conditional Protein Backbone Generation. arXiv preprint arXiv:2405.20313.
questions: questions: See weaknesses.

Besides, it seems GAFL is trained on the smaller dataset (SCOPe-128). I wonder its performance if it is trained on the same dataset (the larger one) used in FrameDiff.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper extends frame-based protein backbone generation with projective geometric algebra. It allows higher-order geometric tensors in frame modeling as seen in EGNNs. Based on FrameFlow, it demonstrates great designability with relatively small increases in computational consumption.
strength_weakness: strength_weakness: - It is unclear whether this Geometric Algebra Extension maintains SE(3) invariance/equivariance and permutation invariance.
- In Tables 1 and 2,  what is the length stride used for generation?
- It would be interesting to see GAFL's performance when trained on the PDB dataset and its comparison with RFDiffusion.
- GAFL appears to be more complex than the original IPA. Information on its training consumption and runtime is missing. Additionally, it is important to know if GAFL is sensitive to the timesteps used for generation.
questions: questions: It’s coherent to see GAFL has better designability and less diversity/novelty since EGNNs are well-suited to model the structure’s geometric nature. I would like to see how this architecture performs on protein function prediction or structure prediction.
limitations: limitations: Limitations are discussed in the paper.
paper_summary: paper_summary: This paper introduces a new architecture based on Geometric Algebra Transformer (GAT) for protein backbone design. By using adapting the protein design IPA architecture with GAT, they train a SE(3) flow matching method to generate protein backbone. The contributions of the papers are essentially on the architecture. They are able to generate designable and diverse (in terms of secondary structure) proteins. They evaluated their proposed architecture on the SCOPe dataset and against standard baseline.
strength_weakness: strength_weakness: The weaknesses I see are ont the performed experiments and ablation study. 

i) I think the authors should have run experiments on the PDB dataset because it makes the comparison between their method and other methods trained on the PDB datasets incomparable (Table 2). As some models were trained on longer proteins, their performance might not be optimal compared to GAFL.

ii) I would like to see the model trained on longer proteins. Currently, the largest protein size is 128 which is very small. Some of the competitors were trained on proteins of much larger size. For instance, FrameDiff and FoldFlow were trained on size up to 384. I wonder if the proposed architecture would still have been performing on longer proteins especially on the generated secondary structure. Maybe it would be harder for it and that would be a valuable input about the limits of the current method. If the authors retrain their method on the PDB dataset, I would encourage them to increase the protein length up to 384.

iii) I would like to see in the ablation study (Table 3) the use of different body order. The body order was set up at '3' and I would like to see the performance of when it is at least '2'.

iv) Some important papers should be discussed. Recently [Mao et al.,] introduced a new architecture to improve IPA with frame-diff. The comparison of GAFL with this method would strengthen the proposed manuscript.


[Mao et al] De novo Protein Design Using Geometric Vector Field Networks, ICLR 2024.
questions: questions: See weakness section. 

While the paper proposes a very interesting idea, I believe some additional experiments, comparisons and ablations are needed for it to pass the acceptance bar at NeurIPS. I am willing to improve my score if the reviewers are able to answer the limits I see in the current manuscript.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper tackles de-novo protein design, focusing on structure generation, where the goal is to generate novel protein backbones. The paper builds on top of FrameFlow, a well-established and popular method that uses flow matching, a residue frame-based protein backbone representation, and equivariant neural networks (invariant point attention and transformers). The proposed method, however, represents the oriented residue frames as elements of a projective geometric algebra. Different orientations and translations of the residue frames are essentially represented through reflections along a set of planes. The advantage of the approach is that it allows to use more complex operations on this algebra and design higher order messages in the message passing of the equivariant network. This can improve expressivity and performance, as the paper shows. The experiments are carried out on standard, relatively small scale, protein generation benchmarks, where the method demonstrates overall slightly favourable performance compared to baselines.
strength_weakness: strength_weakness: Methodologically, the paper seems interesting and novel, but the results are somewhat weak. On the important designability metric, the proposed geometric algebra flow matching method is only incrementally better than the most relevant FrameFlow baseline. This becomes clear especially in Table 5 in the Appendix, where the authors re-trained FrameFlow. On diversity and novelty, the method is not better than any baselines, sometimes actually slightly worse. Hence, the claimed performance boosts seem to be barely significant.

Moreover, the trained models are small and the generated protein backbones short. Whether the proposed approach is scalable remains an open question.

The authors said they kept the number of parameters of the model similar to the FrameFlow baseline, but more details here would be helpful, i.e. which components of the model have how many parameters. Furthermore, it would also be nice to report other properties of the proposed model. Does the proposed architecture converge quicker or slower? How does it affect memory consumption (which affects scalability)?

**Conclusions:** For these reasons, it is overall unclear how practically impactful the proposed method can be. These issues negatively affect the paper's significance. That said, despite these somewhat weak results, I think this is overall a well-written and interesting paper with insightful analyzes, and it proposes novel and original ideas. Hence, I am still leaning towards suggesting acceptance.
questions: questions: The authors do not in detail write down their basic training hyperparameters, like learning rates, batch sizes, etc. I assume these are all taken exactly from FrameFlow? It would be great if the authors could clarify this.
limitations: limitations: Limitations and potential negative societal impact have been briefly, but appropriately discussed. I have no concerns.

Comments: title: Author's response
Reply to Rebuttal
Thank you for the rebuttal
Clarification on FoldFlow values in Table 2 of the original submission
comment: We thank the reviewer for the suggestion of evaluating FrameFlow on the PDB. We will include this evaluation in the camera-ready version together with an ablation study on PDB similar to the one we performed on SCOPe.
Thanks for your response!

The responses address most of my concerns.
I have raised my score.
I would like to acknowledge that I have read the authors' rebuttal. I appreciate the extensive discussion as well as the additional results, in particular the experiments on the PDB. One more thing I would suggest, though, is to also re-train a standard FrameFlow model on the PDB data for the final version and include the results. FrameFlow is the most relevant and comparable baseline here and I believe FrameFlow will perform very well on the PDB, too. Although I would guess that GAFL would still be slightly better.

I have raised my score and suggest acceptance of the work.
I have read the rebuttal. I thank the authors for their answer and I hope they will add the novel experiments and discussions to VFN to their manuscript. I have raised my score.
Thanks for the rebuttal. Hope to see more comprehensive analysis on PDB in the future version. I've raised my score.
Note: The values reported in Table 2 in our original submission contained values for FoldFlow that were obtained without inference annealing.
If evaluated with inference annealing, the backbones sampled by FoldFlow-OT are highly designable (99.7\%) but the model underperforms on all other metrics: The strand content is below 0.5\%, which we interpret as mode collapse towards sampling mostly helical (86\%) structures, which is also reflected in poor diversity (0.49) and novelty (0.82) scores.

We will include these values in the final submission.