Geometric Algebra Attention Networks for Small Point Clouds
Authors: Matthew Spellings
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
geometric algebra
equivariance
geometric deep learning
rotation equivariance
permutation equivariance
chemistry
physics
biology
attention
point cloud

Abstract: Much of the success of deep learning is drawn from building architectures that properly respect underlying symmetry and structure in the data on which they operate—a set of considerations that have been united under the banner of geometric deep learning. Often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present rotation- and permutation-equivariant architectures for deep learning on these small point clouds, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.


Under review as a conference paper at ICLR 2022
GEOMETRIC ALGEBRA ATTENTION NETWORKS FOR SMALL POINT CLOUDS
Anonymous authors Paper under double-blind review
ABSTRACT
Much of the success of deep learning is drawn from building architectures that properly respect underlying symmetry and structure in the data which they operate-a set of considerations that have been united under the banner of geo- metric deep learning. Often problems in the physical sciences deal with relatively small sets of points in two- or drre-ddmenssonal space wherein translation, rota- tion, and permutation equivariance useful in practice. In this work, we present rotation- and permunation-cquivarian architectures for deep learning on these small point clouds, composed of a of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathe- matical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields powerful way to impose permutation equivariance. We demon- strate the usefulness these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.
important even vital for models to be
BACKGROUND
Deep learning has been immensely successful in solving wide range of problems over the last several years, driven in large part by identifying appropriate ways to embed structure of data and symmetry of problems directly into the architecture of the network-an idea the core of geometric deep learning (Bronstemn el all (02TT). Some applications of geometric deep learning include the use of convolutional filters in CNNs attain translational equivariance, graph convolutions in graph neural networks for permutation equivariance." @I Building symmetry into the architecture of deep neural network can improve the network's 's data efficiency and guarantee important analytical properties without having to rely on the model to learn to approximate them from training data.
In this work, we derive a family of architectures that is useful in applications from physics to biology. where problems often deal with relatively small point clouds of labeled coordinates. These could be local environments
particles assembling into a crystal (Dshemuchadss et all, 2021), atoms a molecule interacting with other atoms (Chmiela et all 2017), or coarse-grained beads representing parts of a protein (Marrnn & Licleman, 20T3). In many of these applications without the influence of external field,
interested in modeling attributes of the system- such as the identity of a particle's ' local self-assembly environment, or the potential energy of group of atoms- woomh-which which invariant with respect to rotation of the input coordinates, well as permutation in the ordering of points. As shown in Figure II here we attain rotation invariance by constructing functions from rotation-invariant attributes of geometric products of input vectors from geometric algebra, and per- mutation invariance by using an attention mechanism to reduce the set of vector products.
'In this work, we use the following terms to discuss symmetry of functions f and operations p: f is invariant if does not change when a changes: f o f. f and p commute, then say that f is covariant with respect to the operation of p: f pop = po f (some sources call this equivariance or same- cane-equivarianc:; the more general, but we will only discuss f and p that are endomorphisms in
typical detinition of equivariance the context of covariance). Here we use equivariance to broadly mean considerations of covariance as well as invariance (since scalars of interest are typically invariant to translation and rotation in physical applications) for simplicity of discussion.

---

Under review as a conference paper at ICLR 2022
Σ [e Vay
(FFl os)
6w
Σ Cw(Fiit , vo, My 15) 15)
w
Figure 1: Overall strategy for incorporating rotation and permutation equivariance into deep neural networks using attention mechanisms and geometric products. (a) In the simplest form, our proposed structure uses an attention mechanism over the bond lengths of a cloud points, each of which carries value commonly used in graph neural networks. (b) Geometric products can be used to summarize pairs, triplets, or larger tuples vectors in a systematic and geometrically meaningful way. Rotation equivariance can be attained by producing each layer of a model. Here, attention over pairs of rotation-invariant -covariant quantities in by two-dimensional tiles- used. (c) An attention mechanism reduces the of generated geometric products enforce permutation equivariance; the learned attention maps can provide insights into how models operate. In this case, carbon atom in a naphthalene molecule directs its focus broadly around the aromatic rings in which it is situated, rather than focusing exclusively on its nearest neighbors in the molecular graph. Lighter-colored bonds indicate greater attention weight for the two atoms sharing the bond with respect to the starred atom.
RELATED WORK
"Large" point clouds. Point clouds are ubiquitous data structure found in many domains, even outside of the physical sciences. For the purposes of this work, we focus on relatively small sets of points where each point is more information-rich, for example, carrying information about atom identities, local environments, or other attributes in contrast to those commonly found in computer vision and robotics which may represent the geometry of mesh or be sampled from an object, but otherwise not have as much information content. We refer to Guo et all (2020) for survey of this field, but a few of the develoodd notable approaches include PointNet (Qret 2017), deep sets (Zaheer et all 2017), and kernel point convolutions (Thomaa all 2019).
Geometric approaches for small point clouds. Many architectures have been proposed to incorpo- rate rotation equivariance by augmenting graph neural networks with geometric attributes that are such as bond lengths and angles. SchNet (Schiiti all 2017) learns
known to be distance-based convolution filters which used to propagate signals & Meuwly, 201)) also refines node representations based on bond lengths, while incorporating a learnable attention mechanism. DimeNet (Kliepera all 2019) extends the information used to calculate node-level representations to include angles between bonds. GemNets (Klıсpraa et all 2024) are related to DimeNets, and incorporate angle information on quadruplets of points within a graph message passing scheme using an architecture optimized for molecular datasets. GNNFF (Park all 2021) generates -ovariint results by computing weighted sum of modulated input vectors based on a graph message passing scheme.
graphs. PhysNet (Unke
Group representation- based approaches. These methods take advantage of group representation theory by first transforming inputs into a space in which rotation- naation-qquivariant maps are more eas- ily expressed. This set of methods powerful, having been used the past to design rotation- and cermutation-qquivrrann models (Kondor, 2018; Thomas all 20T8; Anderson et all 201)),
and have even been expanded recently for arbitrary groups (Einzi el all 2021). Attention-bass
models have also been utilized in this area: SE(3) Transformers (Fuchs et all (Л2() extend tensor field networks (Thomas et all 2013 with a self-attention mechanism for increased expressivity by incorporating value- and eeometry-dependeee
attention weights.
Multivector-valued networks. The approach described herein applies geometric algebra to train deep learning models point clouds, but using geometric algebra (also known Clifford algebra)
2

---

Under review as a conference paper at ICLR 2022
to structure the operations of neural networks is not concept. Prior work has introduced architectures which generate multivectors using the mathematical structure of geometric algebra as an extension of complex numbers (Pearson & Bissel 1992; 1994) and directly for geometric applications (Bayro-Corrochano et al., 1996; Buchholz & Sommer 20X Meinyk et all 2020).
The approach present here is similar to many of the ideas presented above; however, rather than specifying particular rotation-invariant quantities to utilize or learning maps that operate on irreducible representations. we leverage the structure provided by geometric algebra calculate rotation-invariant and -covariant quantities are of interest. Finally, we use an attention mechanism to attain permutation equivariance in a flexible manner.
HOMAIN-SPECIIN APPLICATIONS
We use three applications, spanning scientific domains from basic physics to biology, to motivate the development of equivariant models.
Crystal structure identification. From atoms colloidal particles, matter often organizes itself into ordered two- or three-dimensional structures. One of the core ideas of materials science is that structure is one of the key determining factors for material behavior. With this perspective in mind, when studying computational models of self-assembling systems we often first identify what struc- tures, if any, have formed in our simulations-a task complicated by naturally-occurinng thermal noise, crystallographic defects, dataset scale, and potentially the complexity of the structures them- selves. Early efforts to automatically characterize structure led to the widely-used Steinhardt order parameters (Stemnaardi et all, [Α83; MickelLetal. 2013; Boattin et 2019), which are rotationally- invariant magnitudes of sums of spherical harmonies over local particle environments. While Stein- hardt order parameters can be effective when studying phase transitions or distinguishing among small number of phases, determining appropriate consider, spherical harmonic order to use, and thresholds to identify behaviors of interest- can be difficult (Mickel et all, 20TH). For this reason, data-driven approaches to analyzing structure have been the subject of great interest in recent years (Clegg 202I). Here we identify local environments extracted from ordered structures using rotation-invarint classifier built on our attention mecha- nism. While this task is fairly straightforward to perform based on geometry for single-componet systems, suitable methods to apply Steinhardt order parameters to systems with multiple types of particles are less obvious; our deep learning architecture more naturally solves this problem using signals associated with each particle type for every bond.
pperparameters-inncuuee
neighborhood size to
Molecular force regression. One of the most dramatic contributions of deep learning to the field of chemistry lies in constructing fast, accurate approximations of expensive physical calculations (Unke 202T: Behler, 202T). Machine learning models can be many orders of magnitude faster than the methods used to generate their training data, which can bring vastly more detailed and longer-time simulations into the realm of possibility. Central to the applicability of these methods issues of symmetry and equivariance: any imperfection in rotational invariance of a learned potential energy function could ruin the thermodynamic behavior of a system, so care must be taken in model design ensure physical behavior. Here we train models using attention mechanism to predict the atomic forces calculated using ab initio molecular dynamics and density functional theory (Chmiela etall 2017). These models are conservative, cquivariant by construction all crucial attributes when using models in simulation.
contuutttion-nvvriann and rotation-
Backmapping of coarse-graining operators. When simulating large molecules-such as proteins or other polymers-it is common to employ coarse graining: a process by which groups of parti- cles are merged into (fewer) distinct beads, enabling faster simulations by decreasing the number of degrees of freedom of the model (Marrink & Ticleman. 2013), Although data-driven approaches have been highly successful to formulate coarse graining operations in the forward direction (that is, from more-detailed to less-detailed systems), some problems are best solved using the origi- nal, fine-grained system coordinates, which are not directly available in coarse-grained simulations. To demonstrate the potential for our geometric algebra attention mechanism models to predict the coordinates of the heavy atoms that form an amino acid from the centers of mass of the neirest-neighbor amino acids in protein entries found within the Protein Data Bank (Berman et all 2000). Machine learning approaches in the past have treated this problem using sim- ple multilayer perceptrons (An. Deshmukfi 2020) by encoding the geometry of the problem as
this task, we train
3

---

Under review as a conference paper at ICLR 2022
image-type data and learning image translation process (Li all 2020). We treat this problem
much like language translation models using transformers (Naswani et al. 20T2), by "translating"
the point cloud of the coarse- -rassee neighboring amino acid coordinates into set of constituent atom coordinates for a central amino acid.
METHODS
In this work we formulate deep neural networks using learnable functions consisting of two main parts: (I) set of geometric products (from the geometric algebra three spatial dimensions) of input vectors; and (2) a permulation-equivariae reduction over these products using attention mechanism. We describe each these aspects below.
ATTENTION FROM GEOMETRIC PRODUCTS
We begin with description of the bare essentials of geometric algebra used in this work. For more details, we refer the reader to Appendix Al Briefly, geometric algebra provides mathematical structure to deal with geometric objects-such such as points and planes-using a common language for arbitrary numbers of spatial dimensions. The primary objects dealt with using geometric algebra are multivectors, which consist of linear combinations of basis elements; in three dimensions, these basis elements are one scalar component, three vector components, three bivector components, and one trivector component. We can calculate the so-called geometric product of two multivectors to yield new multivector; for example, the geometric product of two vectors yields scalar (that is the dot product of the vectors) plus a bivector (related to the cross product of the vectors). The geometric product is not commutative; for two general multivectors A and B, we denote the product C' as C = AB. In this work, we use the geometric product to combine groups of input vectors and systematically extract rotation-invariant quantities of interest such distances, bond angles, and volumes for use in network layers.
We specify computational complexity (i.e. whether compose calculations from pairs, triplets, quadruplets, or larger groups of points) and, for input point clouds with N points, we construct all N2 possible pairs, N triplets, and on," we call these groups of points tuple here. In addition to coordinate F;, we associate set of values to each point indexed by in some space with given working dimension (we avoid calling these vectors decrease the confusion with geometric vectors; values correspond to the non-geometric attributes of the point, such as type embeddings for atomic systems). To create permutation-covarie of point cloud) permutation-invariann functions (producing single output value for the entire cloud), we make use of a simple attention mechanism based on the rotation-invarian attributes of each tuple. Attention has been used widely in applying deep learning to range of problem domains over the last few years, with particular success in the field natural language processing (Vaswant al 2017). Instead of the dot product self-attention commonly used in language models which operates on pairs of elements-we use geometrically-inforee on each tuple. We specify four functions: value-generating function V, tuple value-merging function M, a joining function that summarizes the geometry and tuple value representations J. and soore-generating function S. The functions have the following uses within the network:
functions (producing a value for each input point
attention mechanism that operates
V produces features in the working dimension of the model from the rotation-invarian ge- ometric quantities associated with each sppllsssciiiilll... rotation-invariant attributes of each geometric product PijA.. = FR.., which include bond distances, angles, volumes, and other geometric attributes. Geometric algebra provides a framework to systematically extract these attributes: for a general multivector, the rotation-invariant quantities are the scalar component, trivector component, the norm of the vector component, and the norm of the bivector component. In the networks presented here, we use MLPs with a single hidden layer for V.
"To avoid polynomial scaling, the set of products could be reduced according to the edges of specified graph, the Voronoi diagram of the point cloud, by randomly sampling tuples of points; however, we leave these extensions future work.
4

---

Under review as a conference paper at ICLR 2022
M merges the 1, 2, 3. working dimension of the model. The form of M could be complex learned function, a linear projection for each tuple position, simply the summation of input tuple values.
values associated with tuple of input points into the
J joins the from M. Like M, it could vary in complexity from a simple sum to a complex learned nonlinear function.
invariant representations from V and the tuple value representations
S generates score logits from the representation of cach tuple, incorporating both geometry and value signals associated with points in the tuple. The results from S, passed through softmax function, will yield the weights for the attention mechanism. We use MLPs with single hidden layer for S in the networks presented here.
We first calculate the multivector geometric products Pijk... of all combinations of input vectors indexed by j. k, and so on, up to the specified rank (for example, pairwise attention would produce a two-dimensional matrix of products Pty). We then use M, J, and S --gooeehher with a function extracting the rotation-invarian attributes of geometric product (which are scalar component, trivector component, and the norms of the vector and bivector components, depending on how many input vectors are joined via the geometric product) into liyk... -s follows for network producing permutation-covarian outputs y; for each input point:
Pijk... Fi.
Pijk... invariant(Ppsjk...
Vijk... I ". Uy ok,...))
Wijk... Softmax(S(ugt....
(I)
jk...
Σ Wisk... Viyk...
=
jk..
If permutation -nviaaattiiinnniaiaantn reduction is desired, then the softmax and final be performed over all tuples of the point cloud simultaneously. While J, V, and M could in principle be used to change the working dimension of the network by stacking permutation-covariann layers, in this work we keep the dimension constant for the sake of easily adding residual connections.
If rotation-covariant (rather than rotation-invariant behavior is needed for the output of the network, the same attention structure can be used with slight modifications; here, we extract a vector from learned linear combination of the input vectors Fi Ty. Fk... and the geometrie product Piyk... (which consists of directly taking the vector component from products of odd numbers of input vectors, or multiplying a bivector by the unit trivector produce vector in the case of even numbers of input vectors). These vectors can be combined with learned vector weights One, scalar rescaling each vector (generated by learned function R), and the attention mechanism to yield
Σ Wifk.. Wisk.. R(Uigh...) (an + ai + a22, + azT +.... jk..
(2)
We use R to provide value- and geometry-dependent rescaling of the output vector, making it casier to generate vectors that lie outside of the convex hull of the input point cloud. Because Wiyk... Vigk... and an notation-invariale and the geometric algebra is coordinate free, the resulting output F/ is also rotation-covariant more details of this are shown in Appendix A
MODEL ARCHITECTRRES
demonstrate the utility of our geometric algebra attention scheme by training deep networks to solve three problems appearing in physics, chemistry, and biology. For simplicity, all of the geo- metric algebra attention models presented here utilize pairwise attention with a working dimension of 32 units. Value functions V, score functions S, and rescaling functions R are simple multilayer perceptrons with a hidden width of 64 units, with layer normalization applied to the hidden layer of V. The network for crystal structure identification uses the mean function for merge functions
5

---

Under review as a conference paper at ICLR 2022
cF4 Cu
c/46
of
cFB
AP2-Mg
c113--8i
c-8-2a8
Figure 2: Unit cells of crystal structure prototypes chosen for the structure identification benchmark. Simple and complex structure types -icluddnn two binary structures are included.
M and join functions J, while the other two applications use learned linear projections. More thor- ough descriptions of the model architectures-inchudii Python code under the MIT license-are are detailed in Appendix B and the Supplementary Information.
Crystal structure identification. We train networks using the 12 nearest neighboring particles around a central particle extracted from a structure. We use the neighbor bond coordinate differ- ences Fij and symmetrized one-hot embedding vector [4 - 14:4 4 Lj] encode particle type information for each bond. After projecting these per-bond signals up to 32 dimensions, we per- form two steps permutation-covarice attention (using Equation (1) before a permutation-invariii reduction over the point cloud and final projection down to class logits.
Molecular force regression. To attain translation invariance, we first convert the set of atomic coordinates F; and one-hot type representations t; into the atom-centered pairwise coordinate differ- ence matrices F, F and symmetrized one-hot type vectors [t, t,.t, t, + ty]. After projecting the value signals up to 32 dimensions, we perform six steps of permutation--ooariiat attention (using Equation M1 before a final pernutation-invarian reduction and projection down to a per-atom energy. summed and we calculate the gradient of the per-molecule energy with respect
These energies to the input coordinates to ensure that a conservative force field is learned. We train models by matching the predicted forces to those found in the training data via mean squared error loss.
Backmapping of coarse-graining operators. We first apply two layers of permutation-covarian attention (using Equation M1) to the center-of-mass coordinates and identities of amino acids sur- rounding a central residue, to incorporate local geometry information into the coarse coarse-grained bead representations. We then apply Iranslation layer convert grarse-grained amino acid coordinates to atomic coordinates, which produces vectors according to Equation A by augmenting the tuple representation Viyk... of Equation with labels corresponding the identity of the atom that should be produced, so that the value is calculated as
Farom.ijk... I (Vatom, V( uk. vk....))
Following this layer, two rotation-covariant layers (using Equation D are applied the atom- centered, fine-grained atomic coordinates to further refine them.
RESULTS
Numerical results are reported using the mean and standard enror of the mean over 5 independent samples. Baseline models described in Appendix B

---

Under review as a conference paper at ICLR 2022
CRYSTAL STRUCTURE IDENTIFICATIO
As shown in Figure D. we select 8 prototypes of single- and two-component crystals from the AFLOW Encyclopedia of Crystallographic Prototypes (Mehlettl) 20T7: Hicks 2019). These structures are chosen to demonstrate that models can learn not only geometric information (cF4- Cu and M.22-Mg are similar structures with different stacking of their llose-packed layers; the clathrates cP46-Si and cF 136-Si are also similar to each other, with different arrangement of many common motifs), but also the information encoded within each point (cP2-CsCII and cF'8-ZnS have identical geometry to e/2 2-2 -- and c/'8-C, respectively, differing only by the type assignments of their particles). For each structure, we rescale the unit cell so that the shortest nearest-neighte distance over the structure is 1 before replicating the unit cell to consist of at least 2048 particles. We then create three samples of each structure by adding Gaussian noise with standard deviation of 10-s, 5 10-2 and 0.1 separately the particle coordinates, in order emulate thermal noise. We find the nearest neighbors of each particle using the freud (Ramasubraman et al, 20200 python library and use these point clouds train models identify the source structure type for each particle.
These networks rapidly learn to identify structures after few epochs, with a final overall accu- racy of 98.98% + 0.08% after training for roughly 50 minutes an NVIDIA Titan Xp GPU, We note that we do not expect this to be difficult task in general: a baseline method using Iratton-invariian spherical harmonic featurization of varying-sizee sized local neighborhoods particles (Spellings & Glotzer, 2018) is also able quickly reach 96.1 % + 0.1% accuracy, although we find encoding particle type information to more natural in the graph neural network-likk setting the geometric algebra attention networks than when using spherical harmonic fcaturizations.
MOLECULE FORCE REGRESSION
In method similar Batzner all (2021),, train models predict the atomic forces calculated using ab initio molecular dynamics and density functional theory available in the MD17 dataset (Chmiela et al, 20оп) for eight different molecules. Consistent with previous benchmarks on this dataset, we train networks using the mean squared error loss for each molecule using 1,000 snapshots of forces each as training, validation, and test data sets. Training a model on an individual molecule's data takes between roughly 20 hours (ethanol and malonaldehyde, with nine atoms each) to 90 hours (naphthalene. with eighteen atoms) on an NVIDIA Titan Xp GPU, Test losses, expressed as the mean absolute error over each force component for each sample, are presented in Table II]
Table I: Mean absolute error of force components (in meV for geometric algebra attention net- works, GemNets (Klicpera et all 2021),, NequlP (Batzner ef all 202T), and SchNet (Schlift et all 2017) architectures.
Molecule
This work
GemNet-Q
NequIP
SchNet
Aspirin
13.5 + 0.55
9.4
15.1
58.5
Benzene
6.68 + 0.082
6.3
8.1
13.4
Ethanol
4.6 + 0.14
3.8
9.0
16.9
Malonaldehyde
8.3 + 0.18
6.9
14.6
28.6
Naphthalene
4.1 + 0.11
2.2
4.2
25.2
Salicylic acid
8.0 = 0.23
5.4
10.3
36.9
Toluene
3.8 = 0.17
2.6
4.4
24.7
Uracil
6.0 = 0.058
4.5
7.5
24.3
Overall, these models perform favorably compared to NequlP and SchNet, while being slightly worse than GemNets. The better performance of GemNets could be due to the operation of GemNet- Q on quadruplets of atoms (the geometric algebra attention networks learned here only operate on pairs of bonds from a central atom), the incorporation of energy in addition force labels in the GemNet training scheme (in this work, we choose to only train on force data, rather than using a
Forr applications of this method to systems at nonzero temperature, we would expect to be better-served by using an architecture that generates output distributions instead of only point values. but we disregard this here for simplicity; in other words, here we are teaching models to memorize training data directly, rather than model thermodynamic ensemble.
7

---

Under review as a conference paper at ICLR 2022
Figure 3: Sample pairwise attention maps for four training data molecules (malonaldehyde, aspirin, benzene, and uracil) after filtering out low-attention pairs. The attention maps indicate how strongly the pair of atoms joined by the line affect the representation of the atom indicated with a star, with lighter lines indicating greater influence. Qualitatively, more complex bonding environments such as those on the right tend to have longer-range attention interactions than the simpler environments on the left.
weighted loss incorporating both quantities), simply a more suitable architecture or hyperparam- eters for the GemNet models.
Another useful attribute of attention based models lies in analyzing learned attention maps (Rogers Stal 2020);; to that end, we present sample attention maps of molecules in the training set in Figure B Notably, the attention weights tend to be more broadly distributed when calculating the forces atoms participating in more complex bonding environments- as aromatic may expect due physical effects, such electron delocalization.
we
PROTEIN COARSE-GRAIN BACKMAPPING
We use 19 protein structures, listed in Appendix C. which have high-resolution structural refine- ments (with resolution error less than equal to 1.0 Å) and were published on the Protein Data Bank (Berman al 2000) between 2015 and 2020 for training data. Because the variety of struc- tural refinement workflows could lead to a large variance in structure fidelity compared to the learn- ing capacity of the models, we use the training set error to characterize model performance instead of measuring generalization capability with typical split of training, validation, and test data We compare geometric algebra attention models to baselines using geometrically-nin naïve transformer models in order to probe the impact of rotation equivariance on performance. This baseline model does not have rotation equivariance built in, and must learn to approximate it through data augmenta- tion. We test a series of models using representations of varying width compare the data efficiency of our geometric algebra attention scheme. In addition to comparing the overall accuracy of model predictions, we record the number of epochs required to reach 75% of the total improvement in training error. As shown in Figure H. equivariant models are typically faster to train and more data- efficient, producing higher-quality predictions using fewer parameters than equivariint models. Furthermore, we are able to train larger equivariant models than geometrically-naie transformers begin to show difficulty converging to a low training error above 64 neurons, while equivariant models do not saturate in performance until roughly 96 neurons in width.
models: basic
DISCUSSION
We find the architectures formulated here to be useful for a variety of tasks. Rather than being lim- ited to operating on bond distances and angles as in SchNet (Schütt et all, 2017), PhysNet (Unke & Meuwly, 2019), and DimeNet (Klicpera et al3, 2019), geometric algebra provides a systematic way to build functions with the desired rotation- and cermutation-equivriann incorporate other types of geometric objects (such as the orientation quaternion commonly used for anisotropic particles in molecular dynamics methods (Kamberay et an, 2005)) into the framework. The attention mechanism presented here provides a simple, powerful method to account for both geo- metric and node- signals. The primitives of our geometric algebra attention scheme distances,
with the flexibility to
'In other applications of this method, we would expect data to be more homogeneous so that typical training. validation, and test data splitting methods can be used; here, we use PDB data as proxy for simulation outputs using typical simulation methods.
8

---

Under review as a conference paper at ICLR 2022
a
b
75% Progress Time
Training MAE/A
Geometric Algebra
Transformer
Geometric Algebra
Transformer
Model Width
Model Size
Figure 4: Performance of rotation equivaaranii geometric algebra attention networks compared to
naive transformer-basee architecture using dot product attention. Rotation-equivarian models typically improve both (a) the speed of convergence during training and (b) final accuracy obtained.
areas, angles, and volumes-and the calculated attention weights naturally lend themselves to inter- pretability, which believe will prove useful in distilling insights from trained models.
Although the architectures presented here work well for the problems we have selected, creating geometric products of vectors only subset of the valid combinations that could be generated. In these cases we have carefully chosen sums and differences of input vectors to respect symmetries we would like to impose on the system such as using the pairwise distance of all input coordinates for the molecular force regression task to impose translation invariance- bustte-buuttt it is possible that incorpo- rating learned linear combinations of inputs even generating intermediate multivector quantities could yield more powerful models.
obvious limitation to using higher-degree correlations lies in the computational complexity and memory scaling of generating tuples, which are both proportional to N for neighborhoods of N coordinates and tuples of length r. Polynomial scaling behavior can be ameliorated by restrict- ing which combinations of input points are considered, essentially treating the attention weights of all other combinations as 0. These combinations could be randomly sampled from all valid indices ijk... more mhysically-elleann restrictions, such utilizing the molecular connectivity graph for molecular force regression or edges derived from the Voronoi tessellation for other applications. If smoothness of model predictions
concern-as may be the case for learning general N-body interaction potentials, for example- the architectures presented here could be augmented by incor- porating weights that decay 0 as bonds are broken in the Voronoi diagram graph (Mickel all 2013)
CONCLUSION
In this work, we have presented strategy for developing rotation- and permutationneqquivrriatt neural network architectures by combining geometric algebra and attention mechanisms. These ar- chitectures operate directly on vector, scalar, and other geometric quantities of interest to produce outputs which respect desirable symmetries by construction. We believe that the mathematical sim- plicity and the insights derived from inspecting attention maps particularly appealing aspects of the algorithms presented here. We hope that these architectures will help a wider range of scientific disciplines reap the benefits of geometric deep learning.
REPRODUCIBILLTY STATEMENT
In addition to the mathematical description the architectures used the Methods section, schematic diagrams and pseudocode to generate models for each task in the style of the tensorflow. keras API are listed in Appendix H. Descriptions of baseline methods for the structure identification and coarse-grain grase backmapping tasks also available in Appendix H. The full code used to train and analyze the new model architectures presented here as well as basclines available in the Supplementary Information.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yaxin An and Sanket A. Deshmukh. Machine learning approach for accurate backmapping of coarse- grained models all-atom models. Chemical Communications, ISSN 1364-548X. doi: 10. 1...3///00C02251D..
330 56(65):9312-9315, August 2020.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Harry P. Austin, Mark D. Allen, Bryon S. Donohoe, Nicholas A. Romrrr, Fiona L. Kearns, Rodrigo L. Silveira, Benjamin C. Pollard, Graham Dominick, Ramona Duman, Kamel El Omari, Vitaliy Mykhaylyk, Armin Wagner, William E. Michener, Antonella Amore. Munir S. Skaf, Michael F. Crowley, Alan W. Thome, Christopher W. Johnson, H. Lee Woodcock, John McGeehan, and Gregg T. Beckham. Characterization and engineering of a plastic-degradin aromatic polyesterase. Proceedings of the National Academy of Sciences, 15(1))E4330 E4357, May 2018. ISSN 0027- 8424, 1091-6490, doi: 11.77//pnas.7718844117
Alexander T. Baker, Alexander Greenshields- Lynda Coughlan, James A. Davies, Hanni Uusi-Kerttula, David K. Cole, Pierre J. Rizkallah, and Alan L.. Parker. Diversity within the aden- ovirus fiber knob hypervariable loops influences primary receptor interactions. Nature Communi- cations, 10(1):741, February 2019. ISSN 2041-1723.
doi: 10. 10.1038844146--019--0599999
Matthias Barone, Matthias Müller, Slim Chiha, Jiang Ren, Dominik Albat, Arne Soicke, Stephan Dohmen, Marco Klein, Judith Bruns, Maarten van Dinther, Robert Opitz. Peter Lindemann, Monika Beerbaum, Kathrin Motzny. Yvette Roske, Peter Schmieder, Rudolf Volkmer, Marc Nazaré, Udo Heinemann, Hartmut Oschkinat, Peter ten Dijke, Hans-Günther Schmalz, and Ronald Kühne. Designed nanomolar small tion impair invasion and extravasation of breast cancer cells. Proceedings of the National Academy of Sciences, 2t073///aaa.00023311111
inhibitors of Ena/VASP EVHI interac-
(177))229644229999 November 2020. ISSN 0027-8424, 1091-6490. doi:
Simon Batzner, Tess E. Smidt, Lixin Sun, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Moli- nari, and Boris Kozinsky. -quivvaaan graph neural networks for data-efficient and accurate interatomic potentials. arXiv:2101.031 [cond-mat, physics:physics] January 2021.
E. Bayro- Coyyoo-Coroocaoo, S. Buchholz, and G. Sommer. A new self-organizing neural network using geometric algebra. In Proceedings of 13th International Conference on Pattern Recognition, vol- ume 4, pp. 555-559 vol.4, August 1996. doi: 10.1 10.1109/1CRR. 1996.547626.
Jörg Behler. Four generations of high-dimensional neural network potentials. Chemical Reviews, March 2021. ISSN 0009-2665. doi: 10. 00.00//acs.chemree00000083
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1):235- 242, January 2000. ISSN 0305-1048. doi: 10. 10.1093/nar/28.1.22
Emanuele Boattini, Marjolein Dijkstra, and Laura Filion. Unsupervised learning for local structure detection in colloidal systems. The Journal of Chemical Physics, 151(11)114490,, October 2019, ISSN 0021-9606. doi: 10. 10.1066//.55 18867.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. (rXiv:2104.13477 [cs, stat], April 2021.
Sven Buchholz and Gerald Sommer. On Clifford neurons and Clifford multi-layer perceptrons. Neural Networks: The Official Journal of the International Neural Network Society, (1):922 935, September 2008. ISSN 0893-6080. doi: 10.
03.004
Stefan Chmiela, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof Schütt, and
Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science Advances, 365)c16 May 2017. ISSN 2378-2548. doi: 10.112266siaadv16000055
Paul S. Clegg. Characterising soft matter using machine learning. Soft Matter, 17(15):3991440 April 2021. ISSN 1744-6848. doi: 10. (0.103//D0MM006800
10

---

Under review as a conference paper at ICLR 2022
William Clifford. Applications of Grassmann's extensive algebra. American Journal of Mathematics, 1(4):350-358, 1878. ISSN 0002-9327. doi: 10.2307/2369379.
Julia Dshemuchadse, Pablo F. Damasceno, Carolyn L. Phillips, Michael Engel, and Sharon C. Glotzer. Moving beyond the constraints of chemistry via crystal structure discovery with isotropic multiwell pair potentials. Proceedings of the National Academy of Sciences, 1 118(21), May 2021. ISSN 0027-8424, 1091-6490. doi: 10.1073/pmas.2022054111.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equiv- ariant multilayer perceptrons for arbitrary matrix groups. 1rXiv:2104:094555 [cs, math, stat], April 2021.
Barbara Franke, Marta Veses-Garcia, Kay Diederichs, Heather Allison, Daniel J. Rigden, and Olga Mayans. Structural annotation of the conserved carbohydrate esterase vb_24B 21 from Shiga toxin-encoding bacteriophage 424B. Journal of Structural Biology, 212(1):107596, October 2020. ISSN 1047-8477. doi: 1..1016/.jsb.2220..00
Karla Frydenvang. Darryl S. Pickering, Giridhar U. Kshirsagar, Giulia Chemi, Sandra Gemma, Desiree Sprogøe, Anne Mette Kærn, Simone Brogi, Giuseppe Campiani, Stefania Butini, and Jette Sandholm Kastrup. Ionotropic glutamate receptor GluA2 in complex with bicyclic pyrimidinedione-basee compounds: When small compound modifications have distinct effects on binding interactions. ACS Chemical Neuroscience, 11(12)1191-11800, June 2020. doi:
10.1022/asschemmeuuo.000000
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. sransfrrrnssso:me:::: 3D roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
Yoshihiko Furuike, Yuka Akita, Ikuko Miyahara, and Nobuo Kamiya. ADP-ribose pyrophos- phatase reaction in crystalline state conducted by consecutive binding of two manganese(ii) ions as cofactors. Biochemistry, 55(12):180--1812, March 2016. ISSN 0006-2960. doi: 1o.1022/acs.boochm..bbbe
Steffen Glöckner, Andreas Heine, and Gerhard Klebe. A proof-of-concept fragment screening of a hit-validated 96-compounds library against human carbonic anhydrase ii Biomolecules, 10(4): 518, April 2020. doi: 10.3390/biom 1040boom00000118... 10045518.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3D point clouds: A survey. IEEE transactions paltern analysis and machine intelligence, 2020.
David Hicks, Michael J. Mehl, Eric Gossett, Cormac Toher, Ohad Levy, Robert M. Hanson, Gus Hart, and Stefano Curtarolo. The AFLOW library of crystallographic prototypes: Part 2. Computational Materials Science, 161:S1-S1011, April 2019. ISSN 0927-0256. doi: (..10666..cmmmttcc..ooo 201........1..............
Y. Hirano, S. Kimura, and Tamada. High-resolution crystal structures of the solubilized domain of porcine cytochrome b5. Acta Crystallographica Section D: Biological Crystallography, 71(7): 1572-1581, July 2015. ISSN 1399-0047. doi: 10. 1..107//11990447559993..
H. Kamberaj. R. J. Low, and M. P. Neal. Time reversible and symplectic integrators for molecular dynamics simulations of rigid molecules. The Journal of Chemical Physics, 122(22):224111 14, June 2005. ISSN 0021-9606. doi: 10. 10.1063/1.1906116
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecu- lar graphs. In International Conference on Learning Representations, September 2019.
Johannes Klicpera, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph neural networks for molecules. arXiv:2 006000003 [physics, stat], June 2021.

---

Under review as a conference paper at ICLR 2022
Risi Kondor. N-body networks: A covariant hierarchical neural network architecture for learning atomic potentials. arXiv:1803.01588 [cs]. March 2018.
Wei Li, Craig Burkhart, Patryeja Polińska, Vagelis Harmandaris, and Manolis Doxastakis. Backmap- ping coarse-grained macromolecules: An efficient and versatile machine learning approach. The Journal of Chemical Physics, 153(4):04110, July 2020. ISSN 0021-9606. doi: 10.1063/5.
0012320.
Alan Macdonald. A survey of geometric algebra and geometric calculus. Advances in Applied Clifford Algebras, 27(1):853-885 891, March 2017. ISSN 0188-7009, 1661-4909, doi: 10.1007/ 500006-016-0665-y.
Siewert J. Marrink and D. Peter Tieleman. Perspective on the Martini model. Chemical Society Reviews, 42(16))6801-6822 July 2013. ISSN 1460-4744. doi: 10.1039/C3CS600933
Shigeru Matsuoka, Shigeru Sugiyama, Daisuke Matsuoka, Mika Hirose, Sébastien Lethu, Hikaru Ano, Toshiaki Hara, Osamu Ichihara, S. Roy Kimura, Satoshi Murakami, Hanako Ishida, Ei- ichi Mizohata, Tsuyoshi Inoue, and Michio Murata. Water-mediated recognition of simple alkyl chains by heart-type fatty-acid- acid-binding protein. Angewandte Chemie International Edition, 54(5): 1508-1511, 2015. ISSN 1521-3773. doi:
Michael J. Mehl, David Hicks, Cormac Toher, Ohad Levy, Robert M. Hanson, Gus Hart, and Stefano Curtarolo. The AFLOW library of crystallographic prototypes: Part 1. Computational Materials Science, 136:$1-S828, August 2017. ISSN 0927-0256. doi: 10. co0mm/s.ommtttsi.2001.
Pavlo Melnyk, M. Felsberg, and Márten Wadenbäck. Embed me if you can: A geometric perceptron. ArXiv, 2020.
Walter Mickel, Sebastian C. Kapfer. Gerd E. Schröder-Turk, Turk, and Klaus Mecke. Shortcomings of the bond orientational order parameters for the analysis of disordered particulate matter. The Journal of Chemical Physics, 138(())044501, January 2013. ISSN 0021-9606, 1089-7690. doi: 00.1063/1.4774085
Akihiko Nakamura, Takuya Ishida, Katsuhiro Kusaka, Taro Yamada, Shinya Fushinobu, Ichiro Tanaka, Satoshi Kaneko, Kazunori Ohta, Hiroaki Tanaka, Koji Inaka, Yoshiki Higuchi, Nobuo Nimura,, Masahiro Samejima, and Kiyohiko Igarashi. "Newton's 's cradle" proton relay with amide-imidic acid tautomerization inverting cellulase visualized by neutron crystallography. Science Advances, 1(7)ee1500263 August 2015. ISSN 2375-2548. doi: 10. 10.1126sscidd, 1500263.
Hiraku Ohno, Kazuki Takeda, Satomi Niwa, Tomotaka Tsujinaka, Yuya Hanazono, Yu Hirano, and Kunio Miki. Crystallographic characterization of the high-potential iron-sulfur protein in the oxidized state at 0.8 À resolution. PLOS ONE, 12(5):e0178183, May 2017. ISSN 1932-6203. doi: 10. 10.11311./// 1 pwn..017811... 0178183.
Renee Otten, Ricardo A. P. Pádua, H. Adrian Bunzel, Vy Nguyen, Warintra Pitsawong, MacKen- zie Patterson, Shuo Sui, Sarah L. Perry, Aina E. Cohen, Donald Hilvert, and Dorothee Kern. How directed evolution reshapes the energy landscape in an enzyme to boost catalysis. Science, 370(6523): 1442-1446, December 2020. ISSN 0036-8075, 1095-9203. doi: 10.1 10.1126/science. abd3623.
Cheol Woo Park, Mordechai Kornbluth, Jonathan Vandermause, Chris Wolverton, Boris Kozinsky, and Jonathan P. Mailoa. Accurate and scalable graph neural network force field and molecular dynamics with direct force architecture. npj Computational Materials, 7(1):1-9, May 2021. ISSN 2057-3960. doi: 02210004154-02--0000
Janet M. Paterson, Amy J. Shaw, Ian Burns, Alister W. Dodds, Alpana Prasad, Ken B. Reid, Trevor J. Greenhough, and Annette K. Shrive. Atomic-resolution crystal structures of the immune protein conglutinin from reveal specific interactions of its binding site with Ilacetyyglucossamine Journal of Biological Chemistry, 294(45): 351X. doi: 10. 10.1077//pc.AA111..000771
November 2019. ISSN 0021-9258, 1083-
12

---

Under review as a conference paper at ICLR 2022
J.K. Pearson and D.L. Bisset. Neural networks in the Clifford domain. In Proceedings 1994 IEEE International Conference on Neural Networks (ICNN'94) volume 3, pp. 1465- 1469 vol.3, June 1994. doi: 10. 10.1109/CCN..1994.34444
Justin Pearson and David Bisset. Back propagation in a Clifford algebra. In I. Aleksander and J. Tay- lor (eds.), Proceedings of the International Conference of Artificial Neural Networks (ICANN), pp. 413-416, 1992.
Marina Plaza- Maza-Garrido, M Carmen Salinas-García. José C. Martínez, and Ana Cámara-Artigas, The effect of engineered ATCUN motif on the structure and biophysical properties of the SH3 domain of c-Sre tyrosine kinase. JBIC Journal of Biological Inorganic Chemistry, 25(4):22-- 634, June 2020. ISSN 1432-1327. doi: 10. 10.1007/500775-000-01111
Charles R. Qi, Hao Su, Mo Kaichun, and Leonidas J. Guibas. PointNet:
Deep learning on point sets for 3D classification and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-83- July 2017. doi: 10.1109/CVPR.201..16
Vyas Ramasubramani, Bradley D. Dice, Eric Harper, Matthew Spellings, Joshua A. Anderson, and Sharon C. Glotzer. Freud: A software suite for high throughput analysis of particle simulation data. Computer Physics Communications, 254:107275, September 2020. ISSN 0010-4655. doi:
11.10166j.cc:222200077771
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Compulational Linguistics, 8:842-
866, 2020. doi: 10. 162/tacla a a_0449.
Matthew Schellenberg, C Denise Appel, Amanda A Riccio, Logan R Butler, Juno M Krahn, Jenna A Liehermann, Felipe Cortés-Ledesma, and R Scott Williams. Ubiquitin stimulated re- versal of topoisomerase 2 DNA-protein crosslinks by TDP2. Nucleic Acids Research, 48(11); 6310-6325, June 2020. ISSN 0305-1048.
doi: 10. 10.1093/nar/gkaa311
K. T. Schütt, P.-. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller. SchNet: A continuous-fiter convolutional neural network for modeling quantum interactions. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 992-1002, Red Hook, NY, USA, December 2017. Curran Associates Inc. ISBN 778-1-510866096-1
Matthew Spellings and Sharon C Glotzer. Machine learning for erystal identification and discovery. AIChE Journal, 64(6):218--224 2206, 2018. doi: 10.1002/aic.1611 16157.
Paul J. Steinhardt, David R. Nelson, and Marco Ronchetti. cund-orientational order in liquids and glasses. Physical Review B, 28(2):784-80) July 1983. doi: 10. 1.11100yysse.B.225 2.2.788
K. Takaba, Y. Tai, H. Eki, H.-A. Dao, Y. Hanazono, K. Hasegawa, K. Miki, and K. Takeda. Sub- atomic resolution X-ray structures of green fluorescent protein. IUCrJ, 6(3):387-400, May 2019. ISSN 2052-2525. doi: 10. 10.1107/5202225511000 9902009
TensorFlow Developers. TensorFlow. Zenodo, August 2021.
Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. KPConv: Flexible and deformable convolution for point clouds. Pro- ceedings of the IEEE International Conference on Computer Vision, 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and eequiaaionn-uuuaariinn neural networks for 3d point clouds. 1rX2vv/802.000
[cs], February 2018.
Wei Tian, Peiqiang Yan, Ning Xu, Arghya Chakravorty, Robert Liefke, Qiaoran Xi, and Zhanxin Wang. The HRP3 PWWP domain recognizes the minor groove of double-strande DNA and recruits HRP3 to chromatin. Nucleic Acids Research, 47(10):5436-544 5448,,44, June 2019. ISSN 0305- 1048. doi: 10. 1093/na//gk7224.
13

---

Under review as a conference paper at ICLR 2022
Oliver T. Unke and Markus Meuwly. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, 15(6):3678- 3693, June 2019. ISSN 1549-9618. doi: 10.102 Je0011...
Oliver T. Unke, Stefan Chmiela, Huziel Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T. Schüitt, Alexandre Tkatchenko, and Klaus-Robert Müller. Machine learning force fields. Chemi- cal Reviews, March 2021. ISSN 0009-2665. doi: 10. 1a.002//ccscchemrre00001111
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017.
Alessandro Vergara, Marco Caterino, and Antonello Merlino. Raman-markers of X-ray radiation damage of proteins. International Journal of Biological Macromolecules, 111:1194-1209 May 2018. ISSN 0141-8130. doi: 10. 1016611/j.100 h.t...iaaaa.......0.1111
Shanshan Wu, Tam T. N. Nguyen, Olga V. Moroz, Johan Turkenburg, Jens E. Nielsen, Keith Wilson, Kasper D. Rand, and Kaare Teilum. Conformational heterogeneity of Savinase from NMR, HDX-MS and X-ray diffraction analysis. PeerJ, 8:e9408, June 2020. ISSN 2167-8359. doi: 10.7717/peerj.94088
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems, 30, 2017.
14

---

Under review as a conference paper at ICLR 2022
A GEOMETRIC ALGEBRA
The geometric algebra was developed in the 19th century and provides a consistent framework for dealing with scalars and other geometric quantities-suce
vectors, areas, and volumes found in three-dimensional space-in in arbitrary dimensions (Clitiord. 1878). Here, we will describe the es- sential parts of geometric algebra related to proposed attention mechanism, and defer to other works, such as Macdonatd (20T7), for more thorough description. The geometric algebra specifies binary operator, the geometric product. that works mullivectors. Multivectors can be expressed as linear combinations of terms from a fixed basis set for given space, such as IR2 or IR:; in three- dimensional space, this yields scalars, vectors, bivectors (which specify signed areas within a plane and have 3 components). and trivectors (which specify signed volumes and have I component) total of linearly independent terms for each multivector. When rotation invariance desired, as components of multivector: scalars, trivectors,
in Equation III we can utilize the rotation-invarian the norms vectors, and the norms of bivectors are general framework that can be used to build up expressive functions as linear combinations and ge- ometric products multivector inputs; in this work, we derive itation-equivarian quantities from geometric products to suit the application and symmetry of our problems of interest.
Geometric algebra provides
Terms for arbitrary geometric products can be calculated using basic mathematical attributes of the geometric product: for two orthogonal vectors é and εq E εσē₂ 1 and é -ē2ē1 As an example, multiplying a vector = aτ₁ + + and vector-bivector combination B = 86 + C uōi6' vē2ēz proceeds follows:
AB = (no + (Ea) 66 +
a(ōē (0122 + vé) (6ēge + (2ge2 pége Eg + v2ge2(g)
a(8 + Ç, + مقz + 22) + 66 (326g - pē E3C3 - VV20303)
aa + (8 + pDz E2 7( 1-00 Çā2τs pē= vē2)
aō
7pوē - 2V06 + 01p aC - 75 75 Eg + ovel
scalar ad
vector (-y ap)
bivector (ac, -6, -(()
trivector oop
More generally, the types of elements produced by geometric product of two multivectors R with the given components summarized in Table 2 below.
Table 2: Terms arising from the geometric product AB = AA,+A, AB B B,) in RR3 three dimensions, multivectors A and B consist of sealars (s), vectors (v), bivectors (b), and trivectors (t).
B
B
Bb
B
b
A
V
8
b
V
V+I
b
Ab
b
b
At
V
S
From Table 2. we can see that successive products of vectors such as Piyk... in Equation MI alternate between producing two types of multivectors: products even numbers vectors yield scalar and bivector ((r + t)v of vectors produce a vector and trivector ((s + b)u = sv + rotation iitationiinvarian quantities from these products described in the main text is the primary applica- tion of geometric algebra in this work, although in general the method could be used to incorporate different types of scalar, vector, bivector, and trivector quantities; for example, rotations could be input as quaternions, which are isomorphic to the scalar-and- and-bivector product of even numbers of vectors.
+ tv (s b) + s b). while products of odd numbers
v + (u+0) +) + v + t). Generating
Multiveetors form vector space: individual components multivectors (bivectors, example) can be directly summed elementwise, but multivector components of different types stay separate and are multiplied using the distributive property of geometric products when needed, producing output types according to Table DI The geometric product has identity of the scalar and is associative; in other words, it forms monoid over multivectors.
15

---

Under review as a conference paper at ICLR 2022
ROTATION EQUIVARIANCE
Because the scalars used in Equation 2 are rotation-invariant and geometric algebra is coordinate- Tree-that is, can perform change of basis replace the basis vectors ετ, é2, éa with rotated versions Qēg Q Qé to transform the point cloud using rotation matrix Q and the vector component of the product Piyk... will transform in the same way-the output of Equation is also rotation-covariam for rotation matrix Q
Q Σ J.....
R (eotor..)) + az/ +
wijk...
Σ Wijk... Sriik...RRuspeee 1a + +..)
jk...
Wigk...
03 Q + +..).
jk..
B MODEL DETAILS
We list python-style pseudocode-usinn the semantics of the Keras API within Tensorflow (Tensor Flow Developers, 2021)- the models used in this work below. also use the following geo- metric algebra attention layers, which are built with MLPs given by the listed make scorefun (S) and make value make_value
(V) functions described in Equation II:
VectorÅttention entin implements the value-genert generating geometric algebra attention calcu- lation described in Equation M It accepts a point cloud and set of values associated with each point, and produces new values.
implements the coordinate--ennrrainng (i.e., rotation- covariant) geometric algebra attention calculation described in Equation It accepts point cloud and set of values associated with each point, and produces new coordinates.
Vector2VectorAttent
LaaeleeVecct. eeeeetoortteeen....... on Implements the lagel-augmented point cloud translation layer described Equation B takes three arguments (a of label values, a set of reference coordinates, and set of reference values) and produces a set of new coordinates.
Listing 1: Sample model-building code for rotation-invarian crystal structure identification. def make__ccraeaae socke__screefu/
return Sequential (Deesse (64, activation=' relu'),
Dropout (0.5), Dense [1)
def make_ aake_valucffn return Sequential ( (Dense (64), LayerNormal wwtiorrmmiiiiiiooo zation (), Activation in ('rell'), Dropout (0.5),
Dense (32)
Inputs are given as N-length sets of coordinates or values for each nearest-neighhe bond r_in Input (None, 3))
v_in Input (None, D_in))
Project values to the desired working dimension last_v Dense (D_ (D_working) (v_in)
Calculate attention a series blocks for in range resiqual_in in last_v
last_v VectarAttention () (r_in, last_v)
last_v Dense (2xD (240_working, activation= (last_v)
16

---

Under review as a conference paper at ICLR 2022
last_v
Dense (D_ working) (last_v) residual in
Pormutation-ivvaiian attention summation (Kx__working)
last_v VectorAttention (osmmtattoon_iiaariimeTTTa)))))) (r_id, last_v)
Pinal MLP
last_v Dense (2*D_ _eok_oo activation='relu") (last_v)
logits Dense tauu_classes, activation='sofeet softmax') (last_v)
model = Model (Ir in, v_in|, logits)
Listing 2: Sample model-building code for conservative, rotation-equivariant neural networks for molecular force regression.
def make scorefun
return Sequential (IDense (64, activation=' swish") )
Dense [1)
def make_ vale_vallfff(). (): return Sequential ( (Dense (64), LayerNormalizatinn ()
Activation ('relu'), Dense (32)
Inputs are given as N-length sets of coordinates
or values for each atom in a molecule
r_in - Input ((None, )
V_in Input (None, D_in))
Convert to atom-centered coordinates (NxNx3) and values (NxNx[2+0_in))
r_ij PaizwiseDifferrecee (r (r_in)
vijj = PairwissSuDifferenee () (v_in)
Project values to the desired working dimension
last_v Dense (D_working) (v_ij)
Calculate attention in a series of blocks for in range(6):
resicual_i last_v
last_v VectorAttention () (r_ij, last_v)
last_v V- Dense (2xD (2xD__wrkke)) working, activation= swish") (last_v)
v Dense (D working) (last_v) v) residual in
Permutation-invarie attention summation (Nx0_working) last_y VectorAttention (permatation_llvvriitt=-oro (r_ij, last_v)
Final MLP and summation
last_v Dense [2tD_working s (last_v)
aatomi_ennerre The energy of the molecule is the sum of each atom's energy mo molecule_energy = NeighoorhoodSuu The force is the negative gradient of the energy with respect input coordinates
Dense (1, use_bias=False) (last_v)
(atemiceenergy)
force_output NegativeGradien () (melecule_energy, r_in) model Model ([r_in, v_in], force_cutput)
Listing Sample model-building code for rotation-covariant coarse-grain operator backmapping. def make_scorefun! :):
return Sequential ( (Dense (64, octivation='relu""
Dense [1)])
def make_valueffe :):
return Sequential (Densse (64), IayerNormalizatioon (),
Activation iedivvation""ellul ('relu"), Dense (32) l)
17

---

Under review as a conference paper at ICLR 2022
Inputs are given as N-length sets of coordinates
or values for each coarse-grained neighbor, and a set
of atomic labels for the contral coarse-grained boad r_in = Input (None, 3
v_in Imput (None, D_in))
child_types_
Input ( (None,))
Project values to the desired working dimension
child_ = Embedding (num_chiui wwwid_ttiteeeeccc (child in)
last_v Dense (D. working) (v. in)
Calculate attention in a series of blocks for in range
resicual last_v
last_y Vectorättention () (r_in, last_v)
last_v Dense (2+D (2*D_working, activation rolu') (last_v)
last_v Dense (D_working) (last_v) + residual_in
Translate coarse-grained to fine-grained coordinates vecs eebeeeeeeecttttteenttti () (r_in, last_v, child_labels)
Cenvert to atom-centered representations for refinement delta_v PairwiseDi ifw.eneeiffereecesuuu (child_labels) delta_v Dense (D_ (D_working) (delta_v)
Refine atomic coordinates
for in 22)ge(2): resicual_in vecs
PairwiseDifference (vees)
vecs vector2eettrrtteettinn
http://atttttttooooooettttttttee (permutation vecs, delta_v)
model Model ([r_in, viin, child_types_in), vecs)
We also present diagrams summarizing the geometric algebra attention model architectures used in this work Figure B.
BASELINE MODELS
Crystal structure identification. We train MLPs on spherical harmonic descriptions of varying- sized local environments of particles as described in SSpllingg & Glotzer (2018). To deal with particle types using this method, we concatenate two of local environment spherical harmonics: one considering only bonds of like-typed particles (i.e. bonds to particles of type A if the central particle is of type A), and another set considering only bonds to differently-typed particles. We consider environments of to the 12 nearest neighbors (disregarding particle types) for each set of spherical harmonics. The MLPs used consist of batch normalization layer followed by two blocks of dense layer 64 neurons in width using a ReLU activation function feeding into dropout layer with dropout rate of 0.5. The final result projected down to class logits.
Backmapping of coarse-graining operators. We train models using dot product attention: vertex representations are concatenated directly with their geometric coordinates and fed into similar architecture to that shown in Figure E. with geometric algebra attention layers replaced by standard dot dot-product attention layers. For the intermediate translation layer, query embeddings associated with each atom type used. Because this baseline model is not rotation-covarian by construction, must learn to generate crration-covarian predictions; to impart this information to the model, we apply random rotation each pair of point clouds during training.
18

---

Under review as a conference paper at ICLR 2022
Position
a
(N, 3)
A Type
Residual partial
Attention reduction
softmax
Class (1)
Dense
Dense
(N, 2N,)
attention
b
Position
(N, N, 3)
A Type
Residual partial attention
Attention reduction
Energy
Dense
Dense
(N, N, 2N,)
(1)
V
Force
(N, 3)
Amino Acid Coordinates
(N (Ncoarser 3)
Vector atomic Pattention reduction
A Type
Residual
Dense
(N coarser 2N,)
partial attention
CA C1
Residual vector
Atom Type Embedding
02
N2
(N D)
atomic
attention
2
Atomic Coordinates
(N 3)
Figure 5: Visual overview of model architectures used this work. Architectures presented are for (a) crystal structure classification, (b) molecular force regression, and (c) coarse-grained backmap- ping.
MODEL TRAINING
Networks for the crystal structure identification and protein coarse-grain backmapping tasks trained for up to 800 epochs using the adam optimizer (Kingma & Ba, 2015); the learning rate decreased by factor of 0.75 after the validation set loss does not decrease for 20 epochs, and training is ended early if the validation set loss does not decrease for 50 epochs. Models trained on the molecular force regression task use the longer training schedule presented in Batzner al., which decreases the learning rate by factor of 0.8 after 1000 epochs of no improvement in the validation loss and ends training after 2500 epochs of improvement, or 50000 epochs in total.
C PDB ENTRIES USED FOR COARSE-GRAINING TASK
3X2L
(Nakamura et all 201
3X32 (Hrrano el all 2015)
4TKJ
(Maisuoka et al 20
3XOJ (Furuike et all 2016
19

---

Under review as a conference paper at ICLR 2022
5WQR
et 2017)
6EQE
Austin et all
2018
6ETK
(erggra et al,
6FJN
(Baker
6JGJ
(Qakaba all
6RYG
(Paterson etall
61IP
(lian
2019)
6Q01
(Schellenberg et al. 2020
6XVM
Plaza-Garrrdo et all 2020
6Y5S
(Wl 2020)
6YP6
(Franke at 2020
7A5M
(Barone et all
(02(
7K4T
et (021
6YK4
Frydenvang cf al, 2020)
6SAY (Glöckner et all 202
D MODEL EVALUATION THROUGHPUT MEASUREMENTS
Typical times required to evaluate models on a single point cloud using an NVIDIA GeForce RTX 2060 GPU, amortized over batch, reported in Table & below.
Table 3: Times required to evaluate models point clouds. Task
Model details
Time (ms/point cloud)
Crystal Structure Identification
0.19
Aspirin
7.6
Benzene
4.2
Ethanol
5.1
Malonaldehyde
5.5
Molecular Force Regression
Naphthalene
6.5
Salicylic Acid
6.1
Toluene
5.3
Uracil
4.4
8-128 neurons
0.67
Coarse-grain Backmapping
192 neurons
1.0
20


Reviews:
[{"review_id": "YzaSgBWXcyM", "reviewer": "Reviewer_vhG8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The proposed method is principled, general, and convenient. It applies to interactions of arbitrary rank.#\n - The method is described in a detailed and reproducible way.\n - The paper is well written and easy to follow.\n - The setting of the paper is clearly defined as learning functions on small point clouds, where there is an emphasis on higher order interactions and respecting symmetries. The chosen evaluation domains form a diverse set of instantiations of this setting.\n\nWeaknesses:\n - The quantitative results are good, but not revolutionary. The model does well on the crystal classification task, but this setting is described as 'not difficult'. It performs slightly worse than GemNet on molecular force regression. On the backmapping task, it is only compared against a naive transformer, as opposed to another rotation invariant model. Another instance of the proposed model outperforming a strong baseline on a difficult task would significantly improve the case the paper is making.\n - While one of the advantages of the proposed method is scaling to arbitrary rank, the experiments are all limited to pairwise attention (not counting the current key element). It would be interesting to demonstrate the effect of scaling up the rank, especially as this is noted as a potential advantage of the GemNet baseline.\n - Group representation-based approaches have not been compared against.", "summary_of_the_paper": "The paper proposes a method for learning functions that take small point clouds as input, such that they are both permutation and rotation equivariant. To aggregate information from a tuple of points in a rotation equivariant way, it utilizes the geometric product of the points' 3D coordinates. Permutation equivariance is guaranteed by the standard attention framework.\nThe model is evaluated on three scientific tasks: Crystal structure identification, Molecular force regression, and backmapping of coarse-grained operators in molecule simulations.", "summary_of_the_review": "Overall, the paper proposes a convenient and principled method to an interesting problem. The empirical evaluation is generally convincing, but could be somewhat more thorough in its analysis, especially given that the paper is a full page under the page limit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QigVYP9udNm", "reviewer": "Reviewer_6SEn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n- Formulation of rotation-equivariance by leveraging generic geometric products.\n- New attention formulation for achieving permutation-equivariance which is important for many deep learning applications in different fields.\n- Performance is evaluated in three applications with different datasets which shows the robustness of the model proposed. The evaluation also showcases the importance of geometric properties in deep learning models for data efficiency, robust prediction, and fast training.\n\n# weaknesses:\n - In table 1, the proposed model performs worse than the baseline GemNet-Q and some possible reasons for that are enumerated by the authors like the use of quadruplets atoms, the incorporation of energy, or even better a better architecture. Can the proposed method be adjusted to have these properties in order to perform a fair comparison with this baseline? However, the results are good and it is fine to present worse results than some baseline finetuned to the application tested.\n - I did not understand footnote 4. Why is the training error used to evaluate the performance instead of the test error in the protein coarse-grain backmapping? How does this measure the generalization performance of the model?\n\n", "summary_of_the_paper": "This paper proposes a rotation and permutation equivariant geometric deep learning model for problems where the data is represented as small points clouds. The equivariance properties are achieved by leveraging geometric algebra formulations. More specifically, rotation-equivariance is accomplished by geometric products of multivectors and permutation-equivariance by using an attention mechanism over invariant terms of these products. This model is evaluated in three different applications showing better or comparable results than existing approaches. These models also offer additional features like the analysis of the attention maps produced.", "summary_of_the_review": "I am not an expert on this topic. But, I think this paper proposes a novel formulation to provide rotation and permutation equivariance for deep learning models for small point clouds which are useful in many applications. They also show improvements over existing approaches in three different applications. Therefore, I suggest accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FuaCOnsp-65", "reviewer": "Reviewer_TD8X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Rotation equivariance based on geometric algebra is novel and impressive. The proposed attention also shows the impressive performance on several real-world tasks. \n\nHowever, I still have several concerns which I will detail below\n\n- The details of the four functions are a little vague to me. For example, what are rotation-invariant geometric quantities for tuples? I recommend elaborating more on the technical details if possible to improve readability. \n\n- Mathematical proof of rotational equivariance is missing. \n\n- While the paper shows an impressive performance in real-world tasks, I wonder about the efficiency in particular compared with other works. \n\n- Missing rotation equivariance experiments are missing. I recommend having an experimental test for the rotation equivariance by comparing with other works -- say rotation equivariance achieved via data augmentation. \n\n- Is it possible to extend work into relatively larger point clouds? Also, I am actually not quite sure what the definition of small point clouds is. Is that determined by the size of point clouds? If so, what’s the threshold then? I would recommend having a simple experiment on 3D point cloud classification -- e.g., the popularly used Modelnet 40. The number of points could be controlled via sampling.\n", "summary_of_the_paper": "The paper proposes a geometric algebra attention network for small point clouds. The attention based on geometric algebra’s multivector is rotation equivariant and permutation equivariant. Specifically, attention is composed of four functions that operate on tuples and respect the desired equivariance. Moreover, the paper validates its proposed geometric algebra attention on three domain-specific applications including crystal structure identification, molecular force regression, and back mapping of coarse-graining operators.\n", "summary_of_the_review": "I am overall impressed by the idea of geometrical algebra-based rotation equivariance. The experimental results are also impressive. However, from my viewpoint, the paper still has several weaknesses in terms of clarity and experimental validation. I would like to hear more during the rebuttal phase. Thus, currently, I vote for weakly accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1XuuCsKTr4k", "reviewer": "Reviewer_sEBH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ It is interesting that the idea of utilizing the theory of geometric algebra to achieve equivariance.\n+ The demonstration broadly covers sample problems relevant to physics, chemistry, and biology.\n\nWeaknesses:\n- The authors do not clearly describe how the geometric product helps to achieve the rotation-invariant attributes. The authors should explain it in the main manuscript since this is one of the main contributions.\n- Geometric products do not appear in both model architectures and appendix B.  The authors are required to exlicitly describe how it is implemented.\n- I think the interpretation of the linear combination in geometric algebra is not trivial. For example, how can we interpret the average of a point and a plane, which are commonly expressed as geometric numbers? Even for the point, a linear combination of some points would be interpreted as a line when the scalar part is 0 (~ eq(2)). Is this appropriate? This paper lacks the interpretation, justification, and discussion about the attention mechanism in geometric algebra.\n- Why is the proposed architecture limited to \"small point clouds\"? If the statement about the computational complexity is the reason, the connection to it should be clarified.\n\nMinor comments:\n- I supposed that this paper uses the projective geometric algebra Cl(3,0,1), but not clarified. Even for 3D, the conformal geometric algebra Cl(4,1,0) might be a candidate. Clarification, justification, and discussion are appreciated.\n- The authors defined four functions, V, M, J, and S. However, another function R is defined later. Why do the authors distinguish R from the others? Are the four functions cover everything sufficiently? Justification is required.\n- y_i in eq(1) seems output, but the statement \"a network producing permutation-covariant outputs for each input point y_i\" mislead y_i to be input point.\n- Is the geometric product commutative? If not, how do the authors achieve permutation-covariance for varying p?\n- Is \"working width\"  in MODEL ARCHITECTURES \"working dimension\"?\n- Is it possible to perform molecule force regression for unknown molecule?\n- The authors stated that GemNets can operate on quadruplets of atoms but the proposed network worked only in pairs of bonds from the central atom. If this is the limitation of the proposed network, clear statement and discussion are required. Moreover, a discussion is appreciated whether the proposed network has technical difficulties to include the force label.\n- I could not find which point \"show difficulty in convergence above 64 neurons\" in protein coarse-grain backmapping.\n\n", "summary_of_the_paper": "This paper presents a deep neural network for processing small sets of points in three-dimensional space with a rotation-invariance and a permutation equivariance. \nThe proposed method is twofold. One utilizes a geometric product of geometric algebra on input vectors to achieve the rotation-invariant attributes. The other is a permutation-equivariant reduction over the geometric products using an attention mechanism.\nThe authors demonstrate applications in physical science such as crystal structure identification, molecular force regression, and backmapping of coarse-graining operators.\n", "summary_of_the_review": "This manuscript has several issues to be accepted as raised in the weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies the problem of building powerful representations of low-dimensional point clouds with permutation and rotational equivariance, with the motivation to tackle applications in the physical sciences. Their main technical contribution is the use of the so-called geometric algebra, a series of operations between scalar and vector quantities that respect rotational symmetries, which the authors then combine with attention mechanisms to provide permutation symmetry. 

Reviewers generally found this work full of interesting ideas, in particular the novel geometric algebra structure to deal with rotational symmetry. However, they also found several issues, such as lack of clarity and somewhat unclear experimental validation. In particular, the authors are encouraged to formalise the rotational equivariance property, and to further address the "small" aspect of the title. Taking all these considerations into account, the AC recommends rejection at this time, but encourages the authors to pursue this exciting line of research.
Decision: Reject