Geometric Attention Networks for Small Point Clouds
Authors: Matthew Spellings
Venue: NeurIPS 2021 Submitted (2021)
Keywords: geometric deep learning
chemistry
physics
biology
equivariance
geometry
attention
point clouds
graph neural networks

Abstract: Much of the success of deep learning is drawn from building architectures that properly respect underlying symmetry and structure in the data on which they operate—a set of considerations that have been united under the banner of geometric deep learning. Often problems in the physical sciences deal with relatively small sets of points in two- or three-dimensional space wherein translation, rotation, and permutation equivariance are important or even vital for models to be useful in practice. In this work, we present an architecture for deep learning on these small point clouds with rotation and permutation equivariance, composed of a set of products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in a systematic way to account for rotation invariance or covariance, while attention yields a powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.

Geometric Attention Networks for Small Point Clouds
Anonymous Author(s)
Affiliation Address email
Abstract
Much of the success of deep learning is drawn from building architectures that properly respect underlying symmetry and structure the data on which they operate-a set of considerations that have been united under the banner of geo- metric deep learning. Often problems in the physical sciences deal with relatively small sets of points in tion, and permutation equivariance are important even vital for models to be useful in practice. In this work, we present architecture for deep learning on these small point clouds with rotation and permutation equivariance, composed of a set products of terms from the geometric algebra and reductions over those products using an attention mechanism. The geometric algebra provides valuable mathematical structure by which to combine vector, scalar, and other types of geometric inputs in systematic way to account for rotation invariance or covari- ance, while attention yields powerful way to impose permutation equivariance. We demonstrate the usefulness of these architectures by training models to solve sample problems relevant to physics, chemistry, and biology.
2
3
three-dimensional space wherein translation, rota-
10
11
12
14
15
Introduction
16
17
Deep learning has been immensely successful in solving wide range of problems over the last several years, driven in large part by identifying appropriate ways to embed structure of data and symmetry of problems directly into the architecture of the network-an idea at the core of geometric deep learning[[]]. Some applications of geometric deep learning include the use of convolutional
18
19
20
21 filters in CNNs to attain translational equivariance, or graph convolutions in graph neural networks
22 for permutation equivariance." Building symmetry into the architecture of a deep neural network improve the data efficiency of the network and guarantee important analytical properties without 24 having to rely on the network to learn to approximate them from training data.
25 In this work, we derive a family of architectures that is useful in applications from physics to biology,
26 where problems often deal with relatively small point clouds of labeled coordinates. These could
27 be local environments of particles assembling into crystal[2], atoms in molecule interacting with other atoms[[]], coarse grained beads representing parts of a protein[E]] In many of these applications without the influence of an external field, we are interested in modeling attributes of the
29
In this work, we use the following terms to discuss symmetry of functions f and operations p: f is invariant if does not change when p changes: f o f. and p commute, then say that f is covariant with respect to the operation of p: f o p po f (some sources call this equivariance or same-equiiei cane-equivariane: the more general, but we will only discuss f and p that are endomorphisms in
typical detinition of equivariance the context of covariance). Here we use equivariance to broadly mean considerations of covariance as well as invariance (since scalars of interest are typically invariant to translation and rotation in physical applications) for simplicity of discussion.
Submitted to 35th Conference Neural Information Processing Systems (NeurIPS 2021). Do distribute.

---

31 of group of atoms-which are invariant with respect rotation of the input coordinates, as well as 32 permutation in the ordering of points. Here we attain rotation invariance by constructing functions Σ 1 uo, ((r))) vo, 0.)
6
w(( No: U. 1)) 15)
Figure 1: Overall strategy for incorporating rotation and permutation equivariance into deep neural networks using attention mechanisms and geometric products. (a) At its simplest level, our pro- posed structure uses an attention mechanism over the bond lengths of a cloud of points, each of which carries a value as commonly used in graph neural networks. (b) Geometric products (or linear combinations thereof) can used to combine pairs, triplets, or larger tuples of vectors atic and geometrically meaningful way. Rotational equivariance can be attained by using invariant or covariant quantities, as desired. (c) An attention mechanism reduces the set of generated geomet- ric products to enforce proper permutation equivariance, and the learned attention maps provide insights into how models operate. For example, here, a carbon atom in a naphthalene molecule (indicated by a blue star) directs its focus broadly around the carbon atoms of the aromatic rings in which it situated, rather than focusing exclusively on its nearest neighbors in the molecular graph. Brighter-colores bonds indicate greater attention weight for the two atoms sharing the bond.
system-
system-suee the identity of particle's s local self-assembly environment, or the potential energy
33 from rotation-invariant components of geometric products of input vectors from geometric algebra,
34 and permutation invariance by using an attention mechanism to intelligently reduce representations
35 over the of vector products.
Related Work
37 "Large" point clouds. Point clouds are ubiquitous data structure and often found in domains outside of the physical sciences. For the purposes of this work, we focus comparatively small sets 39 of points where the points are relatively information-rich forchhhorrree example, carrying information about
40 atom identities, local environments, or other information in contrast to point clouds commonly
41 found in computer vision and robotics which may represent the geometry of a mesh or be sampled
42 from an object, but otherwise not have as much information associated with each point. We refer Guo al. for survey of this field[S]. but few of the include PointNet([6), deep sets and kernel point convolutions(8].
43
notable approaches
45
Geometric approaches for small point clouds. Many architectures have been proposed to incor- porate rotation equivariance by augmenting graph neural networks with geometric attributes that are known to be cnatiioninvariant. such as bond lengths and angles. SchNet[[] learns distance- based convolution filters which are used to propagate signals over graphs. PhysNet[[IJ also refines
46
47
48
49 node representations based on bond lengths, while incorporating learnable attention mechanism. [i extends the information used calculate node-level representations to include angles between bonds. GNNFF[ generates rotation-covarii results by computing a weighted sum of modulated input vectors based on graph message passing scheme.
50
52
53 Group representation-baasee approaches. These methods take advantage of group representation
54 theory by first transforming inputs into a space in which rotation-egue -equiii-quu maps are more eas- 55 ily expressed. This set of methods is powerful, having been used in the past to design rotation- and coonantttioo-eqqivrrii models[IT3 I1S], and have even been expanded recently for arbitrary [[ Attention-based models have also been utilized in this area: SE(3) Transformers

---

58
extend tensor field networks[[a] with self-attention mechanism for increased expressivity by in- corporating value- and geometry-dependen attention weights.
59
60
The approach we present here similar to several of the ideas presented above; however, rather than specifying particular rotation-invariant quantities utilize learning maps that operate on irreducible representations, we leverage the structure provided by geometric algebra to determine which rotation-invariant and -covariant quantities are of interest.
61
63
Geometric Attention Networks
65
In this work we formulate deep neural networks using learnable functions consisting of two parts:
66 (1) a set of geometric products of input vectors; and (2) a permutation-equiiarr reduction over
67 these products using an attention mechanism. We describe each these aspects below.
68
Geometric Algebra
The geometric algebra was developed in the 19th century and provides consistent framework for dealing with scalars and other geometric quantities such vectors, areas, and volumes in three- dimensional space-in in arbitrary dimensions(11)) Here, we will describe the essential parts of geo-
71
72 metric algebra as related to our proposed attention mechanism, and defer to other works for a more
73 thorough description[[1]] The geometric algebra specifics a binary operator, the geometric product,
74 that works on multivectors. Multivectors can be expressed as linear combinations of terms from
75 fixed basis set for given space, such as IR2 R3; in three-dimensional space, this yields scalars, vectors, bivectors (which specify signed areas within plane and have components), and trivec- tors (which specify signed volumes and have component)-a total of 8 linearly independent terms 78 for each multivector When rotation invariance is desired, we can utilize the rotation-invaria invariant components of multivector: scalars, trivectors, the norms of vectors, and the norms of bivectors As an example, the geometric product two vectors yiclds scalar plus a bivector; scalar component is the dot product, and the bivector component related to the cross product the two.
79
80 are
81
82
Geometric algebra provides a general framework that can be used to build up expressive functions as linear combinations and geometric products of multivector inputs; ination-equivarian quantities can then be derived from the products, depending on the application and symmetry of the problem of interest. The types elements produced by geometric product of two multivectors in E with the given components are listed in Table W below.
84
85
86
87
Table Terms arising from the geometric product AB (A.+ Al in R In three dimensions, multivectors A and B consist of scalars (s), vectors (v), bivectors (b), and trivectors ()).
B,)
B
B
Bb
B
A A
t
V
8 b
b
Ab
V+C
S
b
V
At
88
From Table MI, we can see that successive products of vectors alternate between producing two types of multivectors: products of even numbers of vectors yield scalar and bivector ((u+t)v = vettv b), while products of odd numbers of vectors produce a vector and trivector ((s + b)c su + bu = e + (v (o+1) +t) - v + t). Generating rotation-invariant quantities from these products is the primary application of geometric algebra in this work, although in general the method
89
90
(s + b) b
91
92
*Multivectors form a vector space: the individual components of multivectors (any number bivectors, for example) can be directly summed elementwise, but multivector components of different types stay separate and are multiplied using the distributive property of geometric products an identity of the scalar 1 and is associative: in other words, it forms monoid over multivectors.
when needed. The geometric product has

---

93
could be used to incorporate different types of scalar, vector, bivector, and trivector quantities; for example, rotations could be input as quaternions, which are isomorphic to the scalar-and-bivedt product even numbers of vectors.
84
85
96
Attention from Geometric Products
97
For input point clouds with N points, we can construct a series of successively higher-order geo- metric products for all N2 possible pairs, 2 triplets, and so on; these individual points, pairs, or triplets we will call tuple in this context. In addition to coordinate ri. we associate a of values V/ to each point indexed by i in some space with given working dimension (we avoid calling these vectors to decrease the confusion with geometric vectors; these correspond to the non-geometric attributes of the point, such as type embeddings). To create permutation-covariin functions (pro- ducing a value for each input point) or permutation-invarian functions (producing a single output value), we make use of a simple attention mechanism based on the rotation-invann
96
99
102
103
104
attributes of each tuple. Attention has been used widely in applying deep learning to a range of problem do- mains over the last few years, with particular success in the field of natural language processing(20],
105
106
107
Since we are already generating tuple-wise quantities, we choose to utilize a simpler mechanism than the typical dot product self-au attentttennio. We specify four functions: value generateng function V. tuple value-merging function M, joining function that summarizes the invariant and tuple
109
110
representations J, and score-generating function S, The functions have the following uses within the network:
111
V produces features in the working dimension of the model from the invariants associated with each tuple.
114
M merges the 1, 2, more values associated with tuple of input points into the working dimension of the model. The form of M could be complex function, learned linear projection for each tuple position, or simply taking the sum of the tuple values.
117
J joins the invariant representations from V and the tuple representations from M. Like M, could be learned projection or simple sum function.
118
119
S generates score logits from the representation of each tuple, which incorporates invariants associated with the tuple and the values being associated with each point that is part of the tuple. The results from S, passed through a softmax function, will yield the weights for the attention mechanism.
120
121
123
We first calculate the multivector geometric products Piyk... of all combinations of input vectors i. j. k, and so on, up to a specified rank (pairwise attention would produce a two-dimensional matrix of products Pty). We then use V. _M. J, and S S-together with function extracting the rotation-invariant attributes of geometric product (the scalar component, trivector component, and the norms of the vector and bivector components) as follows for network producing permutation- covariant outputs y::
124
125
126
Pijk... F
Lisk... invariants(p.jt..))
Uijk... J (. Uk)...)
Wijk... softmax (S(y.....)
(1)
jk..
Wirk... Vijk...
jk...
129 If a permutation-invariann reduction is desired, then the softmax and final sum can be performed over all tuples simultaneously, rather than for each input point individually. While J, V, and M could in principle be used to change the working dimension permutation-covarian layers stacked on

---

132 top of each other, in this work we keep the working dimension constant for the sake of easily adding
133
residual connections.
134
If rotation-covariant rather than rotation-invane intaiinnn,variant, behavior is needed for the output of the network, the same attention structure can be used with slight modifications; here, we coerce vector from the product Pijk... (which consists of directly taking the vector component from products of odd numbers of input vectors, or multiplying a bivector by the unit trivector to produce a vector- as shown in the last column of Table m-in the case of even numbers of input vectors). These vectors can be combined with a scalar rescaling each vector-generated by learned function and the
135
137
138
139
140
attention mechanism to yield
F!
Σ Wish... Reegsk.. jh...
(2)
Results
141
We demonstrate the utility of our geometric algebra attention scheme by training deep networks to solve three problems appearing in physics, chemistry, and biology. For simplicity, all the models presented here utilize pairwise attention with a working depth of 32 units. Value functions V, score functions S, and rescaling functions R. are simple multilayer perceptrons with a hidden width of 64 units, with layer normalization applied to the output of V. The network for crystal structure identification uses the mean function for merge functions A and join functions J, while the other two applications use learned linear projections. Networks are trained for up to 800 epochs using the adam optimizer(2I]:: the learning rate is decreased by factor of 0.75 after the validation set loss does not decrease for 20 epochs, and training is ended early if the validation set loss does not decrease for 50 epochs. Numerical results are reported as the mean and standard error of the mean over 5 samples. Python code under the MIT license implementing each experimental workflow is included in the supplementary information.
143
144
145
146
147
148
149
151
152
153
154
Crystal Structure Identification
155 On length scales ranging from those of atoms colloidal particles, matter often organizes itself into ordered two- or
156
structures. One of the core ideas of materials science is that struc- ture is one of the major determining factors for material behavior. With this perspective in mind, when studying computational models of self-assembling systems we often first identify what struc- tures, if any, have formed in our simulations-a task complicated by naturally-occurrig thermal noise, crystallographic defects, and potentially the complexity of the structures themselves. Early efforts to automatically characterize structure led to the widely-used Steinhardt order parameters|222
157
158
159
160
161
162
which are rotationally-inaatio environments. While the Steinhardt order parameters can be useful when studying phase transi- tions or distinguishing among small number of phases, determining appropriate hyperparameters- including neighborhood size to consider, spherical harmonic order l to use, and thresholds to iden- tify behaviors of interest-can be difficult[233. For this reason, data-driven approaches to analyzing structure have been the subject of great interest in recent years|25].
sums of spherical harmonic magnitudes over local particle
165
166
187
168
We use geometric attention networks to identify the source structure type of small neighborhoods of particles extracted from bulk crystals. We select 8 prototypes of single- and two-component crystals from the AFLOW Encyclopedia of Crystallographic Prototypes/26. 27]]. These structures are chosen to demonstrate that models can learn not only geometric information (cF'4-Cu and h-P2- Mg are similar structures but with a different stacking of their dlss-ppackd layers; the clathrates P46-Si and cF136-Si are also similar, with a different arrangement of many common motifs), but also the information encoded within each point (eP2-CsCI and eF8-ZnS differ from c72-W and
170
171
172
173
174
175
cF88-C only by their particle type assignments). For each structure, we rescale the unit cell such that the shortest nearest-neighhor distance over the structure is 1 before replicating the unit cell to consist
of at least 2048 particles. We then create three samples of each structure by adding Gaussian noise
5

---

Position
(N, 3)
Type
Residual partial attention
Attention reduction
softmax
Class
Dense
Dense
(N, 2N,)
(1)
b
2
eF4-Cu
cP16-i
cf2-W
cF8-C
hP2-Mg
cP136-Si
cP2-C&CI
cP8-Zn8
(a) Network architecture for crystal structure identification. Coordinates and particle types are passed through two permutation-covarian layers before a final innnutaaionnivvaiian re- duction. (b) Crystal structure prototypes chosen for the structure identification benchmark. Simple and complex structure includinc two binary structures are included.
Figure 2:
178
with standard deviation 10-3 5--00-2, and 0.1 separately to the particle coordinates, in order to emulate thermal noise. For each particle the structure, we find the 12 nearest neighbors and their associated types using the freud[28]] python library, which we feed into the network the pairwise
179
180
distance Fi F F%z and one-hot-enee nneeode symmetrized type vector At. [Ie, I,, It + 4,
where / is the identity matrix of dimension corresponding to the maximum number of types.
189
We train classifiers with 2 permutation-covariint attention blocks before final reduction over the en- particle neighborhood-
184
shown Figure in in order to categorize local particle environments according their source crystal structure type. These networks rapidly learn to identify structures after few epochs, with final overall accuracy of 98.7% 0.2% after training for roughly 45 minutes on an NVIDIA Titan Xp GPU.
185
186
188
Molecule Force Regression
189
One of the most dramatic contributions of deep learning to the field of chemistry lies in constructing fast, accurate approximations of expensive physical calculations 29, G0J. Machine learning models
190
191
can be many orders magnitude faster than the methods used to generate their training data, which
192
can bring vastly more detailed and longer-time simulations into the realm of possibility. Central the applicability of these methods are issues of symmetry and equivariance: any imperfection
198
in rotational invariance of learned potential energy function could ruin the proper thermodynamic behavior of model, for example, so models must be carefully designed ensure physical behavior.
196 In a method similar to Batzner et al. [[I]], we train models to predict the per-atom forces calculated using ab initio molecular dynamics and density functional theory available in the MD17 dataset[3]. As shown in Figure EL we first transform the raw coordinates and types of each atom in given and difference of the coordinates and one- hot type encoding for each atom with respect each other atom, respectively, to fix translation invariance and assign type representations to the pairwise particle bonds. We then perform series of geometric attention calculations, calculating new values per atom, which are finally summed to produce scalar energy. The gradient of this energy with respect to the input coordinates is used to produce the force output of the network, which ensures that conservative force field is learned.
197
198
199
molecule into the pairwise difference and symmetric
201
202
203
204
Consistent with previous benchmarks this dataset, we train networks using the mean squared distance loss for each molecule using 1,000 snapshots of forces each training, validation, and test

---

Position (N, N)
A Type
Residual partial attention
Attention reduction
Energy V.
Force
(N, N, 2N,)
Dense
Dense
(1)
(N,
b
Figure 3: (a) Network architecture for molecular force regression. Coordinates and particle types for all atoms in a molecule are fed into the network as a set of pairwise distances, with the atomic rep- resentations refined through series of geometric algebra attention layers. Six permutation-covarian layers are stacked before reducing the representations with a final, permutation-invariant geometric product attention layer. (b) Sample pairwise attention maps for four training data molecules (mal- onaldchyde, aspirin, benzene, and uracil) after filtering out low-attention pairs. The attention maps indicate how strongly the pair of atoms joined by the line affect the representation of the atom indicated with a star, with lighter lines indicating greater influence. Qualitatively, more complex bonding environments such as those on the right tend to have longer-range attention interactions than the simpler environments on the left.
data sets. We also report results for models trained on all molecules' data simultaneously. or 8,000 snapshots each for training, validation, and testing. Training model on an individual molecule's 's
208
data takes between 30 minutes (ethanol and malonaldehyde, with nine atoms each) to two hours 209
210
(aspirin, with twenty-one atoms) on an NVIDIA Titan Xp GPU, while the all-molecule dataset
211
requires roughly hours train. Test set losses, expressed as the mean absolute error over each force component for each sample, are presented in Table D.
212
Table 2: Mean absolute error of force components (in meV) for geometric algebra attention works, NI]u and SchNer(H) architectures.
Molecule
This work
NequlP
SchNet
Aspirin
37.0 + 1.1
15.1
58.5
Benzene
11.8 + 0.5
8.1
13.4
Ethanol
21.4 + 0.5
9.0
16.9
Malonaldehyde
30.6 + 1.1
14.6
28.6
Naphthalene
23.7 + 1.0
25.2
Salicylic acid
30.2 1.2
10.3
36.9
Toluene
20.5 1.3
4.4
24.7
Uracil
27.4 0.8
7.5
24.3
All molecules
10.7
Our geometric algebra attention networks produce results competitive with SchNet[R], architec- ture using learned radial distance convolution filters. Although the models generated here do not out- perform the Neural Equivariant Interatomic Potentials by Batzner et al. [3]]] we note that our models are trained for a fraction of the time (2 GPU hours and 800 epochs for our method, compared to on the order of GPU days and 2500 epochs for NequIP) and without drastic hyperparameter tuning aside from optimizing the number of residual blocks use in the network architecture. Notably, the models trained on all 8,000 molecular snapshots perform significantly better than almost all of the specialized models, indicating that additional data could likely improve the results presented here without careful hyperparameter optimization.
215
216
217
218
219

---

Amino Acid Coordinates
(Ncaarse 3)
Vector
A Type (Ncorrse 2N
Residual
atomic
Dense
partial attention
attention
reduction
2
CA C1
Residual
Atom Type Embedding
02
vector
N2
(N D)
atomic
attention
Atomic Coordinates
(N 3)
Figure 4: Network architecture for inverting a coarse-grained mapping of a protein. Models are trained predict atomic-resolution coordinates from Neearse 12 neighboring amino acid centers of mass using geometrically-infone scalar-to-scalar attention (blue), scalar-to-vector attention (pur- ple), and vector- cector-to-veeee vector attention (red).
222
Backmapping Coarse-Graining in Proteins
223
When simulating large molecules-Sue such as proteins or other polymers coarse graining: process by which groups of particles are merged into (fewer) distinct beads, en- abling faster simulations by decreasing the number of degrees of freedom of the model[]]. Although data-driven approaches have been highly successful to formulate coarse graining operations in the forward direction (that is, from more-detailed to less-detailed systems), some problems are best solved using the original, fine-grained system coordinates, which are not directly available in coarse- grained simulations. To demonstrate the potential for our geometric algebra attention mechanism
common employ
226
227
228
229
230
on this task, we train models predict the coordinates the heavy atoms that form an amino acid from the centers mass of the nearest-neighhor amino acids. We take protein structures 50]] that have high-resolution structural refinements (with resolution
231
less than equal to 1.0 Á) and were published between 2015 and 2020 from the Protein Data Bank[5]]]. For applications of this method to systems at nonzero temperature, we would expect to be better-served by using an architecture that produces distributions instead of only point values, but we disregard this here for simplicity; in other words, here we are teaching models to memorize the results of structure refine- ment algorithms, which may be different for each PDB entry. For every amino acid each entry, we create point cloud of 12 nearest neighbor amino acid centers of mass, as well as a point cloud of the primary amino acid's atomic coordinates relative to its center of mass. Two layers of permutation-coovaiaa geometric product attention are applied to the coarse-grained amino acid co- ordinates before being passed to layer which produces vector output according to Equation E by augmenting the tuple representation uijk.. of Equation M with labels corresponding to the identity of the atom that should produced, so that the value is calculated as
234
235
236
237
238
242
243
Catom,ijk.... I (Vatom (. Uy Uk....))
244
Following this layer which maps coarse-graie grained coordinates of amino acids to fine-graine coordi- hhych nates of two rotation-covarian layers are applied to the atomic coordinates to further refine
245
246 them, as shown in Figure
Because the resolution of the structural refinement algorithms on the order of 0.5 À or greater, the training set error measure of the learning progress of the models instead of performing standard split of training, validation, and test set data. After training for roughly hours

---

an NVIDIA Titan Xp GPU, models achieve mean absolute error of 0.128 A = 0.002 Á (down
250
251
from approximately 0.5 initially), indicating that they are able to learn to reconstruct atomic-scale coordinates from coarse-graiin gaaneeer positions.
252
Discussion
253
254
Overall, we find the architectures formulated here to be useful for variety of tasks. Rather than being limited to operating on bond distances and angles as in SchNet[B), PhysNet[[O], and DimeNet([L]), geometric algebra provides a systematic way to build functions with the desired with the flexibility to incorporate other types of geomet- ric objects (such as the orientation quaternion commonly used for anisotropic particles molecular dynamics methods|D21). The attention mechanism presented here provides simple yet powerful
255
256
257
rotation- and
258
259
method to incorporate both geometric and node-level signals. The primitives of our geometric alge- bra attention scheme-distances, areas, angles, and volumes-and the calculated attention weights naturally lend themselves to interpretability, which we believe will prove useful in distilling insights from trained models.
262
263
264
Limitations
285
Combination of terms. Although the architectures presented here work well for the problems we have selected, creating geometrie products of vectors only a subset of the valid combinations that could be generated. In these cases have carefully chosen sums and differences input vectors to respect symmetries we would like impose on the system-such using the pairwise distance of all input coordinates for the molecular force regression task to impose translation invariance- but it is possible that more powerful models could be formed by incorporating learned linear combinations of inputs or intermediate multivector quantities. We leave this as a topic of future work.
266
267
268
270
271
Computational scaling and neighborhood definition. An obvious limitation to using higher- degree correlations lies in the computational complexity and memory scaling of generating tuples, which are both proportional to N for neighborhoods of N coordinates and tuples of length Poly- nomial scaling behavior can be ameliorated by restricting which combinations of input points are considered, essentially treating the attention weights of all other combinations as O. These combi- nations could be randomly sampled from all valid indices ijk... or use more physically-relevan re- strictions, such as utilizing the molecular connectivity graph for molecular force regression or edges
272
273
278
277
278
279
derived from the Voronoi tessellation for other applications. smoothness of model predictions concemnas as may be the case for learning general N-body interaction potentials, for example-the
280
architectures presented here could be augmented by incorporating weights that decay bonds broken in the Voronoi diagram graph[23].
Conclusion
284
In this work, we have presented strategy for developing rotation- and permutation-equivarti neural network architectures by combining geometric algebra and attention mechanisms. These architectures operate directly the vector, scalar, and other geometric quantities of interest pro- duce outputs which respect desirable symmetries by construction. We believe that the mathematical simplicity and the insights derived from attention maps are particularly appealing aspects of the algorithms presented here. We hope that these architectures will help wider range of scientific disciplines reap the benefits of geometric deep lcarning.
285
288
289
290
References
291
[1] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learn- ing: Grids, groups, graphs, geodesics, and gauges. arXiv:2104.13478 [cs, stat], April 2021.

---

294
[2] Julia Dshemuchadse, Pablo F. Damasceno, Carolyn L. Phillips, Michael Engel, and Sharon C. Glotzer. Moving beyond the constraints of chemistry via crystal structure discovery with isotropic multiwell pair potentials. Proceedings of the National Academy of Sciences, 118(21),
295
296
297
May 2021.
298
[3]
Stefan Chmiela, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof T. Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science Advances, 3(5)=1660311, May 2017.
289
300
[4] Siewert J. Marrink and D. Peter Tieleman. Perspective on the Martini model. Chemical Society Reviews, 42(16))6801-6822 July 2013.
302
303
[5] Yulan Guo, Hanyun Wang. Qingyong Hu, Hao Liu, Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
304
306
[6]
Charles R. Qi, Hao Su, Mo Kaichun, and Leonidas J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 77-85, July 2017.
307
308
309
77]
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexander J. Smola. Deep sets. Advances in Neural Information Processing Systems, 30, 2017.
310
311
312
[8] Hugues Thomas, Charles R. Qi, Gean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. KPConv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE International Conference on Computer Vision, 2019.
313
314
[9] K. T. Schütt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller. SchNet: A continuous filter convolutional neural network for modeling quantum interactions. In Proceedings of the 31 International Conference on Neural Information Processing Sys- tems, NIPS'17, 17, pages 992-1002, Red Hook, NY, USA, December 2017. Curran Associates Inc.
316
317
318
319
320
[10] Oliver T. Unke and Markus Meuwly. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, 1566)3677-3363, June 2019.
321
322
[11] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, September 2019.
324
325
[12] Cheol Woo Park, Mordechai Kornbluth, Jonathan Vandermause, Chris Wolverton, Boris Kozin- sky, and Jonathan Mailoa. Accurate and scalable graph neural network force field and molecular dynamics with direct force architecture. npj Computational Materials, 7(1):1-9,
326
May 2021.
329
[13]
Risi Kondor. N-body networks: A covariant hierarchical neural network architecture for learn- ing atomic potentials. arXiv:1803.01555 [es], March 2018
330
331
[14] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and cunssaationeeuivarian neural networks for 3d point clouds.
332
333
08 [cs], February 2018
334
[15] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
335
336
[16] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups 19444991 [cs, math, stat], April 2021.
10

---

339
[17] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3) transftransoorm:::: 3D roto-translation equivariant attention networks. Advances in Neural Information Processing Systems,33:10 33:1970-1981, 2020.
340
341
342
[18] William Clifford. Applications of Grassmann's 's extensive algebra. American Journal of Math- ematics, 1(4):350-358, 1878.
343
344
199 Alan Macdonald. A survey of geometric algebra and geometric calculus. Advances in Applied Clifford Algebras, 27(1))853-891, March 2017.
345
348
[00] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc., 2017.
[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
351
352
353
[22]
Paul J. Steinhardt, David R. Nelson, and Marco Ronchetti. Bond-orientational order in liquids and glasses. Physical Review B, 28(2):784 (000))585888811000 July 1983.
354
355
[23]
Walter Mickel, Sebastian C. Kapfer, Gerd E. Schröder- Turk, and Klaus Mecke. Shortcomings of the bond orientational order parameters for the analysis of disordered particulate matter. The Journal of Chemical Physics, 138(4))044501, January 2013.
356
357
358
[24] Emanuele Boattini, Marjolein Dijkstra, and Laura Filion. Unsupervised learning for local structure detection in colloidal systems. The Journal of Chemical Physics, 151(15):154901, October 2019.
359
[25] Paul S. Clegg. Characterising soft matter using machine learning. Soft Matter, 17(15):3991- 4005, April 2021.
362
[26] Michael J. Mehl, David Hicks, Cormac Toher, Ohad Levy, Robert M. Hanson, Gus Hart, and Stefano Curtarolo. The AFLOW library of crystallographic prototypes: Part 1. Computational Materials Science, 136:S1 18288, August 2017.
384
385
366
[27]
David Hicks, Michael J. Mehl, Eric Gossett, Cormac Toher, Ohad Levy, Robert M. Hanson, Gus Hart, and Stefano Curtarolo. The AFLOW library of crystallographic prototypes: Part 2. Computational Materials Science, 161:S1-S1011, April 2019.
367
368
369
[28] Vyas Ramasubramani, Bradley D. Dice, Eric S. Harper, Matthew Spellings, Joshua A. An- derson, and Sharon C. Glotzer. Freud: A software suite for high throughput analysis of particle simulation data. Computer Physics Communications, 254: 107275, September 2020.
370
[29] Oliver T. Unke, Stefan Chmiela, Huziel E. Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T. Schütt, Alexandre Tkatchenko, and Klaus-Robert Müller. Machine learning force fields. Chemical Reviews, March 2021.
374
[30] Jörg Behler. Four generations of high-dimensional neural network potentials. Chemical Re- views, March 2021.
378
377
[1] Simon Batzner, Tess E. Smidt, Lixin Sun, Jonathan P. Mailoa, Mordechai Korbbluth, Nicola Molinari, and Boris Kozinsky. SE(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. vrXiv:2101.03166 [cond-mat, physics:physics), January 2021.
378
379
380
[32] Akihiko Nakamura, Takuya Ishida, Katsuhiro Kusaka, Taro Yamada, Shinya Fushinobu Ichiro Tanaka, Satoshi Kaneko, Kazunori Ohta, Hiroaki Tanaka, Koji Inaka, Yoshiki Higuchi, Nobuo Nimura,, Masahiro Samejima, and Kiyohiko Igarashi. "Newton's cradle" proton relay with amide-imidis acid tautomerization in inverting cellulase visualized by neutron crystallography. Science Advances, 150026,, August 2015.
381
382

---

385
[33] Y. Hirano, S. Kimura, and T. Tamada. High-resolution crystal structures of the solubilized domain of porcine cytochrome b5. Acta Crystallggrahica Section D: Biological Crystallog- raphy, 71(7):1572-15811 July 2015.
386
387
388
[34] Shigeru Matsuoka, Shigeru Sugiyama, Daisuke Matsuoka, Mika Hirose, Sébastien Lethu, Hikaru Ano, Toshiaki Hara, Osamu Ichihara, S. Roy Kimura, Satoshi Murakami, Hanako Ishida, Eicchi Mizohata, Tsuyoshi Inoue, and Michio Murata. Water-mediated recognition of simple alkyl chains by heart-type baddy-acid protein. Angewandte Chemie Interna- tional Edition, 54(5):1508-155 1511, 2015.
389
392
393
[35] Yoshihiko Furuike, Yuka Akita, Ikuko Miyahara, and Nobuo Kamiya. ADP-ribose pyrophos- phatase reaction in crystalline state conducted by consecutive binding of two manganese(i) ions as cofactors. Biochemistry, 55(12):1801-15) March 2016.
394
395
[36] Hiraku Ohno, Kazuki Takeda, Satomi Niwa, Tomotaka Tsujinaka, Yuya Hanazono, Yu Hirano, and Kunio Miki. Crystallographic characterization of the high-potential iron-sulfur protein in the oxidized state at 0.8 Á resolution. PLOS ONE, 12(5)=0017118, May 2017.
398
389
[37] Harry P. Austin, Mark D. Allen, Bryon S. Donohoe, Nicholas A. Rorrer, Fiona L. Kearns, Ro- drigo L. Silveira, Benjamin C. Pollard, Graham Dominick, Ramona Duman, Kamel El Omari, Vitaliy Mykhaylyk, Armin Wagner, William E. Michener, Antonella Amore, Munir S. Skaf, Michael F. Crowley, Alan W. Thome. Christopher W. Johnson, H. Lee Woodcock, John McGeehan, and Gregg T. Beckham. Characterization and engineering of plastic-degrading aromatic polyesterase. Proceedings of the National Academy of Sciences, 115(19)1B4350- E4357, May 2018.
400
401
402
405
406
[38] Alessandro Vergara, Marco Caterino, and Antonello Merlino. Raman-markers of X-ray radia- tion damage of proteins. International Journal of Biological Macromolecules, 111:1194-1205,
407
408
May 2018.
409
[39] Alexander T. Baker, Alexander Greenshields-Watoon Lynda Coughlan, James A. Davies, Hanni Uusi-Kerttla,, David K. Cole, Pierre J. Rizkallah, and Alan L. Parker. Diversity within the adenovirus fiber knob hypervariable loops influences primary receptor interactions. Nature Communications, 10(1):741, February 2019.
412
413
[40] K. Takaba, Y. Tai, H. Eki, H.-A Dao, Y. Hanazono, K. Hasegawa, K. Miki, and K. Takeda. Subatomic resolution X-ray structures of green fluorescent protein. IUCrJ, 6(3):887-400, May 2019.
414
415
[41] Janet M. Paterson, Amy J. Shaw, Ian Burns, Alister W. Dodds, Alpana Prasad, Ken B. Reid, Trevor J. Greenhough, and Annette K. Shrive. Atomic-resolution crystal structures of the immune protein conglutinin from cow reveal specific interactions of its binding site with N- acetylglucosamine. Journal of Biological Chemistry, 294(45): November 2019.
418
419
420
[42]
Wei Tian, Peiqiang Yan, Ning Xu, Arghya Chakravorty, Robert Liefke, Qiaoran Xi, and Zhanxin Wang. The HRP3 PWWP domain recognizes the minor groove of double-stranded DNA and recruits HRP3 to chromatin. Nucleic Acids Research, 37(10)5543655455 June 2019.
421
[43]
Matthew J Schellenberg, C Denise Appel, Amanda A Riccio, Logan R Butler, Juno M Krahn, Jenna A Liebermann, Felipe Cortés-Ledesma, and R Scott Williams. Ubiquitin stimulated reversal of topoisomerase 2 DNA-protein crosslinks by TDP2. Nucleic Acids Research, 48(11):6300-6355, June 2020.
424
425
426
427
[44]
Marina Mlzz--Garride M2 Carmen Salinas-García, José C. Martínez, and Ana Cámara-Artigas. The effect of an engineered ATCUN motif on the structure and biophysical properties of the SH3 domain of e-Sre tyrosine kinase. JBIC Journal of Biological Inorganic Chemistry,
428
25(4):621 664, June 2020.
12

---

[45]
Shanshan Wu, Tam T. T. N. Nguyen, Olga V. Moroz, Johan P. Turkenburg, Jens E. Nielsen, Keith S. Wilson, Kasper D. Rand, and Kaare Teilum. Conformational heterogeneity of Sav- inase from NMR, HDX-MS and X-ray diffraction analysis. PeerJ, 8:e9408, June 2020.
432
433
[46]
Barbara Franke, Marta Veses-Garcia, Kay Diederichs, Heather Allison, Daniel J. Rigden, and Olga Mayans. Structural annotation of the conserved carbohydrate esterase vb 20_24B____ 21 from Shiga toxin-encoding bacteriophage $24B. Journal of Structural Biology, 212(1): October 2020.
425
436
437
[47] Matthias Barone, Matthias Müller, Slim Chiha, Jiang Ren, Dominik Albat, Arne Soicke, Stephan Dohmen, Marco Klein, Judith Bruns, Maarten van Dinther, Robert Opitz, Peter Linde- mann, Monika Beerbaum, Kathrin Motzny, Yvette Roske, Peter Schmieder, Rudolf Volkmer, Mare Nazaré, Udo Heinemann, Hartmut Oschkinat, Peter ten Dijke, Hans-Günther Schmalz, and Ronald Kühne. Designed nanomolar action impair invasion and extravasation of breast cancer cells. Proceedings of the National November 2020.
440
441
442
inhibitors of Ena/VASP EVHI inter-
443
Academy of Sciences, 11
445
[48] Renee Otten, Ricardo A. P. Pádua, H. Adrian Bunzel, Vy Nguyen, Warintra Pitsawong, MacKenzie Patterson, Shuo Sui, Sarah L. Perry, Aina E. Cohen, Donald Hilvert, and Dorothee Kern. How directed evolution reshapes the energy landscape in an enzyme to boost catalysis. Science, 370(6523): 1442-1444, December 2020.
446
447
448
449
[49]
Karla Frydenvang, Darryl S. Pickering, Giridhar U. Kshirsagar, Giulia Chemi, Sandra Gemma, Desiree Sprogøe, Anne Mette Kærn, Simone Brogi, Giuseppe Campiani, Stefania Butini, and Jette Sandholm Kastrup. Ionotropic glutamate receptor GluA2 in complex with bicyclic pyrimidinedione-baseed compounds: When small compound modifications have distinct effects on binding interactions. ACS Chemical Neuroscience, 11(12):1791-1800 June 2020.
450
451
452
454
[50] Steffen Glöckner, Andreas Heine, and Gerhard Klebe. A proof-of-concept fragment screening of a hit-validated 96-compounds library against human carbonic anhydrase Biomolecules, 10(4):518, April 2020.
455
458
457
[51] Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1):235-242, January 2000.
458
459
460
[52]
H. Kamberaj, R. J. Low, and M. P. Neal. Time reversible and symplectic integrators for molecular dynamics simulations of rigid molecules. The Journal of Chemical Physics, 122(22):2241 14, June 2005.
461
462
13

---

Checklist
463
484
1. For all authors...
465
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
466
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
488
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We feel that the methods presented in this paper are sufficiently agnostic to applications that hard to directly address any negative or positive societal impacts of the work.
469
470
471
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
472
473
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
475
(b) Did you include complete proofs of all theoretical results? [N/A]
476
3. If you ran experiments..
477
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] Code and scripts to reproduce results to be included as SI before the appropriate deadline.
478
479
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Details of training and architectures are provided in the Results section, as well as the code that will be included in the SI.
481
482
483
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
485
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
486
487
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
488
(a) If your work uses existing assets, did you cite the creators? [N/A] No existing assets were evaluated.
489
(b) Did you mention the license of the assets? [Yes]
481
(c) Did you include any new assets either in the supplemental material or as a URL?? [Yes] Trained models and workflows will be included in the SI.
492
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
494
495
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
496
497
5. If you used crowdsoureing or conducted research with human subjects...
496
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
499
500
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
501
502
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "msqb0BoVgp7", "reviewer": "Reviewer_pgXS", "summary": "This paper studies attention from a geometric algebra perspective. Geometric algebra provides a useful tool to combine vector, scalar, and other types of geometric inputs to account for rotation invariance/equivariance and permutation invariance/equivariance. In this paper, the attention mechanism is factored into four functions: a value-generating function, a tuple value-merging function, a joining function that summarizes the invariant and tuple representations, and a score-generating function. This new perspective on attention is novel and can motivate more follow-up works on studying high-order attention. On the other hand, however, this paper doesn't introduce new module design beyond interpreting attention and the actual implementation is the same with self-attention. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. This paper proposes a novel view on point-based attention. This new perspective is based on geometric algebra and is easy to generalize to high-order case. I think this is a great attempt to incorporate many useful mathematical tools into a machine learning model. \n2. The rotation equivariant/invariant features are easy to obtain under this new construction. These features are directly related to the corresponding components of multivectors. \n3. Application-wise, this paper tackles important molecule identification/regression problems. \n\n\n======================Post rebuttal====================================\n\nI thank authors for providing response to my questions. However, my concerns are not fully addressed: 1. the performance is worse than the baselines; 2. the learned score function is essentially similar to the the one (dot-product) used in multi-headed attention except this paper uses a \"relation-net\" style formulation. So I am leaning to rejection of this paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. My major concern is the difference between the actual method and the self-attention. Although this paper proposes a good interpretation and motivation to study high-order attention, it does not propose a new module. The implementation is actually an attention module. If this is the major contribution of this paper, what is the new component?\n2. Following 1, I would suggest authors to discussion the difference from Set Transformer and Point Transformer. Essentially, this paper uses exactly the same operation as Set Transformer. \n3. In the experiments, this method only compares to NequIP and SchNet. Why not try PointNet/DGCNN? I understand this paper claims that PointNet/DGCNN work on \"large\" point clouds but they are actually applicable to the \"small\" point cloud case.\n4. Even compared to NequIP and SchNet, this paper doesn't show consistent improvement, which makes me even worry about the empirical value of the proposed method.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "l5AVR1nOpe8", "reviewer": "Reviewer_qUKR", "summary": "The paper presents a new framework for building graph neural networks on point clouds that obey invariance or covariance properties through the use of Geometric Algebra. Geometric Algebra involves the use of a binary operator called the Geometric Product that operates on geometric quantities like scalars, vectors, bivectors and trivectors. \n\nThe proposed framework involves the building neural networks that are composed of geometric products of such quantities together with self-attention. The geometric products could be designed to maintain rotation invariance or covariance as needed, and attention can impose permutation equivariance which is important for point clouds.\n\nThe paper presents experimental evidence of the effectiveness of this method on three different problems taken from physics, chemistry and biology.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces the idea of Geometric Algebra within deep learning to build invariant or equivariant models for building invariant or covariant models for point clouds. This is important in many natural science problems where the point clouds are composed of atoms / molecues / physical objects that obey these properties. Past research has shown that models that obey these properties tend to be significantly more data efficient. \n\nThis paper makes use of Geometric Algebra to come up with a systematic framework for building and reasoning about such models. This is more general than methods like SchNet or Dimenet that only operate on pairwise distances and bond angles.\n\n## Originality\nThe use of Geometric Algebra is truly original as per my knowledge.\n\n## Quality\nThe paper presents three sets of experiments: crystal structure identification, molecular force regression and coarse graining in proteins.\n\nFor the crystal structure and protein tasks, the authors do not present any baseline methods. Without a baseline, it is difficult to understand the complexity inherent in the problem. I would recommend the authors to add at least a simple baseline.\n\nFor the force regression task , the authors compare against SchNet and Nequip. The proposed model performs comparably to SchNet, but significantly worse than Nequip (though its much faster than Nequip). It would be good to discuss why this might be the case. Is there a fundamental limitation to this method that is causing this? If not, would a simple modification / bigger model improve performance? \n\nThe authors do claim that training on all data performs better, indicating that more data could help improve their model's performance. However, this claim does not make sense without a corresponding number for Nequip, since it is possible Nequip will perform even better when trained on all data.\n\nWithout making these changes, it would be hard to definitively conclude that the presented experiments support the claims made by the paper.\n\nOverall, this feels like a very novel work, that needs more work before it can be published.\n\n## Clarity\nThe paper is clearly written and is easy to follow in spite of the authors using a new branch of mathematics. \nA few minor suggestions to improve clarity further:\n* Please add some pictures to explain what bivectors and trivectors mean for the uninitiated.\n* In equations where you use indices `ijk...`, it may be better to use `1:n`.\n\n## Significance\nBeing able to build in equivariance / invariance into NN models is extremely important for many natural science problems. This paper has the potential to impact a large number of future papers. However, the claims need to be validated more thoroughly with experiments before this can happen.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations of their research and propose some future work to possibly address it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "bRW7B5EtylV", "reviewer": "Reviewer_yNSv", "summary": "The authors propose a deep learning architecture for small point clouds that is permutation and rotation equivariant, which is based on geometric algebra. The efficiency of the method is demonstrated in the experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that the writing of the paper is rather confusing. Unfortunately, I am not able to judge the proposed idea, which might be interesting, because I could not understand it after trying to read the paper couple of times. In general, I find the clarity of the paper lacking and it is very hard to get the intuition behind the presented technicalities. In my opinion, the paper should be improved significantly, such that to be accessible from the reader.\n\nFor example:\n1) This is a rather technical paper, with a lot of text. I think the mathematical context should be formulated more explicitly e.g. dimensionalities, examples of multivectors, etc.\n2) The problem of interest is only presented from a high-level point of view. I think more specific examples are necessary.\n3) I could not get the intuition behind the proposed modeling choice. Maybe some images and easy examples are useful to make the presented ideas accessible.\n4) There is a lot of technical terminology (invariant, equivariant, covariant, multivectors, etc.), but I think is not clear exactly what each of them represents.\n5) Figure 1 does not help to understand the concepts.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations of the proposed idea. \n\nAs regards the potential negative societal impact, they argue that it is hard to find any since the method is agnostic to applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "N5MRQ3G9rxp", "reviewer": "Reviewer_f3bU", "summary": "The paper proposes a new attention mechanism based on Clifford Algebra. The main idea is computing features via geometric products, showing this would help to obtain equivariant networks to some symmetries. The method is applied to some significantly relevant tasks from physic and chemistry, with a limited quantitative validation of the method.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PRO\n======\n1) RELEVANCE OF THE TASK \\\nInjecting structural invariances or equivariants into a network is an active research field and has particular relevance in a disparate set of different fields. I believe this topic would be of high interest to the NeurIPS community. \n\n2) THEORETICAL FRAMEWORK \\\nClifford Algebra has been widely used in the past, showing its generality in describing different problems. I think the theoretical foundation is sound and it could gemmate new works, adapting the framework to the need of specific communities. The paper state that the code will be available under the MIT license, which is also a contribution.\n\nCONS\n======\n1) BACKGROUND AND NOVELTY \\\nI think the paper miss a consistent part of its positioning in the literature. Like many other works in deep learning literature, the use of Clifford Algebra was suggested a long time ago [1][2], also discussing the utility of such an approach to deal with invariants [3]. These methods have been re-proposed more recently [4][5]; similarly, other works on geometric algebra and deep learning are [6][7]. Given so, I would suggest including a section in the related work to analyze such works and emphasize the novelty of the proposed method with respect to them. At the moment, such ideas seem already well-known in the deep learning community\n\n2) EXPERIMENTS \\\nThe paper is a bit limited in quantitative analysis of the proposed method: \\\na) In the Crystal Structure Identification is provided only the overall accuracy, without any baseline and comparison. I would suggest considering having at least some baseline that does not exploit the proposed solution (e.g., a standard MLP) to understand the benefit of using the proposed approach \\\nb) In the Molecule Force Regression, the results is more satisfactory; however, the networks are trained for a different amount of time (800 epochs [2 hours] for the method vs 2500 [8 days] for NequIP), and this makes it hard to understand if the comparison is fair. I would suggest training all models for a similar time, or at least add the training curves to highlight the convergence of the networks. If the proposed method converges faster, this is merit and could be emphasized. Also, why in Table 2 All molecules results and standard deviation are reported only for the method of this paper, and not for the others?\\\nc) In Backmapping Coarse-Graining in proteins only a mean absolute error is provided. As above, I would suggest adding some competitors and baselines.\nFrom an analysis perspective, I think that would be good to add some studies on the behaviour of such attention, e.g., showing some examples highlighting where the proposed attention concentrate, and comparing it to a standard dot product attention mechanism. Finally, since the paper claims that different kind of symmetries can be selected by the network, I would suggest showing cases in which the attention mechanism behave in different ways and which invariant\\equivariant properties could be exploited.\n\n3) PRESENTATION \\\nWhile I think that the overall idea is clear, I think few things may be done to improve its presentation:\\\na) an appendix about Geometric Algebra would be useful for the reader; such tools are not common and maybe it is useful a formal recap of the basic principles\\\nb) In the subsection \"Attention from Geometric Products\" I have difficulties following the flow of thoughts. I would suggest defining V, M, J and S by equations. This would help to understand what exactly is going on in the pipeline. Similarly, Equation 1 seems a set of passages that can be well expressed by a diagram, maybe with a graphical example. I think these would be important points that would ease a larger audience in understanding the process.\\\nc) I consider it important to add a better description of the architectures implementations, training losses, hyper-parameters, and everything that would help to replicate the experiments. Such information would fit well into a supplementary material document. Also, a complete analysis of time would be beneficial since (as stated in the limitations) computational complexity and memory scaling is critical. \\\nd) I would suggest making each task of the Results section less verbose and schematizing better its structure with some paragraphs (e.g., for each task specify: data, goal, architecture and loss details, numerical results, analysis\\comment) \n\n[1]: Back Propagation in a Clifford Algebra, Pearson and Bisset, 1992 \\\n[2]: Neural Networks in the Clifford Domain, Pearson and Bisset, 1994 \\\n[3]: A new self-organizing neural network using geometric algebra, Corrochano et al., 1996 \\\n[4]: On Clifford neurons and Clifford multi-layer perceptions, Buchholz and Sommer, 2008 \\\n[5]: An Extended Multilayer Perceptron Model Using Reduced Geometric Algebra, Li and Cao, 2019 \\\n[6]: Embed Me If You Can: A Geometric Perceptron, Melnyk et al., 2021 \\\n[7]: Knowledge Graph Embeddings in Geometric Algebras, Xu et al., 2021 \n\nMINOR QUESTIONS\n===============\nConcerning the technical elements of the method, I have some curiosities:\\\na) by my intuition, this method is considering an attention mechanism that generalizes the standard one; this is achieved by considering a more general mathematical structure that can produce not only scalars but also other objects (vectors, bivector, trivectors). Some of these objects have quantities that have rotation invariance measures (e.g., signed area and signed volumes). Is this reasoning correct? If yes, given that in many cases, the symmetries of interest are known a priori, what is the benefit of the proposed approach w.r.t. other already invariant quantities (e.g., using the norms of point distances as features)?\\\nb) in the Limitations, it is reported that \"geometric products of vectors is only a subset of the valid combinations that could be generated\". But this is true for any proposed feature combinations; e.g., the standard MLP is just one of the many possible ways to recombine the features. So I would not mention this as a proper limitation of the method, since in general, the main approach is to stick to the simplest and more general thing instead of proposing complex features combinations. Or am I missing something?\\\nc) Why do you state that this method is specific for small point clouds? Is it due to the computational limitations, or do you think that such invariants are less useful in the case of larger ones?\n\nRATING JUSTIFICATION\n============\nOverall I like the theory and the idea behind the method. However, it is hard to understand the correct placement of the paper among similar works, both in terms of novelty and quantitative evaluations on the proposed tasks. I am looking for the rebuttal phase.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any particular potential negative societal impact in the proposed method.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see any particular ethical concerns in the proposed method.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for the rebuttal. Unfortunately, the paper is rejected from NeurIPS 2021. 
The reviewers' main criticism was the lacking exposition of the method in the paper, and relation to previous work. I would recommend the authors to improve their exposition of "Geometric Algebra" and the clarity of their respective architectures. Reviewers also complained that some baselines are required to better evaluate the performance of the method in practice. 
Decision: Reject