Gauge Equivariant Transformer
Authors: Lingshen He
Yiming Dong
Yisen Wang
Dacheng Tao
Zhouchen Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Gauge Equivariance
Transformer
Attention Mechanism
Manifolds

Abstract: Attention mechanism has shown great performance and efficiency in a lot of deep learning models, in which relative position encoding plays a crucial role. However, when introducing attention to manifolds, there is no canonical local coordinate system to parameterize neighborhoods. To address this issue, we propose an equivariant transformer to make our model agnostic to the orientation of local coordinate systems (\textit{i.e.}, gauge equivariant), which employs multi-head self-attention to jointly incorporate both position-based and content-based information. To enhance expressive ability, we adopt regular field of cyclic groups as feature fields in intermediate layers, and propose a novel method to parallel transport the feature vectors in these fields. In addition, we project the position vector of each point onto its local coordinate system to disentangle the orientation of the coordinate system in ambient space (\textit{i.e.}, global coordinate system), achieving rotation invariance. To the best of our knowledge, we are the first to introduce gauge equivariance to self-attention, thus name our model Gauge Equivariant Transformer (GET), which can be efficiently implemented on triangle meshes. Extensive experiments show that GET achieves state-of-the-art performance on two common recognition tasks.

Gauge Equivariant Transformer
Lingshen He Yiming Dong Yisen Wang Dacheng Tao* Zhouchen Lin 42,3t
Key Laboratory of Machine Perception (MOE), School of Artificial Intelligence, Peking University "Institute for Artificial Intelligence, Peking University Pazhou Lab, Guangzhou 510330, IJD Explore Academy, JD.com
Lingshenhe@pku. edu yimingdong ml milcutlook com, yisen. edu. en, dacheng
com, 2 zlin@pku edu.
Abstract
Attention mechanism has shown great performance and efficiency in a lot deep learning models, in which relative position encoding plays crucial role. However, when introducing attention to manifolds, there is no canonical local coordinate system to parameterize neighborhoods. To address this issue, propose an equivariant transformer to make our model agnostic to the orientation local coordinate systems (i.e., gauge equivariant), which employs attention to jointly incorporate both position-based and content-based information. To enhance expressive ability, adopt regular field of cyclic groups feature fields in intermediate layers, and propose a novel method to parallel transport the feature vectors in these fields. In addition, we project the position vector of each point onto its local coordinate system to disentangle the orientation of the coordinate system in ambient space (i.e., global coordinate system), achieving rotation invariance. To the best of our knowledge, we are the first to introduce gauge equivariance to self-attention, thus name our model Gauge Equivariant Transformer (GET), which can be efficiently implemented on triangle meshes. Extensive experiments show that GET achieves state-of-ther -hr-ttr-rt performance on two common recognition tasks.
self-
Introduction
Recently, Transformer has dominated the area of Natural Language Processing [48]. Its key advantage over previous methods its ability to attend to the most relevant part in a given context. This is largely attributed to its self-attention operator, which computes the similarity between representations of words in sequences in the form of attention scores. Because the superiority, researchers start to apply Transformer to other learning areas, including Computer Vision 26 53 [4
16
59)
and Graphs
In this work, we aim at applying Transformer to manifolds. Unlike regular data, such as images, where each neighbor owns a clearly quantified relative position to its center in a canonical coordinate system, irregular data do not have a uniquely defined local coordinate system for the neighbors, resulting in problem of orientation ambiguity, which directly obstructs the Transformer to numerically intake the relative position information.
Several works have been proposed to deal with the rotation ambiguity problem, in which a promising way is to exploit gauge equivariance. While most of them are not rotation invariant to global
WWok was done during an internship at JD Explore Academy. 'Equal contribution. Sorted by tossing the coin.
Corresponning author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

coordinate system, all of them are established on convolution, i.e., equal attention to neighboring points and neglect content- information. So is desirable to propose gauge equivariant transformer with the support of rotation invariance.
In this paper, we propose Gauge Equivariant Transformer, named GET for short, which employs multi-head self-attention to simultaneously utilize position-based and content-based information, and both gauge equivariant and rotation invariant. To achieve rotation invariance, we first project zyz coordinates in a global coordinate system onto local coordinate frame, and then design equivariant transformers to overcome the orientation ambiguity problem of local coordinate systems. We adopt the regular field proposed in [3 as feature fields of intermediate layers, since the representation of regular field commutes with element-win wise activation functions. After that, we propose a novel method to accommodate parallel transport of feature vectors in regular field with any rotation angles. Since we adopt regular fields in intermediate layers, we make relaxation such that they are equivariant only for gauge transformations of angles that are multiples of 2T/ /N. Exact equivariance can be guaranteed for gauge transformations at multiples of 2x /N, and an equivariance error bound can be obtained for all other angles. In experiments, our model shows better performance and greater parameter efficiency than all baseline methods. Our contributions can be summarized as follows:
We propose GET, which incorporates attention and achieves both gauge equivariance and rotation invariance with superior expressive power. GET is mathematically proven to be exactly equivariant on angles that are multiples of 2 (N E N and an equivariance error bound is derived for other angles to guarantee the overall approximate equivariance property.
We carefully design the model input ensure that it is irrelevant to the global coordinate system, only depending on the choice of gauge. Our model achieves rotation invariance with the assistance of gauge equivariance.
We propose novel method to parallel transport the feature vectors in the regular field by extending the representation of cyclic group to any angle rotation group. Compared to previous methods using truncation or interpolation, our extension can preserve more geometric information.
We elevate the model performance by designing a new approach which incorporates Taylor expansion in solving the equivariance constraint, which has better approximation ability in local neighborhoods.
We confirm the superiority of our model via extensive experiments. Our model outperforms the HSN model on the SHREC dataset by 3.1% accuracy, and outperforms the MeshCNN model on the Human Body Segmentation dataset by 0.3% accuracy with much fewer parameters, presenting state-of-the-art performance.
2 Related Work
Geometric Deep Learning. Geometric deep learning is an emerging field concerning with adapting neural networks to various data types (7], especially on irregular data. For research works on modeling curved surfaces, common methods include view-bac based methods [46] 61 51] and volumetric methods 32 40 (2). To boost efficiency, some works define convolution on point clouds directly [37 38) but they are vulnerable to pose change since the coordinate inputs are dependent of the global coordinate system. So it is highly desired to develop models that solely intake geometric information of surfaces.
Approaches that merely utilize intrinsic information of surfaces are called intrinsic methods. They use local parameterization to assign each neighboring point with a coordinate for information aggregation. seminal work is Geodesic CNNs (3I), which uses an exponential map to parameterize each local neighborhood and takes the maximum response across multiple choices of local coordinate orientation. While taking the maximum response direction discards the orientation information of feature maps, as an alternative, aligning local coordinate with principle curvature direction is another approach to deal with the ambiguity problem 33 6
But this approach can only be applied in limited cases as the curvature direction may be ill-defined at some points or even areas of curved surfaces. MDGCNN (35) and PFCNN (0 describes features by the so called directional functions. But both of them adopt scalar equivariant kernels, resulting in limited expressive power.

---

Equivariant Deep Learning. Success of CNNs has been attributed to translation equivariance, which inspires researchers implement more powerful equivariant models, including equivariance
of planar rotation [2
15
13
58
54
44
25 3D space rotation 57 19
47
36
55
35
28
2 39].
sphere rotation 99
and so on. All above works are about equivariance on homogeneous
space [29] [0]. Cohen et al. I further extend equívariance to manifolds, in which they identify a
new type of equivariance called gauge equivariance. The models in [4)] are successful extensions
of gauge equivariant CNNs on mesh surfaces. However, their model suffers from changes in the orientation of global coordinate system.
Also, there are works proposed for equivariant attention. Romero et al. equivariant networks, which effectively attends to co-occurring transformations. Romero et al. 44 further propose attentive group equivariant convolutional networks. Besides this, transformers have also been applied to group equivariant networks, where Fuchs et al. [21] do so via irreducible representations, Hutchinson et al. [27] via Lie algebra, and Romero et al. 42 via generalization of position encodings. All the models above are equivariant symmetric groups, while currently gauge equivariant attention is still lacking.
45 propose co-attentive
3 Preliminaries
Unlike regular data, in which coordinates (or pixels) are aligned in a global frame, there is no such specific frame on general manifolds. To begin with, we briefly review and define some mathematical concepts,
3.1 Basic Definitions
We restrict our attention to 2D manifolds in 3D Euclidean space. Consider 2D smooth orientable manifold M. For point p in M, denote its lungent plane as TpM. Each point in T,M can be associated with a coordinate by specifying coordinate system. Namely, we can parameterize the tangent plane Tg M with a pointwise linear mapping Wi TR22 + Tр M, which is defined as the gauge at point p I.I. The gauge of manifold M is the set containing gauges at every point in M.
For planar data, a feature map is the set of features located at differentt positions on plane. Similarly, feature field on surface is a set of geometric quantities at different positions of the surface. Note that these two concepts are similar but not the same. From the perspective of geometric deep learning, a feature map is defined as numerical values of geometric quantities that may be gauge dependent, while feature field refers to geometric quantities themselves that are gauge independent. For example, each point of the surface can be assigned with a tangent vector as its feature vector, all of which form a feature field. As is shown in Figure T the tangent vector v itself is a geomelric quantity, which stays the same regardless of arbitrary gauge selection but takes different numerical values in diffcrent gauges following an underlying rule. We use f to denote the feature field of a manifold, fw M + R" denotes the feature map under the gauge w and (p) denotes the feature map evaluated at point p.
Different gauges can be linked by gauge transformations. The gauge transformation at point p frame transformation: 9p E SO(2). where SO( is the special orthogonal group consisting of all 2D rotation transformation matrices. A new gauge w/, can be produced by applying gauge transformation 9p to the original gauge Wpp i.e., w/, Ip Wpp Gauge transformation is usually characterized by group representations. Group representation is a mapping p G + GL(n, R) where GL(n, IR) is the group of invertible n n matrices, and p meets the condition ((92) = p(9192). where 91, 92 E G are the elements of the group, 9192 are element product defined on the group, and p(gn )p(22) is matrix multiplication. Therefore, after applying the gauge transformation 9p. the feature vector value (p) transforms to (p) = p (g, 1) (p). Here p is a group representation of (2) which is called the rype of the feature vector. If all the feature vectors share the same type p. the feature field is called p-field and p is called the representation type of the field. The above definitions can also be at the manifold level, i.e., for = p(g- The notation kp, where & is a positive integer, refers to the group representation whose output is k-blocks diagonal matrix with each block equals to p. In particular, if the representation of a feature field is p(g) = 1, then the feature field becomes scalar field, denoted as Po-
3

---

3.2 Gauge Equivariance
For function o, its input is a feature map fwo where f is Pin Pin-field, in order to make Φ gauge equivariant, and its output fw should be a feature map, where f is a Pout-field. When ø is a layer of neural network, gauge equivariance implies that does not rely on the gauge in the forward process.
Suppose that there are two gauges w and w' linked by gauge transformation gi w' have fm' Pin f since I is a Pin-field. Gauge equivariance means that the outputs fw fw and fwr olfw]] are linked by the Poat representation of the same transformation g. i.e. J - Pour (g) Finally, we get:
w,
Pora(g 1)0 [f]] =
(1)
To sum up, a function o is gauge equivariant if the above equation always holds for any feature field f. gauge and transformation g.
3.3 Riemannian Exponential Map
Transformers require encoding the relative position to propagate information. Note that in images, there is still a local point parameterization, which is so natural that one even does not realize it. For general manifolds, it non-trivial to establish a parameterization criterion, at least in the local frame. Among many charting-basee methods, the mostly used one is the Riemannian exponential map exp, T M + at point p, which is mapping from the tangent plane to the surface. For a coordinate vector M,M, the output of the Riemannian exponential map is obtained by moving the point p in the direction e along the geodesic curve with distance of ||o|| Denoting the arrival point 9, we have expp(() (v) q. Figure visualizes the exponential map well as some basic definitions introduced in Section 3.1 According to the inverse function theorem, expy is a local diffecomophism avoid metric distortion the point p. The inverse of Riemannian exponential map is the logarithmic map logp" M T,M. Under the gauge Wp. every point @ in the neighborhood of is associated with coordinate wp 19g,(9).
3.4 Parallel Transport
The self-attention operation is essentially an aggregation of local neighboring features. However, the feature vectors different points are from different spaces, thus they need to be parallel transported to the same feature space before being processed. For tangent vector point q. parallel transport it along the geodesic curve to another point with respect to Levi-Civita connection [8], which preserves the norm of the vector. Levi-Civita connection is isometry from T,M to T,.M and determines the parallel transport of s, see Figure
2 In a gauge w, the parallel transport of tangent vector corresponds 2D rotation gh E SO()) which contains the relative orientation of gauges in the neighborhood. For a general feature vector of type, parallel transport can be expressed as s/ pp Sw.
3.5 Self-attention
Attention enables the model to selectively concentrate on the most relevant parts based on their content information [48 53 4 22) Consider set of tokens t= (41,42,-. (T), where I, E R Attention is composed of three parts, namely query, key and value, denoted by Q : R + Re K R + Fx and V IR + g.' respectively. When Q, K and V are from the same source, it is called self-attention. When there are multiple sets of Q, K and V's, it becomes multi-head attention.
The output of a multi-hean self-attentie attention transformer node is the linear transformation of the concatenation of the outputs at all the heads:
MHSA((); = War (ISACOS*)
(2)

---

I.
4 axp
1
M
Eg
Figure Illustration of basic definitions and Rie- mannian exponential map. Here, "p (black) and w/ (blue) are two gauges on the tangent plane T,M and they are linked by the gauge transfor- mation 9p- The coordinate of v takes different numerical values under wp and w/y, as is illus- trated lower part. The exponential map assigns each vector in T.M with corresponding point q on the surface M.
Figure Parallel transport. The tangent vector is parallel transported from g to resulting in new vector at point p. The numerical value change imposed by parallel transport is jointly determined by the geometric property of the surface, the Levi-Civita connection and the underlying gauge w.
where is the vector concatenation operator. The single head attention output at head h is
T Σ j=1
SA())
aij
(b)
(t))
(3)
where (h) is the value function at the head h, and aj (h)
is attention score computed by
(h) S(K (t,). 2
(4)
Ef (() (t,))
where k(h),, Q(h) and S are the key function, query function and score function, respectively.
4
The Proposed GET
4.1 Gauge Equivariant Self-Attention Layers
Suppose that the dimensions of input feature field and output feature field j are Cin and Conts respectively. We define the gauge equivariant multi-head self-attention output at point under the gauge w as
(.)(P) MHSA(f).)) WM (1ISACT)*)))
(5)
where War is the linear transformation matrix. At the head h. the output defined as
SA()())))
a(F) v(h) ( (aw)),
(6)
where u = (uy, TT)TT E R², feature vector from point qu to point p under the gauge w, and V,, is the value function incorporating the position information through an encoder matrix Wy(w) RC XCuutCii, i.e.
exp, (p) flr (gw) is the numerical value of parallel transported
Pa(a = Pin (o," -p (.k.-pp)W (wl Vu Vu(Fll = Wv(w)f((
(7)
5

---

is the attention score incorporating the content information, and is computed as:
(f (frr [ S(K (5) (fw (p)), Q (go))))
a af) p.du
We propose to enforce the attention score to be gauge invariant and the value function to be gauge equivariant, to make the attention layer gauge equivariant. The details of constructing them are presented in Sections 4.3 4.4 respectively.
4.2 Extension of Regular Representation
In our model, the feature fields the intermediate layers are all regular fields (i.e., whose type is regular representation). Regular representation is a special type of group representation of CN- we use O to denote the rotation matrix with angle of k 2 /N, then CN can be expressed as CN {Op, I ON-I For A: = 0, 1, N 1. the regular representation pray (0₂) is an N N cyclic permutation matrix which shifts the coordinates of feature vectors by k steps.
Regular representation provides transformation matrices when rotating by angles of multiples of
2x/ N, but feature vectors can go through any rotation in SO(2) during parallel transport. Figure
3 illustrates this issue by giving example in R with respect to pre, We propose to extend the
regular representation of CN by finding an orthogonal representation PN of SO(2), such that it behaves the same as regular representation for any element in CN, i.e.
AA E CN ON.
Cs Preg (9).
As Cy takes different forms between odd values and even values of N, Theorem I shows that only odd N's are vaild in our model.
Theorem (i) If N is even, there is no such real representation PN of SO(2) that satisfies Egn. (i) If N odd, there representation PN in (ū) is an orthogonal representation.
unique representation PN of SO(2) that satisfies Eqn. 9 (ii) The
Here only show method for constructing Pv in Theorem According group representation theory, regular representation Prey can be decomposed into irreducible representations (irrep for short), i...
Preg (+) A diag (po(O), ". (0))
(10)
where Po.... 1(2-1)/2 are the irreps of CN, and A E GL(N, The irreps of CN take the following form for odd N:
AA (o = pp (ko((k0)
sin(k:)) cos(k-))
(11)
sin
where 8 (0, is the rotation angle of the matrix θ i.e.
cos sin 0 +
(12)
sin 0 cos 0
and h: == 1,.., N-1 We extend the irreps to SO()) as
Toos(k0) (kn(
-sin cos(k0)
AA
= 1,
(13)
where k = 1,..., By substituting the yp's in Equ. 10) with ø's, we get that for V0 SO(2).
(e) AA diag ())( 6 (0), بييم (0))
(14)
Obviously the representation 2N satisfies the condition Eqn. this way, one can apply PN (9%
to feature vector of regular field during parallel transport.

---

4.3 Gauge Equivariant Value Function
Inspired by I1I we choose the value function to be the numerical value of the parallel transported feature vector multiplied by the value encoding matrix. For the value function to be gauge equivariant, the necessary and sufficient condition is that Equ. (15) always holds for any + SO(2):
Wv(e u) = Pout (O " (w).
(15)
We propose a practical method to solve Eqn.
15
We first expand the W, into taylor series:
Wv(u) Wn + Win1 W2
Wan + W4uius + Won Wou.+... +.
(16)
where W; E Cous x Cin 0, 1, .....))
)...)i is the Taylor coefficient. Since we adopt regular representation in this paper, Eqn. (15) only needs to hold for @ E CN. Plugging Eqn. (16) into Eqn. (15). by comparing the coefficients, W,'s 's need to satisfy that for any θ E Cn.
Wo = Pont (0 (1)))
(17a)
())(, - sin(O)W; = Poout (0 1 (otmm(())) (6), Sin(0)W1 + cos(0)W2 = -Pout (0 1 Wtn
(17b)
(17c)
To deal with the issue of having infinite terms in Equ. 16 the Taylor series. We use the second order Taylor expansion and omit higher order terms, i.e. (18)
we may bypass it by simply truncating
Wv(u) A Wo + Win + W222 + Wyu + W4 + Wsu
worth emphasizing that making truncations does not affect the equivariance property in the slightest, as the equations in 17) show the coupling characteristics.
Eqn. (17b is the constraint Wo in the order O, Eqn. (17b) and Eqn. (17c) the constraints on W1 and W2 in order 1, and there are three more equations in Eqn. (17) constraining on W3. Wa and W3 in the order 2. We can see that only the W''s in the same order are coupled together. This coupling property allows us not only to solve the equations in (17) in separate groups, but also to make truncations Eqn. (16) without affecting the equivariance property.
After truncation, solving the first six linear equations in (I7) which are separated into three independent groups, where m is the dimension of solution space. Each W((() is a tuple consisting of six components, W: ,.... wi(6) The details in solving linear equations are provided in supplementary materials. Then, the equivariant matrix basis Wi)) has the following form:
get set of solution bases of Taylor coefficients (W(1).... W(m)) by
(w) = (A) + W 01 + W U2 + W/ + W U1d2 + W. uz
(19)
which satisfies Eqn. 15 for all u. Their linear combination, Σ c; (( still meets Equ. 15 and c 's can be set as learnable parameters during training. With Wy Σ e co the value function in Eqn.
(7) is exactly equivariant to gauge transformations at multiples of 2m/N
It is remarkable that our method of solving the equivariance constraint Eqn. 15) is very general, as the solution process can be applied to any groups, including pin and Pout- Especially, it can avoid solving analytic solutions when the group very complex, like when the case is that the group is a high dimensional orthogonal group. addition, compared to Fourier series used in [54), Taylor series is better approximation in local neighborhoods. The omitted Taylor terms in Eqn. (18) is Ooa²), which is negligible when the radius a is small enough. So GET could achieve the same avoid selecting radial profiles that introduce performance with fewer parameters. In addition, extra hyperparameters.
4.4 Gauge Invariant Attention Score
In implementation, the manifold is discretized to mesh for computer processing. The discretiza- tion details are provided in supplementary materials. Here we set the key and query function to be structurally the same as Graph Attention Network 49). i.e. (fw((( w (p)
7

---

LAP
n
?
W
P
X
Figure 3: Illustration for the reason of exten- sion. fq) is a feature vector of type Preg- which takes numerical value (q) E IR$ under gauge Wg. Applying gauge transformation with an- gle 2 to wny- f((2) takes another value for (7), which is a permutation of (q) The problem
Figure 4: Local coordinate projection. Tp is the position vector in the global coordinate system marked in red. For better illustration it is moved to the local coordinate system, marked in blue. In the local coordinate system xp is projected onto the directions Ups Up and Ry, respectively, and the lengths of three directed line segments (in green) form the input Xp-
here is what value does (q) takes after it par- allel transported to point p.
Qt) (f% w fú (qu), where w E NNCi,, w:(h) RNCi.. The score function is structurally similar to [49]. which takes the following form:
Q()))),(())) (())LU(K(() Q()))).
(20)
Here, ReLU is the Nonlinear Rectified Unit acting each element of the N dimensions, and P: R.W + R is the average pooling function. The linear transformation matrices WK and Wo are
required satisfy the constraint in Eqn. (7a) CN for K and Q be gauge equivariant. After activation and pooling, the final attention score is gauge invariant.
With the gauge invariant attention score and gauge equivariant value function, the single head attention Eqn. (6) is gauge equivariant. For the multi-head attention to be gauge equivariant, the transformation matrix Wa M also needs to satisfy Eqn. (Ta).
4.5 Rotation Invariance
The rotation invariance property of GET accomplished by constructing local coordinate system for every point and making use of the gauge equivariance property. As is shown in Figure 4 assuming that xp the coordinate vector of p E M in the global coordinate system, np is the corresponding normal vector, and the gauge Wp is ascertained by principal axes up and vp. By projecting the raw data Tp onto the local coordinate system, we get the local coordinate of point X, ((tp: Up) (Kip Up) (Ip, "p)). which relies on wp but is invariant to the choice of global coordinate system. The insight is that is actually feature map whose corresponding feature field associated with representation Plocal as:
Cos 0 sin 0
sin 0
0
Plocol (+)
0
0
(21)
If feed the local coordinates into SO()) gauge equivariant model whose outputs scalar fields, the result will be 50(3) rotation invariant.
4.6 Error Analysis
Following the conventions, GET stacks multiple self-attention layers with ReLU activation functions. Even if discretized on triangle meshes, GET is still exactly equivariant to gauge transformations at angles that are multiples of 2n //N.
Theorem Assume a GET v,, whose types of input, intermediate, and output feature fields are Plocals C and pa. respectively, where k; is the number of regular fields in the ith intermediate feature
8

---

field. Denote f as the input feature field on triangle mesh M. and the norm of the feature map is bounded by constant C. Gauges and wf are linked by transformation g. Further suppose that sb Lipschitz continuous with constant L, then we have:
(i) If 9p E CN for every mesh vertex M, then (b(ww) (.w()))
(ii) For general 9p E (O(()), we have (1v((w.)) (f C.
Theorem 2 provides a bound for gauge transformation with respect to any angles. Compared to non-equivariant models, GET decreases the equivariance error by factor of 1/N. In experiments, we empirically show that the performance of our model increases as N increases.
5 Experiments
We conduct extensive experiments evaluate the effectiveness of model. We test the performance of our model on two deformable domain tasks, and conduct parameter sensitivity analysis and several ablation studies make comprehensive evaluation. Note that data preprocessing to precompute some useful preliminary values in order to save training time. The details of preprocessing can be found in supplementary materials.
5.1 Shape Classification
Our model used here is lightweight but powerful. The details of the architecture and training settings are provided in supplementary materials. Under the same setting, we compare our model with HSN
[56]. MeshCNN (24). GWCNN (20), GI [55] and MDGCNN [3], whose results are cited in [56]. As is shown in Table 5.2, our model achieves state-of-the the-trte-rtt performance on this dataset. GET significantly improves the previous state-of-the-an model HSN by 3.1% in classification accuracy.
This may attribute to the attention mechanism and the intrinsic rotation invariance of our model, while all other models are CNNs and directly accepts the raw coordinates zyz as input. Also, HSN the most parameter efficient model among the models we compared with. Our model consumes only 1/7 parameters of HSN (11K vs. 78K).
5.2 Shape Segmentation
widely used task in 3D shape segmentation Human Body Segmentation 30 in which the model is to predict body-part part annotation for each sampled point. The dataset consists of 370 training models from MIT [50], FAUST (5), Adobe Fuse ( and SCAPE [3] and 18 test models from SHREC07 223. The readers may refer to supplementary materials for details of neural network architecture and hyperparameters.
Table reports the percentage of correctly classified vertices across all samples in the test set. The results of comparing models cited from [56]] (60] and [35). Our model outperforms all these models in the segmentation task. GET consumes only about 1/15 parameters compared with MeshCNN (148K vs. 2.28M) but achieves higher performance.
Table 1:
Model results the SHREC dataset. GET performs the best without rotation data aug- mentation. The models trained without rotation augmentation are rotation invariant intrinsically.
Table 2: Segmentation results the Human Body Segmentation dataset. Our GET performs the best even without data augmentation by rota- tions.
Model
Rotation Aug. Acc. (%)
Model
Rotation Aug.
Acc. (%)
MDGCNN 35
82.2
MDGCNN
35
89.5
GI[45]
88.6
PointNet++ 38
90.8
GWCNN 20
90.3
HSN 56
91.1
MeshCNN
91.0
PFCNN 60
X
91.5
HSN 56
96.1
McshCNN 24
X
92.3
GET (Ours)
X
99.2
GET (Ours)
X
92.6

---

5.3 Parameter Sensitivity
Order the Group Cx. The hyperparameter is a key factor to the model equivariance since it controls both the dimension of regular field and the number of angles which the our model exactly cquivariant. Also, Theorem 2 asserts that the equivariance error is bounded by a factor of 1/N compared non-equivariant models. Here study the effect of N on model accuracy while keeping parameter numbers roughly the same. The results the Human Body Segmentation dataset with different N's are shown in Table 3] We can that the model performance improves considerably as increases and stabilizes finally.
Table 3: Model accuracy and the number of parameters in the Human Body Segmentation task with respect different N's.
5
7
9 (Chosen)
Acc. (%)
91.2
92.0
92.4
92.6
92.5
Params.
153K
149K
149K
148K
156K
5.4 Ablation Study
In this section, perform series of ablation studies to analyze individual parts of model. All the experiments are carried out on the Human Body Segmentation dataset under the same setting as in Section 5.2] We evaluate the effectiveness of gauge equivariance, attention, local coordinate and parallel transport method, with the latter two experiments provided in supplementary materials.
Gauge Equivariance and Atfention. To confirm the effectiveness of gauge equivariance property and attention mechanism, we design two baseline models with one not equivariant and the other based on convolution. For the non-equivariant baseline, we use Graph Attention Networks [49]. For the convolution based model, we adopt similar architecture as GET.
Table 4: Model accuracy in the Human Body Segmentation task with two baselines without gauge cquivariance and attention, respectively.
Model
Gauge Equivariance Attention Acc. (%)
GET
92.6
Baseline 1
81.1
Bascline 2
92.3
Table 4 shows see that both properties do contribute to the superiority of the model performance.
shows that GET both benefits from the power of gauge equivariance and attention. We can
6 Conclusion
We propose GET, which firstly incorporates attention in gauge equivariance. GET introduces new input, which is invariant to rotation of the global coordinate system. GET employs a new parallel transport approach, which is plausible for parallel transport between any two points. GET utilizes Taylor expansion in solving equivariant constraints, achieving better approximation ability. GET achieves tfte-of-thheart performances on several tasks and is efficient among the baselines.
Acknowledgment
Zhouchen Lin was supported by NSF China (No.s 61625301 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606) and Project 2020BD006 supported by PKU- Fund. Yisen Wang partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund.
10

---

References
[1] Adobe. Adobe mixamo 3D characters. http:: /www mamoo comm
2016.
[2] Brandon Anderson, Truong- Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. NeurlPS, 2019.
[3] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. SCAPE: Shape Completion and Animation of People. In SIGGRAPH. 2005.
[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[5] Federica Bogo, Javier Romero, Matthew Loper, and Michael Black. FAUST: Dataset and Evaluation for 3D Mesh Registration. In CVPR, 2014.
[6] Davide Boscaini, Jonathan Masci, Emanuele Rodoià, and Michael Bronstein. Learning Shape Correspondence with Anisotropic Convolutional Neural Networks. In NeurlPS, 2016.
[7] Michael M Bronstein, Joan Bruna, Taco S Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arX1v:2104.13778, 2021.
[8] Manfredo Perdigao do Carmo. Riemannian Geometry. Birkhäuser, 1992.
[9] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. ICLR, 2018.
[10] Taco S Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Homogeneous Spaces. NeurlPS, 2019.
[11] Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge Equivariant Convolutional Networks and the Icosahedral CNN. JCML, 2019.
[12] Taco S Cohen and Max Welling. Group Equivariant Convolutional Networks. In ICMI., 2016.
[13] Taco Cohen and Max Welling. Steerable CNNs. ICLR, 2017.
[14] Pim de Haan, Maurice Weiler, Taco S Cohen, and Max Welling. Gauge Equivariant Mesh CNNs: Anisotropic convolutions geometric graphs. ICLR, 2021.
[15] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting Cyclie Symmetry in Convolutional Neural Networks. In ICMI., 2016.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16
words: Transformers for image recognition at scale. ICLR, 2021.
[17] Carlos Esteves, Christine Allen BlanchBBeeeete Ameesh Makadia, and Kostas Danilidii. Learning so (3) equivariant representations with spherical cnns. In ECCV, 2018.
[18] Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin-weighted spherical cnns. NeurlPS, 2020.
[19] Carlos Esteves, Yinshuang Xu, Christine Allen-Blancher and Kostas Daniilidis. Equivariant Multi-view Networks. In ICCV, 2019.
[20] Danielle Ezuz, Justin Solomon, Vladimir G Kim, and Mirela Ben-Chen. GWCNN: A Metric Alignment Layer for Deep Shape Analysis. In Computer Graphics Forum, 2017.
[21] Fabian B Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE (3)-transformers: 3D Roto-transsation Equivariant Attention Networks. NewrIPS, 2020.
[22] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In ICLR, 2021.
[23] Daniela Giorgi, Silvia Biasotti, and Laura Paraboschi. Shape Retrieval Contest 2007: Watertight Models Track. SHREC competition, 8(7), 2007.

---

[24] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen- -O. MeshCNN: a Network with an Edge. TOG, 2019.
[25] Lingshen He, Yuxuan Chen, Zhengyang Shen, Yiming Dong, Yisen Wang, and Zhouchen Lin. Efficient equivariant network. In NeurIPS, 2021.
[26] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018.
[27] Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. ICMI,, 2021.
[28] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-Gordan Nets: a fully fourier space spherical convolutional neural network. NeurIPS, 2018
[29] Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In JCML, 2018.
[30] Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir Kim, and Yaron Lipman. Convolutional Neural Networks on Surfaces via Seamless Toric Covers. TOG, 2017.
[31] Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic Convolutional Neural Networks on Riemannian Manifolds. In ICCV Workshops, 2015.
[32] Daniel Maturana and Sebastian Scherer. Voxnet: A 3D Convolutional Neural Network for Real-time Object Recognition. In IROS, 2015.
[33] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric Deep Learning on Graphs and Manifolds using Mixture Model CNNs. In CVPR, 2017.
[34] Nathanaël Perraudin, Michaël Defferrard, Tomasz Kacprzak, and Raphael Sgier. Deepsphere: Efficient spherical convolutional neural network with healpix sampling for cosmological appli- cations. Astronomy and Computing, 27:130-146, 2019.
[35] Adrien Poulenard and Maks Ovsjanikov. Multi-direccional Geodesic Neural Networks via Equivariant Convolution. TOG, 2018
[36] Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective Rotation-invariant Point CNN with Spherical Harmonics Kernels. 3DV, 2019.
[37] Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation. In CVPR, 2017.
[38] Charles R Qi, Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. NeurIPS, 2017.
[39] Yongming Rao, Jiwen Lu, and Jie Zhou. Spherical fractal convolutional neural networks for point cloud recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 452-460, 2019.
[40] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning Deep 3D Represen- tations at High Resolutions. CVPR, 2017.
[41] David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In ICML, 2020.
[42] David W Romero and Jean Jean-Baptiste Cordonnier. Group Equivariant Stand-Alone Self-Attention For Vision. ICML, 2020.
[43] David W Romero and Mark Hoogendoorn. Co-attentive equivariant neural networks: Focusing equivariance on transformations co-occurring in data. ICLR, 2020.
[44] Zhengyang Shen, Lingshen He, Zhouchen Lin, and Jinwen Ma. PDO-eConvs: Partial differential operator based equivariant convolutions. In ICML, 2020.
12

---

[45] Ayan Sinha, Jing Bai, and Karthik Ramani. Deep Learning 3D Shape Surfaces using Geometry Images. In ECCV, 2016.
[46] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view Convolutional Neural Networks for 3D Sape Recognition. In ICCV, 2015.
[47] Nathaniel Thomas, Tess Smidt, Steven Kearne,, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation- eqquavvrannai neural networks for 3D point clouds. NeurIPS, 2018.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[49] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR, 2018.
[50] Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. Articulated Mesh Animation from Multi-view Silhouettes. In SIGGRAPH. 2008.
[51] Chu Wang, Marcello Pelillo, and Kaleem Siddigi. Dominant Set Clustering and Pooling for Multi-view 3D Object Recognition. BMVC, 2019.
[52] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Ya Yu Sun, and Xin Tong. O-cnm: Octree-based convolutional neural networks for 3D shape analysis. TOG, 2017.
[53] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[54] Maurice Weiler and Gabriele Cesa. General E (2)-equivariant Steerable CNNs. In NeurlPS, 2019.
[55] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNS: Learning Rotationally Equivariant Features in Volumetric Data. NeurIPS, 2018.
[56] Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. CNNs on Surfaces using Rotation- Equivariant Features. TOG, 2020.
[57] Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3D Rotation and Translation. In ECCV, 2018.
[58] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Har- monic Networks: Deep Translation and Rotation equivariance. CVPR, 2017.
[59] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. arXiv preprini Stti::2/06.033444 2021.
[60] Yugi Yang, Shilin Liu, Hao Pan, Yang Liu, and Xin Tong. PFCNN: Convolutional Neural Networks on 3D Surfaces using Parallel Frames. In CVPR, 2020.
[61] Tan Yu, Jingjing Meng, and Junsong Yuan. Multi-view Harmonized Bilinear Network for 3D Object Recognition. In CVPR, 2018.
13


Reviews:
[{"review_id": "zQy89KnHNFV", "reviewer": "Reviewer_5GYK", "summary": "The authors address an important gap in existing literature at the intersection of self-attention networks (transformers) and the equivariant neural networks -- how to apply self-attention to manifolds or meshes (discretized versions of manifolds). This has immediate applications in the area of 3D shapes which are 2D manifolds embedded in 3D. The key challenge is that of gauge-equivariance as there is no canonical coordinate system and we would like the self-attention blocks to be equivariant to the choice of the gauge. To this end, the authors build a provably (approximate) gauge-invariant score function and a gauge-equivariant value function. The experiments show the importance of the proposed methods. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are many strengths in this paper. \n- The authors address an important and challenging problem and effectively advance research in applying self-attention networks on manifolds. The novelty in terms of ideas as applied to this problem appears to be high. \n- They provide an intuitive generalization of the multi-head self-attention (MHSA) for feature-fields on manifolds. \n- MHSA involves computing SA for multiple heads and then linearly combining them. However, the numerical values of the features depend on the gauge which in turn depends on the point on the manifold. Thus in order to compute pairwise self-attention scores at a point p, we need to first transform the numerical values features at a point q to the same gauge as at p and then use parallel transport to move them to p. The various weight matrices themselves need to satisfy constraints such that the score function is gauge-invariant and the value function is gauge-equivariant. This is challenging and the authors do a good job explaining and designing these modules. \n- Theoretical guarantees ensure the desired gauge-equivariance of MHSA.\n- The experiments clearly show the usefulness of the proposed approach. The number of trainable parameters to achieve the same performance as methods like MeshCNNs for shape classification and segmentation is reduced by an order of magnitude. \n\nPoints of clarification:\n- I don't fully follow some of the notation and ideas in Section 4.1:\n   - Why is the subscript $u$ needed in $q_u$? \n   - And shouldn't the value function and the corresponding weight matrix be $V_q$ and $W_q$ instead of $V_u$ and $W_u$? This is because $u$ refers to a point using the gauge at point $p$, but the the weight matrix for the value function acts on the parallel transported point $q_u$. \n\nThe authors should discuss possible limitations of this work. \n\nTable 4 suggests that the attention mechanism itself is not crucial for good performance. Can the authors comment on this?\nAlso, what is this baseline exactly which uses equivariance and not attention? The authors say that this is a convolution based model created by removing the attention scores. How does removing the attention scores make it convolutional. And is the resulting model similar to gauge-equivariant mesh CNNs?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not explicitly addressed the limitations of the work and they say that it's because of space limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "O085Y7qazTc", "reviewer": "Reviewer_3RJ5", "summary": "The paper presents a gauge equivariant transformer for geometric learning. The gauge equivariance aims to solve the orientation ambiguity when doing the convolution or local aggregation operations within a neighborhood. Specifically, this work achieves the gauge equivariant self-attention within the neighborhood defined by mesh data. Different from existing gauge equivariant layers, the paper proposes to accommodate the parallel transport in the regular field. As a result, the gauge equivariance is up to a certain error bound. Further, this work validates the effectiveness on two datasets including shape classification (SHREC) and human body segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From my viewpoint, this work will have a certain impact -- incorporating the gauge equivariance into the recently popular transformer. The main concern I have is about the comparison with other gauge equivariant CNN, especially given that the paper shows the negligible contribution of attention to the performance improvement in ablation study. \n\nOriginality:\nGauge equivariant self-attention is novel. Although it’s a combination between existing gauge equivariance and self-attention, this work closes the gap in the transformer. In addition, the work presents a new way to accommodate parallel transport, which is interesting to me.\n\nQuality:\nThe paper is of high quality in terms of theoretical analysis. But I’m a bit concerned about the experimental validation. \n- Missing baselines related to gauge equivariance. For example, the geodesic CNN [25] also tried to solve the orientation ambiguity problem. Also, there is no other method for comparison in the tables that also incorporate gauge equivariance. \n- I would appreciate it more if the paper could discuss and compare with gauge equivariant mesh CNN [46, 12]. \n\nClarity:\nOverall, the paper is well-written. But I still have several comments. Hope the author could address them during the rebuttal.\n- What’s the motivation for using a regular field for the intermediate features?\n- The gauge equivariance is not strict (up to error bound). I am not sure if the title will be misleading.\n- What’s the efficiency of the proposed method?\n- Adding citations into table 1 and 2 would be more readable.\n\n\nSignificance:\nThis paper injects the gauge equivariance into the recent popular transformer. However, due to the missing comparison with other gauge equivariant CNN, I am not sure if the result is significant. Further, the ablation study shows that the key to the improvement is due to the gauge equivariance. Does this mean that convolution, as used in [46, 12], would be better? I would like to hear more from the authors during the rebuttal. \n\n\n### Post rebuttal\nI have the feedback from the authors and other reviews. Since the feedback addressed my concerns, I would like to improve my rating. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "From my viewpoint, I suggest discussing the limitation in terms of efficiency.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "I47k08zvrmq", "reviewer": "Reviewer_UDQP", "summary": "This paper addresses the main challenge in attention to manifolds, namely, that there are no coordinates to use as input in positional encoding. The network uses regular fields of cyclic groups and a parallel transport method for transporting features to these fields. Invariance to global rotations is achieved by taking only local position vectors. Experiments are performed on SHREC and Human Segmentation using a fraction of the usual transformer capacity.\n\n\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nThis is the first gauge equivariant self-attention on manifolds.\nSeveral contributions of the paper are novel: the parallel transport, the discretization of local rotations and their interpolation, and the approach for solving equivariant constraints. \n\nThe key in the authors' contribution are the cyclic fields obtained by the discretization of the local mesh angles. \n\nFigure 3 is very illuminating in the parallel transport of feature vectors.\n\nThe computation of the value function follows the icosahedral CNN paper by Cohen.\n\nTheorem 2 provides a bound with respect to any angle (not just the discretized ones). \n\nThe appendix is as useful to read as the paper and the derivations are a delight to follow. \n\nThe main puzzle for the reader is in the ablation study. Against the trend, it seems that it is mainly the gauge equivariance in human segmentation that matters rather than the self-attention (performance difference 0.3\\%). The explanation though \n\"Without attention, the convolution-based baseline performs as well as the second best baseline (MeshCNN)\" does not reflect the table. Please elaborate. \n\nIn a similar tone, please explain the challenges of gauge equivariance in self-attention rather than just gauge equivariance in meshes. After following the math and the lack of any difference in performance I am not sure whether the self-attention really matters and whether it is worth writing a paper about it.\n\nPOST-REBUTTAL: My main question was answered here and in the rest of the rebuttal. Stay with initial mark.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BwNlTIcvLQE", "reviewer": "Reviewer_apqm", "summary": "This paper proposes a gauge-equivariant transformer (GET) for manifolds. The challenging issue for self-attention on a manifold is the lack of canonical coordinate systems on a surface, which results in rotation ambiguity. To address it, this work introduces the regular field of cyclic groups as feature fields, proposes a novel parallel transport approach that puts the feature vectors in these fields, and develops the gauge-equivariant self-attention by adopting Taylor expansion in solving equivariant constraints. The proposed method is validated on the tasks of 3D shape classification and segmentation, achieving SOTA on SHREC dataset and the Human Body Segmentation dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n+ This work is the first for gauge-equivariant self-attention (transformer). \n+ The paper is well written with a solid background and the related properties being proven. \n+ The technical contribution is high; a new parallel transport method by extending regular representation,  gauge equivariant value function by Taylor approximation.\n+ The proposed method achieves SOTA on two popular benchmarks. \n\nWeakness\n- Unclear parts. The design of the gauge-invariant attention (Eq.19) needs to be clarified. Why not use the conventional product-based score?  \n- Lack of comparison to group-equivariant transformers.\nWhile group-equivariant transformers [17, 21, 34] are mentioned in the related work, any of them are not compared in the experiments. The benefit of gauge equivariance would be better shown in comparison to the work of group equivariance. I think at least the work of [17] can be compared since they are developed for 3D data. \n- Lack of in-depth comparison with gauge-equivariant CNNs.\nTable 4 shows that the convolution-based gauge-equivariant network performs only slightly worse than GET. Does it imply the marginal gain of the proposed self-attention layer compared to gauge-equivariant convolution? In this sense, a more in-depth comparison with gauge-equivariant CNNs [10, 46, 12] needs to be done to show the advantage of GET. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "I don't find any concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "-3cw9ht9ism", "reviewer": "Reviewer_mwoh", "summary": "This paper considers the problem of introducing attention to data lying on manifolds (without a global symmetry). Since there is no canonical coordinate system available in such cases to parameterize neighborhoods, the authors instead use the notion of gauge equivariance, proposing a so-called gauge equivariant transformer. Some additional innovations in the intermediate layers are introduced to improve expressivity, and equivariance error is characterized. Experiments are reported on SHREC and Human Body Segmentation tasks, showing good performance with significant parameter efficiency compared to prior art. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is concerned with the introduction of attention for operating on data lying on manifolds. General manifolds lack a canonical coordinate system to parameterize local neighborhoods. Since attention-based methods rely on such information for positional encodings, the paper considers a model that is gauge-equivariant (and thus agnostic to the orientation of the local coordinate system), introducing what the authors dub as a \"Gauge Equivariant Transformer.\" \n\nThe starting point is the influential work on gauge-equivariant CNNs, at the core of which is a convolution like operation that is equivariant to local gauge transformations, thus providing principled manifold CNNs. However, the use of convolution also implies that all points in a neighborhood are accorded equal importance. This paper deals with the extension of this machinery to attention mechanisms -- such that the aggregation only depends on the intrinsic geometry. Further, an additional requirement pursued is global rotational invariance (which might not always be the case in gauge-equivariant CNNs for general manifolds). This work can also be considered an addition to a growing body of work on group-equivariant transformer models that have been proposed recently. \n\nAttention is restricted to 2-manifolds (embedded in 3D). The basic definitions and preliminaries are introduced in section 3.1, where quantities such as gauges, tangent vectors, feature fields are defined, followed by explication on gauge transformations. The condition for gauge equivariance is specified in section 3.2, while the exponential and log map are covered in 3.3. Since self-attention amounts to aggregation of neighborhood features, in a general manifold, feature vectors of different points reside in different spaces, and thus have to be parallel transported to the same feature space where they can be processed. This process is described in section 3.4. The usual self-attention formulation is described in 3.5.\n\nFollowing the above background, a gauge equivariant self-attention layer is defined where the value function is required to be gauge-equivariant and the attention score gauge-invariant. The value function is taken to be the numerical outcome of the parallel transported feature vector times the value matrix, subject to the usual gauge equivariant condition. The resulting equation is then reduced to a system of linear equations using a Taylor expansion. Then, using a similar structure as the graph attention network, the key and query are combined to give a gauge-invariant attention score. Finally experimental results are shown on SHREC and Human Body Segmentation datasets (the manifolds are represented as triangular meshes), the results show that the proposed model gives superior performance while using a fraction of parameters (compared to the baseline models). Results are also shown for the choice of N (the order of the cyclic group, which is used for equivariance in the intermediate layers). \n\nWhile I have tried to verify all the details in the paper, my chief concern is that the paper can sometimes be hard to parse (which is also a reason for my lower confidence score). So, even with quite some familiarity with the related literature, I don't feel fully confident about some aspects of the paper (such as solving for the gauge equivariant condition). Nonetheless, the overall formulation seems correct, and the appendix fills in a lot of the missing details. Despite these concerns, I think the paper presents good results on two benchmarks and makes a step towards more principled self-attention networks for data on manifolds. \n\n\nMinor comments:\n\n- The paper keeps using the phrase \"orientation ambiguity problem\", but doesn't quite explain what they mean by it, perhaps due to the presumption that since the paper is on gauge equivariance, it will be clear. However, ii would improve readability if this is clarified concretely in the context of attention right in the beginning. \n- The paper has many grammatical errors and typos, which the authors might want to iron out. I am listing some such examples below.\n- Line 34: \"neglection to content-based information\" --> Neglect of content-based information?\n- Line 49: \"which initiatively\" is this a typo? \n- Line 63: \"emerging field concerning on adapting neural networks on various data types\" --> \"emerging field concerned with adapting neural networks to various data types\"\n- Line 64: \"For researches\" --> For research on modeling/for research works on modeling\n- Line 160: \"selectively concentrating on the most relevant parts\" --> \"selectively concentrate on the most relevant parts\"\n- Line 211: \"feature vector multiply by the value encoding matrix\" --> \"feature vector multiplied by the value encoding matrix\"\n- Line 261: \"the resulted one will be\" --> result will be", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops the tools necessary to construct Gauge-equivariant transformers -- and to apply self-attention to manifolds or meshes with a focus on 2D smooth and orientable manifolds.
The presented methods can be applied to a wide variety of problems.


This paper is clearly written and technically sound. The experiments clearly show the strength of the proposed method (e.g. in comparison to MeshCNNs for shape classification).

Experimental results (Tables 1, 2) also support the idea that directly incorporating equivariances and invariances into the model's architecture is more efficient than using data-augmentation.
Decision: Accept (Poster)